Investigating Decoder-only Large Language Models For Speech-to-text Translation

Abstract

Large language models (LLMs), known for their exceptional reasoning capabilities, generalizability, and fluency across diverse domains, present a promising avenue for enhancing speech-related tasks. In this paper, we focus on integrating decoder-only LLMs to the task of speech-to-text translation (S2TT). We propose a decoder-only architecture that enables the LLM to directly consume the encoded speech representation and generate the text translation. Additionally, we investigate the effects of different parameter-efficient fine-tuning techniques and task formulation. Our model achieves state-of-the-art performance on CoVoST 2 and FLEURS among models trained without proprietary data. We also conduct analyses to validate the design choices of our proposed model and bring insights to the integration of LLMs to S2TT.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyhuang2024investigating

Related papers

• On Decoder-only Architecture For Speech-to-text And Large Language Model Integration (2023)0.00
• Zero-resource Speech Translation And Recognition With Llms (2024)3.58
• Making Llms Better Many-to-many Speech-to-text Translators With Curriculum Learning (2024)7.31
• From Hype To Insight: Rethinking Large Language Model Integration In Visual Speech Recognition (2025)0.00
• Llast: Improved End-to-end Speech Translation System Leveraged By Large Language Models (2024)10.67
• Chain-of-thought Prompting For Speech Translation (2024)6.34
• Long-form Speech Translation Through Segmentation With Finite-state Decoding Constraints On Large Language Models (2023)0.00
• Hearing To Translate: The Effectiveness Of Speech Modality Integration Into Llms (2026)0.00