Blending Llms Into Cascaded Speech Translation: Kit's Offline Speech Translation System For IWSLT 2024

Abstract

Large Language Models (LLMs) are currently under exploration for various tasks, including Automatic Speech Recognition (ASR), Machine Translation (MT), and even End-to-End Speech Translation (ST). In this paper, we present KIT's offline submission in the constrained + LLM track by incorporating recently proposed techniques that can be added to any cascaded speech translation. Specifically, we integrate Mistral-7B\footnote\{mistralai/Mistral-7B-Instruct-v0.1\} into our system to enhance it in two ways. Firstly, we refine the ASR outputs by utilizing the N-best lists generated by our system and fine-tuning the LLM to predict the transcript accurately. Secondly, we refine the MT outputs at the document level by fine-tuning the LLM, leveraging both ASR and MT predictions to improve translation quality. We find that integrating the LLM into the ASR and MT systems results in an absolute improvement of \(0.3%\) in Word Error Rate and \(0.65%\) in COMET for tst2019 test set. In challenging tes

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keykoneru2024blending

Related papers

• Kit's Low-resource Speech Translation Systems For IWSLT2025: System Enhancement With Synthetic Data And Model Regularization (2025)0.00
• Hearing To Translate: The Effectiveness Of Speech Modality Integration Into Llms (2026)0.00
• Llast: Improved End-to-end Speech Translation System Leveraged By Large Language Models (2024)10.67
• Zero-resource Speech Translation And Recognition With Llms (2024)3.58
• Simuls2s-llm: Unlocking Simultaneous Inference Of Speech Llms For Speech-to-speech Translation (2025)3.58
• Chain-of-thought Prompting For Speech Translation (2024)6.34
• Transformer-based Cascaded Multimodal Speech Translation (2019)0.00
• Large Language Model Can Transcribe Speech In Multi-talker Scenarios With Versatile Instructions (2024)11.23