TTA: Transcribe, Translate And Alignment For Cross-lingual Speech Representation

Abstract

Speech-LLM models have demonstrated great performance in multi-modal and multi-task speech understanding. A typical speech-LLM paradigm is integrating speech modality with a large language model (LLM). While the Whisper encoder was frequently adopted in previous studies for speech input, it shows limitations regarding input format, model scale, and semantic performance. To this end, we propose a lightweight TTA model specialized in speech semantics for more effective LLM integration. With large-scale training of 358k hours of speech data on multilingual speech recognition (ASR), speech translation (ST) and speech-text alignment tasks, TTA is capable of producing robust cross-lingual speech representations. Extensive evaluations across diverse benchmarks, including ASR/ST, speech retrieval, and ASR-LLM performance assessments, demonstrate TTA's superiority over Whisper. Furthermore, we rigorously validate the interplay between cross-lingual capabilities and ASR/ST performance. The model

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyliu2025tta

Related papers

• Large Language Model Can Transcribe Speech In Multi-talker Scenarios With Versatile Instructions (2024)11.23
• LI-TTA: Language Informed Test-time Adaptation For Automatic Speech Recognition (2024)3.58
• Zero-resource Speech Translation And Recognition With Llms (2024)3.58
• TASU: Text-only Alignment For Speech Understanding (2025)0.00
• Alignformer: Modality Matching Can Achieve Better Zero-shot Instruction-following Speech-llm (2024)6.77
• Transducer-llama: Integrating Llms Into Streamable Transducer-based Speech Recognition (2024)3.58
• Investigating Decoder-only Large Language Models For Speech-to-text Translation (2024)0.00
• Ideal-llm: Integrating Dual Encoders And Language-adapted LLM For Multilingual Speech-to-text (2024)5.24