Fastadasp: Multitask-adapted Efficient Inference For Large Speech Language Model

Abstract

In this study, we aim to explore Multitask Speech Language Model (SpeechLM) efficient inference via token reduction. Unlike other modalities such as vision or text, speech has unique temporal dependencies, making previous efficient inference works on other modalities not directly applicable. Furthermore, methods for efficient SpeechLM inference on long sequence and sparse signals remain largely unexplored. Then we propose FastAdaSP, a weighted token merging framework specifically designed for various speech-related tasks to improve the trade-off between efficiency and performance. Experimental results on WavLLM and Qwen-Audio show that our method achieves the state-of-the-art (SOTA) efficiency-performance trade-off compared with other baseline methods. Specifically, FastAdaSP achieved 7x memory efficiency and 1.83x decoding throughput without any degradation on tasks like Emotion Recognition (ER) and Spoken Question Answering (SQA). The code will be available at https://github.com/yich

Stats

• citations0
• S2 citations—
• github stars17
• HF likes0
• heat score2.51
• arxiv keylu2024fastadasp

Related papers

• SPADE: Structured Pruning And Adaptive Distillation For Efficient LLM-TTS (2025)0.00
• Discrete Multimodal Transformers With A Pretrained Large Language Model For Mixed-supervision Speech Processing (2024)0.00
• Multilingual And Fully Non-autoregressive ASR With Large Language Model Fusion: A Comprehensive Study (2024)0.00
• Low Frame-rate Speech Codec: A Codec Designed For Fast High-quality Speech LLM Training And Inference (2024)5.24
• Scaling Spoken Language Models With Syllabic Speech Tokenization (2025)0.00
• Spidr-adapt: A Universal Speech Representation Model For Few-shot Adaptation (2025)2.68
• SLM-TTA: A Framework For Test-time Adaptation Of Generative Spoken Language Models (2025)0.00
• Towards Efficient Speech-text Jointly Decoding Within One Speech Language Model (2025)0.00