What Makes A Good Speech Tokenizer For Llm-centric Speech Generation? A Systematic Study

Abstract

Speech-language models (SLMs) offer a promising path toward unifying speech and text understanding and generation. However, challenges remain in achieving effective cross-modal alignment and high-quality speech generation. In this work, we systematically investigate the role of speech tokenizer designs in LLM-centric SLMs, augmented by speech heads and speaker modeling. We compare coupled, semi-decoupled, and fully decoupled speech tokenizers under a fair SLM framework and find that decoupled tokenization significantly improves alignment and synthesis quality. To address the information density mismatch between speech and text, we introduce multi-token prediction (MTP) into SLMs, enabling each hidden state to decode multiple speech tokens. This leads to up to 12\(\times\) faster decoding and a substantial drop in word error rate (from 6.07 to 3.01). Furthermore, we propose a speaker-aware generation paradigm and introduce RoleTriviaQA, a large-scale role-playing knowledge QA benchmark

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyfan2025what

Related papers

• Towards Efficient Speech-text Jointly Decoding Within One Speech Language Model (2025)0.00
• Dm-codec: Distilling Multimodal Representations For Speech Tokenization (2024)3.53
• Scaling Spoken Language Models With Syllabic Speech Tokenization (2025)0.00
• Investigating Neural Audio Codecs For Speech Language Model-based Speech Generation (2024)2.26
• Evaluating Text-to-speech Synthesis From A Large Discrete Token-based Speech Language Model (2024)0.00
• TASTE: Text-aligned Speech Tokenization And Embedding For Spoken Language Modeling (2025)0.00
• Improving Robustness Of Llm-based Speech Synthesis By Learning Monotonic Alignment (2024)0.00
• Discrete Multimodal Transformers With A Pretrained Large Language Model For Mixed-supervision Speech Processing (2024)0.00