Principled Coarse-grained Acceptance For Speculative Decoding In Speech

Abstract

Speculative decoding accelerates autoregressive speech generation by letting a fast draft model propose tokens that a larger target model verifies. However, for speech LLMs that generate acoustic tokens, exact token matching is overly restrictive: many discrete tokens are acoustically or semantically interchangeable, reducing acceptance rates and limiting speedups. We introduce Principled Coarse-Graining (PCG), which verifies proposals at the level of Acoustic Similarity Groups (ASGs) derived from the target model's embedding space. By splitting each token's probability mass across the overlapping groups that contain it, we define an overlap-aware coarse-grained distribution and perform rejection sampling on the resulting group variable. This yields an exactness guarantee at the group level while allowing the accepted draft token to stand in for any member of the group in practice. On LibriTTS, PCG increases acceptance and throughput relative to standard speculative decoding and prior

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyyanuka2025principled

Related papers

• Fast And High-quality Auto-regressive Speech Synthesis Via Speculative Decoding (2024)5.24
• Accelerating Codec-based Speech Synthesis With Multi-token Prediction And Speculative Decoding (2024)4.52
• Speculative Speech Recognition By Audio-prefixed Low-rank Adaptation Of Language Models (2024)3.58
• What Makes A Good Speech Tokenizer For Llm-centric Speech Generation? A Systematic Study (2025)0.00
• Generative Error Correction For Code-switching Speech Recognition Using Large Language Models (2023)0.00
• Livespeech: Low-latency Zero-shot Text-to-speech Via Autoregressive Modeling Of Audio Discrete Codes (2024)5.84
• SPADE: Structured Pruning And Adaptive Distillation For Efficient LLM-TTS (2025)0.00
• Towards Efficient Speech-text Jointly Decoding Within One Speech Language Model (2025)0.00