Analyzing And Mitigating Inconsistency In Discrete Audio Tokens For Neural Codec Language Models

Abstract

Building upon advancements in Large Language Models (LLMs), the field of audio processing has seen increased interest in training audio generation tasks with discrete audio token sequences. However, directly discretizing audio by neural audio codecs often results in sequences that fundamentally differ from text sequences. Unlike text, where text token sequences are deterministic, discrete audio tokens can exhibit significant variability based on contextual factors, while still producing perceptually identical audio segments. We refer to this phenomenon as \textbf\{Discrete Representation Inconsistency (DRI)\}. This inconsistency can lead to a single audio segment being represented by multiple divergent sequences, which creates confusion in neural codec language models and results in omissions and repetitions during speech generation. In this paper, we quantitatively analyze the DRI phenomenon within popular audio tokenizers such as EnCodec. Our approach effectively mitigates the DRI ph

Stats

• citations5
• S2 citations—
• github stars0
• HF likes0
• heat score5.84
• arxiv keyliu2024analyzing

Related papers

• Codec Does Matter: Exploring The Semantic Shortcoming Of Codec For Audio Language Model (2024)15.02
• Investigating Neural Audio Codecs For Speech Language Model-based Speech Generation (2024)2.26
• Audiolm: A Language Modeling Approach To Audio Generation (2022)18.91
• Language-codec: Bridging Discrete Codec Representations And Speech Language Models (2024)4.64
• Dm-codec: Distilling Multimodal Representations For Speech Tokenization (2024)3.53
• Almtokenizer: A Low-bitrate And Semantic-rich Audio Codec Tokenizer For Audio Language Modeling (2025)0.00
• Code Drift: Towards Idempotent Neural Audio Codecs (2024)2.26
• Wavtokenizer: An Efficient Acoustic Discrete Codec Tokenizer For Audio Language Modeling (2024)6.22