Language-codec: Bridging Discrete Codec Representations And Speech Language Models

Abstract

In recent years, large language models have achieved significant success in generative tasks related to speech, audio, music, and other signal domains. A crucial element of these models is the discrete acoustic codecs, which serve as an intermediate representation replacing the mel-spectrogram. However, there exist several gaps between discrete codecs and downstream speech language models. Specifically, 1) Due to the reconstruction paradigm of the Codec model and the structure of residual vector quantization, the initial channel of the codebooks contains excessive information, making it challenging to directly generate acoustic tokens from weakly supervised signals such as text in downstream tasks. 2) numerous codebooks increases the burden on downstream speech language models. Consequently, leveraging the characteristics of speech language models, we propose Language-Codec. In the Language-Codec, we introduce a Masked Channel Residual Vector Quantization (MCRVQ) mechanism along with i

Stats

• citations0
• S2 citations—
• github stars209
• HF likes0
• heat score4.64
• arxiv keyji2024language

Related papers

• Codec Does Matter: Exploring The Semantic Shortcoming Of Codec For Audio Language Model (2024)15.02
• Lscodec: Low-bitrate And Speaker-decoupled Discrete Speech Codec (2024)0.00
• Single-codec: Single-codebook Speech Codec Towards High-performance Speech Generation (2024)9.23
• Hifi-codec: Group-residual Vector Quantization For High Fidelity Audio Codec (2023)0.00
• Secodec: Structural Entropy-based Compressive Speech Representation Codec For Speech Language Models (2024)2.26
• Investigating Neural Audio Codecs For Speech Language Model-based Speech Generation (2024)2.26
• Msr-codec: A Low-bitrate Multi-stream Residual Codec For High-fidelity Speech Generation With Information Disentanglement (2025)2.35
• Repcodec: A Speech Representation Codec For Speech Tokenization (2023)8.82