Repcodec: A Speech Representation Codec For Speech Tokenization

Abstract

With recent rapid growth of large language models (LLMs), discrete speech tokenization has played an important role for injecting speech into LLMs. However, this discretization gives rise to a loss of information, consequently impairing overall performance. To improve the performance of these discrete speech tokens, we present RepCodec, a novel speech representation codec for semantic speech tokenization. In contrast to audio codecs which reconstruct the raw audio, RepCodec learns a vector quantization codebook through reconstructing speech representations from speech encoders like HuBERT or data2vec. Together, the speech encoder, the codec encoder and the vector quantization codebook form a pipeline for converting speech waveforms into semantic tokens. The extensive experiments illustrate that RepCodec, by virtue of its enhanced information retention capacity, significantly outperforms the widely used k-means clustering approach in both speech understanding and generation. Furthermore

Stats

• citations14
• S2 citations—
• github stars0
• HF likes0
• heat score8.82
• arxiv keyhuang2023repcodec

Related papers

• Secodec: Structural Entropy-based Compressive Speech Representation Codec For Speech Language Models (2024)2.26
• Freecodec: A Disentangled Neural Speech Codec With Fewer Tokens (2024)4.52
• Language-codec: Bridging Discrete Codec Representations And Speech Language Models (2024)4.64
• Lscodec: Low-bitrate And Speaker-decoupled Discrete Speech Codec (2024)0.00
• Dm-codec: Distilling Multimodal Representations For Speech Tokenization (2024)3.53
• Socodec: A Semantic-ordered Multi-stream Speech Codec For Efficient Language Model Based Text-to-speech Synthesis (2024)6.34
• Codec-asr: Training Performant Automatic Speech Recognition Systems With Discrete Speech Representations (2024)6.77
• Semanticodec: An Ultra Low Bitrate Semantic Audio Codec For General Sound (2024)10.97