Secodec: Structural Entropy-based Compressive Speech Representation Codec For Speech Language Models

Abstract

With the rapid advancement of large language models (LLMs), discrete speech representations have become crucial for integrating speech into LLMs. Existing methods for speech representation discretization rely on a predefined codebook size and Euclidean distance-based quantization. However, 1) the size of codebook is a critical parameter that affects both codec performance and downstream task training efficiency. 2) The Euclidean distance-based quantization may lead to audio distortion when the size of the codebook is controlled within a reasonable range. In fact, in the field of information compression, structural information and entropy guidance are crucial, but previous methods have largely overlooked these factors. Therefore, we address the above issues from an information-theoretic perspective, we present SECodec, a novel speech representation codec based on structural entropy (SE) for building speech language models. Specifically, we first model speech as a graph, clustering the s

Stats

• citations1
• S2 citations—
• github stars0
• HF likes0
• heat score2.26
• arxiv keywang2024secodec

Related papers

• Repcodec: A Speech Representation Codec For Speech Tokenization (2023)8.82
• Socodec: A Semantic-ordered Multi-stream Speech Codec For Efficient Language Model Based Text-to-speech Synthesis (2024)6.34
• Semanticodec: An Ultra Low Bitrate Semantic Audio Codec For General Sound (2024)10.97
• Lscodec: Low-bitrate And Speaker-decoupled Discrete Speech Codec (2024)0.00
• Language-codec: Bridging Discrete Codec Representations And Speech Language Models (2024)4.64
• Codec Does Matter: Exploring The Semantic Shortcoming Of Codec For Audio Language Model (2024)15.02
• Single-codec: Single-codebook Speech Codec Towards High-performance Speech Generation (2024)9.23
• Freecodec: A Disentangled Neural Speech Codec With Fewer Tokens (2024)4.52