Semanticodec: An Ultra Low Bitrate Semantic Audio Codec For General Sound

Abstract

Large language models (LLMs) have significantly advanced audio processing through audio codecs that convert audio into discrete tokens, enabling the application of language modelling techniques to audio data. However, traditional codecs often operate at high bitrates or within narrow domains such as speech and lack the semantic clues required for efficient language modelling. Addressing these challenges, we introduce SemantiCodec, a novel codec designed to compress audio into fewer than a hundred tokens per second across diverse audio types, including speech, general sound, and music, without compromising quality. SemantiCodec features a dual-encoder architecture: a semantic encoder using a self-supervised pre-trained Audio Masked Autoencoder (AudioMAE), discretized using k-means clustering on extensive audio data, and an acoustic encoder to capture the remaining details. The semantic and acoustic encoder outputs are used to reconstruct audio via a diffusion-model-based decoder. Semant

Stats

• citations28
• S2 citations—
• github stars0
• HF likes0
• heat score10.97
• arxiv keyliu2024semanticodec

Related papers

• Codec Does Matter: Exploring The Semantic Shortcoming Of Codec For Audio Language Model (2024)15.02
• Almtokenizer: A Low-bitrate And Semantic-rich Audio Codec Tokenizer For Audio Language Modeling (2025)0.00
• Socodec: A Semantic-ordered Multi-stream Speech Codec For Efficient Language Model Based Text-to-speech Synthesis (2024)6.34
• Low Frame-rate Speech Codec: A Codec Designed For Fast High-quality Speech LLM Training And Inference (2024)5.24
• Lscodec: Low-bitrate And Speaker-decoupled Discrete Speech Codec (2024)0.00
• Msr-codec: A Low-bitrate Multi-stream Residual Codec For High-fidelity Speech Generation With Information Disentanglement (2025)2.35
• Wavtokenizer: An Efficient Acoustic Discrete Codec Tokenizer For Audio Language Modeling (2024)6.22
• Secodec: Structural Entropy-based Compressive Speech Representation Codec For Speech Language Models (2024)2.26