ESC: Efficient Speech Coding With Cross-scale Residual Vector Quantized Transformers

Abstract

Neural speech codecs aim to compress input signals into minimal bits while maintaining content quality in a low-latency manner. However, existing neural codecs often trade model complexity for reconstruction performance. These codecs primarily use convolutional blocks for feature transformation, which are not inherently suited for capturing the local redundancies in speech signals. To compensate, they require either adversarial discriminators or a large number of model parameters to enhance audio quality. In response to these challenges, we introduce the Efficient Speech Codec (ESC), a lightweight, parameter-efficient speech codec based on a cross-scale residual vector quantization scheme and transformers. Our model employs mirrored hierarchical window transformer blocks and performs step-wise decoding from coarse-to-fine feature representations. To enhance bitrate efficiency, we propose a novel combination of vector quantization techniques along with a pre-training paradigm. Extensive

Stats

• citations5
• S2 citations—
• github stars0
• HF likes0
• heat score5.84
• arxiv keygu2024esc

Related papers

• Freecodec: A Disentangled Neural Speech Codec With Fewer Tokens (2024)4.52
• Optimizing Neural Speech Codec For Low-bitrate Compression Via Multi-scale Encoding (2024)0.00
• ERVQ: Enhanced Residual Vector Quantization With Intra-and-inter-codebook Optimization For Neural Audio Codecs (2024)6.34
• Latent-domain Predictive Neural Speech Coding (2022)12.15
• Enhancing Into The Codec: Noise Robust Speech Coding With Vector-quantized Autoencoders (2021)10.21
• Codec-asr: Training Performant Automatic Speech Recognition Systems With Discrete Speech Representations (2024)6.77
• Disentangled Feature Learning For Real-time Neural Speech Coding (2022)0.00
• Neural Feature Predictor And Discriminative Residual Coding For Low-bitrate Speech Coding (2022)6.77