Fewer-token Neural Speech Codec With Time-invariant Codes

Abstract

Language model based text-to-speech (TTS) models, like VALL-E, have gained attention for their outstanding in-context learning capability in zero-shot scenarios. Neural speech codec is a critical component of these models, which can convert speech into discrete token representations. However, excessive token sequences from the codec may negatively affect prediction accuracy and restrict the progression of Language model based TTS models. To address this issue, this paper proposes a novel neural speech codec with time-invariant codes named TiCodec. By encoding and quantizing time-invariant information into a separate code, TiCodec can reduce the amount of frame-level information that needs encoding, effectively decreasing the number of tokens as codes of speech. Furthermore, this paper introduces a time-invariant encoding consistency loss to enhance the consistency of time-invariant code within an utterance and force it to capture more global information, which can benefit the zero-shot

Stats

• citations21
• S2 citations—
• github stars0
• HF likes0
• heat score10.07
• arxiv keyren2023fewer

Related papers

• Clam-tts: Improving Neural Codec Language Model For Zero-shot Text-to-speech (2024)0.00
• Freecodec: A Disentangled Neural Speech Codec With Fewer Tokens (2024)4.52
• Tacolm: Gated Attention Equipped Codec Language Model Are Efficient Zero-shot Text To Speech Synthesizers (2024)0.00
• Latent-domain Predictive Neural Speech Coding (2022)12.15
• Single-codec: Single-codebook Speech Codec Towards High-performance Speech Generation (2024)9.23
• VALL-E R: Robust And Efficient Zero-shot Text-to-speech Synthesis Via Monotonic Alignment (2024)0.00
• Vec-tok Speech: Speech Vectorization And Tokenization For Neural Speech Generation (2023)0.00
• Optimizing Neural Speech Codec For Low-bitrate Compression Via Multi-scale Encoding (2024)0.00