Vec-tok Speech: Speech Vectorization And Tokenization For Neural Speech Generation

Abstract

Language models (LMs) have recently flourished in natural language processing and computer vision, generating high-fidelity texts or images in various tasks. In contrast, the current speech generative models are still struggling regarding speech quality and task generalization. This paper presents Vec-Tok Speech, an extensible framework that resembles multiple speech generation tasks, generating expressive and high-fidelity speech. Specifically, we propose a novel speech codec based on speech vectors and semantic tokens. Speech vectors contain acoustic details contributing to high-fidelity speech reconstruction, while semantic tokens focus on the linguistic content of speech, facilitating language modeling. Based on the proposed speech codec, Vec-Tok Speech leverages an LM to undertake the core of speech generation. Moreover, Byte-Pair Encoding (BPE) is introduced to reduce the token length and bit rate for lower exposure bias and longer context coverage, improving the performance of L

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyzhu2023vec

Related papers

• Spark-tts: An Efficient Llm-based Text-to-speech Model With Single-stream Decoupled Speech Tokens (2025)8.08
• Vec-tok-vc+: Residual-enhanced Robust Zero-shot Voice Conversion With Progressive Constraints In A Dual-mode Training Strategy (2024)3.58
• Fewer-token Neural Speech Codec With Time-invariant Codes (2023)10.07
• Freecodec: A Disentangled Neural Speech Codec With Fewer Tokens (2024)4.52
• Speechx: Neural Codec Language Model As A Versatile Speech Transformer (2023)11.85
• Delightfultts 2: End-to-end Speech Synthesis With Adversarial Vector-quantized Auto-encoders (2022)9.23
• Vec2wav 2.0: Advancing Voice Conversion Via Discrete Token Vocoders (2024)0.00
• Preliminary Study On Using Vector Quantization Latent Spaces For TTS/VC Systems With Consistent Performance (2021)0.00