Acoustic BPE For Speech Generation With Discrete Tokens

Abstract

Discrete audio tokens derived from self-supervised learning models have gained widespread usage in speech generation. However, current practice of directly utilizing audio tokens poses challenges for sequence modeling due to the length of the token sequence. Additionally, this approach places the burden on the model to establish correlations between tokens, further complicating the modeling process. To address this issue, we propose acoustic BPE which encodes frequent audio token patterns by utilizing byte-pair encoding. Acoustic BPE effectively reduces the sequence length and leverages the prior morphological information present in token sequence, which alleviates the modeling challenges of token correlation. Through comprehensive investigations on a speech language model trained with acoustic BPE, we confirm the notable advantages it offers, including faster inference and improved syntax capturing capabilities. In addition, we propose a novel rescore method to select the optimal synt

Stats

• citations7
• S2 citations—
• github stars0
• HF likes0
• heat score6.77
• arxiv keyshen2023acoustic

Related papers

• Discrete Audio Representation As An Alternative To Mel-spectrograms For Speaker And Speech Recognition (2023)8.60
• Dmel: Speech Tokenization Made Simple (2024)0.00
• Acoustic Data-driven Subword Modeling For End-to-end Speech Recognition (2021)6.77
• DASB - Discrete Audio And Speech Benchmark (2024)0.00
• Audio Language Modeling Using Perceptually-guided Discrete Representations (2022)0.00
• Audiolm: A Language Modeling Approach To Audio Generation (2022)18.91
• Generative Spoken Language Model Based On Continuous Word-sized Audio Tokens (2023)3.58
• Speech Rhythm-based Speaker Embeddings Extraction From Phonemes And Phoneme Duration For Multi-speaker Speech Synthesis (2024)3.58