Text-free Prosody-aware Generative Spoken Language Modeling

Abstract

Speech pre-training has primarily demonstrated efficacy on classification tasks, while its capability of generating novel speech, similar to how GPT-2 can generate coherent paragraphs, has barely been explored. Generative Spoken Language Modeling (GSLM) \cite\{Lakhotia2021\} is the only prior work addressing the generative aspects of speech pre-training, which replaces text with discovered phone-like units for language modeling and shows the ability to generate meaningful novel sentences. Unfortunately, despite eliminating the need of text, the units used in GSLM discard most of the prosodic information. Hence, GSLM fails to leverage prosody for better comprehension, and does not generate expressive speech. In this work, we present a prosody-aware generative spoken language model (pGSLM). It is composed of a multi-stream transformer language model (MS-TLM) of speech, represented as discovered unit and prosodic feature streams, and an adapted HiFi-GAN model converting MS-TLM outputs to

Stats

• citations38
• S2 citations—
• github stars32213
• HF likes0
• heat score20.95
• arxiv keykharitonov2021text

Related papers

• Generative Spoken Language Model Based On Continuous Word-sized Audio Tokens (2023)3.58
• Generative Pre-trained Speech Language Model With Efficient Hierarchical Transformer (2024)5.96
• Paralinguistics-enhanced Large Language Modeling Of Spoken Dialogue (2023)0.00
• SLMGAN: Exploiting Speech Language Model Representations For Unsupervised Zero-shot Voice Conversion In Gans (2023)0.00
• Generative Pre-training For Speech With Flow Matching (2023)0.00
• Phone-level Prosody Modelling With Gmm-based MDN For Diverse And Controllable Speech Synthesis (2021)9.23
• Audiolm: A Language Modeling Approach To Audio Generation (2022)18.91
• Long-form Speech Generation With Spoken Language Models (2024)0.00