Generative Pre-trained Speech Language Model With Efficient Hierarchical Transformer

Abstract

While recent advancements in speech language models have achieved significant progress, they face remarkable challenges in modeling the long acoustic sequences of neural audio codecs. In this paper, we introduce \textbf\{G\}enerative \textbf\{P\}re-trained \textbf\{S\}peech \textbf\{T\}ransformer (GPST), a hierarchical transformer designed for efficient speech language modeling. GPST quantizes audio waveforms into two distinct types of discrete speech representations and integrates them within a hierarchical transformer architecture, allowing for a unified one-stage generation process and enhancing Hi-Res audio generation capabilities. By training on large corpora of speeches in an end-to-end unsupervised manner, GPST can generate syntactically consistent speech with diverse speaker identities. Given a brief 3-second prompt, GPST can produce natural and coherent personalized speech, demonstrating in-context learning abilities. Moreover, our approach can be easily extended to spoken cro

Stats

• citations1
• S2 citations—
• github stars70
• HF likes0
• heat score5.96
• arxiv keyzhu2024generative

Related papers

• Text-free Prosody-aware Generative Spoken Language Modeling (2021)20.95
• Lauragpt: Listen, Attend, Understand, And Regenerate Audio With GPT (2023)0.00
• Frame-stacked Local Transformers For Efficient Multi-codebook Speech Generation (2025)0.00
• Hierarchical Multi-grained Generative Model For Expressive Speech Synthesis (2020)8.60
• Speechformer++: A Hierarchical Efficient Framework For Paralinguistic Speech Processing (2023)14.43
• Foundationtts: Text-to-speech For ASR Customization With Generative Language Model (2023)0.00
• Hifi-sr: A Unified Generative Transformer-convolutional Adversarial Network For High-fidelity Speech Super-resolution (2025)10.81
• Generative Pre-training For Speech With Autoregressive Predictive Coding (2019)14.73