Audiolm: A Language Modeling Approach To Audio Generation

Abstract

We introduce AudioLM, a framework for high-quality audio generation with long-term consistency. AudioLM maps the input audio to a sequence of discrete tokens and casts audio generation as a language modeling task in this representation space. We show how existing audio tokenizers provide different trade-offs between reconstruction quality and long-term structure, and we propose a hybrid tokenization scheme to achieve both objectives. Namely, we leverage the discretized activations of a masked language model pre-trained on audio to capture long-term structure and the discrete codes produced by a neural audio codec to achieve high-quality synthesis. By training on large corpora of raw audio waveforms, AudioLM learns to generate natural and coherent continuations given short prompts. When trained on speech, and without any transcript or annotation, AudioLM generates syntactically and semantically plausible speech continuations while also maintaining speaker identity and prosody for unseen

Stats

• citations331
• S2 citations—
• github stars0
• HF likes0
• heat score18.91
• arxiv keyborsos2022audiolm

Related papers

• Continuous Audio Language Models (2025)0.00
• Uniaudio: An Audio Foundation Model Toward Universal Audio Generation (2023)5.56
• LM-VC: Zero-shot Voice Conversion Via Speech Generation Based On Language Models (2023)0.00
• Audio Language Modeling Using Perceptually-guided Discrete Representations (2022)0.00
• Uniaudio 1.5: Large Language Model-driven Audio Codec Is A Few-shot Audio Task Learner (2024)0.00
• From Alignment To Advancement: Bootstrapping Audio-language Alignment With Synthetic Data (2025)2.26
• C3LLM: Conditional Multimodal Content Generation Using Large Language Models (2024)0.00
• Audiotoolagent: An Agentic Framework For Audio-language Models (2025)2.60