Melnet: A Generative Model For Audio In The Frequency Domain

Abstract

Capturing high-level structure in audio waveforms is challenging because a single second of audio spans tens of thousands of timesteps. While long-range dependencies are difficult to model directly in the time domain, we show that they can be more tractably modelled in two-dimensional time-frequency representations such as spectrograms. By leveraging this representational advantage, in conjunction with a highly expressive probabilistic model and a multiscale generation procedure, we design a model capable of generating high-fidelity audio samples which capture structure at timescales that time-domain models have yet to achieve. We apply our model to a variety of audio generation tasks, including unconditional speech generation, music generation, and text-to-speech synthesis---showing improvements over previous approaches in both density estimates and human judgments.

Stats

• citations0
• S2 citations—
• github stars0
• HF likes0
• heat score0.00
• arxiv keyvasquez2019melnet

Related papers

• Wavenet: A Generative Model For Raw Audio (2016)0.00
• Audiolm: A Language Modeling Approach To Audio Generation (2022)18.91
• Inspiremusic: Integrating Super Resolution And Large Language Model For High-fidelity Long-form Music Generation (2025)6.26
• Gansynth: Adversarial Neural Audio Synthesis (2019)0.00
• Uniaudio: An Audio Foundation Model Toward Universal Audio Generation (2023)5.56
• Samplernn: An Unconditional End-to-end Neural Audio Generation Model (2016)0.00
• Adversarial Generation Of Time-frequency Features With Application In Audio Synthesis (2019)0.00
• Hifi-gan: Generative Adversarial Networks For Efficient And High Fidelity Speech Synthesis (2020)0.00