Masked Diffusion Transformer for Music Generation

Abstract

Text-to-music (TTM) generation using diffusion models faces challenges due to the scarcity of high-quality paired datasets, resulting in limited output diversity and fidelity. To address these challenges, we propose a novel framework that incorporates a Masked Diffusion Transformer (MDT) with quality-aware training. The MDT architecture dynamically adapts to varying input quality and effectively models complex temporal and harmonic dependencies in music.

Additionally, a multi-stage caption refinement process ensures enhanced alignment between textual inputs and musical outputs. Our approach improves musical coherence and provides a scalable solution for generating high-fidelity music from text descriptions. Extensive experiments conducted on diverse datasets, including MusicCaps, Song-Describer, and a custom Hugging Face compilation, demonstrate state-of-the-art performance in metrics such as Fréchet Audio Distance (FAD), KL Divergence, and Inception Score (IS). This work significantly advances TTM research and contributes to practical generative music applications.