Continuous Latent Diffusion Language Model

Problem
This paper addresses the limitations of existing autoregressive language models in generating high-quality text without being constrained to a fixed left-to-right order. The authors highlight that current alternatives struggle to balance generation efficiency, scalable representation learning, and effective global semantic modeling. This work is presented as a preprint and has not yet undergone peer review.

Method
The authors introduce Cola DLM, a hierarchical latent diffusion language model that employs a novel approach to text generation through hierarchical information decomposition. The architecture consists of three main components:

1. Text VAE: A Variational Autoencoder (VAE) is utilized to establish a stable text-to-latent mapping.
2. Block-Causal DiT: A diffusion model is employed to model a global semantic prior in a continuous latent space, using a block-causal structure to maintain causal relationships.
3. Conditional Decoding: Text generation is performed through a decoding process conditioned on the learned latent representations.

The diffusion process is framed as latent prior transport, which decouples global semantic organization from local textual realization. This design allows for a flexible non-autoregressive inductive bias, enabling semantic compression and prior fitting in continuous space. The model is trained on a dataset that includes approximately 2 billion parameters, with scaling experiments conducted up to around 2000 EFLOPs.

Results
Cola DLM was evaluated across four research questions and eight benchmarks, demonstrating competitive performance against matched autoregressive and LLaDA baselines. The results indicate that Cola DLM achieves superior generation quality and scaling behavior, suggesting that these metrics may better reflect model capability than likelihood alone. Specific headline numbers were not disclosed in the abstract, but the authors emphasize the effectiveness of their configuration and the strong scaling behavior observed during experiments.

Limitations
The authors acknowledge that while Cola DLM presents a promising alternative to token-level language modeling, it may still face challenges in fully capturing the nuances of language generation compared to autoregressive models. Additionally, the reliance on a VAE for text-to-latent mapping may introduce limitations in the expressiveness of the latent space. The paper does not address potential issues related to the interpretability of the latent representations or the computational overhead associated with the diffusion process.

Why it matters
The introduction of hierarchical continuous latent prior modeling represents a significant advancement in the field of language modeling, offering a principled alternative to traditional autoregressive methods. This work not only enhances the understanding of text generation dynamics but also suggests pathways for unified modeling across discrete text and continuous modalities. The implications extend to various applications in natural language processing, where improved generation quality and efficiency can lead to more robust and versatile AI systems.

Authors: Hongcan Guo, Qinyu Zhao, Yian Zhao, Shen Nie, Rui Zhu, Qiushan Guo, Feng Wang, Tao Yang et al.
Source: arXiv:2605.06548
URL: https://arxiv.org/abs/2605.06548v1