Continuous Diffusion Model for Language Modeling

We sincerely thank you for your constructive and helpful comments. We appreciate your positive feedback that our work is novel and reasonable, and the mathematical derivation is solid. We initially address your comments below:

---

C1: I think the method in this paper can not scale to Qwen3 and Kimi-chat, and it is a little overclaim to say 'approaches the performance of autoregressive models.'

R1: Thank you for your valuable feedback. We would like to clarify that our work proposes a novel language diffusion framework aimed at improving previous diffusion models for language modeling. Our primary focus is on advancing the modeling approach rather than scaling to very large modern LLMs such as Qwen3 or Kimi-chat.

Regarding the comparison to autoregressive models, this comparison is made within the context of models of similar scale. We do not claim that our method matches the performance of state-of-the-art large-scale autoregressive models, but rather that it approaches the performance of autoregressive models at comparable model sizes. We will clarify this point in the revised manuscript to avoid any overclaim.

---

C2: Detailed time-cost of RDLM compared to other discrete diffusion models and DDPM is needed.

R2: RDLM does not require additional computational cost during sampling compared to the discrete diffusion models in terms of the number of model evaluations. Specifically, to generate samples, RDLM simulates the generative process modeled by the SDE in Eq. (6) (please see Algorithm 3 in the Appendix), and simulating Eq. (6) with a fixed step solver (e.g., Euler-Maruyama) requires the same number of model evaluations as simulating the discrete diffusion process. Therefore, in terms of sampling time-cost, RDLM is comparable to discrete diffusion models, as both approaches rely on an equivalent number of model evaluations per sample generation.

---

C3: In CIFAR-10 experiments, the comparison to classical DDPM/EDM/rectified flow matching is beneficial.

R3: CIFAR-10 experiment in our work addresses a fundamentally different task compared to the image generation tasks tackled by DDPM or EDM models. Our CIFAR-10 experiment focuses on pixel-level image modeling framed as an order-agnostic set modeling problem of discrete tokens, each drawn from a vocabulary of size 256, following the approach in MD4 (Shi et al., 2024). This formulation intentionally removes spatial information about pixel proximity, which is a key inductive bias exploited by classical image diffusion models through architectures like U-Net. Given this distinct setting, we excluded DDPM and EDM models from our baselines, as their assumptions and modeling strategies do not directly apply.

---

C4: Experiments on scalability should be given.

R4: Our experiment on the LM1B dataset, which features a sufficiently large vocabulary of size 30,522, demonstrates strong potential for scaling to even larger vocabularies. Notably, RDLM is designed to scale to higher dimensions efficiently due to its simulation-free training scheme (as detailed in Section 4) and the dimension splitting technique (described in Section 5).

Thanks for your detailed rebuttal.

My point of view has not changed; I still value the novelty of this paper and question its practical usage. My score is pretty marginal, and I keep my score between 3 and 4.

We sincerely thank you for your constructive and helpful comments. We appreciate your acknowledgement of our key contributions, in particular, that our work offers a principled route to exploit iterative refinement without losing signal at state jumps. We initially address your comments below:

---

C1: Missing related works

R1: We appreciate the reviewer for providing additional related works on text generation. We would like to clarify that DiffuSeq is a diffusion model designed for sequence-to-sequence tasks, whereas our work addresses general unconditional text generation. Moreover, TEncDM and LD4LG are continuous diffusion models that adapt image diffusion models to text, and they are conceptually similar to Diffusion-LM, Plaid, or Bayesian Flow Network, which we have included as baselines in our comparisons. We will add discussion with these works in the revision.

---

C2: Difference with GIDD

R2: We appreciate the reviewer for bringing to our attention the very recent work GIDD, which is a contemporaneous work (appeared on arXiv after March 1st). GIDD is a discrete diffusion model that interpolates masked and uniform diffusion by mixing masking and uniform noise. On the other hand, our RDLM is a continuous diffusion model using Riemannian diffusion process on hypersphere, which fully exploits the power of iterative refinement.

In particular, our mixture path formulation (Eq. (8)) can be seen as a continuous generalization of interpolating diffusion process in GIDD. Extending our Proposition 3.1., the mixture path built from mixing masked diffusion and uniform diffusion corresponds to the trajectory of transition distribution of GIDD process. Moreover, Eq. (8) allows interpolating more than two processes, while GIDD is limited to mixing masked and uniform noise.

During the rebuttal, we could not make a quantitative comparison with GIDD as GIDD does not have checkpoints for the benchmarks used in our work, for example, Text8, LM1B, CIFAR-10, or DNA sequence design. We will include a detailed discussion with GIDD and add comparison in the final revision.

---

C3: Language results are limited to Text8 and LM1B and other benchmarks such as RocStories and OpenWebText, are not covered

R3: Text8 and LM1B are among the most widely used datasets for evaluating diffusion models in recent literature, especially for discrete diffusion models. Notable recent works such as SEDD (Lou et al., 2024), MDLM (Sahoo et al., 2024), and MD4 (Shi et al., 2024) also use these datasets for benchmarking. These works do not typically use RocStories as a benchmark. We believe that experiments on Text8 and LM1B, as well as image generation and DNA sequence design extensively validate the effectiveness of our framework. While we agree that results on OpenWebText would further strengthen our work, we could not run the experiments due to limited resources. We will make every effort to include OpenWebText results in the final revision.

---

C4: Evaluation on more challenging tasks, including conditional generation, would be necessary

R4: Unconditional generation of discrete data using diffusion models is itself a challenging task, with a notable performance gap compared to autoregressive models. Recent works such as MDLM, MD4, and Discrete Flow Matching, have also focused primarily on unconditional generation tasks without extending to conditional generation. In our paper, we focus on validating the effectiveness of our new continuous diffusion framework for modeling language and discrete data in an unconditional setting. While conditional generation tasks are important, addressing them is beyond the scope of the current work.

---

C5: The work claims that the continuous approach offers advantages for controllable generation but provides no evidence on conditional text generation tasks

R5: We would like to clarify that we did not claim our method to have advantage for controllable generation. In introduction lines 29–30, we explained that previous works have adapted continuous diffusion models for discrete data by their advantage on controllability. For example, Diffusion-LM uses classifier-guidance to diffusion model for controllable text generation. While our continuous diffusion process can similarly incorporate classifier-guidance, we have not explored this direction in the current work.

---

C6: The bridge approximation is empirically validated with MMD, yet its impact on final likelihood in extreme dimensions is not thoroughly analysed.

R6: MMD results show that our approximation on the transition distribution is accurate. This leads to superior results on test perplexity in the LM1B dataset, which has a high dimension of 30522.

---

C7: Comparison with Gaussian diffusion on simplex or sphere

R7: First, we would like to clarify that Gaussian diffusion are ill-defined on non-Euclidean space like simplex or sphere. This is because the Gaussian distribution itself is not well-defined in these spaces, and diffusion process must be redefined to account for the underlying geometry and metric of the manifold.

While Gaussian diffusion models can be applied in the ambient Euclidean space, recent literature on generative modeling over manifolds, for instance, RDM (Huang et al., 2022) or RSGM (De Bortoli et al., 2022), demonstrate that Riemannian diffusion outperforms Gaussian diffusion when modeling data that reside on non-Euclidean manifolds. Our experimental results further support this, showing that our method significantly outperforms Diffusion-LM, a Gaussian diffusion model designed for text data.

Furthermore, prior works used as baselines in our paper, for example, DirichletFM (Stark et al., 2024) and Fisher-Flow (Davis et al., 2024), show that naively adapting flow matching to simplex, i.e.., LinearFM, underperformed Riemannian approaches. We will include this discussion in our revision and add LinearFM as our baseline.

---

C8: What happens if you use only the masked or only the uniform path?

R8: When using only the masked or the uniform path, we achieved test perplexity of 1.40 and 1.41 on the Text8 test set, respectively, while using the mixture path achieved 1.32.

We sincerely thank you for your constructive and helpful comments. We appreciate your positive comments that our work is based on solid theoretical guarantees, achieve excellent results, and each component is discussed and designed in detail. We initially address your questions below.

---

C1: It would be better to add more visualizations for an easier understanding of the RDLM pipeline.

R1: Thank you for your valuable feedback. Due to the page limit, we were unable to include additional visualizations in the current version. We agree that visual aids would enhance understanding, and we will add concept figures illustrating the RDLM pipeline in the final revision.

---

C2: Comparison of mixture path and masked or uniform path

R2: When using only the masked or the uniform path, we achieved test perplexity of 1.40 and 1.41 on the Text8 test set, respectively, while using the mixture path achieved 1.32. This shows that using the mixture path results in significantly improved performance.

---

C3: As this work shows the potential of RDLM, would it be possible to scale up RDLM to achieve great performance?

R3: Indeed, RDLM has the potential to achieve better performance when scaled up with larger models. Larger models can more accurately learn the transition distribution $p_{T|t}(X_T|X_t)$, which is critical for the effectiveness of RDLM. Our empirical results on Text8 and CIFAR-10 datasets support this observation, where we observe improved performance with increased model size. We believe that further scaling of RDLM could lead to even greater performance gains.

We sincerely thank you for your constructive comments. We appreciate your positive comments that our paper tackles an important topic in current AI research, and the proposed method is elegant and neat. We initially address your comments below:

---

C1: While the proposed method is elegant and neat, it is unclear whether there is real benefit for large-scale language modeling.

R1: We believe our method offers benefits for large-scale language modeling, both in terms of performance and the inherent advantages of a continuous diffusion framework.

First, superior results on the LM1B dataset, which has a sufficiently large vocabulary set of size 30522, demonstrate strong potential for scaling to large-scale language tasks. Importantly, RDLM can scale to even higher dimensions due to its simulation-free training scheme (Section 4) and dimension splitting technique (Section 5).

Furthermore, our continuous diffusion approach brings additional practical advantages that have been successfully leveraged in other continuous diffusion models. These include controllable generation and compatibility with advanced techniques such as self-conditioning. It also supports scalable and efficient sampling strategies like DPM-Solver, which can further improve inference speed and quality.

---

C2: If we consider the ambient space $\mathbf{R}^d$ in which the statistical manifold $P(\mathcal{X})$ and the sphere $\mathbb{S}^{d-1}$ are embedded, the gap between these two diffeomorphic manifolds becomes larger as $d$ increases. Intuitively, this makes transitions from some point to a token (i.e., $e_k$) much farther on the sphere than on the simplex.

R2: We would like to clarify that the diffeomorphism between the statistical manifold and the sphere preserves their intrinsic geometric structure, and the notion of gap or distance between points on these manifolds is preserved under this mapping. In particular, the transitions from a point on the statistical manifold to a token $e_k$ on the simplex correspond directly to transitions on the sphere via the diffeomorphism, and the distances do not become inherently larger on the sphere compared to the simplex as $d$ increases.

---

C3: Line 135: It says "providing numerous opportunities to correct wrong predictions during the process.", but being continuous can also lead to error accumulation, while discrete transitions hold the possibility of entirely denoising the accumulated error.

R3: Our continuous diffusion process is an iterative refinement process, where every intermediate step progressively improves the prediction to the correct target. This iterative nature effectively mitigates error accumulation, which makes the diffusion model a powerful generative model across diverse domains.

In contrast, discrete transitions attempt to reach the correct target in a single jump, which is inherently more challenging and prone to irreversible errors. This is especially critical in masked diffusion models, where incorrect predictions cannot be easily recovered without additional correction mechanisms. Our experimental results consistently show that the continuous diffusion approach outperforms discrete diffusion models, highlighting the advantages of gradual and continuous refinement over discrete jumps.

---

C4: Eq. (10): The left-hand side should be $p_{\theta}(X_T)$?

R4: In Eq. (10), we parameterize the model $p_{\theta}$ to approximate the transition distribution from current state $X_t$ to final state $e_k$, i.e., $p_{T|t}(e_k|X_t)$. The model takes as input $X_t$ and time $t$, and outputs the the probabilities $p_{T|t}(e_k|X_t)$ for all tokens $e_k$. Therefore, the left-hand side should be $p_{\theta}({X}_t, t)$ as in the paper. We will add a more detailed explanation to clarify this point.