We want to thank the reviewer for their constructive and detailed feedback.

Concern 1: Transition kernel and the loss landscape

We emphasize that while the $\arg \max$ operator maps Gaussian marginals to discrete marginals, it also preserves the transition dynamics and Markov property. In anon. link [3], we show the marginal evolution follows: $\frac{d}{dt} q_t = -\frac{\mathcal{T}'(\tilde{\alpha}_t)}{K \mathcal{T}(\tilde{\alpha}_t)} \left[{\mathbf{1}} {\mathbf{1}}^\top - K \mathbf{I}\right] q_t$, where $\mathcal{T}'(\tilde{\alpha}_t)$ is the time derivative of $\mathcal{T}(\tilde{\alpha}_t)$. This implies$\frac{d}{dt} q_t = Q_t q_t$, with $Q_t = -\frac{\mathcal{T}'(\tilde{\alpha}_t)}{\mathcal{T}(\tilde{\alpha}_t)} \left[{\mathbf{1}} {\mathbf{1}}^\top - K \mathbf{I}\right] \in \mathbb{R}^{n \times n}$ representing the transition kernel for a Markovian discrete diffusion process (Anderson, 2012). Schiff et al., 2024 (Supp C, Eq. 50) confirm this is the transition dynamics of USDMs with diffusion parameter $\mathcal{T}(\tilde{\alpha}_t)$.

Loss Landscape: However, transforming a Gaussian diffusion into a discrete one does not preserve the loss landscape. In fact, the loss landscape for the discrete diffusion process is much more desirable because it induces a tighter bound on the likelihood as stated in Theorem 3.1.

We stress that the equivalence betweeen Gaussian diffusion and the USDMs is deep and fundamental without any approximations whatsoever.

Concern 2: This method is an approximation

As established above, the connection between Gaussian diffusion and USDMs is absolute and without any approximations. The softmax approximation to argmax introduced in Eqn(16), the training loss, leverages this fundamental relationship to design a low variance curriculum learning training scheme.

Concern 3: Two orders sampling speedup

After distillation, DUO achieves a Gen. PPL of $79.24$ and entropy of $5.25$ with $T = 8$, closely matching the undistilled DUO with $T = 1024$, which has a Gen. PPL of $72.05$ and the same entropy ($5.22$). This reduction in steps comes without sacrificing entropy or degradation in sample quality (see anon. link [3]).

Concern 4: Novelty of Softmax annealing trick

The core novelty of this work is establishing a fundamental connection between discrete and Gaussian diffusion via the argmax operator. We show that the softmax annealing trick, originally proposed for backpropagating through argmax [1, 2], can be repurposed to design a low-variance training curriculum for USDMs.

Concern 5: 2x faster training

Variance reduction leads to better generalization, as shown by DUO achieving lower val. PPL with Curriculum Learning compared to without it. See our response to Concern 1 for Reviewer 1i8M.

Concern 6: Curriculum Learning

Please refer to discussions in the response for concern 2 for the Reviewer 1i8M

Concern 7: Weak AR Baselines

The comparison with the AR baseline is fair—DUO, diffusion baselines [5, 7, 8], and the AR Transformer in Tabs. 1–3 all use the same architecture and dataset, with the only difference being causal attention for AR and bidirectional for diffusion models. Notably, in Tab. 3, where DUO outperforms AR on 3 out of 7 benchmarks, both use the same Transformer architecture. Omninet and Transformer-XL results in Tab. 2 are included for reference only.

Concern 8: Experimental Design

We’d like to emphasize that our experimental setup is consistent with the current literature in the field of diffusion modeling [5, 7, 8]. Our experiments isolate the effect of the training method by fixing the Transformer architecture. While training with modern architectures like GPT-3.5, PaLM, or LLaMA on datasets like Pile or C4 would be valuable, it would require retraining all baselines—an infeasible task for an academic lab.

Concern 9: Sampling speed

Sampling from a distilled DUO model is significantly faster than an AR model. See responses for concern 2 for Reviewer J6SU.

Other concerns:

Missing citations: We’ll cite [4] and add a detailed discussion for [1, 2, 4] in the next version of the paper.

Qualitative Analysis: We provide more samples in the anon. link [3] where we observe that the distilled DUO model outputs significantly better quality samples than the distilled MDLM model at lower sampling steps.

---

References

[1] Jang et al., 2016

[2] Maddison et al., 2016

[3] Anon. link: https://docs.google.com/document/d/e/2PACX-1vR0uKDuQHl4bBuC8KokEQhHNMvGdxbIskJm_SfXO_L6haSzWEqjPtL9wkVmg_yacNzMei2DAk21J5XX/pub

[4] Anderson, Continuous-time Markov chains: An applications-oriented approach.

[5] Schiff et al., 2025 “Simple Guidance Mechanisms for Discrete Diffusion Models”

[6] Luo, Simian, et al. "Latent consistency models"

[7] Sahoo et al., 2024 “Simple and Effective Masked Diffusion Language Models”

[8] Lou et al., 2024 “Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution”

We want to thank the reviewer for their constructive feedback. We address the reviewers comments and questions below.

Concern 1: USDMs still lag MDMs

As mentioned in the response to Concern 1, USDMs lag MDMs only when measured in terms of perplexity which isn’t necessarily the best metric for comparisions with MDMs. USDMs are preferred over MDMs on few-step generation (Fig. 3) where a distilled version of DUO significantly outperforms a distilled version of MDLM at $T=8$

As mentioned in lines 16-27 (right), the design space of Discrete Diffusion models remains largely under explored as compared to Gaussian Diffusion models. In this work we establish a core property of USDMs — they emerge from Gaussian diffusion. This opens up new avenues of future research which would leverage this connection to improve USDMs by borrowing techniques from Gaussian diffusion— a connection that doesn’t exist for MDMs.

Concern 2 : Motivation for using USDMs over MDMs

USDMs are preferable to MDMs for tasks like guidance (Schiff et al., 2025) and fast sampling (this work). The reason USDMs allow for faster sampling than MDMs is that they can fix their mistakes. This allows USDMs to make mistakes fast, but fix them later.

We also note that while perplexity is a useful sanity check, it does not account for speed. Perplexity only captures how sample quality with a high number of sampling steps.

In the table below, we present new experiments that show that USDMs can achieve strong sample quality quickly: Either with low latency, the time to produce a whole sequence, or high throughput, the rate of parallel generation.

In this table, we train AR models on OWT for 1M steps with a context length of $1024$, varying the number of Transformer layers to control parameter count; exact architecture details can be found in anon link [2]. We do not use a KV cache for the AR baselines. We measure latency (sec) for generating batch size (BS) = 1 and find that DUO is significantly faster than even the smallest AR model. We also measure throughput (tokens/sec) using the largest BS (multiple of 16) that fits in memory. The distilled DUO outperforms the 17M-parameter AR model in tems of speed. All experiments were run on a single A100-SXM4-80GB. The mean and std are computed over 100 batches.

Concern 3: MDLM number different from original in Tab. 2

Please refer to our response for the reviewer maKg, Concern 4.

---

References

[1] Schiff et al., 2025 “Simple Guidance Mechanisms for Discrete Diffusion Models”

[2] Anon. link : https://docs.google.com/document/d/e/2PACX-1vR0uKDuQHl4bBuC8KokEQhHNMvGdxbIskJm_SfXO_L6haSzWEqjPtL9wkVmg_yacNzMei2DAk21J5XX/pub

We want to thank the reviewer for their constructive feedback. We address the reviewers comments and questions below.

Concern 1 : Generating trajectories using DDIM.

As noted in lines 250–252 (left), the deterministic Gaussian trajectories assume an optimal denoiser: given clean data $\mathbf{x}$ and a latent $\mathbf{x}\_t \sim q_t(. | \mathbf{x})$, the optimal denoiser is defined as $\mathbf{x}^*_\theta(\mathbf{x}_t, t) = \mathbf{x}$ for all $t \in [0, 1]$. These DDIM trajectories (Eq. 17), under an optimal denoiser, preserve the Gaussian diffusion marginals (Song et al., 2022; Sec. C.1 in Zhou et al., 2025) and, when mapped to discrete space via $\arg \max$, align with the marginals of uniform state diffusion, as shown in Sec. 3.

Concern 2: Training loss vs Eval loss

We'd like to clarify that while Eq. 14 and Eq. 6 are equivalent, Eq. 16 which corresponds to a biased estimate of the true ELBO is used for training. The $\arg \max$ in Eq. 14 is approximated by a tempered-softmax in Eq.16. This approximation reduces training loss variance and improves generalization. For a detailed explanation, please see our response to Concern 2 from Reviewer 1i8M.

Concern 3: Perplexity for UDLM in Tab 2 and Tab 3.

The numbers for SEDD Absorb and Plaid are taken from Sahoo et al., 2024 and Lou et al., 2024. At the request of the reviewer, we conduct new experiments by training UDLM [3] on OWT as follows:

Zero shot perplexities:

The conclusions remain unchanged—DUO is the state-of-the-art among USDMs.

Concern 4: Reported numbers for MDLM and UDLM in Tab. 2

Lines 258–264 (right) clarify the discrepancy caused by differences in dataset preprocessing. The LM1B dataset contains short sentences (~30 tokens each with the GPT-2 tokenizer). In prior work [1, 3, 4], each datapoint in a batch is a single sentence padded to 128 tokens. This results in each sentence being considered in isolation.

In contrast, we follow the sentence-packing scheme from Austin et al. (2021) [5], where sentences are concatenated before batching. This results in sentences potentially being split between batches, with additional and potentially noisy context added. As a result, the packed data has a higher (worse) perplexity than padded data.

At the reviewer’s request, we conduct new experiments where we trained our model using the preprocessing from [1, 3, 4], and report the results below. The conclusions remain unchanged—DUO is state-of-the-art among USDMs and approaches MDM performance.

Concern 5: Low sample diversity

To make the curves in Fig. 3 more readable, the corresponding entropy values are provided separately in Tab. 4. As clarified in lines 351–358, the entropies of MDLM, SEDD Absorb, and SEDD Uniform align with that of an autoregressive model without nucleus sampling (approximately 5.6). In contrast, DUO’s entropy matches that of an AR model with nucleus sampling (p = 0.9), around 5.2. Qualitative samples (anon. link [6]) further show that DUO produces significantly higher-quality outputs than both distilled and undistilled MDLM at lower sampling steps $T=8$.

Comments: Clarification on lines 768-769 in the appendix

We meant to say the following: "We empirically verify the equivalence of the USDM ELBO with discrete latents (Eqn. 6) and Gaussian latents (Eqn. 14). To do this, we trained UDLM [3] on LM1B using the true ELBO from (6). We then evaluated the model using Gaussian latents (Eqn. 14), and recovered the same perplexity (36.71) as when using discrete latents. For each datapoint $\mathbf{x}$, we used $1000$ Monte Carlo samples for $t$ sampled using antithetic-sampling, with a linear schedule for $\tilde{\alpha}_t = 1 − t$."

---

References

[1] Sahoo et al., 2024 “Simple and Effective Masked Diffusion Language Models”

[2] Zhou et al., 2025 “Inductive Moment Matching”

[3] Schiff et al., 2025 “Simple Guidance Mechanisms for Discrete Diffusion Models”

[4] Lou et al., 2024 “Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution”

[5] Austin et al., 2021 “Structured denoising diffusion models in discrete state-spaces”

[6] Anon. link: https://docs.google.com/document/d/e/2PACX-1vR0uKDuQHl4bBuC8KokEQhHNMvGdxbIskJm_SfXO_L6haSzWEqjPtL9wkVmg_yacNzMei2DAk21J5XX/pub

We want to thank the reviewer for their constructive feedback. We address the reviewers comments and questions below.

Concern 1: Interpreting training loss (Fig 2) vs Tab 2

We will clarify Fig 2 by highlighting the key takeaway: DUO’s training loss exhibits significantly lower variance than both MDLM and UDLM’s. We thank the reviewer for pointing out that the biased training loss of DUO distracts from this key takeaway, and will revise this figure to emphasize the empirical variance of the loss, as well as gradient variance (more details in Concern 2).

We note that in the current Fig. 2 (paper), DUO’s curve lies below MDLM’s because its training loss (Eq. 16) is a biased approximation of the true ELBO (Eqs. 6 and 14). This bias causes the training loss to be lower. More formally, for the true ELBO (Eq. 6), the discrete diffusion parameter $\tilde{\alpha}_t$ must lie in $[0, 1]$. However, during training with the approximation (Eq. 16), it suffices to choose the Gaussian diffusion parameter $\tilde{\alpha}_t \in [0.85, 0.97]$, which maps to $\alpha_t = \mathcal{T}(\tilde{\alpha}_t) \in [\approx 0, \approx 1]$ for $\tau \to 0^+$ (see Fig. 4, appendix). Since, Gaussian latents contain more signal than noise, while training with Eq. 16 with $\tau > 0$ the denoising model finds reconstructing the input easier, resulting in a lower training loss.

Tab. 2 (paper) reports the validation perplexity computed using the true ELBO (Eq. 6 for DUO). For clarity, we report the validation perplexities of various methods below at different training steps and observe that DUO consistently outperforms UDLM and approaches the performance of MDLM. We thank the review for pointing out the potential conflicting interpretations of Fig 2 and Tab. 2, and believe revising Fig 2 to focus on variance will alleviate this.

Note: UDLM is equivalent to DUO w/o Curriculum Learning

Concern 2: Gradient Variance

As requested by the reviewer, we report gradient variance across model weights. We compute the variance of each weight’s updates over 10 steps on LM1B at different training steps. With Curriculum Learning (CL), gradient variance is significantly lower—especially in early training (RTab 1; below). Among the 100 weights with the highest variance (RTab 2; below), CL reduces variance by an order of magnitude early on, which gradually diminishes over time. This reduction appears beneficial, as it accelerates learning in the initial phases, reflected in improved validation loss; see above.

(RTab 1) Sum of all variances:

(RTab 2) Sum of highest 100 variances:

We observe a similar trend in the variances of the loss as well; see below.

(RTab 2) Sum of highest 100 variances:

We will replace Fig 2 (the loss curves) with these findings.

Other comments

During distillation, the student and teacher networks share the same architecture and differ only in weights—the teacher uses EMA weights, while the student uses the current model parameters. Both networks factorize the joint distribution independently as $p_\theta(\mathbf{x}\_0 | \mathbf{z}\_t) = \prod_i p_\theta(\mathbf{x}^i_0 | \mathbf{z}\_t)$. Although Xu et al. (2024) propose a more expressive factorization with an energy term,$p_\theta(\mathbf{x}\_0 | \mathbf{z}\_t) = \prod_i p_\theta(\mathbf{x}^i_0 | \mathbf{z}\_t) e^{E_\phi} / Z_\phi$, which leads to a better fit to the data, the independent assumption performs well in practice [2, 3, 4]. Modeling the energy function $E_\phi$ and the partition function $Z_\phi$ is also non-trivial, often requiring an additional network and complicating both training and sampling. We believe incorporating this approach could enhance our model’s expressiveness and benefit both DUO and MDLM, which we leave for future work.

---

References

[1] Xu et al., 2024, ENERGY-BASED DIFFUSION LANGUAGE MODELS FOR TEXT GENERATION

[2] Sahoo et al., 2024 “Simple and Effective Masked Diffusion Language Models”

[2] Schiff et al., 2025 “Simple Guidance Mechanisms for Discrete Diffusion Models”

[3] Lou et al., 2024 “Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution”