Distillation of Discrete Diffusion through Dimensional Correlations

Thank you for your positive evaluation of the paper. Let us answer your questions. We will also correct typos you have pointed out.

[q-1] Why we chose SDTT model as teacher

Why do you opt to apply your method on top of another specific distilled model? Does the proposed method work well on top of a vanilla teacher model compared to the other distillation method?

We chose the SDTT model as the teacher for the following reasons. Firstly, we wanted to see if our method works even on top of a well-distilled model. While the teacher models (sdtt-6 / sdtt-7) went through many rounds of distillation, their modeling remains dimensionally independent. So, we wanted to confirm that there is still room for improvement by introducing dimensional correlations. Secondly, we were not aware of any other distillation methods for discrete diffusion (even though SDTT is just for masked diffusions) with which we compare our methods. By examining their performance gain (i.e., sdtt-6 -> sdtt-7), we could see how our methods work compared to another distillation method at the same time (please also see [q-2] for the actual comparison).

[q-2] Generative perplexity and diversity

It is known that the generative perplexity has crucial flaws [1,2], so it would be great to consider additional metrics for the text generation task.

Generative perplexity can indeed be easily hacked as it does not consider the diversity of the generated samples. Following our teacher model's work, we measured the Self-BLEU metric (similar to the sentence entropy used in [2] from your reference list, but lower is better) in addition to generative perplexity and plotted their trade-off curve in Figure 4(b). Our finding is that our distillation method (sdtt-6 -> sdtt-6+di4c(^2)) does not worsen the Self-BLEU compared to a round of SDTT (sdtt-6 -> sdtt-7).

[q-3] Relation to an advanced fast sampler

How does the proposed method compare to advanced samplers for discrete diffusion models such as [1]?

Thank you for pointing out this relevant work. We understand that the fast sampling method in the reference [1] (from your reference list) suggests that, in masked diffusion models, we can reduce the size of categorical sampling by first choosing the index (or indices in the parallel decoding variant) to unmask. In their parallel decoding variant, they do not model the dimensional correlations, so our mixture modeling can be combined to further improve sampling quality.

[q-4] Time step scheduling

Does the time step schedule (i.e., the choice of $s,u,t$) play an important role as in the consistency models [2,3]?

We suppose so, as the quality of our distillation target can vary depending on the time step scheduling. While we did not optimize the teacher sampling steps in our work, there is research optimizing the time step schedules in discrete diffusion [Par25], which can be combined with our method. With this approach, we can also optimize the step schedule after training the model (i.e., the step schedule in few-step sampling).

References

[Des25] Deschenaux and Gulcehre. Beyond Autoregression: Fast LLMs via Self-Distillation Through Time. ICLR 2025.

[Par25] Park et al. Jump your steps: Optimizing sampling schedule of discrete diffusion models. ICLR 2025.

Thank you for your feedback and positive comments. Let us respond to your questions.

[d-1] On the reference distributions

the method relies on reference distributions for $r_\delta$ and $r_t$, which must either be sampled from real training data or approximated using multiple teacher steps.

It is a valid point that our method has limitations on the source of reference samples. However, in the relatively large-scale experiments (masked image/language modeling), we used only a small portion of the original dataset. For example, we used 200K out of 9M samples in OpenWebText (similarly for ImageNet). Additionally, in our current implementation, each sample $x_0$ generates only a single $x_t$ (with a single $t$) throughout the training run. This suggests that more efficient batch designs could reduce the essential number of training samples. That said, we acknowledge that accessing or constructing reference samples is not always straightforward. Future research should explore the use of partial samples or samples from alternative distributions to address this limitation.

[d-2] Regarding $\tau$LDR models

Why is the result of the teacher model using 40 steps in Table 1 already close to 8?

This discrepancy arises from the different sampling methods used. As detailed in Table 3 in Section F.2.2, the $\tau$-leaping sampler does not work well with 40 steps. Instead, we used the analytical sampler to evaluate the teacher model. We suppose this difference stems from the non-time-homogeneous nature of the forward diffusion. Specifically, $\tau$-leaping approximates the transition rate using a constant matrix over a time interval of length $\tau$, which fails to accurately capture the actual transition rates when $\tau$ is large and the rates vary significantly within the interval. In contrast, the analytical sampler avoids this issue, although it still does not account for dimensional correlations.

Does it indicate that the proposed distillation method is not compatible with the predictor-corrector sampler.

We can use predictor-corrector (PC). However, PC requires additional network evaluations (NFE; each as expensive as one step of denoising), and does not perform well in the small NFE regime [Cam22, Fig 4]. Below are the FIDs (based on 10K images; IS omitted due to the character limit) for different PC settings under total NFE of 20. 'n+2*m' means we used m corrector steps before each of the final 2 out of n denoising steps (imitating [Cam22]).

[d-3] Inference of other number of steps

It would be more informative to include results of other steps in a graph similar to Figure 3. For example, I would like to see how many times of acceleration the proposed distillation method can achieve to maintain a similar performance (in FID/IS) to the teacher model using 20 inference steps.

Here, we have computed additional FID/IS for our CIFAR-10 experiment using 10K samples (fewer than the 50K samples used in the paper, so the numbers may be slightly worse) for 2-20 steps:

FID:

IS:

In terms of FID, our method achieves approximately 1.4 times acceleration in the 10–20 step range. However, it does not perform well in very few steps (e.g., 2–4 steps).

I feel there is still a huge gap between the proposed method and its counterpart for continuous diffusion.

To further reduce the number of steps, it is necessary to model high-dimensional discrete distributions more efficiently, as we lack a deterministic formulation like probability-flow ODEs (available in the continuous case). While mixture modeling provides some improvement, further optimization is required, including adjustments to the architecture, initial distribution, and dimensionality of $\lambda$. As you noted, our work represents a first step in building the theoretical foundation for this goal.

[d-4] Sampling distribution of $\lambda$

How should the sampling distribution of $\lambda$ be chosen? Have the authors ever considered other common distributions than uniform?

We prioritized ablations of different components over exploring alternative distributions for $\lambda$. As noted at the end of [d-3], this remains an area for future investigation.

References

[Sah24] Sahoo et al. Simple and effective masked diffusion language models. NeurIPS 2024.

[Cam22] Campbell et al. A continuous time framework for discrete denoising models. NeurIPS 2022.

Thank you for your review. Let us reply to your comments.

[1-1] Added metrics beyond FID/IS

the authors mainly compare their model with the teacher model using FID/IS scores, which is inherited limited

We have additionally computed precision and recall metrics for the ImageNet experiment (see the table below), following established practices in the current literature (e.g., [Tian24]). Due to character limits, we present the results for each model at the classifier-free guidance (CFG) coefficient that achieves the best FID (see Table 6). We can see di4c models achieve better scores than the teacher model with the same number of steps.

[1-2] DDMs are not "fast and approximate" in general

It is confusing why the authors insist on distilling DDM, which is already a fast approximate model, rather than slow high-quality models? Is it possible to use the model for distilling original or modified diffusion models of higher quality, or training from a scratch?

The characterization of DDMs as approximations of 'original diffusion models' (as implied in your latter comment) is not typically correct. Indeed, DDMs are not "approximate" or "fast" in general.

• Not Approximate: The teacher DDMs were generally trained from scratch (except for SDTT models we used in language modeling) without relying on any other diffusion models. While the parallel decoding (analytical sampling) in teacher DDM is indeed an approximate inference (with guarantees given in Theorem 1), a similar approximation (unimodal Gaussian approximation) is commonly employed in continuous diffusion models, as noted in [Li24]. Therefore, DDMs are not particularly "approximate" compared to continuous diffusion models.

• Not Fast: DDMs typically require tens to hundreds of sampling steps, depending on the complexity of model/data. For the experiment on Discretized Gaussian Diffusion, the teacher model requires 30-40 steps to reach FID<10, which we believe is no longer "very few steps". For comparison, in the continuous case, while EDM on CIFAR-10 requires 35 steps [Kar22], notable distillation methods [Son23] start from EDM and distill it into a few-step model. MaskGIT(-pytorch) is one of few examples that achieves a small number of steps (8 steps to reach FID~7.0), enabled by some heuristics including confidence-based sampling and CFG. We tested our method in this scenario because we believe it is worth investigating if our distillation methods can also work in combination with such heuristics.

Thus, we believe our method can be applied to "original or modified diffusion models". Learning dimensional correlations in DDM from scratch by incorporating the consistency training of [Son23] into our method is an interesting future work.

[1-3] We mostly compute multiple metrics for each dataset

each time only single metric is used for one dataset, and no statistical variance analysis was given.

It is not accurate to state that "each time only single metric is used for one dataset". In fact, we computed both FID and IS for CIFAR-10 and ImageNet experiments. For the OpenWebText experiment, we computed Gen. PPL for unconditional generation, and three different metrics (Gen. PPL, Self-BLEU, and MAUVE) for conditionally generated texts with prompts from the WebText dataset.

Regarding statistical variance analysis, we provided variance values for the latency analysis in Table 2. While it is ideal to retrain and resample everything 5-10 times, the large number of training checkpoints and experimental data points makes it common practice in the literature to report numbers from a single run (as seen in most references cited in this reply).

[1-4] We compare our method with SDTT distillation

most important, the method is not compared against other distilling methods.

While there are few distillation methods for discrete diffusion available (except for some concurrent works), we did compare our method with one round of distillation by SDTT [Des25] in the language modeling experiment (Figure 6). Specifically,"sdtt-7" is obtained after one round of SDTT distillation upon "sdtt-6", while "sdtt-6 + di4c" (or di4c^2) is obtained by Di4C training using the same teacher (sdtt-6). We explicitly compare them in Section 5.3.

References

[Tian24] Tian et al. Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction. NeurIPS 2024.

[Li24] Li et al. Soft mixture denoising: Beyond the expressive bottleneck of diffusion models. ICLR 2024.

[Kar22] Karras et al. Elucidating the design space of diffusion-based generative models. NeurIPS 2022.

[Son23] Song et al. Consistency models. ICML 2023.

[Des25] Deschenaux and Gulcehre. Beyond Autoregression: Fast LLMs via Self-Distillation Through Time. ICLR 2025.