Democratized Diffusion Language Model

Thank you for your review. While performing an early exit is a well-established approach in Deep Learning, performing it for text diffusion was not previously explored. Furthermore, since the generation is performed in categorical space, we were able to study novel exit methods for diffusion early exiting.

• From Table 1 main results, we can see the choice of steps also provides very marginal impact to the final performance. It might be beneficial for the research to delve deeper into how this method stands out from or builds upon existing techniques in the field of diffusion models.

This is not actually correct. As seen with the Unconditional generation setup, reducing the total number of steps could lead to a 0.2-0.3 drop in the AR-NLL metric, which is not marginal for fixed dist-N values. Furthermore, based on our experiments, early exiting could be used in two scenarios – when we want to achieve a better quality of samples with a given number of generation steps or when we want to reduce the number of generation steps for a given quality of samples. Thus studying number of generation steps is orthogonal to early exiting methods.

With our paper, we aimed to understand the behavior of DLMs, which is different from continuous diffusion models. We established new sampling methods and provided insights into reasons for such behavior. Understanding this behavior makes it possible to develop connections to other models and approaches, but without precisely studied groundings, it is impossible.

• LLM as Judge has some known weakness, from the provided prompts [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena], it seems the prompt is very simple and does not consider LLM bias. Can author comment on this?

Our task is significantly easier than usually accessed with LLM as a Judge. We do not want to select the best answer across a set of responses (e.g., alignment side-by-side setup). Instead, we used LLM to compare a generation from a specific step with the final generation (grammar, similar words, etc.). We observed that LLM provided a valuable estimation of such differences.

Thank you for your review!

• The innovation of the paper is insufficient. The main contribution is to reproduce the CDCD structure and analyze the existing DLMs, without proposing new models or methods.

The main aim of our paper is to understand the generation process of DLMs. With novel methodology in this direction, we observed that existing DLMs allow performing early exiting. We proposed several methods for performing early exiting (See Sections 4.2, 4.3). This work proposes faster sampling algorithms for DLMs and also lightens future research directions to further understanding and analysis of DLMs, which could lead to more new methods.

• The length of the trained model is limited to 64, and it is not clear whether there will be different conclusions for longer lengths. The length of 64 is still a bit far from actual application.

We will add experiments with longer sequences to the rebuttal revision soon. Please consider these new experiments when reevaluating your score.

• We still care about the performance of pre-trained models on downstream tasks, and the paper did not select some downstream tasks for evaluation.

Please note that we do not propose new DLMs with our paper. Instead, we studied sampling algorithms for existing models. These sampling algorithms are not connected to downstream task performance.

• (1) The main contribution of the paper, such as the analysis of DLMs, is not given in the title. The writing style is a bit messy.

We desired to highlight our trained DDLM model. Also, we wanted to have a connection with early exiting algorithms that allow faster sampling, thus making DLMs closer to practical usage by other people

• (2) It is recommended to add the model parameter quantity to the comparison in Table 1.

Thank you for this proposal; we have added the number of model parameters in the latest revision.

• Does the model have the ability to output the </s> token? The sentences in Appendix D are all truncated results.

Sentences in the Appendix D section were sampled with a sequence length of 64. Thus, they are truncated. All used in our experiments model could produce </s> tokens. For example, it is possible to set it as a condition at any specific point within the sequence, or the model itself could generate it for long enough sequences.

Thank you for the clarification. But my concerns are merely addressed.

(1) I still harbor doubts regarding the accuracy of the title, and the innovativeness of the paper may not meet the standards set by ICLR.

(2) Despite the authors' claim of not proposing a new DLM, the downstream tasks remain a subject worthy of investigation.

(3) Experiments with longer text have yet to be provided.

Thank you for your review!

• It is good to see analysis of sampling between different diffusion models, however, no further explanation about the deep reason to cause these differences.

Differences between different diffusion models are specific to different training objectives and could be caused by noise scheduling. Experiments with varying noise scales ground this hypothesis. However, we did not dig into this and focused on proposing and understanding sampling methods for different DLMs and establishing an experimental setup for such research direction. Understanding various ways to perform an early exit for various models first could provide more evidence and grounding for further research on this topic rather than making solid assumptions without proper evaluation.

• The advantage of DDLM is to early exit and speedup the generation process. However, compared with some faster ODE solvers (e.g. DPM-solver[1]), early exit of DDLM maybe not superior than them.

First, let us clarify that DPM-solver is an second-order method, which implies two model’s forward passes during one step. Also note that DPM-solver is very similar to Heun sampler from [1]. In our experiments this type of samplers do not outperform Euler sampler with similar number of forward passes.

[1] https://arxiv.org/abs/2206.00364

• Early exit leads to the downgrade of generation diversity.

We did not observe such behavior during our experiments. We hypothesize that you refer to Table 2 with experiments on noise scaling to allow faster convergence of the sampling algorithm. Lower diversity is expected since trimming the noise scale leads to “more deterministic” sampling. We will add plots with sampling diversity akin to Figure 5 in Section 4.3 to prove that early exiting does not reduce samples diversity in the rebuttal revision. Please refer to it for this weak point when reevaluating your score.

• Why after your findings that using a noise scale of 0.9 is optimal, you still use a scale of 1.0 in later experiments

The purpose of experiments with noise scaling is to understand the underlying behavior of DLMs, leading to emerging capabilities to perform an early exiting. However, we observed that performing generation with a slightly reduced noise scale without hurting AR-NLL and Dist values, such a setup is not conventional. In our case, this finding could provide insights on early exiting for DLMs for future works, but performing experiments in such configuration for other models is not conventional.

• In table 2, it is weird AR-NLL=0.44 and dist_1=0 with noise=0, do you have generation examples?

This is caused by AR-NLL metric issues since it is evaluated with external LM. It is conventional to obtain small values for repetitive samplings (e.g., one word repeated 64 times). Because of the issues with this metric, we also incorporated other metrics, such as a number of distinct tokens and self-bleu. E.g. Table 2 shows that the Dist value reached 0, indicating that AR-NLL does not provide helpful information.

• The GPT-Score is a relative value, with step-1000 as the reference text. However, the reference text may not be fluent. Is it possible to obtain the absolute value?

It is possible, although we observed that absolute value without side-by-side comparison with GPT-4 had a large variation. Performing a side-by-side comparison is established by other works (e.g., in Direct Preference Optimization), and to the best of our knowledge, no works used absolute evaluation with GPT-4 (doing so is an exciting research idea itself). In our case, we used it to understand the convergence of sampling algorithms. Even though the final sampling could not be fluent, this sampling is the one that is obtained with the original models as is. So, this experiment shows that early exiting samples after a specific step do not differ from the final sampling (see Appendix Section D).

• Can you also compare with discrete diffusion models?

We are unaware of discrete diffusion models capable of generating unconditional text. Existing unconditional general-purpose text generation models are continuous, so we performed experiments with them.

• Why in Fig5, the NLL of Plaid (c) (<3.6) is lower than the DDLM (a) (>3.68)? This contradicts to the main table.

For Figure 5, we used 1000 samples to evaluate the models, while the main table used 5000 samples. This led to a slight variation in AR-NLL values. We added an explanation to the latest revision.