Thank you for your thorough and thoughtful review and suggestions! In our new draft, we incorporated your suggestions for clearer exposition, more experimental baselines, and connecting our method to flow matching. We answer your four questions while describing how we incorporated your suggestions below.

(W1) I am not convinced by the primary hypothesis outlined in the abstract that says "...we propose that masking diffusion dominates due to knowledge of when corruptions occur." Unless I am misunderstanding something, masking diffusion models have no knowledge of when an event occurs, only that an event occurred. To put a different way, the identity of the state (masked or not) tells us whether it has been noised. However, we have no idea when the noising occurred. This seems like an important point for motivating the use of noise schedules as input.

(Q1) Can the authors clarify my point about masking diffusion models not having knowledge of when corruptions occur?

Indeed, in Eqn. 2, the schedule conditioned model is shown to predict $\mathrm{prev}(x^d)$ using $S$, the set of transition times. Masking diffusion only sees whether a token has been corrupted but not the full set of times $S$, so how can it be a scheduled conditioned model?

Note in Eqn 4, the schedule conditioned model actually only makes its predictions using $s\_t$ the number, but not the times, of transitions up to time $t$. We only pass the number of transitions as this is a sufficient statistic for $S$ – the transition times do not help us better predict the previous token; this is proven in Prop. 4.2 . This is also the case for masking diffusion, which also doesn’t need to take in the exact transition times. Notice we proved this formally in Sec. 5.1. Here we further show that an indicator of whether $s\_t$ is larger than $0$ is a sufficient statistic – this sufficient statistic is the mask.

Another way to convince oneself that masking diffusion has baked in the known distribution of transitions is by noting that if we were to recreate the plots in Fig. 3 for a masking diffusion model, we would see no error. During the backwards process in masking, tokens are denoised according to their probability of being noised in the forward process regardless of the prediction $\tilde x\_0$; therefore the noising rate forward and backward are identical. This is not true for uniform for example, where errors in predicting $\tilde x\_0$ can speed up or slow down transitions in the backwards process.

(W2) The primary weakness for me was the experimental results. In particular, the paper seems to be lacking several key details about the experiments. I will reiterate this in the "Questions" section, but I did not see any explanation for why the authors chose to model Cifar10 with both 128 and 256 pixel values. Based on the lack of explanation, I am left feeling that the B=128 experiments were done simply because this was a setting where the authors found they can get Gaussian SCUD to improve over Masking. However, to me, this makes the results appear cherry-picked. Furthermore, in Figure 2, it appears Gaussian is already outperforming Masking, without SCUD (although perhaps not with statistical significance), diminishing the novelty of the result. There should be some commentary on this since.

(Q2) Why were experiments done with B=128 and B=256? Was is due to the fact that only for B=128 did SCUD outperform masking?

For the CIFAR10 experiments, we used 128 only because we had to run a large number of experiments and replicates for Fig. 2 and we were limited by compute. We didn’t anticipate results to change substantially at full scale.

To confirm this intuition, in the new draft, we replicate the result with 256 in table 1 (we summarize the result in the global response). Note 1) with B=128 Gaussian classical diffusion performs similarly to masking; with B=256 however masking diffusion outperforms Gaussian classical diffusion; and 2) in this case, as with B=128, SCUD beats both masking and diffusion.

Thank you for your thoughtful review! In our new draft, we’ve incorporated your suggestions to make the paper more easily readable. We describe how we’ve incorporated each of your suggestions below.

(Q1) Could the authors clarify the role of delta t in discrete Markov process? This is not clearly defined in the “Background” section, nor are there references provided for further reading.

Every time-homogenous Markov process in discrete space can be simulated by what’s known as the “Gillespe algorithm”. This algorithm chooses when to jump and where to jump by sampling exponential random variables. Simply put:

While true:
    For each b ≠ x_t:
        Sample Δt_b ~ Exp(L_{x_t, b})
    Δt = min_b(Δt_b)
    x_{t+Δt} = argmin_b(Δt_b) 
    t = t + Δt

We’ve made the description of the process clearer and added a citation to the Gillespe algorithm to Sec. 2.

​(W1) Lack of Motivation Explanation: The authors seem to focus heavily on explaining how SCUD works, with less emphasis on why this approach is chosen and what its ultimate goal is. This may make it difficult for readers to follow the authors’ line of reasoning.

To make this roadmap and motivation more clear, we’ve added a paraphrase of the outline below at the end of the introduction, the last paragraphs of Sec. 3, and the first paragraph of Sec 4.

In the current paper, Sec. 3 introduces the issue we are trying to solve – discrete diffusion models cannot easily learn the transition schedule – and we quantify the problem in Eqn. 2. We note Eqn 2 suggests a method that solves this problem and goes on to develop this method – SCUD – rigorously in Sec. 4. The main challenge we want to solve is the intractability of $p((x\_t)|x\_1, S)$ and Sec. 4.1 solves this by introducing the concept of events rather than transitions. Sec. 5-7 are dedicated to comparing SCUD to previous methods.

(W2) Over-Reliance on Appendix for Proofs: Many key proofs and details are placed in the appendix, which disrupts the main text’s coherence and could interfere with readers’ logical understanding.

(Q2) Would it be possible to simplify some of the proofs to make them more accessible in the main text?

We appreciate your suggestion to better integrate our results in the main text.

Of the proofs and details we’ve left to the appendix all either describe minor modifications of standard techniques described elsewhere which we’ve only included for completeness (details in Sec. 4.3), are lengthy but standard algebra (Prop. 3.1, 4.2, 4.3), or require the introduction of formal mathematical techniques and definitions not necessary for understanding the main ideas of the paper (Prop 4.1, 4.4, 5.1, 5.2). However, we’ve added hyperlinks from every result to its proof and from the proof back to the result, and we’ve added an intuitive description of the proof of every result in the paragraph preceding.

​(W3) Inadequate Training/Sampling Procedure Description: The description of the training and sampling processes is not sufficiently detailed, making it difficult to understand SCUD’s training and sampling mechanics without referring to supplementary materials or external resources.

(Q3) Why didn't provide the complete training and inference process in algorithmic form in the paper?

While details to replicate our training and sampling are available in the text and code of the paper, we’ve made these clearer so that a reader skimming the paper can immediately identify how to train and sample. We’ve also included both sampling and loss calculations as algorithms.

In the current paper, section 4.3 describes how to sample; As described in this section, while the procedure is simple, it uses some extra notation that, while standard (for example, see Campbell et al., 2022 and Zhao et al. 2024), are not necessary for understanding the paper; we’ve put these details in App B.2 for the interested reader. To App. B.3, by your suggestion, we’ve also added the sampling procedure in algorithmic form.

For inference, as described in Sec. 4.2 under the section “Efficient loss”, we perform minibatch gradient descent on the loss in Eqn. 6; we evaluate this quantity using the identities in Sec. 4.3. By your suggestion, we’ve highlighted Eqn. 6 as the loss we are optimizing and added the procedure to calculate the ELBO in algorithmic form in App B.1.

Conclusion

We hope you will look at our global response and updated draft, which contain new experimental results, and we ask that you consider raising your score if your major concerns have been addressed. If you have any remaining questions, we hope you will ask them so that we can offer clarifications.

Thank you for your thorough and thoughtful review! In our new draft, we incorporated your suggestions, 1) arguing and demonstrating that SCUD does not add significant computational overhead, 2) demonstrating in new experiments that SCUD can generate high quality images, and that SCUD is not sensitive to the choice of the rate parameter $\gamma$. We answer your questions while describing how we incorporate your suggestions below.

(W1) The text of the main paper exceeds the 10-page limit. Please move your remarks regarding Reproducibility to the appendix, ICLR will likely be strict about enforcing the 10-page limit, exceeding the limit may lead to your work being rejected down the line.

We strongly appreciate your giving us a chance to fix such a potentially large formatting mistake! Our placement of the Reproducibility statement was informed by the following instructions from the ICLR author’s guide: “Authors are strongly encouraged to include a paragraph-long Reproducibility Statement at the end of the main text (before references) [sic] ... “ and “This optional reproducibility statement will not count toward the page limit, but should not be more than 1 page.”. Checking other papers, they seem to have the reproducibility statement in a similar place. Again thank you for bringing this to our attention!

(W2) The paper's complexity could limit practical adoption. SCUD requires intricate parameterizations and careful handling of components like the rate parameter and the transition matrix in particular, potentially increasing computational cost.

(W3) While SCUD reduces training samples, the discussion on costs associated with complex matrix operations, schedule conditioning, and increased dimensionality in high-dimensional data is minimal.

(Q1) Can you provide O-Notation w.r.t. the data-dimensionality for the added computational cost arising from ? Furthermore mentioning the GPU hours of the different methods in your work could help put the computational cost of different methods into perspective.

(Q2) In addition to 1.: Could you provide a quantitative comparison of the computational costs associated with SCUD versus standard discrete diffusion methods in terms of memory usage and processing time?

To App. D.5 of the new draft we have added the below thorough discussion of considerations for 1) parameterization, 2) computational complexity from matrix computations and using $S$. We have included descriptions in big-O notation. We describe the strategies we used so that SCUD has minimal theoretical additional overhead for parameterization and computation compared to classical discrete diffusion, and in some cases allows new strategies for more efficient computation.

Additionally, we’ve also added the GPU hours required to run SCUD and our implementations of masking and classical discrete diffusion to App. D.2,3,4 (all large models were trained for roughly 2 days each on 2 A100 GPUs on an academic cluster). In practice, we use virtually identical amounts of compute for our implementations of masking, SCUD, and classical discrete diffusion (maximum difference was classical diffusion was 20% faster than SCUD on images) and our GPU hours are comparable to previous methods with differences likely due to differences in hardware and implementation.

1. Parameterization:

To specify a SCUD model, one needs to specify (1) the parameterization of the predictor $\tilde x_{0, \theta}$, (2) the rate function $\beta_t$, (3) a transition matrix $K$, (4) the amount to condition on the transition schedule $\gamma$. This is actually similar to what one must specify for classical discrete diffusion models: these need the parameterization and function (1) and (2); and, similar to (3), they must specify the infinitesimal generator $\mathcal L$. The major difference with SCUD is that $\gamma$ is set to $0$. Note below we show that SCUD is not very sensitive to the choice of $\gamma$, so there is little extra complexity involved in specifying a SCUD model compared to a classical discrete diffusion model.

Thank you for your thorough and thoughtful review! We have incorporated your writing and experimental advice into the new draft. We respond to your individual suggestions below.

(W1) The formal notion of event schedule is only introduced in Proposition 4.2. This should instead be moved to before Proposition 3.1, so the readers already know what the event schedule captures, and the corresponding equations become easier to follow.

We have moved this definition to the sentence before Prop. 3.1.

(W2) There are claims in the paper regarding SCUD outperforming masking, but evidence of this is not visible in the experiments. It is unclear whether the SCUD conditioning with Gaussian / Uniform will outperforming existing discrete diffusion (e.g. D3PM) with masking, given equal compute.

(Q1) In light of the point 2 in weaknesses, could the authors run the D3PM model with masking diffusion with the same number of training samples as SCUD? -- The reviewer hopes this is feasible within the available compute budget of the authors. An alternative would be to also consider small versions of D3PM and SCUD that can be trained for equivalent number of samples.

In our paper we claim that SCUD generalized masking diffusion and this generalization allows it to incorporate inductive biases in the forward process that improve model performance. We showed mathematically that SCUD with a uniform forward process with $\gamma=1-1/B$ is masking. We showed explicitly in Fig. 2 that SCUD outperforms masking. In the rest of the experiments, we compared SCUD with structured forward processes with SCUD with a uniform forward process; this later model we argued was similar to masking, which we see evidence of in Fig. 2.

To properly justify our claims comparing SCUD and masking diffusion, in new results on images and proteins in the global response, we run our results for longer and replace the SCUD uniform baseline with explicit masking diffusion as in Fig. 2. We indeed see that SCUD with structured forward processes outperforms our reimplemented masking diffusion.

(Q2) How is B defined in the language and protein experiments?

For the protein experiments, $B$ is defined as the number of amino acids $20$ plus $11$ special tokens that appear in the data representing ambiguous amino acids. For language, $B$ is the number of text tokens, which is 30522 for the BERT tokenizer. We have added this into the description of experiments in Sec. 7.

We thank all reviewers for a productive review session! SCUD makes an important theoretical contribution to discrete diffusion models by explaining the dominance of masking diffusion. In addition, we use this insight to build models that outperform masking in practice. In experiments we showed that by combining schedule conditioning and a structured forward process, SCUD outperforms 1) our careful implementations of classical discrete diffusion and masking diffusion, and 2) previously reported numbers for classical discrete diffusion methods. Previously reported numbers for masking diffusion came from models that were trained for weeks or months of GPU time which we could not achieve.

Reviewers suggested that it would strengthen the paper to beat these numbers as well. Therefore, throughout this review session we have run our methods for much longer. Tab 1, 2 show that when trained for sufficiently long SCUD indeed outperforms all previously trained models as we hypothesized. FigS2 also shows that the sample quality of SCUD again greatly improves with longer training, now containing recognizable vehicles and animals. With this new set of results, SCUD achieves state-of-the-art likelihoods among discrete diffusion models on two modalities.

Table 1

Table 2