Constrained Discrete Diffusion

We thank the reviewer for their insights, particularly their acknowledgment of our robust experimental results as well as the timeliness of our work. Below, we provide our answers to their questions and comments.

1. The notation used in Sections 3 and 4 was confusing in my opinion.

We are happy to clarify. The variables represent vectors in the interior of the simplex at higher noise levels. As $t \rightarrow 0$, these vectors transition to one-hots. For instance, when MDLM unmasks a token, it transitions to a one-hot; the masking probability decreases with $t$, so the probability of a given position being a one hot changes as dictated by the masking schedule.

We can expand further on the notation in the final version of the paper and we are certainly also receptive to the reviewer's suggestions they may provide during rebuttal.

2. It would be important to add entropy to tables in Figure 3 and Table 1 to ensure the generative perplexity value is not being “gamed”, an issue several works have noted with this metric, e.g., Zheng et al [2].

Excellent point. To check whether the true generative perplexity may be "gamed" we have ran some additional experiments during rebuttal and include an LLM-as-a-Judge as an evaluation metrics. Interestingly, it shows no degradation in generation quality.

In the table above, we have show results for further experimentation with the entropy-based evaluation as described in [A] and report that CDD preserves predictive entropy relative to its base diffusion model (MDLM in this case) with at most a ∼ 1% decrease in average entropy. This demonstrates that CDD preserves diversity even while enforcing toxicity constraints. The last row of the table reports yet another model we ran during rebuttal, the baseline of PPLM for Autoregressive models, which sees a much larger decrease in token diversity!

We are happy to include and extend these results in the final version of the paper.

3. In Appendix A, the authors note that there are no open-source large scale discrete diffusion models, however there have been several released recently, specifically Llada [3] and Dream [4] which are both 7-8B parameter models

We are quite excited to see the significant progress that researchers have recently made in the development of discrete diffusion for language generation! Indeed, we will modify this statement to indicate that no open-source large scale discrete diffusion models were available at the time of development. In fact, Dream [A] was released in April, and Llada [B] appeared in late February, with the code being released several weeks later.

We are excited to extend CDD to these larger architectures in our future work, and we are currently actively working on it.

3. Along those lines, the drop in quality (as measured by PPL) in the table in Figure 3 between MDLM and CDD is quite stark.

While we acknowledge that CDD does incur a perplexity decrease relative to the base MDLM, there are many scenarios where this as an acceptable trade-off for strict constraint satisfaction, especially in safety-critical settings where any violation is unacceptable. Furthermore, controllable generation baselines like PPLM and FUDGE demonstrate an even more substantial perplexity degradation while still resulting in constraint violations.

As shown in Response (2), while other controllable generation methods may be "gaming" the perplexity metric, CDD certainly is not. Thus, (and as you also acknowledged) the perplexity metric is not a perfect gauge of the model's true performance. As result we further use LLM-as-a-Judge as a quality metric in which CDD actually benefits with a minimal improvement, while PPLM and FUDGE report large degradation. Additionally, when measuring entropy, another important quality metric, we report negligible decrease in token diversity.

4. On Line 152, the citation format for “Xu et al” seems to be inconsistent with the “[xx]” used throughout the rest of the manuscript. (Same issue on line 843) | Line 297 is missing the word “Appendix” before “Appendix C”. | There is a typo on line 335, “scientifi tasks” should be “scientific tasks”.

Thank you very much! We have updated our paper with these fixes.

We hope the clarifications above resolve the remaining questions about our contributions, and we are happy to elaborate further as needed. Thank you!

[A] Jiacheng, Ye et al. “Dream 7B”

[B] Nie, Shen, et al. "Large language diffusion models." arXiv preprint arXiv:2502.09992 (2025).

[C] Sahoo, Subham, et al. "Simple and effective masked diffusion language models." Advances in Neural Information Processing Systems 37 (2024): 130136-130184.

[D] Lou, Aaron, Chenlin Meng, and Stefano Ermon. "Discrete diffusion modeling by estimating the ratios of the data distribution." arXiv preprint arXiv:2310.16834 (2023).

Thank you for the additional clarifications.

On this point:

Although our custom projection module accurately recovers the ground-truth counts, it does this without access to the labels

So the counting module is an external (differentiable?) classifier that takes in the sequence and returns the count of the letters?

Thank you for the comments. We respond to your questions below and welcome further discussion.

1. Undefined terms: Equation (5) appears central to understanding the proposed method. However, several critical terms in the equation such as $\mathbf{x}'$, $**y**$, and $t(\ell)$ are not defined.

This is something that might have been overlooked. All the symbols are defined in the paper, e.g., $x_{t(\ell)}'$ is defined in Equation (3), and $\mathbf{y}$ is defined in line 177, it's the minimizer variable in the $\arg\min$ (the projected sequence).

2. The paper relies on concrete score matching as the core denoising framework, but it is not explicitly described and justified. In contrast, the diffusion-based method using $\mathbf{x_0}$ prediction is explicitly explained. Why not use a more common $\mathbf{x_0}$-prediction-based method?

Please notice that we describe and justify the use of score matching in the first paragraph of Section 4.

To elaborate further, projected diffusion sampling processes have recently emerged for constrained generation within continuous domains. These approaches have adopted score-based Langevin samplers due to the theoretical perspective that is enabled by this sampling approach. Our work extends this to discrete domains, offering both SOTA empirical results and novel strong theoretical guarantees. Concrete score matching, in particular, enables presenting the theoretical results in a clear way.

However, we'd like to stress that $\mathbf{x_0}$ predictive models implicitly capture concrete score matching! We'd encourage the reviewer to check Appendix C of [D], where the connection between MDLM (the model we employ) and concrete score matching is explicitly drawn.

Also note that our experimental validation adopts $\mathbf{x_0}$-prediction-based generation. CDD is parameterization-agnostic, and the projection step only requires the per-step categorical token distributions, which are available under both $\mathbf{x_0}$-prediction and Concrete Score Matching parameterizations. MDLM uses $\mathbf{x_0}$-prediction on masked positions while previously unmasked positions remain fixed, resulting in the distributions that CDD projects onto with a KL-based augmented-Lagrangian optimization.

3. The $\beta$ -prox-regularity assumption in Theorem 4.1 is used without definition or intuition, only a reference. Readers unfamiliar with this property are left without the tools to evaluate its implications.

The definition of $\beta$-prox regularity is from [C] (Def. 13.27) as specified already in the statement of Theorem 4.1. This is a common notion in convex optimization theory, but we will be happy to add additional details in the final version of the paper. In short, this condition is a relaxation of typical convexity assumptions; it implies that for each viable point and normal direction, small perturbations still project uniquely and smoothly back to $\mathbf{C}$. This is used instead of a convexity assumption, as it allows us to broaden the applicability of the theory to constraint sets that are not strictly convex.

4. Unspecified experimental choices: The noise distribution $\nu$ used in experiments is never mentioned. How the choice of noise affect the model's behavior (and which noise choice are even possible within the method is not discussed)

This might have been overlooked by the reviewer. Please note that the reference distribution $\nu$ is the fixed categorical corruption used in the forward process as discussed in lines~115--121. As noted we discuss two standard choices: (i) mask corruption, where $\nu$ is a one-hot on [MASK] as done in MDLM [A] (better for larger vocabularies such as natural language); and (ii) uniform corruption, where $\nu$ is uniform over the vocabulary as in UDLM [B] (better for smaller vocabularies such as SMILES).

In our experiments, natural-language tasks use the corruption choice introduced in MDLM, in which $\nu$ is the one-hot [MASK], while molecule generation uses the corruption introduced in UDLM where $\nu$ is uniform. CDD’s augmented-Lagrangian projection operates on the token-probability vectors returned by the reverse sampler after each denoising step and is therefore compatible with any fixed $\nu$ the base diffusion model is trained with.

5. Missing Discussion of Connections to RL. The paper does not position the work in the context of related RL-based constraint enforcement methods. There is no comparative discussion or evaluation against RL baselines in either the related work section or experiments.

RL-methods differ fundamentally from our approach in both goal and methodology:

1. Our method enforces hard, per-sample constraints at inference time in discrete diffusion, without requiring retraining. In contrast, RL methods operate during training and optimize expected reward across samples and do not offer per-sample control at test time.

2. Our approach projects each diffusion step back onto the feasible set, providing theoretical guarantees and ensuring zero constraint violations in our experiments. RL-based methods rely on policy optimization and do not provide such guarantees during sampling.

CDD intervenes in the sampling process, and the augmented Lagrangian method enforces constraints by projecting the sample onto the constraint set, rather than implicitly through reward maximization. Recall also that CDD achieves zero violations in our experiments without retraining.

Because of these fundamental mechanisms and objective mismatch, RL-based approaches are deemed outside the scope of this work. We will be happy to add a discussion.

6. The proposed method is not applied to maximize validity [...] this raises concerns that the method may not be effective for enforcing structural validity. At least, the authors should discuss this point.

Our framework can handle arbitrary constraints, so we see this as an exciting opportunity. However, this paper did not focus solely on synthetic chemistry experiments, but prioritized a breadth of applications, as pointed out also positively by other reviewers.

Please notice our evaluation indeed included validity and shows that our method provides state-of-the-art results for generating valid and novel molecules even if validity is not enforced as a constraint. In our evaluation, we outperform the prior SOTA dramatically, reporting a 48% increase in valid and novel generations. This increase is even more substantial when conditioning on drug-likeliness, producing more than three times valid and novel samples as compared baselines.

To show how CCD could handle additional constraints we ran additional experiments during rebuttal in which we extended the molecule generation setting to handle an instability related constraint. Instability undermines potency and developability. The structural properties of 'three‑membered heterocycles' has been shown to exacerbate instability and reactivity [E]. For this application, we use the RDKit library [F] (the same library we use to check for chemical correctness) to flag molecules violating this structural property and implement a projection to ensure that generated molecules do not contain this structure. The projection operator consists of multiple structural editing stems. First, it expands the ring by inserting an atom so it becomes a 4- or 5-membered ring; if that isn’t possible, it then opens the ring by breaking one bond so the fragment is linear. If still flagged, it then removes the entire three-membered ring and directly reconnects the substituents.

Empirically we report $0\%$ violation rate in the molecules generated with the CDD method while all the baselines report large violations.

These results show how important this technique can become for applications such as synthetic chemistry and beyond.

7. The proposed method includes an optimization step at each denoising step, which suggests a potentially significant computational overhead. This limitation is not discussed in the paper.

This limitation is acknowledged and discussed in detail in the paper (limitation section, deferred to Appendix A). We also encourage the reviewer to check Appendix E, where we analyze and report the runtime. Here, we find that CDD is orders of magnitude faster than other controllable generation methods.

Furthermore, analysis in Appendix D explores several hyperparameters that can be adjusted to control the tradeoff between quality and runtime as seen in Table 3, Figure 8, and Figure 9. Also, we would like to highlight Table 4 where we compare the overhead with the baselines and observe that CDD provides the least additional overhead as compared to other controllable generation baselines.

Finally, in many settings, such as scientific tasks, generating feasible and physically plausible material largely outweigh latency considerations, given that physically unreliable content might not be even synthesizable.

Thank you for your feedback. We hope our response addressed all your concerns; We also provided new results and noted missed items. Given your comments that these would strengthen the work, we hope that you could re-evaluate it, in light of our rebuttal.

[A] Simple and effective masked diffusion language models

[B] Simple guidance mechanisms for discrete diffusion models

[C] Variational analysis

[D] Constrained synthesis with projected diffusion models

[E] Flavin-enabled reductive and oxidative epoxide ring opening reactions

[F] G. Landrum et al. RDKit 2024.09.5 (Q3 2024) release. Zenodo, 2025

[G] Grammars and reinforcement learning for molecule optimization

Thank you, again, for your feedback during the response period. We believe that our response has addressed your remaining concerns, but would like to inquire whether any outstanding questions remain.

Again, we appreciate the constructive feedback provided and assuredly will include these organizational revisions in a camera ready version of our paper. Thank you for your time and consideration of our work!

We thank the reviewer for their insights, and in particular the acknowledgement of the strong empirical results and well motivated theoretical support. We provide our answers to their questions and comments.

1. The distinction between ULDM and MDLM should be discussed in greater detail, especially in the context the limitations of available and design of constraints. [...]

Thank you for the opportunity to clarify the use of UDLM [A] and MDLM [B] and their interactions with constraints. MDLM and UDLM share the same discrete-diffusion backbone, in which they corrupt a clean sample with a schedule and then iteratively denoise to learn the target distribution. However, they use different noise kernels; MDLM uses an absorbing $`[MASK]`$ process, where once a token is unmasked it is not further updated, and UDLM uses uniform categorical noise in which any token may be revised at further states. We chose UDLM for the molecular experiments as empirically in [A] UDLM has been shown to perform better for small vocabularies, and MDLM has better performance for larger vocabularies, which covers natural language generation tasks.

Crucially, in both cases the sampler exposes a full $L \times |V|$ probability tensor at every reverse step, and therefore CDD always operates on the whole sequence rather than a partial generations. The reviewer's point that some constraints are enforceable on partial generations (e.g., counting or lexical constraints) is insightful, but in many application we need global consistency. This cannot be achieved with the level of control provided with current autoregressive architectures. This is highlighted well by our molecule generation experiments, where autoregressive models report the poorest performance in generating molecules which are both novel and valid. This occurs as the models often reach intermediate states where no valid and novel continuations remain. Consequently, they become "stuck," and either reproduce known samples (violating the novelty constraint) or generate invalid structures (violating molecule validity).

Even in Section 5.3, where it may be possible to enforce constraints on partial generations, we find that there are distinct benefits to imposing the constraints in a non-autoregressive paradigm, as evidenced by better PPL and LLM-as-a-Judge scores. Specifically, lexical constraints benefitted from non-autoregressive generation as tokens both preceding and following the constrained positions can be simultaneously adjusted, a flexibility unavailable to strictly autoregressive models.

As a final note, the LLM-as-a-Judge is evaluation-only on the post-generation. During sampling we use a differentiable surrogate models (or other differentiable constraint functions) to evaluate the entire sequences, not partial generations. After applying the Gumbel SoftMax, we then embed these distributions directly into autoregressive LLM-based surrogates (using the same tokenizer as the base model), creating a continuous, fully differentiable computational graph. In scenarios involving hard constraints, we leverage hand-coded rule-based constraint functions, enabling us to avoid relaxation and embedding, and directly evaluate the argmax-decoded sequences. More details on these implementations are provided in Appendix C.

We thank you for bringing up this point and will add a concise subsection clarifying these points and explicitly justify the base model choice per task.

2. Does the number of steps used for diffusion decoding make a significant difference in the application of constraints? Do you expect this method to work with low NFE methods such as consistency-distilled discrete diffusion models?

While the number of denoising steps effects efficiency and sample quality, it does not prevent CDD from enforcing constraints. As the projection operates directly on token‑level probability outputs, CDD is fully compatible with low‑NFE samplers, including consistency‑distilled discrete diffusion models, without architectural changes.

The performance on such methods could be extrapolated from our analysis in Appendix D, where we analyze the effects of reducing the number of projections (by taking more diffusion steps between each projection). The only difference would be that low‑NFE samplers would take a single large step between projections, whereas in our analysis many small steps were taken between projections. We observed that performance only degraded slightly when a higher number of steps occurred between projections, and, based on this result, we believe that this suggestion could be a promising area to explore in future work as consistency-distilled discrete diffusion models become more readily available. Thank you for pointing it out!

3. As the paper explicitly requires the constraints to be defined and evaluated at runtime, a number of other baselines, such as standard decoding + rejection sampling for the constraint should be considered. This may further highlight the benefits of the constrained diffusion approach, as the number of samples required could lead to drastically longer runtimes for AR models or diffusion models without explicit sampling constraints.

Indeed, rejection sampling is implicitly included as one of our baselines. In all experiments we report the percentage of samples which violate the constraint set. These samples can be treated as those which would have been 'rejected'. We'd be happy to highlight this point in the main text.

4. Could Constrained Discrete Diffusion be used to edit already completed generations (e.g. from the output of a AR model)?

This is an interesting suggestion and could be an intriguing application of CDD. In fact, our research team has already been exploring this idea for continuous diffusion models; in such settings, we found that there are some significant benefits that can be realized, which very likely extend to discrete settings as well. In short, CDD could be applied in this way by initializing the input sequence with an existing sequence. Deviation from the original sequence could be controlled by tuning the masking schedule (and how many diffusion steps to take). However, if the original output is quite far from the constraint set (e.g., imposing toxicity constraints on a very harsh sequence), this may result in a largely changed sequence. Thank you again for this suggestion!

5. Could the be a larger discussion about “constraint start ablation” and associated runtimes in the main body of the paper. Greater signposting in Appendix D could be helpful. Related, it may be interesting to explore the extent to which late-starting of constraints could be applied and the minimum number of steps required to enforce constraints. This may heavily differ between MDLM and UDLM but would make for a different level of usability.

We appreciate this suggestion. It adds to the clarity and we will implement these suggestions on the updated manuscript. To the latter point, note also that the question of late-starting of constraints and the minimum number of steps required to enforce constraints is a result we have characterized theoretically (Theorem 4.1). We are happy to pair this explanation with the empirical analysis conducted in Appendix D in the updated version of our paper.

We thank you for your suggestion and are happy to include your suggestions in the final version and to engage further if there are any additional questions. Thank you!

[A] Schiff, Yair, et al. "Simple guidance mechanisms for discrete diffusion models." arXiv preprint arXiv:2412.10193 (2024).

[B] Sahoo, Subham, et al. "Simple and effective masked diffusion language models." Advances in Neural Information Processing Systems 37 (2024): 130136-130184.

We thank the reviewer for their comments, and, in particular, the acknowledgement of the rigorous theoretical grounding and novelty of our work. Below we provide our answers to their questions.

1. Since $**C**$ represents the constraint set and $**y**$ denotes the projected probability distributions in Eq.5, the meaning of $\arg\max(**y**) \in **C**$ is quite confusing. What does it mean in the task of natural language toxicity mitigation?

In Equation~(5), the constraint is placed not on the probability tensor itself but on the decoded sequence $**y**^{\star}=\arg\max\bigl(**y**\bigr)$ where $**y**\in\mathbb{R}^{L\times |V|}$ is the projected probability distribution. As greedy decoding picks the highest–probability token at each position, the condition $\arg\max(**y**)\in**C**$ means the decoded sequence must be feasible. Specifically for toxicity control $\mathbf{C}=\lbrace\mathbf{y}\in V^{L} : g(\mathbf{y}^{\ast})\le \tau \rbrace,$ where $g(\cdot)$ is a differentiable toxicity surrogate and $\tau\in[0,1]$ is the threshold. Therefore, in the task of natural language toxicity mitigation, $\arg\max(**y**)\in**C**$ is the set of sequences which are less toxic than $\tau$ as score by the surrogate model. Since $\arg\max$ is non-differentiable, the projection optimizes an augmented–Lagrangian objective using a Gumbel Softmax relaxation $\tilde{\phi}(**y**)$ and penalizes violations of $g\bigl(\tilde{\phi}(**y**)\bigr)\le \tau$. In the reported benchmarks, this procedure yielded $0\%$ violations at the tested thresholds without retraining the base model!

2. Surrogate models are trained on clean data but applied to noisy intermediate samples during diffusion. How does this affect their reliability?

This is an interesting question, which is closely connected to ongoing discourse within the context of classifier-based guidance methods [A-C]. Specifically, recent works exploring "training-free" guidance methods have analyzed how pretrained classifiers, trained exclusively on clean data, could be directly adopted for diffusion model guidance without explicit finetuning on noisy samples. The trade-off here is well established: employing pretrained models directly on noisy inputs significantly reduces implementation complexity, albeit at the cost of slightly reduced performance due to distribution shift. Our adoption of surrogates trained exclusively on clean data can thus be understood within this well-established trade-off; indeed, we observed strong empirical results even without noisy-domain finetuning.

Next, it is important to clarify what "reliability" means in this context. We interpret this as the accuracy with which the surrogate models the constraint set. Let us stress that the combinatorial set of feasible generations defined by our surrogate remains unchanged regardless of the input noise. As the surrogate model defines a closed set of allowable generations, the noise level of the measured sequences does not compromise the integrity of the surrogate's prediction of set membership. At worst, the noisy data may lead to slightly less accurate gradients (as documented by training-free guidance literature), introducing slightly longer runtime for the convergence of the ALM projection. However, as we have theoretically shown (Theorem 4.1), CDD effectively converges towards the training data distribution while simultaneously converging to the constraint set.

3. While Table 4 shows CDD’s runtime overhead, how does the frequency of projection steps impact sample quality?

Thank you for bringing up this important point. Indeed, we examine this in Appendix D, where we ablate the projection frequency over various frequencies $\{1,5,10,20,50,100\}$ and report the average perplexity and runtime in Figure 8. We also include sensitivity studies for Augmented Lagrangian hyperparameters in Table 3 and for the projection start time in Figure 9. Notably, across all settings, constraints remain fully satisfied (0% violations), underscoring the robustness of our method. In practice this method comes with practical knobs for practitioners which allows users to control compute, speed, and generation quality while preserving feasibility.

4. Designing task-specific projection operators adds overhead, limiting plug-and-play usability. The paper could better discuss generalizability to unseen constraints.

We appreciate the reviewer highlighting this point. Indeed, this concern closely relates to the earlier discussion in Response (2) about training-free guidance. In that context, we noted that our approach explicitly avoids reliance on surrogate models pretrained on noisy diffusion outputs. Consequently, training a surrogate model for arbitrary, unseen constraint sets is relatively straightforward, requiring only standard supervised training procedures without specialized diffusion-domain fine-tuning. This is consistent with prior work on attribute-controlled diffusion using surrogate models where constraint-specific loss functions or guidance models are similarly required [D,E]. Thus, the method naturally generalizes to a wide variety of constraints with minimal overhead.

Furthermore, we have also demonstrated that the approach generalizes to other representations of constraint (e.g., symbolic, black-box, neural surrogate), which are treated as modular inputs to the optimization procedure. For a given setting, it is only necessary to devise a differentiable constraint function -- which, again, can take many forms. Therefore, rather than requiring "task-specific projection operators," our method only necessitates defining differentiable "constraint violation functions." We emphasize that defining these constraint functions has a long tradition in optimization and should not be seen as a barrier to generalizability.

We would be glad to explicitly incorporate additional explanation into the paper, emphasizing the distinction between task-specific projection operators (which our method avoids) and differentiable constraint representations.

Thank you for your detailed review! We hope that our response addressed all of your questions. We are also happy to provide any other requested details.

[A] Sadat, Seyedmorteza, et al. "No training, no problem: Rethinking classifier-free guidance for diffusion models." arXiv preprint arXiv:2407.02687 (2024).

[B] Ye, Haotian, et al. "Tfg: Unified training-free guidance for diffusion models." Advances in Neural Information Processing Systems 37 (2024): 22370-22417.

[C] Shen, Yifei, et al. "Understanding and improving training-free loss-based diffusion guidance." Advances in Neural Information Processing Systems 37 (2024): 108974-109002.

[D] Schiff, Yair, et al. "Simple guidance mechanisms for discrete diffusion models." arXiv preprint arXiv:2412.10193 (2024).

[E] Gruver, Nate, et al. "Protein design with guided discrete diffusion." Advances in neural information processing systems 36 (2023): 12489-12517.

Thank you for the helpful review, and, in particular, for the acknowledgement of the rigorous and extensive empirical evaluation and performance. We address below the outstanding questions.

1. In what respect does your paper differ substantially from NOS?

Our method aims to solve a different mathematical problem and has a different behavior and guarantees when compared to NOS. Specifically, while NOS appends a gradient term to the denoising step for soft guidance, our method implements an augmented Lagragian projection operator that enforces all samples to lie in the constraint feasibility set $**C**$. Within the inner loop, our method solves the projection problem: find the nearest point to the original sample that lies in the constraint set $**C**$. Differently from NOS, the KL term here is used as distance measure in the projection objective: it keeps the new sample as close as possible to the model’s current score distribution. Another novelty of our framework is the theoretical ground provided: Feasibility (as convergence to a constrained set) is reported in Theorem 4.1. Additionally, empirically, CDD shows zero constraint violations in every experiment tested.

In contrast, in NOS the KL term acts a regularizer between the current and original logits with the $\lambda$ term regulating the trade‑off between distance and reward. The guidance is encoded only through this scalar reward and depends on tuning $\lambda$. It also provides no convergence guarantees. Qualitatively this technique is largely outperformed by our method. For example, [B] assesses the NOS guidance scheme for the same molecule domain we study in this paper, adapting it to MDLM and UDLM. Below we provide a side-by-side comparison for the novelty setting:

Note how adding NOS to UDML (second row) actually increases the violations by ~10% with respect to its baseline while adding our constrained projection method (last row) reduces these violations to 0. Hence, we hold that our comparison in Figure 4 effectively illustrates significant improvement over the state-of-the-art, illustrating for the first time such drastic constraint reductions in these challenging domains.

We appreciate the question and will make sure to explain these differences in an updated Related Work section.

2. Theoretically, the paper is relatively weak. Moreover, while the proof of Theorem 4.1 seems relatively sound, I fail to follow a few parts which could, I believe, use a bit of clarification. Namely, "$\forall x, y$ in the neighbourhood of $**C**$ ": what does this mean? which neighbourhood? (Maybe in “a” neighbourhood?) What is $\beta$-prox regularity?

First, let us highlight that to the best of our knowledge, CDD provides the first framework for discrete sequence modeling which incorporates theoretical guarantees for non-asymptotic feasibility, while also providing a formulation which enables modeling of arbitrary constraint sets.

As specified in Theorem 4.1, we adopt the notion of $\beta$-prox regularity as defined by [A] (Def. 13.27). This is a classical tool in convex optimization theory which implies that for each viable point and normal direction, small perturbations still project uniquely and smoothly back to $**C**$. The "neighborhood" referenced is defined in this sense. It indicates the region in which the prox‑regularity inequalities hold. The fact that our framework, relying on a principled methodology, allows us to derive theoretical guarantees should be seen as a strengths in our view. We appreciate your suggestion and will explicitly present this definitions in the final version of the paper.

3. While the method works well, it requires a great deal of additional runtime (which, I understand, can be a cost that is absolutely fine to pay, in some contexts). Do you believe your method could be sped up in any simple way?

Yes, there are ways to accelerate this method, although they lie outside the scope of our initial investigation. First, however, let us notice that, when compared to other controllable generation baselines, CDD provides the least additional overhead (see Table 4).

Our analysis in Appendix D explores a few of the "knobs" that can be adjusted when controlling the tradeoff between quality and runtime. Most notably, by adjusting the projection frequency and the timestep at which projection steps begins, one can control sampling speed and quality. Our results in Table 3, Figure 8, and Figure 9 show that starting to projection at later steps, alongside less frequent projections, results in faster sampling times while still satisfying the desired constraints.

Finally, as you mention, in high-assured context (e.g., scientific tasks) certificates of constraint satisfaction outweigh latency considerations. Indeed, providing outputs that violate critical constraints and physical principles make them unsuitable for production and testing, potentially slowing down, instead of accelerating, the AI-driven discovery pipeline.

4. section 4 should be broken down into subsections

Thank you for this suggestion. We agree and will update the final version. In particular, we will include subsections covering the following concepts:

1. Projected Diffusion Sampling
2. Differentiable Projection
3. Augmented-Lagrangian Projection
4. Theoretical justification

Thanks again!

5. Have you tried methods such as ReinMax [C], that allow for differentiable discrete sampling instead of Gumbel softmax?

This is an interesting suggestion, and we certainly will consider this as a direction for future work. However, a few concerns were uncovered by our early investigation during the rebuttal phase. First, ReinMax depends on multiple forward-backward passes which introduce further overhead; this will be especially challenging for the high-dimensional problems (e.g., we use a ~60K vocabulary). In fact [C] only tests on toy problems (with independent 128 bernulli variables), thus there a significant concern as to the scaling of this approach. Second, we suspect that ReinMax's reliance on Taylor approximations may introduce bias which could impact the projection dynamics of CDD. That said, we look forward to test it in our future work.

6. Have you attempted your method in combination with other SOTA sampling methods, such as P2 [D]?

We appreciate the suggestion, which may constitute an interesting follow-up study. For this paper, we have not provided such a comparison for a few primary reasons. First, our experimental analysis is designed to ablate the performance of our proposed method; to accurately assess the efficacy of projected discrete diffusion sampling processes, we validate this approach in isolation, rather than introducing variance through a composite approach. As such variations of traditional sampling techniques are yet to be adopted as de facto sampling schemes, it is most practical to assess our approach using the most common sampling processes.

Second, while CDD and P2 are not fundamentally incompatible, as [D] reorders the denoising trajectory while CDD enforces feasibility by projecting each intermediate distribution (orthogonal objectives), there are several practical limitations of P2 that make us hesitant to pursue this particular direction. One concern is that the additional overhead that is induced by P2 will compound with the inherent complexity introduced when certifying constraint satisfaction. Another challenge is that P2 may not generalize well to the domains that we explore in this paper, as the robustness of different planners has yet to be fully tested under domain shift. Hence, while we believe this certainly could be an interesting direction to explore, the work would likely fit the scope of an independent investigation.

Thank you for your detailed review! We hope that our response addressed all of your questions. We are also happy to provide any other requested details.

[A] Rockafellar, R. Tyrrell, and Roger JB Wets. Variational analysis. Berlin, Heidelberg: Springer Berlin Heidelberg, 1998.

[B] Schiff, Yair, et al. "Simple guidance mechanisms for discrete diffusion models." arXiv preprint arXiv:2412.10193 (2024).

[C] Bridging Discrete and Backpropagation: Straight-Through and Beyond (2023). Liyuan Liu, Chengyu Dong, Xiaodong Liu, Bin Yu, Jianfeng Gao.

[D] Path Planning for Masked Diffusion Model Sampling (2025). Fred Zhangzhi Peng, Zachary Bezemek, Sawan Patel, Jarrid Rector-Brooks, Sherwood Yao, Avishek Joey Bose, Alexander Tong, Pranam Chatterjee.