We thank the reviewer for their thorough and insightful review. We address each point in detail below, quoting relevant comments, and will incorporate all suggestions into revision.

Due to the paper's density, we aimed to balance presenting our method and providing essential background information. We omitted some standard practices for brevity, the concerns arising from these omissions can be clarified below with minor revisions, as they do not reflect flaws in our method or experiments.

Q5

Inconsistent Fig 1 baseline numbers

cited numbers are different from original paper. calls into question soundness of methodology or reported numbers.

The discrepancy comes from different experimental setups. Fig 1 shows models trained on 26B OpenWebText tokens, making scores incomparable to original papers. We mentioned this in Sec 4.2 (L301) and Fig 1's caption, but agree it needs more emphasis. See response to Q2 of vK3t for more details.

Q1

How to compute likelihood?

We use same ELBO methods as prior works:

• Masked diffusion: MD4/MDLM approach (Eq.4 in [1])
• Uniform diffusion: DFM method (Eq.7.32 in [3])

Are Eq(4,5) likelihood bounds?

Writing is overly general (is sampling t from arbitrary $\omega(t)$ necessary?)

Masked diffusion models have ELBO as weighted loss based on noise schedule, the source of the scaling factor in TCSM isn't clear.

$\omega(t)$ provides necessary weighting for valid ELBO and scaling factors, following diffusion literature (Eq.7 in [5], [4 Sec2.1]). For masked diffusion, ELBO weighting is $\lambda(t) = \frac{d\alpha_t/dt}{1-\alpha_t}$, so $\omega(t) \propto \lambda(t)$. We sample $t$ uniformly and apply $\lambda(t)$, equivalent to $t \sim \omega(t)$. Eq (4, 5) are likelihood bounds only when $\omega(t)$ matches the ELBO weighting scheme.

Q2

Refer to Q1 of vK3t.

Q3

Eq.10 comes from Eq.9 using conditional independence $p^{\theta}_{1|t}(\mathbf{x}_1 | \mathbf{x}_t) = \prod\_{i=1}^{L} p^{\theta}\_{1|t} (x^i_1 | \mathbf{x}_t)$. Thus $p^{\theta}\_{1|t}(\mathbf{x}\_1 | \mathbf{x}\_1^{\neq i}, \mathbf{x}\_t)= p^{\theta}\_{1|t}(\mathbf{x}\_1 | \mathbf{x}\_t)$, yielding Eq.10.

Q4

In pre-training, $h_{1|t} = p_{1|t}$ sampling from true data distribution, noted in Prop. 4.1 (L195) and L239.

Q6

Table shows OWT val PPL from OWT-trained models. We'll add main text references.

Where is data-prediction (Eq.12) from?

"Data-pred" follows DDPM conventions, used alongside "noise-prediction" or "v-prediction". It refers to predicting clean data $\mathbf{x}_1$ from noisy $\mathbf{x}_t$, versus predicting noise. See Eq. 109 in [4] for more.

Q7

In Sec 4.2, we discuss gradient variance to motivate using parametric models for faster convergence. Prior works (e.g., Fig 2 in [6]) shows high gradient variance can slow convergence in diffusion models. While we don't directly measure variance reduction, our experiments demonstrate TCSM's faster convergence, addressing this core issue. We'll clarify in the revision that sample efficiency is our direct objective and demonstrated benefit.

Using existing LLMs (e.g. LLaMa) as parametric models avoids pretraining costs. Though we trained from scratch on OWT for fair comparisons, results show potential for leveraging pretrained LLMs.

Q8

Simplification comes from using $p_{1|t}$ as $h_{1|t}$ and forward KL. We'll add complete notation: $l^i\_{\text{distrib}} = - \mathbb{E}\_{p_{1|t}(x\_1^i | \mathbf{x}\_t)} \log p\_{1|t}^{\theta}(x^i\_1| \mathbf{x}\_t) + C$.

Q9

RHS of Eq.19 repeated twice

RHS terms differ by $\mathbf{x}_1^{\neq i}$ positions.

RHS of Eq.20 is identical to Eq.19

The proof shows equivalence between two concrete score views: Eq.19's LHS gives conditional score $p(x_1^i | \mathbf{x}_1^{\neq i}, \mathbf{x}_t)$, while Eq.20's LHS gives $i$-th component of joint score $p(\mathbf{x}_1 | \mathbf{x}_t)$. They're equal as normalization constants cancel in ratio-based concrete score. Though the RHS expressions appear identical, the key difference lies in their LHS interpretations

Q11

See response to Q5 of vK3t.

Q12

Prop 3 insight, Prop 4.1 Proof

Prop 3 enables practical TCSM objective estimation from data. Since $\mathbf{c}\_{p\_{1|t}}$ is unknown, it reformulates the objective using KL and IS divergences, leading to the objective in Prop 4.1 used in experiments. It thus connects theory to practical estimation.

We'll include the proof of Prop 4.1 in appendix, which straightforwardly writes out the explicit forms of the KL and IS divergences.

Q13

Discussed divergence not used in exp

Gen. KL used in "TCSM Absorb $L_{score}$" (Table 4). Forward KL used in $\ell_{\text{distrib}}$ in "TCSM Absorb $L_{distrib}$" (Tables 3-4). Post-training uses f-divergence, giving objective (Eq. 11), tested in Tab 5-6 and App E.2.

Q14

Table 2 not elaborated

Please refer to Q4 of vK3t.

Q15

Samples

Will be added in the revision.

Ref

[1] 2406.04329 [2] 2406.07524 [3] 2412.06264 [4] 2303.00848 [5] 2011.13456 [6] 2503.09573

We deeply appreciate the reviewer's thorough, high-quality, and insightful feedback. We'll address all suggestions and correct typos in our revision.

We are particularly encouraged by the reviewer's positive assessment of our work as "innovative", "compelling", and "representing a very big step forward." We also appreciate their willingness to increase the score once the concerns on experimental details are addressed. To directly address these concerns, we have organized our response to focus first on the experimental details.

Below we respond to each point (Q=Question, W=Weakness)

Experimental Details Concerns

[W3, Q1, Q2, Q3]

• Sec 4.1 introduces TCSM using only data samples from training dataset, same as prior works (SEDD, MD4)
• Sec 4.2 presents an alternative method using auxiliary parametric models (AR, BERT, Hollow) to estimate Target Concrete Score (TCS) via $p_1(x_1^i|\mathbf{x}_1^{\neq i})$

They describe orthogonal techniques within TCSM.

W3

Experiments following Sec 4.1 use only data-only methodology, without auxiliary models from Sec 4.2.

Q1

Parametric models in Sec 4.2 are used only during training to estimate TCS. They're not needed for inference/generation. After training, standard sampling techniques are used. (Please refer to Q1 of vK3t for more details).

Q2

Our procedure:

1. Initialize parametric model from scratch and train on same dataset
2. Use this trained model (frozen) to provide TCS estimates

"Pre-trained" means auxiliary model is trained before being used in diffusion model training.

Q3

Yes, it requires additional forward passes during training. However:

1. The overhead applies only during training, not inference, preserving generation speed
2. We observed just a ~20% increase in training time
3. As Fig 4 shows, these models significantly accelerate convergence, requiring fewer total training steps (and FLOPs) to reach target performance compared to data-only baselines

Q5

• Pre-training: $h_{1|t}$ is true data posterior $p_{1|t}(x_1 | x_t)$
• Post-training with reward: $h_{1|t}$ is pre-trained denoising model $p_{1|t}^{pre}$

Q8

Yes, we re-trained both baselines with same training budget as TCSM. Please refer to Q2 of vK3t for more details.

Q10

"Reimpl." refers to our re-implementation of EDLM NCE. As TCSM with BCE in Eq.11 is equivalent to EDLM's NCE loss and their code isn't public, we implemented it ourselves (detailed in App E.3, L1105-L1110).

The MDLM number in Tab 6 comes from Table 2 in the EDLM paper. Thanks for noting this discrepancy. We'll update with the original number from Sahoo et al. 2024.

Q11

Like SEDD Sec 5.2.3, we divide text into fixed-length, non-overlapping segments. AR models use sliding window approach.

W2

Comparisons are fair. All TCSM models were trained from scratch on identical datasets and evaluated under the same conditions using the same validation set. TCSM was not initialized from any pre-trained model trained on any extra data.

Baseline models were trained to convergence across multiple epochs, TCSM's gains come from our density ratio parameterization (see W5).

W4

We implemented AR baseline with 26B tokens, achieving 22.51 PPL. We'll add to Fig 1.

W5

Tab 6 gains come from our density ratio model $r_{1|t}^{\theta}$, not extra training. This model captures token dependencies other approaches miss with conditional independence assumption.

W6

Please refer to Q3 in vK3t.

W7, W8

All experiments use linear probability paths. We built on MD4's codebase, maintaining identical architectures, sizes, training steps, and optimizers for fair comparison. Code will be released for reproducibility.

Other Concerns

Q4

Eq.4 and 5 are indeed specific instances of the general Eq.3, simplified by using the 1-Hamming neighborhood. This adds no additional assumptions on the divergence operator, as the goal of matching model and TCS can use any divergence measure. The 1-Hamming structure only defines what scores we match, not how we measure their differences through the divergence.

Q6, Formatting issues 6

'Multiset' in Def 3.1 refers to how we construct the concrete score matrix (L114, RHS) by duplicating the original sequence. This enables transition from score space to normalized probability distribution space (Sec 3, L185-L219, 'Target Concrete Score' paragraph). Our approach parallels concepts in prior work (Proposition 1, Eq. 9 in MD4).

Q7

We agree "pseudo-likelihood" in Prop 4.1 isn't used in exact sense. This term was chosen because minimizing $l_{pseudo}$ directly corresponds to maximizing standard pseudo-likelihood. We'll revise.

Q9

As in Table 5, when implementing Eq. 11, we treat samples from $p^{ref}$ as "negative samples" for training model $\theta$. We don't backpropagate through sampling from $p^{ref}$.

W1

Please refer to Q4 of vK3t.

Extra References

Thank you for pointing out the extra references. We will add all suggested references.

We sincerely thank the reviewer for their positive review and valuable feedback. We greatly appreciate the opportunity to address the questions and provide additional clarifications about our work. In the following sections, we carefully respond to each point raised by the reviewer, quoting the relevant comments to ensure clear and direct responses. We have also included additional explanations that we believe will help further clarify our approach and results. Please don't hesitate to let us know if you have any follow-up questions or need further details on any aspect of our work.

Experimental Details Clarifications

the paper could benefit from a discussion of the experimental results.

We agree that more discussion of the experimental results would be helpful. We layout following additional details on the implementation details and experimental results.

Q1 Sampling

A discussion on sampling methods from masked diffusion models is missing

We agree that including explicit introduction of sampling methods used in this work would be helpful. TCSM's flexibility allows it to work with any diffusion model parameterization, compatible with standard sampling techniques like ancestral sampling and reverse CTMC simulation via Euler method.

In our experiments:

• Masked diffusion models (Sec 5.2, Fig 3): Used ancestral sampling
• Uniform diffusion models (Fig 5): Used reverse CTMC simulation with Euler method

These are standard approaches briefly noted in App A. We'll provide more details in the revision.

Q2 Fig 1 evaluation

Fig 1 shows how parametric models (Sec 4.2) enhance sample efficiency during training. We aimed to test if this approach accelerates convergence vs data-only baselines. To clearly show performance differences, we deliberately restricted all models to 26 billion tokens, creating a controlled comparison within this limited training budget. In contrast, the results reported by Sahoo et al. (23.21 for MDLM, 24.10 for SEDD) baseline were obtained using full training data. Consequently, the PPL scores in our Fig 1 are not directly comparable to those specific results from the original papers. For fair comparisons under identical conditions (with full training convergence), see Tab 3 and 6.

Further Clarifications

Q3 AR to Diffusion Distillation

We placed distillation in post-training to align with LLM community where distillation is typically a post-training technique.

Distillation section highlights a key TCSM advantage: it's the only discrete diffusion framework enabling effective AR-to-diffusion distillation—valuable given today's many pre-trained LLMs. We developed novel techniques for this process (Top-K, first-order approximation) that enhance scalability beyond Sec 4.2.

Q4 Table 2 details

While the key connections summarized in Table 2 are currently present in the manuscript, they are distributed across different sections. For instance, we establish that our simplified $\ell_{distrib}$ objective (Eq. 9) corresponds to the standard cross-entropy loss employed by MD4/MDLM/DFM. Furthermore, the relationship with EDLM (2410.21357) is detailed in App E.3 (L. 1122), linking their objective (Eq. 10) to the BCE-based Bregman divergence shown in Table 2 (row BCE (ii)).

To improve readability, we will add a consolidated paragraph in the revised manuscript. This new paragraph will explicitly outline these connections and clearly explain the significance and utility of Table 2.

Q5 SEDD, SDDM comparison

SEDD was included as a baseline in all language modeling experiments, with results shown in Tables 3, 4 and Fig 1.

For SDDM, we followed prior discrete diffusion studies by excluding it, as its focus is image generation. Comparable language modeling results and official code were unavailable, preventing direct comparison.

We selected DFM for Sec 2 due to its general framework covering various discrete diffusion approaches, including SEDD (score parameterization) and MD4/MDLM (mask source distribution). This makes DFM ideal for introducing core concepts before our specific contributions.

Extra references

We will make sure to include all the references pointed out by the reviewer.