Diffusion Beats Autoregressive in Data-Constrained Settings

Thank you for your constructive feedback and for carefully reviewing our work.

Below, we address each of your points in turn:

---

Q4.1: Further experiments validating the hypothesis on why MDMs are more data-efficient.

To better understand why MDMs outperform ARMs in low-data regimes, we conducted a series of ablations on ARM models to test whether exposing them to more diverse prediction tasks, which is similar to what MDMs inherently do, could close the gap or interpolate between the two methods.

We hypothesized in the paper:

“MDM is exposed to diverse token orderings and prediction tasks, unlike the fixed left-to-right training of AR models. This enables them to extract more information per example.”

To validate this, we conducted the following experiments:

(a) Random Token Masking in ARM

Token masking generally degrades validation loss, likely due to over-regularization.

(b) Attention Dropout in ARM

Moderate dropout provides marginal gains but degrades performance at higher levels.

(c) Randomizing Token Order (Sigma-GPT [1] / RandAR [2])

In order to show the full impact, including the impact on overfitting points, we used a setting of 100 epochs here. Therefore, the original result for AR is different from the previous two experiments (50 epochs).

Randomized token ordering shows the most promising improvement, supporting our hypothesis that MDMs benefit from diverse prediction tasks. We implemented RandAR (Any-Order AR) on top of our ARM baseline for this experiment.

---

Q4.2: Motivation of internet-scale data is not aligned with small-scale experiments.

We agree that current LLMs are trained on trillions of tokens, which is larger than the data regimes we study. However, we believe our results remain meaningful:

• Our experiments demonstrate that objectives inducing randomized ordering (MDM) are more data-efficient.
• This insight can motivate future hybrid approaches, where diffusion-like objectives are applied on scarce high-quality data, while AR is applied on abundant lower-quality data.
• Domains with inherently limited data (e.g., robotics and healthcare) would particularly benefit from MDM-like data efficiency.

Since submission, we have scaled our experiments to 500M unique tokens using our critical compute scaling law (Appendix C), We train:

• 2.3B MDM trained for 130 epochs (terminated due to compute limits)
• Best ARM configuration via grid search for the same data budget

We report the results below:

Validation Losses:

• MDM: 3.08573
• ARM: 3.20136

---

Q4.3: Authors should discuss Llada [2], which shows more comparable scaling laws using downstream metrics.

Thank you for pointing this out. We will include a discussion of Llada [2] in the final version. Our scaling law analysis focused on validation loss, which is standard in scaling law studies and provides a stable signal.

To complement this, we conducted additional downstream evaluations, which confirm that MDM’s lower validation loss translates to better generalization:

Table 2: Accuracy (%) on downstream evaluation tasks at different unique token scales.

Note: Best results shown in bold.

---

Q4.4: Disagreement with lines 202–205 claiming MDMs are more compute-inefficient than ARMs.

We appreciate the clarification. We agree that MDM computes loss on only the masked tokens, which provides less direct supervision per update and higher gradient variance than ARM, where every token is supervised.

However, we disagree that processing unmasked tokens is “wasted compute”:

1. Unmasked tokens contribute to the loss indirectly via gradient propagation.
2. Unmasked inputs act as registers, gathering and passing information that improves predictions at masked positions.

Our conclusion that MDMs are more compute-inefficient is empirically grounded: achieving the same validation loss as ARM requires more FLOPs as a result more GPU-hours, as is also shown in prior works [16, 24]. .
Future work may mitigate this, e.g., Arriola et al. [1] show that tuning the masking schedule can reduce gradient variance and improve compute efficiency. We will add further clarification in the final version of the paper on this.

---

Q4.5: Other Minor Comments /Suggestions

Thank you for these helpful suggestions. We will incorporate the fixes in the final version of the paper

---

Q4.6: Difference between Table 1a and Table 1b? Why is the R² so poor in Table 1b?

• Table 1a represents scaling law fitting in the single-epoch setting (Chinchilla-style compute/data scaling).
• Table 1b represents scaling law fitting in the multi-epoch, data-constrained setting (Muennighoff et al. [15]) and estimates the constants $R_D^*$ and $R_N^*$.

The initial submission showed lower R² in Table 1b due to the inclusion of:

1. Configurations with <20 epochs for MDMs
2. ARM models with <45M parameters

These configurations produced high-variance validation loss, making regression noisier.

Since submission, we have refit the scaling laws, excluding these high-variance points, resulting in more stable and interpretable fits:

Updated Table 1:

(a) Initial fit

(b) Second-step fit with extracted scaling parameters

---

Q4.7: In Figure 2, isn’t the NLL relatively poor compared to Table 2 in MDLM [3]?

The comparison is not directly fair because:

• MDLM Table 2 is trained on 262B unique tokens, while our experiments use 100M unique tokens.
• If we assume 100 epochs of repetition is the same as unique tokens ≈ 10B effective tokens, this is still 26× smaller than MDLM.

Further we would like to point, our best ARM validation loss (3.71) in the 100M regime (Figure 2) closely matches 3.72 reported by Muennighoff et al. for the same scale (Figure 3 of their paper).

---

Q4.8: Authors are missing a scaling factor for the MDM loss.

Thank you for pointing it out. Yes, this is a typo and will be fixed in the final version.

---

Q4.9: There is no additional pre-training loss to enable CFG in Nie et al. [6]?

Correct. We acknowledge this and will remove the line in the final version. Thanks for pointing it out!

---

Q4.10: In Figure 2, is the colorful “Chinchilla-optimal” star for 1 epoch of training?

Yes, the star corresponds to 1 epoch of training under Chinchilla-optimal scaling.

---

Q4.11: Why are there different model sizes in the legends of Figure 5 for ARM vs. MDM?

We fixed the training compute to $1 \times 10^{19}$, $3 \times 10^{19}$, and $1 \times 10^{20}$ FLOPs.
We then used the compute-optimal scaling law to determine model sizes for each family:

• For $1 \times 10^{19}$ FLOPs → ARM ≈ 217M, MDM ≈ 117M

Hence, model sizes differ because compute-optimal configurations are different for ARMs vs. MDMs.

---

Q4.12: What does “showing masking, not attention, is key to their advantage when data is scarce” mean?

We agree that this sentence was unclear and will remove it in the final version.
Our updated experiments (see Q4.1) clarify that randomized token orderings are the main driver of MDM’s data efficiency.

References:

[1] Arriola, E., et al. Improving compute efficiency of diffusion models via masking schedules.
[2] Llada, J., et al. Downstream scaling laws for generative models.

Thank you for your constructive feedback and for carefully reviewing our work.

Below, we address your points in turn:

---

Q3.1: Extensions to other datasets and model architectures.

Thank you for this suggestion. We agree that validating our findings across different datasets and model architectures will strengthen the paper.
We are currently running experiments on SlimPajama (a large, high-quality text dataset) and on T5 (encoder-decoder architecture). These experiments are ongoing, and we plan to include the results in the discussion phase or in the final version of the paper.

---

Q3.2: Varying optimizer settings for ARM. Chose hyperparameters might be particularly well-suited for MDMs

We followed the exact hyperparameter settings from the NeurIPS data-constrained Scaling Law paper (Muennighoff et al. [15]), which uses hyperparameters commonly used for training autoregressive language models. We believe these hyperparameters may slightly favor ARMs, as these were originally tuned for that family.

Importantly:

• Our best ARM validation loss in the 100M unique token setting is 3.71 (Figure 2 of our paper), which closely matches the 3.72 reported by Muennighoff et al. (Figure 3 of their paper).
• Our MDM achieves a validation loss of 3.55, which is unlikely to be matched by ARM even with hyperparameter tweaks.

That said, we are currently running additional experiments with varied learning rates for ARM to confirm this point and will include results in the discussion/final version.

---

Q3.3: More downstream experiments.

To assess whether MDM’s lower validation loss translates to better generalization, we evaluated the best-performing MDM and ARM models on a range of language understanding tasks. Across all tasks and data scales, MDM consistently outperforms ARM.

Table 2: Accuracy (%) on downstream evaluation tasks at different unique token scales.

Note: Best results shown in bold.

---

Additional Updates

We have also conducted additional experiments since our submission. We summarize these updates below:

---

i) Why is MDM Better in Data-Constrained Settings?

To better understand why MDMs outperform ARMs in low-data regimes, we conducted a series of ablations on ARM models to test whether exposing them to more diverse prediction tasks, which is similar to what MDMs inherently do, could close the gap or interpolate between the two methods.

We hypothesized in the paper:

“MDM is exposed to diverse token orderings and prediction tasks, unlike the fixed left-to-right training of AR models. This enables them to extract more information per example.”

To validate this, we conducted the following experiments:

(a) Random Token Masking in ARM

We observe that token masking in AR degrades validation loss, especially at higher ratios, likely due to over-regularization.

(b) Attention Dropout in ARM

Moderate dropout yields marginal improvements but leads to degradation at higher levels.

(c) Randomizing Token Order (Sigma-GPT [1] / RandAR [2])

In order to show the full impact, including the impact on overfitting points, we used a setting of 100 epochs here. Therefore, the original result for AR is different from the previous two experiments (50 epochs).

Randomized orderings show the most promising gains, supporting our hypothesis that MDMs benefit from diverse prediction structures. For this experiment, we implemented RandAR (Any-Order AR) on top of our ARM baseline.

---

ii) Scaling to 500M Unique Tokens Using the Critical Compute Equation

In Appendix C, we derive an analytical equation that estimates the critical compute at which MDM matches the validation loss of ARM, given a fixed unique data budget. Using this, we scaled the training set to 500M unique tokens and performed a grid search over model sizes and total data based on the predicted compute to identify the configuration predicted by the scaling law to achieve the optimal validation loss as close as possible. Therefore, we trained a 2.3B parameter MDM using the predicted critical compute budget. After training for 130 epochs, we observed that the validation loss of MDM is consistently lower than that of ARM.

For the ARM baseline, we use the same method to find out optimal setting under the same data budget. We report the empirical validation losses of the trained models below:

Validation Losses:

• MDM: 3.08573
• ARM: 3.20136

---

iii) Improved Fitting Metrics (in Appendix)

We refit the scaling laws after removing configurations with a small number of training epochs (<20) for MDMs and excluding very small model sizes (<45M parameters) for ARMs. These configurations often exhibited high variance in validation loss due to insufficient optimization, introducing noise into the regression. Excluding them resulted in significantly more stable and interpretable fits.

Updated Table 1:

(a) Initial fit

(b) Second-step fit with extracted scaling parameters

---

References:

[1] Pannatier, Arnaud, Evann Courdier, and François Fleuret. σ-GPTs: A new approach to autoregressive models.

[2] Pang, Ziqi, et al. RandAR: Decoder-only autoregressive visual generation in random orders.

Dear Reviewer,

We ran additional experiments as per your suggestions:

1. SlimPajama Dataset (100M Unique Tokens)

To test the generality of our findings, we ran experiments on SlimPajama, which is more compressed and information dense than C4, therefore smaller models get preferred.
We searched around the optimal ARM and MDM settings based on the C4 dataset, varying both model size and number of training epochs.

ARM Sweep (7 configurations):

MDM Sweep (Only 2 configurations):

Even though we tried 7 configurations for ARM and only 2 for MDM, we find MDM outperforms the best ARM by a clear margin. This reinforces our core finding that MDMs do better in data-constrained settings. We expect even stronger MDM performance if a full sweep is conducted on SlimPajama.

---

2. Optimizer Sensitivity for ARM

We explored several learning rate and optimizer settings for ARM to check if MDM’s advantage could be attributed to suboptimal AR hyperparameters. Below are the results:

Across all configurations, we find that our default setup continues to yield the best validation loss. We believe this is because the optimizer settings we adopted - based on [1] - were already well-tuned for training ARMs under data-constrained settings.

[1] Muennighoff et al., "Scaling Data-Constrained Language Models", NeurIPS 2023.

Thank you for your constructive feedback and thoughtful suggestions.

Below, we address each of your points in turn:

---

Q2.1: Limited technical novelty.

We agree that our work does not introduce a new algorithm. However, we believe our empirical findings are novel and important for the generative modeling community. Our work is the first to systematically characterize the data vs. compute trade-offs between MDMs and ARMs across a wide range of model and data scales, and to establish a closed-form scaling law for the critical compute threshold at which MDMs outperform ARMs.

These insights, while empirical, can provide a foundation for future algorithms that can better interpolate between the compute-efficiency of ARMs and the data-efficiency of MDMs.

---

Q2.2: More analysis on why diffusion is better in data-constrained settings.

To better understand why MDMs outperform ARMs in low-data regimes, we conducted a series of ablations on ARM models to test whether exposing them to more diverse prediction tasks, which is similar to what MDMs inherently do, could close the gap or interpolate between the two methods.

We hypothesized in the paper:

“MDM is exposed to diverse token orderings and prediction tasks, unlike the fixed left-to-right training of AR models. This enables them to extract more information per example.”

To validate this, we conducted the following experiments:

(a) Random Token Masking in ARM

We observe that token masking in AR degrades validation loss, likely due to over-regularization.

(b) Attention Dropout in ARM

Moderate dropout yields minor improvements, but quickly becomes harmful at higher levels.

(c) Randomizing Token Order (Sigma-GPT [1] / RandAR [2])

Randomized token orderings show the most promising improvements, supporting our hypothesis that MDM’s exposure to diverse prediction structures enables greater information extraction per example. For this experiment, we implemented RandAR (Any-Order AR) on top of our ARM baseline.

---

Q2.3: Factorizations for AR and MDM in Lines 27 and 32 lack rigor.

Thank you for pointing out. We agree that the factorizations for both AR and MDM were oversimplified and may be misleading, and will revise the text to refer readers directly to the method section, where the model factorizations are explained rigorously and unambiguously.

In particular:

• For AR models, we will clarify that the joint distribution is factorized as: $p(x) = \prod_{t=0}^{T-1} p(x_t \mid x_{<t})$ , rather than the simplified and potentially misleading Markovian example currently shown.

• For MDM, we acknowledge that writing the joint distribution in a factorized form is not theoretically grounded. Unlike AR or any-order AR models, MDMs do not explicitly define a joint factorization. To avoid confusion, we will remove the equation in Line 32 and use the following sections to explain in detail.

---

Q2.4: Can higher learning rates mitigate MDM’s compute inefficiency?

We do not expect this to be the case. Multiple prior works have noted that MDMs require more compute than ARMs to reach similar validation losses (e.g., [1, 16, 24]). We believe these inefficiencies arise not from optimization difficulties, but from fundamental differences in supervision density and variance in gradient (see Q2.6).

That said, we are currently running additional experiments with higher learning rates to test whether optimization plays any non-negligible role. We will include updated results in the discussion/final version once available.

---

Q2.5: Does this research convey a pessimistic signal that MDMs are not suitable as the next generation of large language models?

We do not believe so. As Ilya Sutskever aptly said, "Compute is growing at a rapid pace, but data is not—we have just one internet." Our findings suggest that MDMs are significantly more data-efficient than ARMs, which may make them better suited for future regimes where high-quality unique data is scarce.

We envision a number of exciting research directions that build on our work:

• Interpolating between MDM and ARM objectives to achieve better trade-offs between compute and data efficiency.
• Modality- or quality-adaptive objective, where MDM is used on scarce, high-quality data and ARM is used on abundant, low-quality data.
• Applications in domains such as robotics and healthcare, where one doesn't have billions/trillions of tokens to begin with and data-efficient generative models are critical.

Rather than pessimistic, we view our work as an invitation to rethink model design in the post-internet era.

---

Q2.6: Further elaboration on effective tokens seen by MDM.

Thank you for highlighting this. We agree that in MDM training, loss is computed on only a subset of tokens (in expectation, 50%), compared to ARM where every token is supervised. We believe this could contribute to the compute inefficiency of MDMs and have added the following discussion in our paper:

Why are autoregressive (AR) models more compute-efficient than diffusion models?
We hypothesize the following as contributing factors:
(i) Stronger supervision per update: In AR training, every token in a sequence is a supervised target, and the causal dependency structure enables dense, low-variance gradient updates.
(ii) Sparser supervision in MDM: In contrast, MDM loss is computed on a subset of masked tokens, and while gradients propagate through the entire sequence, each update carries less direct learning signal.
Arriola et al. [1] demonstrate that tuning the masking schedule can help reduce gradient variance and improve compute efficiency in MDMs.

We thank the reviewer again for this useful clarification.

---

Additional Updates

We have also conducted additional experiments since our submission. We summarize these updates below:

---

i) Scaling to 500M Unique Tokens Using the Critical Compute Equation

In Appendix C, we derive an analytical equation that estimates the critical compute at which MDM matches the validation loss of ARM, given a fixed unique data budget. Using this, we scaled the training set to 500M unique tokens and performed a grid search over model sizes and total data based on the predicted compute to identify the configuration predicted by the scaling law to achieve the optimal validation loss as close as possible. Therefore, we trained a 2.3B parameter MDM using the predicted critical compute budget. After training for 130 epochs, we observed that the validation loss of MDM is consistently lower than that of ARM.

For the ARM baseline, we use the same method to find out optimal setting under the same data budget. We report the empirical validation losses of the trained models below:

Validation Losses:

• MDM: 3.08573
• ARM: 3.20136

---

ii) More Downstream Results

We further evaluated the best-performing MDM and ARM models on a range of language understanding tasks to assess whether the gains in validation loss translate to better generalization. Across a diverse set of tasks and data scales, MDMs consistently outperform ARMs.

Table 2: Accuracy (%) on downstream evaluation tasks at different unique token scales.

Note: Best results shown in bold.

---

iii) Improved Fitting Metrics (in Appendix)

We refit the scaling laws after removing configurations with a small number of training epochs (<20) for MDMs and excluding very small model sizes (<45M parameters) for ARMs. These configurations often exhibited high variance in validation loss due to insufficient optimization, introducing noise into the regression. Excluding them resulted in significantly more stable and interpretable fits.

Updated Table 1:

(a) Initial fit

(b) Second-step fit with extracted scaling parameters

References:

[1] Pannatier, Arnaud, Evann Courdier, and François Fleuret. σ-GPTs: A new approach to autoregressive models.

[2] Pang, Ziqi, et al. RandAR: Decoder-only autoregressive visual generation in random orders.

Thank you for your constructive feedback and for spending the time to read our paper in detail.

Below we address your concerns:

---

Q1.1: The primary takeaway of the paper is that MDMs offer benefits only when the number of unique tokens is 100M or fewer, which feels somewhat underwhelming.

We respectfully disagree with this characterization. The primary contribution of our work is to demonstrate that diffusion models consistently outperform autoregressive models in data-constrained regimes. To test this, we conducted experiments across a range of unique token budgets—25M, 50M, and 100M—in the original submission.

Since submission, we have extended our experiments to a 500M unique token setting, where we again observe that MDM outperforms ARM when both are trained under the same data budget, as shown in the updated results below.

While we agree that it would be valuable to validate these trends at even larger scales (e.g., 10B unique tokens), doing so would require training for trillions of tokens to maintain the epoch count (e.g., 500 epochs × 10B = 5T tokens), which is far beyond our current compute budget. That said, we believe the consistency of our results across multiple unique token scales, including up to 500M, could validate our claims.

---

Q1.2: Reference [21] actually trains AR and MDM in a multi-epoch setting.

Thank you for pointing this out. We apologize for the oversight and will correct this in the final version.

---

Additional Updates

We have also conducted additional experiments since our submission. We summarize these updates below:

---

i) Why is MDM Better in Data-Constrained Settings?

To better understand why MDMs outperform ARMs in low-data regimes, we conducted a series of ablations on ARM models to test whether exposing them to more diverse prediction tasks, which is similar to what MDMs inherently do, could close the gap or interpolate between the two methods.

We hypothesized in the paper:

“MDM is exposed to diverse token orderings and prediction tasks, unlike the fixed left-to-right training of AR models. This enables them to extract more information per example.”

To validate this, we conducted the following experiments:

(a) Random Token Masking in ARM

We observe that token masking in AR degrades validation loss, especially at higher ratios, likely due to over-regularization.

(b) Attention Dropout in ARM

Moderate dropout yields marginal improvements but leads to degradation at higher levels.

(c) Randomizing Token Order (Sigma-GPT [1] / RandAR [2])

In order to show the full impact, including the impact on overfitting points, we used a setting of 100 epochs here. Therefore, the original result for AR is different from the previous two experiments (50 epochs).

Randomized orderings show the most promising gains, supporting our hypothesis that MDMs benefit from diverse prediction structures. For this experiment, we implemented RandAR (Any-Order AR) on top of our ARM baseline.

---

ii) Scaling to 500M Unique Tokens Using the Critical Compute Equation

In Appendix C, we derive an analytical equation that estimates the critical compute at which MDM matches the validation loss of ARM, given a fixed unique data budget. Using this, we scaled the training set to 500M unique tokens and performed a grid search over model sizes and total data based on the predicted compute to identify the configuration predicted by the scaling law to achieve the optimal validation loss as close as possible. Therefore, we trained a 2.3B parameter MDM using the predicted critical compute budget. After training for 130 epochs, we observed that the validation loss of MDM is consistently lower than that of ARM.

For the ARM baseline, we use the same method to find out optimal setting under the same data budget. We report the empirical validation losses of the trained models below:

Validation Losses:

• MDM: 3.08573
• ARM: 3.20136

---

iii) More Downstream Results

We further evaluated the best-performing MDM and ARM models on a range of language understanding tasks to assess whether the gains in validation loss translate to better generalization. Across a diverse set of tasks and data scales, MDMs consistently outperform ARMs.

Table 2: Accuracy (%) on downstream evaluation tasks at different unique token scales.

Note: Best results shown in bold.

---

iv) Improved Fitting Metrics (in Appendix)

We refit the scaling laws after removing configurations with a small number of training epochs (<20) for MDMs and excluding very small model sizes (<45M parameters) for ARMs. These configurations often exhibited high variance in validation loss due to insufficient optimization, introducing noise into the regression. Excluding them resulted in significantly more stable and interpretable fits.

Updated Table 1:

(a) Initial fit

(b) Second-step fit with extracted scaling parameters

---

References:

[1] Pannatier, Arnaud, Evann Courdier, and François Fleuret. σ-GPTs: A new approach to autoregressive models.

[2] Pang, Ziqi, et al. RandAR: Decoder-only autoregressive visual generation in random orders.