Unifying Autoregressive and Diffusion-Based Sequence Generation

We thank the reviewer for acknowledging the novelty of our contributions, as well as raising an important point and providing typo fixes.

I would like to know more about the evaluation of quality-diversity trade-offs [...]

We acknowledge that perplexity, while a common metric for language models, doesn't always align with human perception of sample quality. We include it for consistency with prior diffusion baselines, but recognize its limitations.

To provide a more comprehensive evaluation, we complement perplexity with the MAUVE score. MAUVE assesses the divergence frontier between generated and human text in embedding space and, critically, has been shown to have a strong correlation with human judgments of both text quality and diversity. As depicted in Figure 4 (right) in the paper, our hybrid models consistently achieve a superior perplexity-MAUVE Pareto frontier compared to all baselines. Furthermore, the qualitative samples in Appendix G demonstrate that these improvements in perplexity and MAUVE translate into more coherent and varied outputs.

The camera ready will clarify the limitations of perplexity, and that combining it with MAUVE provides a more balanced evaluation framework.

Action point for camera ready: Discuss the limitations of perplexity and why we include MAUVE evaluations.

[1] Pillutela et al., MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers

We would like to thank the reviewer for their thoughtful feedback and for recognizing the theoretical contributions of our work.

I think the missing part is the analysis on the training [...]

Regarding training overhead, please refer to our detailed response to Reviewer NmzT: as shown in Appendix B.1, our two-stage pretraining and fine-tuning schedule requires no more time than other diffusion-based LMs of comparable size (110 M parameters).

For inference overhead, we leverage our hyperschedule custom attention masks with KV-caching. Appendix F.7 (Table 11) demonstrates that our latency is lower than MDLM’s fastest caching strategy. Furthermore, Tables 7, 9, and 10 show how varying the generation rate ρ (tokens generated per network call) can further accelerate inference—an emergent capability of diffusion-based generators that lets users trade off speed and quality on the fly.

For unconditional generation:

• the total number of times that the model is called is $O(d/\rho + \omega)$ (where $d$ is the sequence length, $\rho$ is the hyperschedule's token generation rate and $\omega$ is its window width);
• without KV-caching, the total number of tokens to be processed (including re-processings of already-known tokens) is $O(d^2/\rho)$; and
• with KV-caching, the total number of tokens to be processed is: $O(\omega d/\rho)$.

Action point for camera ready: Add inference complexity analysis to Appendix B.4.

Finally, we wish to emphasize that our contribution introduces many different models and variations which, we concede, may seem complex altogether. However, each specific instantiation of our framework is much simpler.

We thank the reviewer for acknowledging the importance of our contribution, as well as for their questions, which should help us clarify our manuscript.

You design attention masks to support KV-caching for efficient inference, but the training cost of your hybrid models is not discussed. Could you provide more details about this?

Our hybrid models follow a two-stage training approach, detailed in Appendix B.1. Initially, we pretrain models using the $\tau\_{flat}$ hyperschedule: approximately 14 days on 8×A100-80GB GPUs for OpenWebText (OWT) and 7 days on 2×A100-80GB GPUs for LM1B. We then fine-tune for each configuration over an additional 150K steps -- ~3 days (8×A100 GPUs) on OWT and ~36 hours (2×A100 GPUs) on LM1B. This two stage approach helps us get as many runs as possible with our limited compute budget and also proves that the hyperschedule variants are robust enough to improve performance while being limited to the late-stage fine-tuning.

The table below compares training throughput across different models (single A100-80GB GPU, no custom kernels, largest batch size divisible by 2; 64 for most methods and 32 for “Ours Efficient” due to its processing strategy). It highlights the efficiency gains achieved by our method through the use of customized attention masks. Here, we set the window size as ω = d/4; using smaller values of ω further amplifies the efficiency benefits of our attention masks during training.

By leveraging our efficient attention masks and doubling context length through concatenation (clean and noisy sequences), we achieve nearly twice the active token throughput compared to our non-efficient training, thus significantly reducing overall training cost. This confirms that our efficient training paradigm allows our hyperschedule variants to achieve an effective throughput comparable to other diffusion baselines.

Action point for camera ready: Add efficiency throughput table and discussion to the appendix.

[1] Sahoo et al., Simple and Effective Masked Diffusion Language Models

[2] Lou et al., Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution

All experiments use relatively small transformer models (110M). Have you attempted scaling your method to larger models (e.g., 1B+)? If not, what are the expected challenges or limitations when applying at that scale?

While fully scaling to 1B+ models lies beyond this paper’s scope, we are actively exploring this direction: early experiments (intended for future work) on a 350M–parameter prototype confirm that our hybrid models follow the expected, near-linear scaling trends in perplexity and per-token training time [3]. To further cut training time at scale, we’re integrating flex-attention [4] custom kernels, which dramatically reduce the overhead of our windowed attention masks. Moreover, recent work [5] has successfully scaled diffusion-based language models up to 8B parameters without introducing fundamental architectural bottlenecks, reinforcing our belief that our method remains practical and may even further benefit from larger-scale deployment.

Action point for camera ready: Add a short discussion about scaling to appendix.

[3] Nie et. al, Scaling up Masked Diffusion Models on Text.

[4] https://pytorch.org/blog/flexattention/

[5] Nie et. al, Large Language Diffusion Models.

(Continued below.)

We thank the reviewer for acknowledging the novelty of our work, as well as for providing valuable inputs that allowed us to identify specific action points that should improve the overall clarity of our contribution.

It appears that the authors evaluate their models using the training objectives described in Appendix B.2. As a result, the perplexity (PPL) values reported in Tables 1 and 2 may be incorrect or misleading.

We first wish to clarify that the training objectives described in Appendix B.2, as well as the hyperschedule and other details of the underlying curriculum (masking and/or uniform noise), are not directly involved in our "Test PPL" metric. We now realize that this may not be clear in the present state of our Appendix C.1.

All Test PPL scores computed by us, including both $\gamma$-Hybrid and $\epsilon$-Hybrid, as well as baseline diffusion models (unless mentioned "reported in..."), are all evaluated with the same code implementing Algorithm 1, using the loss listed in Appendix C.1 (i.e., not the one described in Appendix B.2). This algorithm estimates an upper bound for the perplexity, reflecting how good the score/logit-predicting neural network -- not the "full inference strategy" -- is at "language modeling" (in the sense of "predicting the probability for the masked tokens"). The inference-time specificities of the model could actually make it effectively better at language modelling than this metric captures.

The training objectives described in Appendix B.2, the choice of hyperschedule and the value of $\epsilon$ or $\gamma$, can indirectly affect the Test PPL metric through their impact on the training of the neural network's weights. The camera ready will clarify these points.

Action point for camera ready: Add clarifications about the "Test Perplexity" metric.

(Continued below.)

Dear Reviewer WiDd,

We wish to respectfully remind you that the author-reviewer discussion period is scheduled to end tomorrow, June 10th. This is the final opportunity for us to address any further questions you may have.

Thank you for your time and consideration of our paper.

Sincerely,

The Authors