Distilled Decoding 2: One-step Sampling of Image Auto-regressive Models with Conditional Score Distillation

• Thank you for the insightful question! Please see our detailed response below. We’re happy to continue the discussion if any concerns remain.

Weakness 1: Including comparative results between the CSD loss function and other score distillation losses

• Thank you for your insightful suggestion!
• As we mentioned in Sec 3.2.2, our CSD loss is compatible with any valid score distillation loss function (e.g., DMD loss or SiD loss). In our main experiments, we used SID loss with $\alpha=1.0$ as the default setting.
• In our understanding, the reviewer is suggesting we compare against other types of score distillation losses. To address this, we provide additional results below comparing different loss functions. We choose SID loss with varying $\alpha$ values and DMD loss as options. Due to time constraints of the rebuttal period and the limtied computational resources, all experiments are conducted under the default setting with 40,000 guidance iterations and 8,000 generator iterations.

• It can be seen that our method is relatively robust to different values of $\alpha$ in the SiD loss. For DMD loss, while it performs worse on the VAR model, it achieves performance comparable to SiD loss on the LlamaGen model. Overall, we recommend our default setting based on SiD loss with $\alpha=1.0$.

Weakness 2: It would be beneficial to include a more high-level flowchart in the main Methods section

• Thank you for the helpful suggestion! We agree that including a high-level flowchart would improve the clarity of our presentation.
• We have already illustrated the main components of our method in Fig 5. However, due to the limitation of space, we were unable to include the multi-stage training pipeline and other implementation details in this figure. We put them in the appendix instead. We will carefully consider your suggestion and revise the figure accordingly in the final version.

• Thank you for the insightful questions! Please see our detailed response below. We’re happy to continue the discussion if any concerns remain.

Weakness 2 & Question 1: MAR architecture

Our response is organized into three sub-questions:

1. Why do the student network and the guidance network both use the MAR architecture?
   • The generator must use an MAR-style structure (i.e., using an MLP to output continuous embeddings instead of probability vectors), because a one-step generator must should directly produce continuous samples. Token-wise probabilities (as in AR models) lead to independent sampling at each position and prevent gradient from back propagating through the CSD loss. Thus, a differentiable MLP head is essential.
   • The guidance model is trained to predict the conditional score of the generator’s distribution, which is also a continuous vector in the codebook embedding space. Therefore, it also requires an MLP architecture similar to MAR to operate in this continuous domain.
2. Is AR-diffusion training necessary?
   • As explained above, AR-style models (outputting token probabilities) are unsuitable as generator or guidance networks in our framework. Hence, we cannot use a shared architecture between student and teacher. In this case, AR-diffusion initialization becomes important (see Tab.4), and the proposed GTS loss proves effective (Tab.5).
3. Is the performance gain simply due to using MAR’s continuous representations?
   • We believe that the performance improvement is not brought by the MAR architecture. DD1 also employs an MLP head that outputs continuous representations like MAR (see Appendix B of [1]) for details). Since DD2 outperforms DD1 with similar architectural, we argue that the improvement stems from the new training strategy, not the continuous representations.

[1] Liu et al., Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models with Flow Matching.

Weakness 1: Distinction with diffusion score distillation

• Thank you for your thoughtful question! We would like to take this opportunity to clarify the novelty and contributions of our work.

• Technical Contribution/Novelty:

  • We discuss the high level non-trivial differences between our methods and diffusion score distillation & DD1 below:
    • Compared to Diffusion Score Distillation:
      • The problem we addressed is fundamentally different. Diffusion score distillation focuses on reducing the number of sampling steps in diffusion models. In contrast, our work targets one-step sampling for pre-trained AR models, which is under explored despite AR models achieving competitive or even superior performance compared to diffusion models[2][3][4]. We believe this makes our direction independently meaningful.
      • To the best of our knowledge, no prior work has directly applied score distillation to AR models. We are the first to enables this application by introducing a way to compute conditional scores with AR models. Accordingly, our guidance network is redesigned from a diffusion-based model to an AR-diffusion model, which brings a non-trivial architectural and conceptual shift.
      • From a high-level perspective, our method can be seen as decomposing score distillation into a sequence of AR steps. As discussed in Sec. 3.2.2, DD2 can be viewed as applying score distillation to each token position sequentially following the AR order, essentially splitting the original global distillation task into a series of simpler sub-tasks. This token-wise perspective is unique to our approach.
    • Compared to DD1:
      • While DD1 introduced a way to compute conditional velocities, it mainly used them to construct a static mapping and did not leverage this information in a more principled way. In contrast, DD2 explicitly integrates this information into the training objective, leading to more effective guidance.
  • Furthermore, directly applying score distillation to AR models is not feasible without several key adaptations. Beyond our high-level idea, we have made multiple technical innovations that serve as standalone contributions:
    • As discussed before, one-step generators must output generated samples directly, rather than probability distributions. Therefore, we replace the classification head of the pretrained AR model with an MLP that outputs continuous embeddings.
    • We identify the sensitivity of diffusion-style distillation to initialization. To address this, we propose adapting the pre-trained discrete AR model into an AR-diffusion model as an initialization. This modification has proved important (see Tab 4).
    • We observed that standard AR-diffusion loss is not efficient. Thus, we propose the GTS loss, which trains the AR-diffusion model using ground-truth scores as targets and converges much better (see Tab 5).

• Significant Performance Gains:

  • As far as we know, DD1 is the only prior work that successfully compresses AR models into one-step generation. DD2 achieves significantly better performance and lower training cost compared to DD1, thus pushing the boundary of one-step AR generation by a large margin.

• Broader Impact:

  • Our strong results offer a new perspective on training one-step generative models. Previously, the dominant strategy for training such models focused on diffusion-based frameworks. In contrast, our method demonstrates that following the AR schedule is also a highly competitive approach. As shown in the results below (adapted from the Shortcut model paper [5]), our method surpasses several representative and recent diffusion distillation techniques, offering an effective and scalable alternative.

  	DD2-VAR-d16	DD2-VAR-d20	DD2-VAR-d24	PD	CD	CT	Reflow	Shortcut
  FID	6.21	5.43	4.91	35.6	136.5	69.7	44.8	10.6

[2] Tian et al., Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction.

[3] Sun et al., Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation.

[4] Han et al., Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis.

[5] Frans et al., One Step Diffusion via Shortcut Models.

Weakness 3: Wrong symbol

• Thank you for pointing this out! You're right — we should have used a different letter, such as $\Psi$, to represent the AR-diffusion model, since $\psi$ is already used for the guidance network. We will revise all related notations and double-check the formulas in the revision.

Question 2: Why did you choose to use DMD for these experiments instead of validating it on CSD?

• We have already validated the effectiveness of AR-diffusion initialization under our CSD loss setting in the ablation studies. As shown in Tab 4, omitting initialization for either the generator or the guidance network leads to performance degradation or even training collapse.
• In Sec 3.3, we intentionally use the original DMD setup to demonstrate that the sensitivity to initialization arises from the nature of score distillation itself, not from our specific modifications. This strongly motivates our introduction of AR-diffusion tuning as an effective initialization strategy.

Question 3: The benefits of eliminating pre-defined mapping

• We provide the following intuitive explanation for why removing the pre-defined mapping from DD1 leads to better training:

  • More effective utilization of knowledge from the pre-trained model: The pre-defined mapping in DD1 provides only an end-to-end training signal. Due to the complexity of the mapping construction process, this signal is difficult for the model to learn. In contrast, DD2 directly aligns the score at each token position, providing a much richer and more localized supervisory signal.
  • Reduced accumulation of error:
    • DD1 constructs the mapping autoregressively, where the target token at each position depends not only on the noisy token at that position, but also on previously generated tokens. If the model fails to fit the mapping at one position, the errors will propagate to later positions, leading to cumulative error.
    • In DD2, the model aligns the conditional distribution of the generator (given previous tokens) with that of the teacher at each position. Thanks to the strong generalization ability of the pretrained teacher model, it can still provide a meaningful conditional distribution even when the previous tokens are imperfect. This behavior is also evidenced by our teacher-involved sampling results shown in Table 2. As a result, DD2 suffers from significantly less error accumulation.

• In addition, more generally, training generative models without relying on pre-defined mappings allows the model to discover smoother latent representations of the target data distribution automatically. This not only benefits training (because smoother representation is easier to learn), but also improves the model's ability to downstream tasks such as latent interpolation, which is an important downstream task in generative modeling[6].

  • To quantify this property, we adopt the Perceptual Path Length (PPL) metric proposed in [6], where a lower value indicates smoother interpolation in latent space. We evaluate this metric on both DD2 and DD1.

  	DD1	DD2
  PPL	18437.6	7231.9

  • The results are shown below, and clearly demonstrate that DD2 achieves significantly smoother latent interpolation than DD1. This is very likely because the pre-defined mapping in DD1 itself is not smooth, which inherently limits the smoothness of the learned latent representation.

[6] Karras et al., A Style-Based Generator Architecture for Generative Adversarial Networks.

I thank the authors for their answers. I suggest that the authors explicitly explain in the final version of the paper why their method is fundamentally different from diffusion score distillation, as well as clarify the advantages of the statement in the abstract, "Compared to DD1, DD2 eliminates the pre-defined mapping." This will help readers better understand the contributions of the paper. Based on this, I'll raise my score.

• Thank you for your positive feedback! Please see our detailed response below. We’re happy to continue the discussion if any concerns remain.

Weakness 1: Lack of large-scale experiments

• Thank you for the suggestion! We agree that larger-scale experiments would further strengthen the paper.
• Due to the limited time and computational resources available during the rebuttal period, we are unable to conduct larger-scale experiments at this time. However, we plan to include them in the revision and future work.
• Additionally, we would like to offer the following clarification to address your concern: ImageNet-256 is a sufficiently large and diverse dataset, and generation on this dataset is considered challenging. It has been widely adopted as the main benchmark in many recent works on new generative paradigms [1][2][3][4][5]. Our choice of dataset follows the setting in DD1. Therefore, we believe that using ImageNet 256 provides a meaningful and representative evaluation of our method.

[1] Tian et al., Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction.

[2] Sun et al., Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation.

[3] Frans et al., One Step Diffusion via Shortcut Models.

[4] Liu et al., Distilled Decoding 1: One-step Sampling of Image Auto-regressive Models with Flow Matching.

[5] Geng et al., Mean Flows for One-step Generative Modeling.

• Thank you for the insightful questions! Please see our detailed response below. We’re happy to continue the discussion if any concerns remain.

Weakness 1: Whether weighting schemes can be applied to different token positions

• Thank you again for raising this insightful point!
• In our default setting, we adopt a weighting scheme similar to SID[1], where each data sample is weighted by the norm of its score function. We use different weights for different data samples, which is different from your suggestion, where different weights are applied to each token position. To investigate this idea, we experimented with your proposed approach, where each token position is weighted by the norm of its score function.
• In addition, to further explore the impact of token-position-dependent weighting schemes on training, we also explored two static, token-wise weighting strategies:
  • The weights linearly decrease from 3 to 1 across token positions from the first to the last.
  • The weights linearly increase from 1 to 3 across token positions from the first to the last.
• We conducted experiments using the VAR-d16 model. Due to time constraints of the rebuttal period and limtied computational resources, we trained under the default setting with only 40,000 guidance iterations and 8,000 generator iterations. The results are summarized below: | Schedule | Default | Token Wise Norm | Static from 3 to 1 | Static from 1 to 3 | |:----------:|:---------:|:-----------------:|:--------------------:|:--------------------:| | FID | 8.35 | 10.85 | 6.91 | 10.44 |
• The results show that adjusting the token-wise loss weighting schedule is indeed beneficial. Assigning higher weights to tokens at the beginning of the sequence leads to much better performance compared to our default setting.
  • Possible reason: During training, the generator's objective at each token position is to align the conditional distribution of that token (given the previously generated tokens) with the corresponding distribution provided by the pretrained autoregressive model. This implies that tokens earlier in the sequence are responsible for providing accurate conditioning for subsequent tokens. If these early tokens are not well-trained, the learning targets for later tokens may also become unreliable. Therefore, increasing the loss weight for early tokens can help improve their learning quality and, in turn, enhance overall training effectiveness.

[1] Zhou et al., Score identity Distillation: Exponentially Fast Distillation of Pretrained Diffusion Models for One-Step Generation.

Weakness 2: Why not use Eq. (5) to train the conditional guidance network?

• Thank you for your careful reading!
• The loss in Eq. (6) is used to train a model to output the conditional score of the teacher (i.e., the pre-trained AR model) distribution. It relies on the pretrained AR model to provide the ground truth conditional probability of each token given all its previous tokens. This allows us to explicitly compute the ground-truth score for each token.
• In contrast, Eq. (5) defines the training objective for the guidance network, which aims to learn the conditional score of the one-step generator distribution. Since the one-step generator directly outputs the generated sample without explicitly modeling the conditional distribution of each token given the previous ones as the pretrained AR model does, we are unable to compute the exact ground-truth score for the generator.
• As a result, we follow the common practice in traditional diffusion and AR-diffusion frameworks and train the guidance network by approximating the score function using a Monte Carlo estimatation for each token's score, as formulated in Eq. (5).

Limitation 1: The quality of the one-step generation can still be improved.

• This is because our training process relies solely on the information provided by the pretrained AR model. Thus, the performance is inherently bounded by that model. Nevertheless, DD2 has already achieved a relatively small performance loss compraed to the pretrained AR model. We believe that incorporating real samples in the training process with adversarial objectives (e.g., GAN loss as used in [2][3]) could further overcome this limitation, which we leave for future work.

[2] Kim et al., Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion.

[3] Yin et al., Improved Distribution Matching Distillation for Fast Image Synthesis.

Limitation 2: The advantage compared to the one-step diffusion model is not clear.

• We compared several representative and recent methods in diffusion distillation on the ImageNet 256×256 benchmark, as shown below. Results of diffusion-based are taken from [4]. As can be seen, DD2 achieves significantly better performance compared to these methods.

• We would like to emphasize that the problem we aim to address is fundamentally different from diffusion distillation. Specifically, we focus on pre-trained AR models, while diffusion distillation is based on pre-trained diffusion models, which is a fundamentally different type of generative models. As a result, existing diffusion distillation methods cannot be directly applied to AR models.

• Notably, AR models have already achieved performance comparable to, or even surpassing, that of diffusion models [8][9][10]. Therefore, addressing the distillation of AR models presents an independently meaningful and valuable research direction.

[4] Frans et al., One Step Diffusion via Shortcut Models.

[5] Salimans & Ho, Progressive Distillation for Fast Sampling of Diffusion Models.

[6] Song et al., Consistency Models.

[7] Liu et al., Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow.

[8] Tian et al., Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction.

[9] Sun et al., Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation.

[10] Han et al., Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis.