We sincerely thank you for the valuable feedback. We address each comment below.

The paper needs proofreading as it contains many typos and grammatical issues

Thank you for your valuable comments. We will revise those typos and grammatical issues in the revision.

The limitations section is quite limited, I think the paper could benefit from further limitations discussion including images of failure cases.

Thank you for your great suggestion. We will discuss our limitations in greater detail in our revision. Specifically, we will add some failure cases and corresponding analysis.

What are the results of Reward-Instruct over more complex prompts such as the ones presented in Figure 2?

Our model also performs well on complex prompts. For example, the prompts used in Figure 1 have some complex cases: "A alpaca made of colorful building blocks, realism", "A professional portrait photo of an anthropomorphic capybara wearing fancy gentleman hat and jacket walking in autumn forest". We will add more samples on complex prompts in revision, but we cannot provide it in the rebuttal stage due to the policy.

“Figure 6: The prior distillation-based reward maximization methods collapse when the reward is chosen to be HPS v2.1.” – what happens for prior approaches vs. Reward-Instruct when not using HPS v2.1? (e.g. using HPS v2.0 [60] or Image Reward [61] as in their original approach?)

Great question. The results in Figure 5 and Table 1 for RG-LCM and DI++ are obtained by using the reward in their original approach. It can be seen that our proposed Reward-Instruct achieves better performance.

Figure 7 – there is a mild improvement when increasing the number of rewards used, but it seems more like enhancing the gradient step because the improvement is generally toward the same direction. My question is – what happens if you use a higher weight for one of the regularization methods instead of using the weighted average?

We argue that the performance improvement brought by multiple rewards is not merely due to the enhancement of gradient step. We presented the results using weak regularization in Figure 8 (small weight regularization coefficient and without using random eta sampling), equivalent to a higher weight of rewards. From Figure 8, it can be observed that using a single reward leads to severe artifacts, while ensembling through multiple rewards can generate reasonably high-quality images.

Besides, we conduct an extra study on using a large weight regularization coefficient (5 times larger). The results are shown below:

It can be seen that with large regularization and multiple rewards, the performance on rewards is degraded while the zero-shot FID becomes better. And its performance is way better than using only single reward.

In addition to text proofreading that is needed, in some figures it is not clear what are the prompts for the examples, e.g. Figure 8, or it looks like the wrong prompt/images, e.g. in Figure 7 the "squirrel" example which shows bird images. These should be fixed.

Thank you for your effort! We will add the prompt descriptions in Figure 8 and will revise the wrong prompts (which should be "bird") in Figure 7. We will also carefully check typos and improve the clarity of the presented figures.

We sincerely thank you for the valuable feedback. We address each comment below.

Generalization of the final model is unclear. The authors seem to maximize the combination of HPS, ImageReward, and CLIP scores (I would appreciate it if the authors could confirm this and also clarify which HPS reward model in particular). However, their evaluations measure the scores from these same reward models plus the LAION Aesthetics model. Thus, it’s unclear if the preference alignment generalizes outside of these reward models– e.g., how do the results compare if measuring Pick-a-Pic reward model score, or a different version of HPS rm, or ideally if using human annotators?

Great suggestion. We maximize HPS V2.1, ImageReward, and Clip Scores. To fully confirm the effectiveness of our Reward-Instruct in generating high-quality samples, we conduct a user study between the base model (50 NFE) and Reward-Instruct (4 NFE). The user study is conducted by presenting users with two anonymous images generated by different models and asking them to select the sample with higher image quality and better prompt alignment. We randomly selected 20 prompts for image generation. In total, we collected approximately 20 user responses on 20 pairs.

It can be seen that Reward-Instruct is able to generalize to unseen rewards, and also achieves better performance regarding user preference compared to the pre-trained base models.

Comparisons to reward-centric approaches. Many of the reported baselines, e.g. ReFL, optimize their model w.r.t to one reward signal whereas the benefit of ensembling rewards is a key finding of this paper. I understand that re-training such baselines may be infeasible, however I think it should be noted that the method benefits from multiple signals whereas many other baselines do not, when in fact their performance may improve with multiple signals too.

Great suggestion. This is a very interesting direction. Indeed, some other RL methods that integrate multiple rewards could also potentially further improve performance. We will discuss this point in the revision and further explore it in future work. Nevertheless, it is worth noting that our reward-centric approach directly produces few-step samplers, without the need for diffusion distillation. In this sense, ReFL (standard diffusion sampling) is not a direct baseline in our setting.

Comparisons to offline approaches. Given the prevalence of offline approaches like Diffusion-DPO, how does Reward-Instruct compare to SD-1.5 aligned via Diffusion-DPO or Diffusion-KTO?

We note that the DPO-like methods belong to another kind of method. We are also glad to provide the comparison between DPO and our Reward-Instruct below:

We find that DPO typically aligns less with human preference compared to reward-centric approaches, but performs better regarding the FID. We believe that a possible reason is that DPO training still relies on curated data, which benefits the FID. However, it is clear that RI is not in conflict with DPO in that one can readily use a DPO-aligned diffusion model as a base model and apply RI after it to get the advantage of both. However, in this paper, we evaluate RI without the influence of DPO to clearly study its pros and cons.

Could the authors please elaborate on the design decisions when selecting reward models? The justification for reward ensembling highlights that maximizing the reward based on shared data modes across different reward models can help avoid reward hacking. In the paper however, the reward models used are somewhat similar in that they are optimized for some combination of image aesthetics and/or image-text alignment. Does this mean the reward models must be “sufficiently similar” in terms of what they reward? As a hypothetical example, if you included a safety classifier as one of the reward models would the training still be stable and, most importantly, effective?

We believe that rewards do not need to be sufficiently similar to each other, and that the model may benefit from combinations of different types of rewards. In Table 3 of the Appendix, we maximize HPS, CLIP scores, and AeS, where each reward represents a different type. In particular, HPS represents human preference, CLIP score represents image-text alignment, and AeS represents pure aesthetics. The results in Table 3 demonstrate that this combination can achieve similar rewards as well as notably better zero-shot FID, suggesting better avoidance of reward hacking. To further verify this, we took your valuable suggestion and incorporated a "safety classifier" into our training.

Results in above show that after integrating the safety classifier into the training, we can still achieve comparable performance and slightly better zero-shot FID.

Hi,

I would like to thank the authors for their work and answers to my questions. However, regarding my first question about generalization, I still have some reservations about the provided results. First, wrt the automated metrics (HPS, etc.), HPS v2.1 is trained using the HPSv2.0 dataset so it would seem that maximizing HPSv2.1 would maximize HPSv2.0 score. However, the improvement in PickScore does seem promising as most papers usually show ~+0.2 improvement. My biggest issue is that the results do not provide enough information to gauge statistical significance which is important given that these are probabilistic models. The user study only compares 20 images which seems very small and it's hard to tell if the win-rate is significant. Also, this user study seems to have only gathered 1 response per image pair. I believe the questions about the generality of this method still remain.

I would like to hear the authors' thoughts on these questions before I finalize my review. Thank you.

We sincerely thank you for the valuable feedback. We address each comment below.

Since Reward-Instruct directly maximizes rewards over a few-step sampling process without explicit diffusion distillation, can the resulting outputs still be interpreted as valid diffusion score functions? Or should the model be viewed simply as a general epsilon-conditioned generator lacking traditional diffusion properties?

The generator trained by Reward-Instruct is no longer diffusion score functions. We note that almost all distilled diffusion models are also no longer diffusion score functions, since their objectives are not for learning scores.

Without explicit diffusion distillation, how can RI theoretically or empirically guarantee meaningful preservation of the original pretrained diffusion model’s trajectories?

Currently, there are usually two mainstream diffusion distillation approaches: one is distribution matching, which aims to align the student and teacher at the distribution level (such as SD-Turbo [48] and Diff-instruct [28]); the other is trajectory distillation, which aims to simulate the original diffusion model’s trajectories with fewer steps (such as Consistency Models [5] and CTM [21]). Only the latter aims to preserve the original diffusion model’s trajectories, while the former performs much better than the latter and is the most popular approach currently. Our method, different from traditional diffusion distillation approaches, is a novel reward-centric approach. Note that, similar to distribution matching, our approach achieves strong performance without preserving the original diffusion model’s trajectories.

It is also worth emphasizing that there has recently been a paradigm shift from supervised training (data-centric) to RL training (reward-centric). In the LLM field, the community is actively exploring reward-centric post-training approaches that can demonstrate powerful emergent capabilities, e.g., the GRPO adopted in DeepSeek R1. Our work represents a systematic study of the first reward-centric approaches to post-train pre-trained diffusion models into powerful few-step generators. By our reward-centric RI, a simple yet elegant approach that doesn't rely on traditional diffusion distillation techniques, can achieve surprisingly good few-step generation performance.
We hope our investigation is helpful for the community to make further explorations along this novel path.

Why exactly does Reward-Instruct provide increased robustness against reward hacking compared to methods incorporating explicit diffusion distillation losses (e.g., DI++, RG-LCM)? Could similar robustness be achieved simply by applying RI-style reward maximization after completing standard diffusion distillation training?

Our work aims to explore a reward-centric approach to achieve fast generation without the need for tricky yet expensive diffusion distillation. To achieve this, we investigate several simply yet effective regularizations to alleviate reward hacking, which could also potentially benefit the distillation-based reward maximization for better performance. It is our experimental observation that our RI delivers surprisingly robust performance with simple regularization. However as you suggested, we agree that it is exciting to further explore the theoretical properties of regularizations introduced with RI in the future.

Interestingly, we can explore combining reward-centric RI training with traditional distillation in our future work to produce more powerful models.

Starting from the distilled model is a much stronger initialization compared to pre-trained diffusion models. We can directly initialize from the distilled model, and replace the prior with the weight of the distilled model to construct RI for fine-tuning the distilled model. We show results below:

It can be seen that RI can effectively fine-tune the distilled models with better performance compared to initializing from pre-trained models.

Several closely related recent works addressing reward-guided few-step diffusion methods are missing: PSO [a], LaSPO [b], DSPO [c].

Thank you for bringing our attention to these interesting works. We discuss the mentioned works below:

• DSPO, PSO, and LaSPO focus on essentially different tasks compared to us. DSPO focus on fine-tuning pre-trained diffusion models, which is not for constructing few-step models. PSO and LaSRO aim to fine-tune the well-trained few-step models, which is a different and much easier setting compared to our method and our baselines. The notable contribution and finding of our work is that: we show that reward-centric approach can fine-tuning form pre-trained diffusion models to few-step generator.

• PSO and DSPO belong to DPO-like methods, which are essentially different from RL methods that aim to maximize a certain reward. In their papers [a, c], there is no comparison between DPO-like methods and RL-based methods.

• However, we are also glad to provide a comparison below:

It can be seen that applying reward-instruct for post-training distill model (LCM) is more effective than starting from pre-trained diffusion models, while reward-instruct init w/ LCM outperforms the prior works in fine-tuning LCM regarding both reward metrics and FID.

[a] Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization

[b] Reward Fine-Tuning Two-Step Diffusion Models via Learning Differentiable Latent-Space Surrogate Rewards

[c] DSPO: Direct Score Preference Optimization for Diffusion Model Alignment

How sensitive is RI to variations in reward scales or reward noise? Can RI reliably maintain stability and image quality if rewards become noisy or biased?

For reward scales, we have applied gradient normalization to balance different rewards, which effectively balances contributions from different rewards regardless of their individual scales. Hence, our method is robust to reward scales. We have verified the effectiveness of gradient normalization in Figure 9.

In the case of biased rewards, we expect RI with multiple rewards to be also relatively robust. On one hand, if only one reward is biased, its negative effect will not be significant since the supervision is an average of all rewards. On the other hand, even if all rewards are biased, we believe that as long as the biases are not consistent (e.g., biases are random with zero mean), their average effect should be neutral.

It would be an interesting future work to conduct exact experiments about this matter. However, such an investigation is difficult to conduct in a rigorous manner since, in the current community, it is difficult to obtain a large pool of rewards with different sources of quantifiable biases.

Inconsistent Naming Conventions

RI is short for Reward-Instruct, and R0 is a typo. Sorry for the confusion. We will clean up these inconsistent abbreviations and use RI as our abbreviation.

Unclear CFG Loss Description

It has been pointed out in [27] that CFG can be viewed as an implicit reward function. Using it in our RI incurs a slight difference compared with other rewards, since it does not have an explicit value.

However, we can compute its gradient. Hence, the CFG loss is a surrogate loss that has the same gradient of $-E_{x_t,t}\log p_\psi(\mathrm{text}|x_t) = -E_{x_t,t}\log \frac{p_\psi(x_t|\mathrm{text})}{p_\psi(x_t|\emptyset)}$. We will add a detailed description in our revision.

Missing Algorithm Description for Reward-Instruct+

Thank you for your suggestion. We have detailed our RI+ in Section 3.4, but didn't summarize it into an algorithm. We will add an algorithm description for Reward-Insturct+ in revision.

SD3-Medium results are presented without comparison to existing methods, reducing interpretability and value.

We note that the multi-step SD3-Medium itself is a strong baseline. The comparison between multi-step SD3-Medium and 4-step RI shows that RI, as a reward-centric approach, is capable of post-training powerful flow-based models to few-step generators with better reward metrics and comparable zero-shot FID. This highlights RI's effectiveness and its wide application.

Lack of SDXL Comparison.

We compare to the base model (SDXL) and distillation-based RL methods, i.e., RG-LCM and DI++.

The results shown above show that the proposed Reward-centric approach - Reward-instruct also has advantages over prior distillation-based RL methods on the SDXL backbone.

loss = reward_function(model, target)
loss.backward()
# Detailed SGLD implementation here

loss.backward()
for param, param_0 in zip(model.parameters(), model_pretrained.parameters()):
    with torch.no_grad():
        if param.grad is not None:
            # SGLD noise:
            noise_para = torch.randn_like(param.grad.data) * noise_scale 
            # weight regularization:
            reg_para = (param - param_0) * reg_scale
            param.grad.data.add_(noise_para)
            param.grad.data.add_(reg_para)
optimizer.step()

We sincerely thank you for the valuable feedback. We address each comment below.

Many of the provided insights are only underpinned by a handful of cherry-picked examples (e.g. Fig. 4, Fig. 6, Fig. 7, Fig. 8, Fig. 11). The paper would benefit from corroborating these findings with quantitative experiments

We provide quantitative results about removing one technique at a time to show the effectiveness.

The results, presented in above, demonstrate the critical role of each component:

1. Multiple rewards: Improves both rewards and FID, mitigating artifacts and reward hacking.
2. Random-$\eta$-sampling: Maintains similar reward performance but significantly improves FID, aiding to find better mode with fewer artifacts.
3. Weight regularization: Trades slight reward gains for better FID, ensuring the generator stays within the image manifold.

Use of LAION Aesthetics classifier v1 instead of v2, despite the improved version being readily available

We clarify that AeS v2 is used in our experiments.

Inferring improved convergence from Fig. 10 (Reward Instruct+) is a bit of stretch. No improvement for HPS for example

Reward-instruct only provides supervision at the endpoint and requires backpropagation to flow through the entire generator (multiple denoising steps), while Reward-instruct+ provides signal supervision during the process, which intuitively enables faster convergence and higher training efficiency, as we only allow RI+'s gradients to backpropagate one step rather than through the entire generator. Reward-instruct+ can converge faster than Reward-instruct regarding most rewards. Although it doesn't show significant gains on HPS, RI+'s training is cheaper (each iteration requires only 65% of Reward-instruct's time), which makes it still have clear advantages over Reward-instruct overall. Exploring how to make RI+ have more consistent gains would be an interesting future direction.

The presentation can be improved. Especially the sudden introduction of Reward Instruct+ over more substantive experiments may confuse some readers

Thank you for your valuable comments. We will add more motivating analysis and illustrations for the introduction of Reward-Instruct+ in our revision.

The paper would immensely benefit from a human user study on actual improvements in quality, since automated metrics are known to be faulty (and as discussed in the paper, there may be reward hacking)

Great suggestion. We provide a user study to compare the base model and our reward-instruct. The user study is conducted by presenting users with two anonymous images generated by different models and asking them to select the sample with higher image quality and better prompt alignment. We randomly selected 20 prompts for image generation. Due to the limited time in the rebuttal phase, we collected approximately 20 user responses on 20 pairs in total.

Results in above shows that the proposed RI achieves better user preference compared to the base models, indicating the effectiveness of the proposed reward-centric approach in aligning human preference.

Could you elaborate further on the intuition $\epsilon_\eta$ behind (L146)? Is this routed in prior literature on DM sampling?

This refers to an interpolation between the Consistency Model (CM) sampler and the DDIM sampler. The former is a purely stochastic sampler ($\eta=0$), while the latter is a deterministic sampler ($\eta$ = 1). There is a notable difference in both the sampling paths and the style of the resulting samples generated by these two distinct methods. Our experiments in Figure 4 further show that different $\eta$ leads to different styles. This discrepancy drives the strategy of interpolating between them by randomizing the eta parameter during sampling.

What is the importance of randomly sampling $\eta$ at different diffusion steps? Could you provide further analysis of the impact of that design choice?

Randomly sampling $\eta$ at different diffusion steps can augment the generator's output distribution and broaden the scope of its exploration, since different $\eta$ deliver different sampling trajectories and styles. Consequently, we believe this can force the generator to learn a more robust policy by maximizing the rewards across these varied sampling paths, which ultimately improves its robustness.

If we fixed $\eta$ in sampling during training, the generator only needs to learn to maximize the reward under a specific sampling trajectory and style, which makes it easier for the generator to hack the reward.

The results above show that fixed $\eta$ leads to inferior results in both FID and reward metrics.