Improving Video Generation with Human Feedback

We sincerely appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one by one in what follows.

[W1] Clarify the motivation for introducing three strategies for flow-based models within a reward model.

Thank you for your insightful comments. The three alignment algorithms are not isolated; they all stem from a unified reinforcement learning framework that aims to maximize reward with KL regularization. In other words, these strategies are three different implementations of the same optimization objective.

These strategies were introduced to address the different practical needs in applying human preference alignment to flow-based models. Specifically, Flow-DPO and Flow-RWR are designed for training time, with our results showing Flow-DPO to be more effective. Flow-NRG, on the other hand, is designed for inference time scenarios where retraining is impractical.

[W2] Limited Preference Aspects: Additional aspects, such as temporal coherence, should be considered, or the authors could provide further justification for focusing solely on these three aspects.

In our paper, we primarily focus on three aspects: Visual Quality (VQ), Motion Quality (MQ), and Text Alignment (TA). Section F provides a detailed explanation of the specific factors considered under each aspect. Regarding temporal coherence, we treat it as a component of Motion Quality (see Table 7). This categorization is inspired by the design of image reward models like ImageReward [1], which typically consider VQ and TA. Building on this, we introduce MQ to account for the temporal dimension of video preferences. We believe this three-aspect formulation is reasonable, and we agree that adopting a more fine-grained decomposition could be a valuable direction for future work.

[1]. Imagereward: Learning and evaluating human preferences for text-to-image generation. NeurIPS 2023.

[W3] Selecting video pairs from the same VDM (with different random seeds) might result in more comparable positive and negative videos, enabling finer-grained preference modeling

Thank you for the valuable suggestion. We agree that sampling pairs from the same VDM could yield finer-grained preferences. In our current setup, although the pairs are drawn from different VDMs, the models are comparable in capability, with similar mean MOS scores (see the table below). Here, MOS refers to the mean human opinion score, where annotators rate individual videos on a scale from 1 to 5. So this choice does not, in practice, prevent effective preference modeling. We acknowledge that intra‑model pairing is a promising direction and will explore it in future work.

[W4] Broader evaluations on more base models are necessary.

As shown in Section C and Figure 7, our base model follows a standard VDM recipe—a 3D VAE coupled with a Transformer-based latent diffusion model. Most contemporary T2V models (e.g., Sora [2], Hunyuan Video [3], Wan2.1 [4]) adopt the same formulation. Additionally, our method is not tied to any architecture‑specific components. We believe that the results on this architecture are representative, and our approach can be applied to other T2V models without modification.

[2]. Sora: Video generation models as world simulators.
[3]. Hunyuanvideo: A systematic framework for large video generative models.
[4]. Wan: Open and Advanced Large-Scale Video Generative Models.

[W5-1] Reference or Experimental Support: Line #27 regarding previous reward models.

Previous reward models (e.g., VideoScore [5], LIFT [7]) were trained on datasets consisting primarily of low resolutions (often $\le$ 480p) and short durations (2s), whereas ours cover recent T2V models (most $\ge$ 720p) with higher resolutions and longer clips (3s -6s). The statistics are listed below. We will add these statistics in the revision.

Furthermore, as shown in Table 2, while prior reward models perform well on GenAI-Bench, they perform significantly worse on VideoGen-RewardBench compared to ours. GenAI-Bench videos are shorter (2–2.5 seconds) and lower in quality, while VideoGen-RewardBench features longer clips (4–6 seconds) and higher resolution generated by modern video models.

[5]. Videoscore: Building automatic metrics to simulate fine-grained human feedback for video generation. EMNLP 2024.
[6]. Videodpo: Omni-preference alignment for video diffusion generation. CVPR 2025.
[7]. LiFT: Leveraging Human Feedback for Text-to-Video Model Alignment. arXiv 2024.

[W5-2] Reference or Experimental Support: Line #251 regarding the effects of full fine-tuning vs. LoRA fine-tuning

Thank you for your helpful suggestion. Regarding line #251 ("full-parameter updates were observed to hurt quality or even collapse the model"), our preliminary ablation shows that full-parameter fine-tuning more often degrades overall quality and is more susceptible to reward hacking issues than LoRA-based tuning. Recent works like SD3 [8] also favor LoRA over full fine-tuning in their post-training stage. We will add the supporting evidence in the revision.

[8]. Scaling rectified flow transformers for high-resolution image synthesis. ICML 2024

We sincerely appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one by one in what follows.

[W1]: Limitations of using model-generated data for reward model training.

We appreciate the reviewer’s observation and agree that current SOTA video generation models still fall short of producing video clips that match the fidelity of real-world videos. However, we would like to clarify the reasoning behind our choice of using model-generated data for training the reward model:

1. The goal of the reward model is to capture human preferences among outputs with similar generation quality, rather than to distinguish between real and generated videos. As the reviewer rightly pointed out, given the current gap between real and generated videos, assessing the quality difference between real and generated videos is often a trivial task. This distinction may not provide meaningful learning signals for the fine-grained preference modeling required in our setting. Constructing such incomparable pairs might hinder the model's ability to learn subtle preference differences.
2. Our reward model is primarily used to guide post-training algorithms such as Flow-DPO or other RL-based methods. In this context, the reward model needs to capture preference distributions among outputs that are within the capability range of the base generation model, as these are the samples it will be asked to evaluate during alignment. For this reason, we intentionally construct the reward model training set using generations from recent, high-quality T2V models. This ensures the reward model can provide relatively accurate reward signals during the alignment phase.

[Q1] Do you plan to open-source your dataset and pre-trained reward model?

Yes. We will open‑source our reward model, including the model weights as well as the code for training and evaluation. The dataset is subject to a stricter internal review; we are actively working through that process and will update it once approval is obtained.

[Q2] Challenges of Real-Data Training in DPO/GRPO Settings?

Yes, we have explored the use of real videos in DPO training, for example, by using real videos as the chosen samples and generated videos as the rejected ones. However, this setup often leads to undesirable effects, such as over-sharpened outputs and the emergence of grid-like patterns in generated frames. We attribute this to the significant distributional gap between real and generated videos produced by current T2V models. Such a mismatch may result in suboptimal alignment outcomes.

To mitigate this, possible solutions include (1) incorporating an auxiliary supervised fine-tuning (sft) loss to stabilize training and (2) increasing the $\beta$ coefficient in the DPO loss to enforce stronger KL regularization. Additionally, if the primary failure is degraded visual quality (assuming DPO focuses on aligning other aspects such as motion naturalness), we have verified that a short sft stage (around 50 steps) on high-quality real videos after DPO training can effectively restore visual fidelity. In summary, if DPO leads to noticeable degradation in visual quality, a subsequent SFT stage on high-quality data can recover fidelity with very few steps, while still preserving the alignment benefits of DPO.

We sincerely appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one by one in what follows.

[W1] Is it necessary to differentiate between Diffusion-DPO and Flow-DPO?

Thanks for asking this clarification question! A straightforward way to extend Diffusion-DPO to flow matching is indeed to replace the diffusion's noise prediction term with the FM's velocity prediction term. However, this heuristic lacks theoretical grounding, so whether it works is unpredictable.

We first utilize the connection between diffusion and flow matching and derive a vanilla Flow-DPO objective that naturally carries a timestep-dependent $\beta_t$. Our analysis shows that this form induces a timestep-dependent KL constraint, which biases alignment towards high-noise (early) steps and triggers noticeable reward hacking on certain dimensions. Motivated by the DDPM simplification from Eq.12 to Eq.14, we then test a variant that replaces $\beta_t$ with a time-independent constant $\beta$; this version performs better overall and we term it as "Flow-DPO".

Thus, although Flow‑DPO looks similar to Diffusion‑DPO in form, we provide the theoretical rationale and empirical evidence for why the constant‑$\beta$ variant is preferable to the timestep‑dependent one. Throughout the paper, we stress that Flow‑DPO is an extension of Diffusion‑DPO to flow‑matching models. Given that leading image generation models (e.g., SD3 [1], FLUX [2]) and video generation models (e.g., Sora [3], Wan2.1 [4], Seedance [5]) now adopt flow matching in place of traditional diffusion, we believe this distinction is necessary to keep the theory reasonable and making the framework more intuitive for the community.

[1]. Scaling Rectified Flow Transformers for High-Resolution Image Synthesis.
[2]. FLUX.
[3]. Sora: Video generation models as world simulators.
[4]. Wan: Open and Advanced Large-Scale Video Generative Models.
[5]. Seedance 1.0: Exploring the Boundaries of Video Generation Models.

[W2] The novelty of the paper is relatively limited.

We respectfully disagree with the assessment that the novelty of our work is limited. While it is true that some components were built upon existing paradigms in other fields, we emphasize that human preference alignment in video generation remains in its early stage, with several initial challenges yet to be adequately addressed. These include: (1) the mismatch between existing preference datasets and modern T2V models; (2) the underutilized potential of VLM-based reward modeling; and (3) the limited applicability of alignment algorithms like DPO. Our work aims to propose a comprehensive alignment pipeline tailored to flow-based T2V models and validate its effectiveness in improving video generation.

Importantly, our contributions go beyond straightforward adaptations. For example, we show that directly extending Diffusion-DPO to flow-based models by leveraging the connection between diffusion and flow matching leads to reward hacking and suboptimal performance. Our Flow-DPO addresses this with a constant KL constraint, resulting in more stable and effective alignment. While Flow-DPO is similar to Diffusion-DPO in form, we provide both theoretical rationale and empirical evidence to support its effectiveness. Likewise, our proposed Flow-NRG offers a novel inference-time method for distribution alignment without additional training. Beyond the alignment algorithms, our work also contributes insights into dataset construction and reward model design, offering a comprehensive pipeline to preference alignment for video generation.

[W3] The performance difference between CFG and Flow-NRG.

Thank you for your question, and we apologize for the confusion. Noisy Reward Guidance and classifier-free guidance (CFG) are orthogonal techniques. In all our experiments, the model first uses CFG to obtain an initial velocity, followed by Noisy Reward Guidance to refine it.

As shown in Table 5, we directly compare the full method (CFG + Noisy Reward Guidance) with the CFG-only baseline. The reported win rates (%) reflect the performance gains brought by Noisy Reward Guidance on top of CFG. We will clarify this point more explicitly in the revision to avoid further confusion.

[W4] VBench results show no significant overall improvement over the pre-trained model.

While the VBench score shows a modest increase from 83.19 to 83.41 after applying DPO (Table 3), this improvement is more significant than it appears. We argue that in generative AI, relying solely on automatic metrics to assess model capability remains challenging and can mask perceptual gains. Beyond VBench, our user study (Fig. 5) shows 44.0% wins for the DPO‑tuned model versus 27.0% for the pretrained baseline (29.0% ties). These consistent advantages indicate that DPO significantly improves visual quality and alignment despite the modest increase in the VBench score. We encourage the reviewer to view the video examples in the supplementary materials for a visual comparison of the improved generation quality.

[Q1] Clarify reward training on noisy latents or pixel space

We apologize for the confusion caused by the lack of detail. For the "Efficient Reward on Noisy Latents" section, the loss is computed on noisy latents. As noted in line 208, we train a time-dependent reward model $r_\theta(·, t)$ directly in latent space to avoid the costly backpropagation through the full VAE decoder required for computing ∇r in pixel space.

Thanks for the further discussion. It has resolved my concerns about the limited performance gains. However, although the authors state that the goal is to "align modern, state-of-the-art flow-based video models with human preferences," the technical novelty and theoretical explainability of the work are still limited. Considering both the strengths and weaknesses of the paper, I will maintain my original score and leave the final decision to the ACs.

We sincerely appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one by one in what follows.

[W1 & Q1] The quantitative evaluation for Flow-NRG.

Table 5 in the paper reports Flow‑NRG’s quantitative evaluation (via quantitative) on VideoGen‑Eval under custom reward weights. For convenience, the numbers are reproduced below. The three columns (VQ, MQ, TA) are the win rates (%) against the vanilla model when we steer the generator with different reward-weight triplets $w_{\text{VQ}}:w_{\text{MQ}}:w_{\text{TA}}$, where $r=w_{\text{VQ}}*r_{\text{VQ}}+w_{\text{MQ}}*r_{\text{MQ}}+w_{\text{TA}}*r_{\text{TA}}$.

The results show Flow‑NRG reliably steers generation toward the weighted objectives and improves alignment. Compared with other methods in Table 1, Flow‑NRG achieves performance comparable to SFT and somewhat below Flow‑DPO, which is reasonable given that Flow‑NRG is purely inference‑time.

[Q2] The visual manifestations of reward hacking

Thank you for the suggestion! We observe that Flow-DPO with time-dependent $\beta$ is more prone to hacking issues, including abnormal color shifts, unnatural brightness and contrast, and reduced diversity (e.g., the model tends to enlarge the main subject to avoid penalties from missing local details). These artifacts degrade the general quality yet still yield higher rewards. We will add qualitative examples of these hacking issues in the revision.