Hierarchical Fine-grained Preference Optimization for Physically Plausible Video Generation

Thank you to all the reviewers for your thoughtful and constructive comments! We are truly encouraged to see that the reviewers appreciate several positive aspects of our work, such as strong performance (Reviewer epiP, EaEb, JHEX, R9Wj), comprehensive & convincing experiments (Reviewer epiP, R9Wj), novel & effective framework (Reviewer R9Wj, EaEb), valuable & practical data selection pipeline (Reviewer R9Wj, EaEb, JHEX), well-motivated methodology (Reviewer epiP, JHEX), and clear writing (Reviewer epiP).

Your expertise and thoughtful review have been instrumental in improving our manuscript. Below, we address your comments point-by-point to clarify and strengthen our work.

---

Weakness 1: Hierarchical modeling may increase training complexity or cost. There is no clear discussion in the main text regarding the computational demands for the hierarchical DPO approach relative to vanilla DPO or SFT approaches.

Thank you for pointing out this important aspect. We acknowledge that hierarchical modeling increases computational complexity, but it also brings significant performance gains. Balancing this trade-off presents a challenge and an opportunity for developing more efficient, scalable approaches, which we will also further discuss in future versions.

---

Weakness 2: The approach employs LLM/VLM-based scoring and captioning for data selection and semantic preference, but does not thoroughly assess how model choice impacts reliability.

Thank you for highlighting this issue! Different foundation models indeed impact downstream performance. Here we compare several VLM in our pipeline:

Table A: Performance Comparison between Different VLMs on VideoPhy

Qwen2.5, a widely used open-source model, outperforms others, emphasizing the importance of choosing suitable LLMs and prompts for task-specific pipelines. Variations in architecture and pretraining data explain these differences, with models like Qwen2.5 excelling in multimodal reasoning tasks.

---

Question 1: Although improvements are shown, the main text lacks systematic analyses of failure cases. where does PhysHPO still fail to produce physically plausible videos, or what are the remaining challenges?

Thank you for pointing this out. PhysHPO, as a strategy to enhance the physical plausibility of VDM-generated videos, has demonstrated strong performance and deeper learning from real-world videos. However, a systematic analysis of failure cases reveals certain limitations. Specifically, high-speed human action videos, such as dancing or gymnastics, often fail to capture precise motion dynamics and physical plausibility. This is also largely due to limitations in the base model’s ability to represent complex, rapid movements, as well as potential gaps in the quality and quantity of the training data used.

These challenges highlight areas for improvement and remain open questions for further exploration by the research community. We recognize the importance of thoroughly analyzing failure cases and will include a more detailed discussion in a future version of this work.

---

Question 2: Does the filtering pipeline's effectiveness depend heavily on threshold choices, and if so, how significantly do these parameters impact final model performance?

Yes, the effectiveness of the filtering pipeline does depends on threshold choices, which could impact the quality and relevance of selected data. While Figure 3 provides a preliminary comparison, here we present a more detailed analysis:

Table B: Diversity Selection Threshold Analysis on PhyGenBench

It clearly demonstrates how selecting appropriate thresholds is crucial for ensuring high-quality data selection and improving downstream task performance.

---

Question 3: Does the framework maintain its effectiveness for non-physical domains (e.g., stylized content), and are these edge cases systematically evaluated in the experiments?

Thank you for your thoughtful question! We have considered this important evaluation criterion and used VBench as the benchmark for non-physical domains, including 16 metrics such as Appearance Style (AS), as shown in below:

Table C: Full Dimensions of VBench Evaluation

The results demonstrate that PhysHPO not only significantly enhances the physical plausibility of videos generated by the base model but also maintains or even improves its performance in non-physical (general) domains.

---

We hope this rebuttal sufficiently addresses your concerns and clarifies the points raised. We are open to further discussion and greatly appreciate your thoughtful feedback.

Dear Reviewer epiP,

This is a gentle reminder that the discussion phase will end in less than 2 days, and we have not yet received your feedback on our rebuttal. We fully understand the demands on your time and sincerely appreciate your efforts in reviewing our work. To facilitate communication, we’ve summarized our key responses below:

• Computational Complexity (Weakness 1): We acknowledged the increased training complexity of hierarchical modeling, emphasizing the trade-off between computational demands and significant performance gains. We will explore optimization methods in future work.
• Model Reliability (Weakness 2): We compared several LLMs in our pipeline, demonstrating that Qwen2.5 excels in the reasoning tasks.
• Failure Cases (Question 1): We systematically analyzed failure cases, identifying challenges in high-speed human action videos (e.g., dancing, gymnastics).
• Threshold Sensitivity (Question 2): We analyzed the impact of threshold choices on the filtering pipeline’s effectiveness.
• Effectiveness in Non-Physical Domains (Question 3): Using VBench, we demonstrated that PhysHPO maintains or improves performance in non-physical domains, ensuring robustness across diverse content types.

Thank you once again for your thoughtful insights and dedication throughout the review process. Your insights deeply inspired us, and we are genuinely committed to addressing your concerns with the utmost care. We would be greatly grateful if you could let us know whether our rebuttal has sufficiently addressed your concerns.

Thanks and Regards,

The Authors

Thank you to all the reviewers for your thoughtful and constructive comments! We are truly encouraged to see that the reviewers appreciate several positive aspects of our work, such as strong performance (Reviewer epiP, EaEb, JHEX, R9Wj), comprehensive & convincing experiments (Reviewer epiP, R9Wj), novel & effective framework (Reviewer R9Wj, EaEb), valuable & practical data selection pipeline (Reviewer R9Wj, EaEb, JHEX), well-motivated methodology (Reviewer epiP, JHEX), and clear writing (Reviewer epiP).

Your thoughtful review has been invaluable in helping us enhance our manuscript. Here, we address your comments thoroughly to ensure clarity and accuracy.

---

Weakness 1: How does PhysHPO align with the original definition of DPO, given its different setting? Specifically, how is $p_{ref}()$, defined as the ground truth video that always wins comparisons, consistent with DPO?

Thank you for raising this important point! We would like to respectfully clarify that PhysHPO indeed belongs to the DPO paradigm. In PhysHPO, $p_\theta$ represents the policy video diffusion model, and $p_{ref}$ denotes the reference model, as outlined in the Preliminaries section of the paper. The key modification in PhysHPO lies in constructing new positive and negative sample pairs to explore physical commonsense. This does not alter the standard selection process between $p_\theta$ and $p_{ref}$, ensuring PhysHPO remains consistent with the core DPO methodology.

---

Weakness 2: How can the writing be improved to better connect the data selection section with the methodology, especially given that this is not a standard DPO method?

Thank you for the feedback! We will revise the writing to better clarify how the data selection ties into the methodology and explicitly highlight the connection to PhysHPO as a DPO adaptation. This will improve the flow and make the methodology clearer.

---

Weakness 3: Why does PhysHPO work when it assumes the ground truth video always wins and the losing video is generated through heuristic manipulations, such as replacing frames (state-level), masking prompt tokens (instance-level), or rephrasing prompts (semantic-level)?

Thank you for your comment! The design of PhysHPO assumes that the ground truth video always wins the comparison, while losing videos are generated by introducing physical or semantic inconsistencies. Operations at instance, state, and semantic level are designed to simulate common errors in physical reasoning or semantic alignment. Specifically,

• State-level (e.g., replacing frames) disrupts physical continuity, mimicking unrealistic state transitions such as sudden object disappearance or appearance.
• Instance-level (e.g., masking prompt tokens) introduces semantic deviations, helping the model learn the relationship between the prompt and the video.
• Semantic-level (e.g., rephrasing prompts) creates subtle semantic inconsistencies, further challenging the model to align video generation with the intended meaning.

Although these designs may appear heuristic, they are grounded in physical commonsense and semantic reasoning principles.

Additionally, beyond the ablation studies in Table 3, which remove each level alignment progressively, we conducted further ablation experiments focusing on the contribution of each level:

Table A: Additional Ablation Study on Level Losses on VideoPhy

The results show that every level of preference alignment contributes to improving the physical plausibility of video generation. We hope this further clarifies our rationale.

---

Weakness 4: A lot the details need to be explained: (1) What is $u_{motion}$ in line 248? (2) How is $y_w$ (ground truth video) used, given it isn’t generated by the reference model? Did you add noise and use denoising outputs for loss calculation? (3) Should the equations be explicitly written, like in the original diffusion DPO paper (e.g., eq. 14)? (4) Without human preference annotations, how was Vanilla DPO implemented in Table 2? Was the winning sample assumed to be the ground truth video, and how were the losing samples generated?

Thank you for your detailed questions and suggestions, and we apologize for any potential confusion caused!

1. $u_{motion}$ refers to the motion-level loss calculation function for positive and negative video structure pairs.

2. Regarding $y_w$, it is indeed the ground truth video, but we add noise to it before inputting it into the reference model. This approach aligns with prior works like VideoDPO[1], where winning videos are pre-generated by the reference model and then used as training data, rather than being directly generated during training.

3. We appreciate your suggestion on improving the clarity of the equations and will revise the formulas in future versions to ensure they are more explicit.

4. For Vanilla DPO in Table 2, we implemented it using our dataset (i.e., "Vanilla DPO w/ Our Data") for fair comparison. In this setup, the winning sample corresponds to $y_w$ (ground truth video), while the losing sample corresponds to $y_l^{err}$, which is generated using the caption associated with $y_w$ as the prompt.

---

Thank you again for your detailed feedback. We hope this rebuttal clarifies the points raised and addresses your concerns. We are happy to continue discussing ways to improve the paper.

---

[1] VideoDPO: Omni-Preference Alignment for Video Diffusion Generation [CVPR’25]

Thank you to all the reviewers for your thoughtful and constructive comments! We are truly encouraged to see that the reviewers appreciate several positive aspects of our work, such as strong performance (Reviewer epiP, EaEb, JHEX, R9Wj), comprehensive & convincing experiments (Reviewer epiP, R9Wj), novel & effective framework (Reviewer R9Wj, EaEb), valuable & practical data selection pipeline (Reviewer R9Wj, EaEb, JHEX), well-motivated methodology (Reviewer epiP, JHEX), and clear writing (Reviewer epiP).

Thank you for your insightful feedback and expertise, which have guided us in improving the clarity and quality of our paper. Below, we provide detailed responses to your comments.

---

Weakness: How does the method ensure that the semantic loss doesn’t dominate and lead to sub-optimal solutions, while the motion loss effectively penalizes the network, given their similar weights (0.3 for motion and 0.2 for semantic)?

Thank you for raising this important concern! We acknowledge that Motion Loss and Semantic Loss are optimized simultaneously during training, which could theoretically lead to the model overfitting to Semantic Loss and neglecting Motion Loss. Below, we provide a detailed explanation of how our framework mitigates this issue:

• Independent Loss Design: Motion Loss and Semantic Loss operate on distinct inputs, where Motion Loss targets video structural information (i.e., optical flow), while Semantic Loss focuses on text-video alignment. The physical characteristics captured by Motion Loss could not be "cheated" by optimizing Semantic Loss.

• Positive-Negative Sample Design: Both losses rely on strict positive-negative sample comparisons. Even if Semantic Loss is over-optimized, Motion Loss penalizes discrepancies in physical motion between positive and negative samples, ensuring motion accuracy remains a priority.

• Experimental Validation: Here we conduct an ablation on impossible prompts (IPV-TXT in Figure 5) to evaluate the impact of Motion Loss and Semantic Loss individually.

  Table A: Robustness Testing on IPV-TXT

  	Visual Quality	Prompt Following
  PhysHPO (Ours)	58.82	47.09
  w/o $L_\text{Motion}$	54.06	44.93
  w/o $L_\text{Semantic}$	55.39	43.75
  Baseline	36.54	30.12

  Results show that removing either loss significantly reduces the model’s ability to generate impossible videos. Both are critical for handling complex prompts, demonstrating their complementary roles in the optimization process.

Overall, the framework is designed to ensure that Motion Loss remains critical for physical motion accuracy while Semantic Loss complements it without dominating the optimization process. We appreciate your thoughtful feedback and hope this clarifies our approach!

---

Question: Can the authors discuss how and why these metrics are good enough to evaluate the physics of the video? To my understanding, they are heavily based on human evaluation and motion accuracy, but I am unclear if they can efficiently evaluate the accuracy of the physics in the video.

Thank you for your thoughtful question! Our work utilizes Semantic Adherence (SA) and Physical Commonsense (PC) as core metrics to evaluate the physics of the generated videos, which are also widely adopted in recent works [1-3]. PC goes beyond simple motion accuracy by assessing adherence to real-world physical laws and intuitive commonsense, such as conservation of mass or gravity, while SA ensures alignment between the video content and the textual description. Together, these metrics provide a comprehensive framework for evaluating physical plausibility and semantic accuracy.

Importantly, our evaluation uses VideoCon-Physics from VideoPhy for automated assessment, ensuring scalability and objectivity. However, we acknowledge that automated metrics have certain limitations, such as challenges in fully aligning with human preferences. To address this, benchmarks often separate different physical phenomena for evaluation to improve accuracy. Additionally, we emphasize the importance of qualitative results (Figure 1,4,6,7,15-18) and user studies (Figure 5) to complement automated metrics. To mitigate potential biases, we also perform quantitative tests across multiple benchmarks (i.e., VideoPhy, PhyGenBench, VBench, IPV-TXT).

We hope this clarifies how our metrics are designed and their role in evaluating the physics of generated videos!

---

Suggestion: The metrics used in the paper can be explained in more detail. Some lines from the supplementary can be moved to the main paper to explain about the metrics used.

Thank you for the suggestion! We will move key details about SA and PC from the supplementary to the main paper for better clarity in our revision.

We appreciate your insightful comments again and hope this rebuttal addresses your questions. Please feel free to reach out with further suggestions to improve clarity.

---

[1] PhyT2V: LLM-Guided Iterative Self-Refinement for Physics-Grounded Text-to-Video Generation [CVPR’25]

[2] Articulated Kinematics Distillation from Video Diffusion Models [CVPR’25]

[3] VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Prior [ICCV’25]

Thank you to all the reviewers for your thoughtful and constructive comments! We are truly encouraged to see that the reviewers appreciate several positive aspects of our work, such as strong performance (Reviewer epiP, EaEb, JHEX, R9Wj), comprehensive & convincing experiments (Reviewer epiP, R9Wj), novel & effective framework (Reviewer R9Wj, EaEb), valuable & practical data selection pipeline (Reviewer R9Wj, EaEb, JHEX), well-motivated methodology (Reviewer epiP, JHEX), and clear writing (Reviewer epiP).

We greatly appreciate your detailed review and valuable insights, which help us refine our manuscript. Here, we respond to your comments systematically to address your concerns.

---

Weakness 1: While the performance improvements shown in the tables are often substantial, the stochastic nature of video generation and automated evaluation metrics means that the robustness of these gains is not statistically validated.

Thank you for your thoughtful question! You are correct that the stochastic nature of video generation and automated evaluation metrics may not fully capture human preferences. This is precisely why presenting sufficient qualitative results (Figure 1,4,6,7,15-18) and conducting User Studies (Figure 5) are critical to complement quantitative analyses. Additionally, we have evaluated our framework across multiple benchmarks (i.e., VideoPhy, PhyGenBench, VBench, IPV-TXT) to minimize potential biases introduced by these limitations.

---

Weakness 2: It would strengthen the paper to more clearly disentangle the gains from the novel instance-level formulation and the gains from the additional hierarchical levels (State, Motion, Semantic).

Thanks for your thoughtful suggestion! Table 3 progressively removes each Level loss to show their contributions. While our initial version didn’t compare individual Level alignments to vanilla DPO due to time and computational costs, we now provide this ablation:

Table A: Additional Ablation Study on Level Losses on VideoPhy

---

Weakness 3: (1) The cost, reproducibility, and potential biases of this "LLM-as-a-Judge" paradigm are not deeply discussed. (2) The selection process is defined by a fixed set of 17 physical phenomena from a prior work, which may limit the breadth of physical concepts the pipeline can identify. A discussion on the sensitivity to the choice of LLM or the physical taxonomy would be beneficial.

(1) Dependency on the "LLM-as-a-Judge" Paradigm: Using LLMs significantly improves efficiency compared to human annotators, as they can rapidly process video captions and score across 17 dimensions. To address potential biases, we employ an in-depth evolving prompting strategy (Figure 8), which guides the LLM through step-by-step reasoning and requires it to output both scores and justifications. This reduces hallucinations and ensures effective scoring. Furthermore, as shown in Figures 10 and 11, our pipeline achieves over 90% consistency with human annotations while being cost-effective.

(2) Scope of the 17 Physical Phenomena: While LLMs can analyze more granular phenomena, the 17 selected phenomena comprehensively represent core Dynamic, Thermodynamic, and Optic categories. This taxonomy balances breadth and clarity, ensuring interpretability for both LLMs and human annotators without introducing excessive complexity. Here, we show a comparison between our 17 phenomena and the categories generated by the LLM itself:

Table B: Adjacency Accuracy ($\pm$ 0.3) of Different Scoring Strategies with Human Scoring (Following Figure 11's Setting)

The results demonstrate that our taxonomy effectively captures the most critical aspects while avoiding overly fragmented or ambiguous classifications. We hope this addresses your concerns and welcome further feedback.

---

Question 1: Does the "Vanilla DPO" baseline also use this dual-negative sampling strategy, or does it use a more standard approach of generating a single negative sample from the prompt?

Thank you for your detailed review and thoughtful question! "Vanilla DPO w/ Our Data" uses the standard one-positive vs. one-negative paradigm, where the positive video is $y_w$ and the negative video is $y_l^{err}$ in our data. The comparison with "Only w/ $L_\text{instance}$" highlights the additional contribution of modeling the semantic gap ($y_l^{gap}$) to performance improvements.

---

Question 2: The "in-depth evolving prompting" strategy for physical fidelity scoring is a central part of your data selection pipeline. This process seems sensitive to the capabilities of the LLM used (Qwen2.5). Could you provide more insight into the robustness of this selection process?

Thank you for your insightful question! The capabilities of the base model indeed play a significant role in the robustness of the physical fidelity scoring process. As shown in Table C, we compare the impact of using different LLMs on PhysHPO’s performance, highlighting the variability introduced by different base models.

Table C: Performance Comparison between Different LLMs on VideoPhy

Furthermore, Table D demonstrates how both Qwen2.5 and DeepSeek-V3, combined with our in-depth evolving prompting strategy, achieves better alignment with human evaluations. This strategy leverages specific contextual comparisons to guide the model towards more professional and detailed assessments of physical fidelity.

Table D: Human Align Accuracy with Qwen 2.5 and DeepSeek-V3

---

Question 3: The finding that PhysHPO generalizes better to "impossible prompts" is fascinating and suggests a deeper understanding beyond rote memorization of physics. You hypothesize that enhancing physical fidelity equips the model with a stronger ability to generalize. Could you elaborate on this?

Thank you for your thoughtful question! Our experiments show that PhysHPO generalizes better to "impossible prompts". We believe this may stem from physical fidelity going beyond simple text-pixel alignment, enabling the model to learn deeper, more abstract relationships between physical and semantic contexts.

By modeling the boundaries of physics, PhysHPO can creatively explore these limits while maintaining semantic coherence. Its hierarchical alignment strategy further integrates physical understanding with context, allowing logical and consistent generation even in implausible scenarios.

This suggests that enhancing physical fidelity optimizes the model’s knowledge structure, strengthening its ability to reason and generalize beyond rote memorization.

We appreciate your interest in this aspect of our work!

---

Thank you again for your valuable feedback. We hope this rebuttal resolves your concerns and enhances the clarity of our work. We remain open to any further suggestions for improvement.