Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation

Q1: The PhyGenBench is a dataset with 160 text prompts...

A1: Thank you for your suggestion. We emphasize that PhyGenBench focuses on the most fundamental physical laws and simple scenarios. It undergoes rigorous screening and quality control. Experiments reveal that even for these basic physical scenarios, current video generation models struggle to produce videos that align with physical commonsense. We provide a more detailed explanation in General Response-Part1.

Q2: In PhyGenEval, the overall score is set on a four-point scale...

A2: We apologize for any confusion caused (We have explained this in the caption of Table 2 in the main text.). And We provide detailed clarifications below:

1. Normalization of Scores
   The results presented are normalized scores (out of a maximum score of 3). We emphasize this in the table captions in the paper to ensure clarity.

2. Distinguishing Capability of the Method
   The method demonstrates a clear distinction between open-source and closed-source models. For example:

   • Among open-source models, the best-performing Vchitect2.0 achieves 0.45, while the best-performing closed-source model, Gen-3, achieves 0.51. This reflects a notable gap of nearly 29 points in total scores, highlighting the performance difference between open-source and closed-source models.
   • Within open-source models, the total score difference between CogVideo2b and CogVideo5b is also approximately 29 points, showcasing the impact of scaling laws on models' ability to understand physical realism.

3. Challenges of the Benchmark
   The challenging nature of the benchmark results in relatively low scores across models. Generating physically accurate videos is inherently difficult, as it requires balancing smooth video generation with physical correctness. While current video generation models aim to achieve physical realism, their performance still falls short. We hope that PhyGenBench and PhyGenEval can support the community in further advancing this direction.

Thank you again for your thoughtful suggestions. If you have further questions or need additional clarification, please feel free to contact us.

Q3: Since the topic is related to evaluating the physics...

A3: Thank you for your valuable feedback. We have added the related work for this section in Appendix A. If you have further questions or need additional clarification, please feel free to contact us.

We would like to extend our heartfelt gratitude for the invaluable suggestions provided for this paper. In light of your feedback, we have diligently provided comprehensive and elaborate explanations.

1. Reproducibility and Model Replacement: We have discussed the reproducibility of the method in detail in Response to Reviewer PPCZ (Q2), as well as the potential replacement with Open-Model in Generat Response-Part2. Experimental results confirm that the method is highly reproducible.
2. Human Evaluation of GPT-4o generated questions: We conducted a human evaluation to assess the questions generated by GPT-4o, which is shown in Response to Reviewer wXRw (Q1) Table 2 . The results show that, despite some hallucinations from LLMs, the simplicity of PhyGenBench ensures that the generated questions are highly reliable.
3. Effectiveness of the Three-Stage Method: In Appendix D.3 (The Component in PhyGenEval on physical commonsense alignment evaluation), we explain that all three stages of our method contribute to the final results. By combining these stages, we effectively reduce the hallucinations that occur when using a single VLM, which also enhances the robustness of the method.
4. Robustness Discussion in Appendix E: In Appendix E (The robustness of PhyGenBench and PhyGenEval), we further discuss the robustness of our method. When using video quality enhancement modules like VEnhancer that do not affect physical correctness, the results of PhyGenEval remain nearly unchanged, highlighting the robustness of the method.

Therefore, we believe that both PhyGenBench and PhyGenEval demonstrate strong reproducibility and robustness. If you have any further inquiries, we are more than delighted to offer additional clarifications and conduct further experiments.

As today marks the final day of the discussion period, we would like to sincerely thank you for your thoughtful feedback on our paper. In response, we have addressed each of your points with thorough and detailed explanations. If you have any further questions or would like additional clarifications, we are more than happy to provide further insights and conduct additional experiments. We kindly ask if you would reconsider the score, and we look forward to hearing your thoughts.

Q1: The paper includes extensive comparisons to demonstrate PhyGenEval’s effectiveness...

A1: Thank you for your thoughtful feedback. We provide our detailed responses below:

1. Clarification on alternative open-source models.
   The alternative open-source models refer to replacing the GPT-4o used in the Physics Order Verification and Overall Naturalness Evaluation stages with open-source VLMs. For question generation, we consistently use GPT-4o. We have further clarified this point in the paper.

2. Specification of the GPT-4o version.
   We specify in our anonymous repository that the version of GPT-4o used is gpt4o-0806, and we have explicitly stated this in the paper as well.

3. Validation of GPT-4o-generated questions.
   we conduct a detailed human evaluation, which demonstrates that the questions generated by GPT-4o are highly reliable. For detailed information, please refer Reponse to Reviewer wXRw (Q1)

4. Replacing GPT-4o with open-source VLMs

   We provide a detailed discussion on replacing GPT-4o with open-source VLMs in General Response-Part2. Notably, even without using GPT-4o, the method achieves reasonably good performance (the Pearson correlation is over 0.7), demonstrating its robustness and flexibility.

Thank you again for your feedback. If you have further questions or need additional clarification, please feel free to contact us.

Q2: Certain T2I models may perform poorly on specific prompts...

A2: Thank you for your thoughtful feedback. We provide detailed responses below:

1. Filtering difficult prompts in PhyGenBench.
   During the construction of PhyGenBench, we deliberately filter out overly challenging prompts that lead to extreme distortions in generated videos. These prompts make it meaningless to evaluate physical realism. Instead, we focus on retaining prompts that enable the generation of basic scenes while exposing physical issues. This high-quality prompt design allows PhyGenEval to perform efficient and meaningful evaluations.

2. Incorporation of semantic alignment scores.
   We design a semantic alignment (SA) score to evaluate whether the generated videos align with the prompts. As detailed in Appendix D.2 (Quantitative result about semantic alignment), the results show that due to the high quality of our prompts, all tested models achieve high SA scores in both machine and human evaluations.

3. Evaluation of low-quality videos.
   We manually filter videos with low SA scores (≤ 1) and identify a total of 50 such low-quality videos. For these videos, we calculate their PCA scores. Since these videos are inherently highly distorted, it is nearly impossible to assess their physical realism. As shown in the Table 5 below (after norm), these low-quality videos also achieve extremely low PCA scores, further demonstrating that PhyGenEval is capable of identifying such low-quality cases effectively.

Thank you again for your valuable feedback. If you have further questions or need additional clarification, please feel free to contact us.

Table 5: SA and PCA scores on the selected video, the scores are normalized to 0-1

Q3: The issue of hallucination in large language models...

A3: Thank you for your insightful feedback. As noted in our Response to Reviewer wXRw (Q1), thanks to the extensive world knowledge of GPT-4o, we are able to generate highly reliable questions. We verify this through human evaluation to ensure their accuracy. Human evaluation shows that the questions generated by GPT-4o are physically consistent and highly reliable. Furthermore, we plan to refine this component further and release it as an open-source resource to enable standardized testing for others.

Thank you again for your suggestions. If you have further questions or require additional clarification, please feel free to contact us.

Q4: The author promised more human evaluation results...

A4: Thank you for your suggestion. We have made the necessary revisions. For the instructions used in human annotations, we provide a detailed explanation in Appendix D.1 (Human evaluation details) and include examples of these instructions in Figure 10. Thank you again for your valuable feedback. If you have further questions or need additional clarification, please feel free to contact us.

Q5: What is "the final score is calculated as 0 according to 4.2..

A5: Yes, the three-stage scores are 0, 1 (only $Question_1$ is correct), and 0. The final score is calculated as 0 because the average of these scores is 0. In this example, since the egg bounces off the rock like a rubber ball, the human annotation score is also 0.

Q6: It seems like the entire evaluation rely...

A6: Thank you for your feedback. We provide detailed responses below:

• For the questions generated by GPT-4o, we conduct a thorough human review, as explained in our Response to Reviewer wXRw (Q1). This demonstrates that the generated questions are highly reliable. We plan to further refine and release these questions as an open-source resource to facilitate standardized testing.
• Regarding GPT-4o's use as a VLM for evaluation, we provide a detailed explanation in our General Response-Part2, where we discuss replacing GPT-4o with open models. The results show that open models can also achieve a Pearson correlation coefficient above 0.7 with human ratings.
• Finally, we plan to further explore the use of other open-source LLMs and work towards implementing a more end-to-end evaluation approach. Thank you again for your valuable suggestions! If you have further questions or need additional clarification, please feel free to contact us.

We would like to extend our heartfelt gratitude for the invaluable suggestions provided for this paper. In light of your feedback, we have diligently provided comprehensive and elaborate explanations. If you have any further inquiries, we are more than delighted to offer additional clarifications and conduct further experiments. We eagerly anticipate your response!

As today is the final day of the discussion period, we would like to express our sincere thanks for your valuable feedback on our paper. In response, we have carefully considered your suggestions and provided detailed explanations. Should you have any further questions or require additional clarifications, we are more than willing to offer further insights and conduct additional experiments as needed. We kindly ask if you could reconsider the score and we look forward to your response.

Q1: As PhyGenEval uses VLMs for scoring ... more comprehensive understanding of the proposed evaluation pipeline.

A1: Thank you for your insightful feedback. We provide our detailed responses below:

1. Directly using Video VLMs is insufficient for evaluating physical realism in videos.
   We test evaluations using InternVideo or GPT-4o independently, as shown in Table 8 in the Appendix. Results indicate that even GPT-4o achieves a Pearson correlation of only 0.21 with human ratings when directly evaluating video physical realism. This demonstrates that current VLMs are not capable of effectively evaluating physical realism in videos on their own.

2. The design of each stage in PhyGenEval is necessary.
   Based on the characteristics of fundamental physical laws, we design a three-stage evaluation framework. As shown in Table 9 in the Appendix, using a single stage (or a single VLM) or combining two stages results in poorer performance compared to the full three-stage PhyGenEval. This analysis underscores that the proposed multi-stage evaluation pipeline is essential and well-justified.

3. The pipeline performs well with both open-source and closed-source models.
   We discuss the effectiveness of incorporating open-source and closed-source models in the pipeline in the General Response-Part2. Notably, even without using GPT-4o, the method achieves reasonably good performance, demonstrating its robustness and flexibility.

Thank you again for your valuable suggestions. If you have any further questions or need additional clarification, please feel free to contact us.

Q2: As the evaluation pipeline needs a lot of retrieval...

A2: Thank you for your thoughtful question. We provide detailed explanations below:

1. Retrieval accuracy is integrated into the method design.
   In the Key Physical Phenomena Detection and Physics Order Verification stages, we include retrieval operations by designing $VLM(I_j, P_r)$ , which checks whether image after the retrieval $I_j$ matches the retrieval prompt $P_e$. This ensures that key phenomena occur at the correct frame. The retrieval accuracy is also factored into the score calculation for these two stages, functioning similarly to a regularization term to account for retrieval correctness.

2. PhyGenBench emphasizes quality control, leading to high retrieval success rates.
   The careful construction of PhyGenBench ensures that models generate semantically aligned images while exposing issues with physical realism. As a result, retrieval operations achieve a relatively high success rate. We report the average $VLM(I_j, P_r)$ scores for different models in Table 4 below, showing that even Lavie achieves a retrieval score above 0.75, indicating that retrieval accuracy is generally high.

Thank you again for your insightful feedback. If you have further questions or need additional clarification, please feel free to contact us.

Table 4: Retrieval accuracy scores of different models

Thank you very much for your insightful suggestions on this paper. We have responded to each of these in great detail. If you have any other questions, we are more than happy to provide additional clarification as well as experiments, and look forward to your reply !

As today marks the final day of the discussion period, we would like to express our sincere gratitude for your insightful feedback on our paper. We have taken the time to thoroughly address your concerns and provide detailed explanations in response. If there are any remaining questions or if further clarifications are needed, we would be more than happy to assist and conduct additional experiments if necessary. We kindly request that you reconsider the score, and we eagerly await your reply.

Q2: The pipeline relies on proprietary models such as GPT-4o...

A2: Thank you for your valuable feedback. We have carefully considered your comments and provide our detailed responses below:

1. First, we emphasize that although our method uses GPT-4o, it is both cost-effective and efficient. As shown in Table 1 of the Response to Reviewer wXRw (Q1), the resource costs for our method are low in terms of both time and computational resources.

2. Second, we highlight that our method is reproducible. Specifically:

   • For LLaVA-Interleave, we disable the do_sample operation to ensure determinism.
   • For VQAScore, CLIP, and InternVideo, these methods are inherently non-random.
   • For GPT-4o, we use the default parameter configuration and gpt4o-0806 to ensure consistent results.

To demonstrate this, we perform five repeated experiments on Kling, originally scoring 0.49. As shown in Table 3 below, the results indicate that our method is stable and reproducible. Additionally, to facilitate testing by others, we provide the question files generated by GPT-4o for different stages of our evaluation.

In addition, we explore the use of various open-source models to enhance the performance of PhyGenEval. Experiments show that even with fully open-source models, the framework achieves a high correlation with human evaluations (approximately 0.7). We provide a detailed explanation of these results in All A2.

Thank you again for your suggestions. If you have any further questions or need clarification, please feel free to contact us.

Table 3: Results of five replicate experiments with Kling of PhyGenEval

Q3: I believe PhyGenBench is an excellent contribution for the research community...

A3: Thank you for your insightful suggestions. We would like to explain why we present PhyGenBench and PhyGenEval together in our paper:

1. We acknowledge that human evaluation is the most effective method, but it is difficult to scale up. Our goal is to provide PhyGenEval as a means for efficiently testing different video generation models on PhyGenBench.

2. Thanks to the automated testing provided by PhyGenEval, machine scores can be computed quickly and serve as a valuable reference for human evaluations.

3. While we recognize that PhyGenEval is not perfect, it achieves a relatively good alignment with human evaluations. We provide a detailed explanation of this in Response to Reviewer PPCZ (Q1).

Thank you again for your valuable feedback. If you have further questions or require additional clarification, please feel free to contact us.

Q1: The pipeline is not reliable enough...

A1: Thank you for your insightful comments! We have carefully considered your suggestions and provide our responses below:

1. The tasks in PhyGenBench are inherently challenging.
   The benchmark involves nuanced physical reasoning across diverse domains, which makes evaluation complex even for human reviewers. For instance, scenarios like "A timelapse captures the transformation of arsenic trioxide as it is exposed to gradually increasing temperature at room temperature" require an understanding of arsenic trioxide’s physical and chemical properties at different temperatures and the ability to interpret its gradual changes. This level of complexity poses significant challenges even for highly educated university students.

2. PhyGenEval already achieves strong human alignment compared to related work.
   While we acknowledge room for improvement, PhyGenEval demonstrates competitive or superior results compared to benchmarks like VideoScore (0.77) and T2V-CompBench (0.5-0.6). With a Pearson correlation of approximately 0.8, PhyGenEval validates its effectiveness in assessing complex physical realism in this domain.

3. Future Work.
   We acknowledge the potential to refine PhyGenEval further. In future work, we will focus on improving task-specific modules and exploring novel alignment techniques to enhance both accuracy and robustness, making the pipeline more refined and effective.

Thank you again for your valuable suggestions! If you have further questions or require additional clarification, please feel free to contact us.

We sincerely appreciate your questions and would like to provide further clarifications:

• First, while our method leverages GPT-4o, it remains reproducible and resource-efficient, as evidenced in Response to Reviewer PPCZ (Q2) and Response to Reviewer wXRw (Q1)

• Second, our method can fully eliminate the need for GPT-4o by only using Open models:

  • Regarding VLM evaluation, as shown in Table 1 below, we have already evaluated the framework using only open-source VLM models. Details can be found in General Response-Part2. Although the performance is slightly weaker compared to closed-source models, it still achieves a Pearson correlation coefficient of 0.72, demonstrating the high reliability of PhyGenEval. Furthermore, as stronger and larger open-source models are used, the alignment between PhyGenEval and human evaluation improves. This reinforces our confidence that, with the continued development of open-source models, our framework will no longer require GPT-4o. Additionally, we provide further evaluation results using only open-source VLMs in Table 2 below.

    Similar to Table 2 in the main text, these results highlight the performance gap across models (e.g., Gen3 achieves the best performance, while Lavie performs the worst). Moving forward, we will release a leaderboard and inference code fully based on open-source VLMs to further enhance the stability and reproducibility of our framework.

  • Regarding the generation of questions using LLMs, we have validated the reliability of GPT-4o-generated questions through human evaluation as shown in Response to Reviewer wXRw (Q1) . These questions, similar to the prompts in PhyGenBench, are an integral part of the benchmark. We will continue to refine and clean these questions and release them as open-source to enable standardized and convenient testing for the research community.

• Regarding the use of GPT-4o in the main text, the primary reasons are:
  i) Accessing the API is more convenient and has a lower entry barrier compared to deploying models.
  ii) As shown in Response to Reviewer PPCZ (Q2), our approach is computationally efficient, making the cost of using the API lower than the GPU resources required for deployment.

We sincerely thank you again for your insightful questions. We believe our response sufficiently addresses your concerns regarding GPT-4o. However, if you still have further questions or require additional clarification, we would be happy to provide further explanations at any time.

Table 1: Comparison of PCA correlation results using close source or open source models in PhyGenEval.

Table 2: Evaluation results of PCA with the proposed PhyGenEval with open VLMs in videos generated by several models .

I would like to thank the authors for their detailed responses to my questions. While I believe the benchmark dataset is a very useful contribution to the community, I share concerns with the other reviewers on the automated approach to evaluation with GPT-4o. I stand by my rating.

We express our sincere appreciation for the valuable suggestions regarding this paper. In response, we have provided thorough and detailed explanations. If there are any further questions, we are delighted to offer additional clarifications and conduct further experiments. We eagerly look forward to your reply!

Q1: The paper lacks decent novelty in terms of benchmark and evaluator itself...

A1: Thank you for your valuable feedback. We have carefully considered your comments and provide detailed responses below:

1. Novelty and Practical Significance
We are among the first to evaluate the physical realism of AI-generated videos. This topic is both novel and highly practical, as understanding physical realism is fundamental for treating video generation models as world simulators. Notably, Reviewer eSnR and Reviewer QELL highlighted the great novelty of our work, and Reviewer ZEcL and Reviewer PPCZ emphasized the importance of PhyGenBench to the community.

2. Contribution of PhyGenEval Framework
We acknowledge that training an evaluator is a significant contribution, but it is not the only approach. While we do not train a new evaluator, we propose the PhyGenEval framework, which integrates multiple Vision-Language Models (VLMs) to achieve better results. We believe this approach fundamentally aligns with the goal of reducing VLM hallucination in recognizing physical realism. Many excellent works in the community, such as VBench and DEVIL, also build upon existing VLMs. Thus, we argue that our approach represents a novel contribution by demonstrating the utility of this framework.

3. Simplicity of Our Method
Although our method consists of multiple stages, it is not overly complex. We provide resource consumption statistics in Table 1 below, which demonstrate that our evaluation process is both efficient and cost-effective. This efficiency underscores the practicality of our approach. We also add this content in Appendix F

4. Reliability of GPT-4o in Question Generation
When generating questions for different stages using GPT-4o, we do not simply rely on direct outputs. Instead, we incorporate few-shot examples and carefully designed prompts. Thanks to GPT-4o's extensive world knowledge, the generated questions are highly reliable.

To validate this, we conducted human annotations. Specifically, we recruited five senior undergraduate students, assigning each question to all five annotators for evaluation. Their task was to assess the physical correctness with 0 or 1 of each GPT-generated question. For the Overall Naturalness Evaluation stage, our criteria required that each level of description demonstrate progressive improvements in correctness and distinguishability. As shown in Table 2 below, the results indicate that questions generated by GPT-4o align strongly with physical realism.

To further enhance quality, we plan to refine the question set and release the cleaned dataset publicly for testing.

5. Effectiveness of the Multi-Stage Design
Each stage of our method is carefully designed and necessary. The multi-stage design aims to mitigate VLM hallucinations by analyzing fundamental physical laws and creating a three-stage evaluation framework. As shown in Appendix Table 9, using only one stage (or one VLM) or combining just two stages performs worse than the full three-stage PhyGenEval. This design effectively reduces hallucinations without causing error propagation. Therefore, the multi-stage structure is both essential and effective.

Thank you for your suggestions. If you have further questions or require additional clarification, please feel free to reach out.

Table1: Resource consumption of models used in PhyGenEval.

Table2: Human evaluation for GPT-4o generated questions

Q3: The number of prompts engaged in this paper are limited...

A3: Please refer to the detailed answer in General Response-Part1

Q4: What is the efficiency of using your auto-evaluator...

A4: We provide resource consumption statistics in Table 1 in the Response to Reviewer wXRw (Q1), which demonstrate that our evaluation process is both efficient and cost-effective. This efficiency underscores the practicality of our approach. We also add this content in Appendix F

Q5: Could you provide some error analysis on the bad cases...

A5: Thank you for your valuable feedback. We have carefully addressed your comments and provide our responses below:

We visualize some error cases in Error Case Analysis in Appendix E, where both PhyGenEval and comparison methods like DEVIL fail to correctly identify the physical realism of videos. These error cases are often caused by confusing but iconic physical phenomena in the videos that do not align with the correct progression of physical processes (e.g., in the erroneous case of the "burnt bread" experiment, black coloration appears but does not align with the expected phenomenon), leading to misjudgments. However, even in these cases, PhyGenEval remains closer to human ratings compared to other methods.

We plan to focus on addressing this issue in our future work, including but not limited to the following directions:

• Training our own evaluator
• Designing a more refined evaluation framework that leverages deeper video features, such as optical flow, to better assess physical realism and avoid being misled by visually smooth videos.

Thank you again for your constructive suggestions. If you have further questions or require additional clarification, please feel free to contact us.

Q2: The comparison with other baselines is unfair...

A2: Thank you for your valuable feedback and constructive comments. We carefully address your concerns and provide detailed responses below:

1. Scope and Motivation: Our primary objective is to design a benchmark (PhyGenBench) capable of clearly reflecting physical commonsense through simple, explicit prompts. While constructing this benchmark, we observe that existing metrics are inadequate for measuring physical realism, especially when applied to PhyGenBench. This leads us to design PhyGenEval, an evaluation metric tailored to address this shortcoming.

2. PhyGenBench and PhyGenEval appear in pairs: As we emphasize throughout the paper, PhyGenBench and PhyGenEval are designed to be used together, forming a cohesive framework for assessing physical commonsense in video generation models. The focus of this work is not on general-purpose evaluators but on addressing the specific gap in evaluating physical commonsense using a paired benchmark and metric.

3. PhyGenEval outperforms existing metrics on PhyGenBench: The design of PhyGenEval explicitly considers the key physical laws and phenomena incorporated into PhyGenBench. Therefore, it achieves higher consistency with human evaluations on this benchmark compared to other metrics. Specifically, PhyGenEval attains an overall Pearson’s correlation coefficient of 0.81 and Kendall’s Tau of 0.78 on PhyGenBench, significantly outperforming other metrics such as VideoScore (Pearson’s 0.19, Kendall’s 0.17) and DEVIL (Pearson’s 0.18, Kendall’s 0.17).

Thank you again for your thoughtful comments. If you have any further questions or require additional clarifications, please feel free to contact us.

We sincerely thank you for your insightful suggestions on this paper. In response, we have carefully addressed your points with detailed explanations. Should you have any additional questions, we would be happy to provide further clarifications or conduct additional experiments. We look forward to your feedback!