VideoPhy: Evaluating Physical Commonsense for Video Generation

We thank the reviewer for their encouraging comments. We are very excited to see that the reviewer finds our work (a) well-written and easy to follow, (b) high-quality benchmark for evaluating physical commonsense for video generation, (c) comprehensive in analysis, (d) reliable in terms of the automatic evaluation, and (e) relevant for boosting performance of automatic evaluation in the future.

Q: Human evaluation of VideoCon-Physics

• We clarify that Table 4 indicates the agreement (ROC-AUC) between the videocon-physics predictions and the human annotations on the unseen prompts and videos in the test set. The results highlight that the agreement of Videocon-physics is high in comparison to the other baselines.
• Table 5 highlights that the VideoCon-physics is not biased towards annotations that were taken for specific video models. Specifically, we show that the Videocon-physics can reliably perform judgments for the video models that were unseen in the training on the unseen prompts.
• Further, we highlight that the videophy annotations were performed by 14 workers where the load was shared uniformly across the annotators. It is unlikely that the model will be biased towards the judgments of a specific annotator.
• Since the goal of automatic evaluation is to align with human judgment, we will publicly release the human annotations and automatic evaluation scores as a valuable resource for the community.

We thank the reviewer for their diligent feedback. We are motivated to see that the reviewer finds our work (a) relevant as it addresses a gap in existing video evaluation datasets, (b) interesting in terms of dataset construction which is motivated from the computer graphics perspective, (c) comprehensive in conducting experiments for a wide range of T2V models, and (d) insightful for further improvements in T2V models.

Q: Category definitions and category ratios

• We believe it appropriate to group various types of solids into a single, unified 'solid' category, for our purpose. In theory, all solids can be prescribed as a universal constitutive model governed by the conservation of mass and momentum. Nevertheless, graphics researchers typically model certain materials using simplified constitutive models to lower computational costs. For instance, a rigid body, in reality, is a deformable body with a very high stiffness. In fact, exact rigidity does not exist in the real world. Similarly, researchers propose other solid constitutive models (e.g. fabrics, granular materials) to simplify computation for some specific material behaviors. Our choice of solid, as a broad categorization, serves as a basis for generalization beyond specific simulation techniques used in graphics research. Our benchmark is designed to capture a wide spectrum of physically plausible behaviors, rather than isolating specific graphics sub-domains.
• Interactions between solid-solid pairs primarily focus on resolving contact constraints to prevent penetration. In contrast, solid-fluid interactions exhibit more diversity. For example, water can be repelled by an umbrella but can be absorbed by a sponge. Such behaviors, including permeability, adhesion, and absorption, are not available in solid-solid interactions. Given the diversity of solid-fluid interactions, we choose to balance the sample counts for solid-solid and solid-fluid interactions to ensure our benchmark does not overemphasize simpler contact-based interactions while underrepresenting more complex and varied solid-fluid dynamics.
• We appreciate the reviewer's suggestion of categorizing based on interaction types or specific physical properties. This is an insightful point, and we agree that such an approach could provide additional granularity. However, our current choice prioritizes simplicity and broad applicability, especially for non-experts who may use the benchmark across various disciplines. Future work can consider more fine-grained categorization to better measure the ability of generative models to handle specific properties (e.g., plasticity, viscosity) if desired. We will add this discussion in the revised paper.

Q: Training set of VideoCon-Physics

• Since the training and testing set of the VideoPhy serve different purposes, we had to balance the resource allocation in the limited academic budget for human annotations. Specifically, the testing set was used to create a reliable leaderboard based on human judgements. In this case, we sample 1 video per test prompt and use three annotators to provide their judgements to ensure highest quality.
• However, the role of the training set was to train a deep neural network based automatic evaluator which is more data hungry. Hence, we decided to sample 2 videos per train prompt which increases the diversity of the data, and got 1 annotator to judge it for text adherence and physical commonsense evaluation (12000 annotations in total).
• We clarify that the task of semantic adherence and physical commonsense judgments is inherently subjective. Prior work such as ImageReward [1] or AlpacaEval [2] are widely adopted to study human preferences in generative models and achieve human agreement close to 65%. In this regard, our human agreements of 70%-75% are quite reasonable.
• We respectfully disagree with the reviewer that the training set is not trustworthy. In fact, our empirical findings suggest that VideoCon-Physics achieves the highest agreement with the human judgments on the test set (Table 4). In addition, Table 5 shows that the VideoCon-Physics decisions align with the human judgements for unseen video models too. This would not have been possible if the training dataset was noisy.
• Ideally, we agree that having more human judgements will benefit the data quality but it will significantly increase the data collection expenses, which goes beyond our budget. We will add this discussion in the limitations section (Appendix B).

[1] ImageReward: https://arxiv.org/pdf/2304.05977
[2] AlpacaEval: https://github.com/tatsu-lab/alpaca_eval

Q: Binary feedback for evaluating semantic adherence and physical commonsense

• We highlight that binary feedback (0/1) is quite popular in aligning generative models such as large language models [1]. Further, we observe that binary feedback is much easier to collect at industrial scale by big generative model providers (e.g., ChatGPT). For instance, we note that the ChatGPT user interface asks binary preference after generating the response to a simple query (https://ibb.co/6nHV85c). Similar extensions exist in the field of text-to-image generative models [2,3]. Hence, the binary feedback protocol is quite powerful in studying and improving the generative models.
• The ability to assign a score to generated content is the common way to assess the model performance across various benchmarks [4,5]. In addition, it makes it easier for us to collect large-scale human annotations under a limited financial budget.
• While the automatic evaluator is trained with the binary feedback, it can provide us with a continuous score between [0,1] which can be useful for fine-grained video assessment.
• In addition, we agree with the reviewer that a dense feedback system would capture more nuanced mistakes of the video generative models (e.g., completing 8 movements versus 6 movements). However, designing such prompts is non-trivial, and evaluating the generated videos in such scenarios is much more challenging, labor-extensive, and expensive in the limited academic budget.
• We firmly believe that binary feedback can provide a lot of interesting insights too. For instance, we uncovered the ability of the video generative models to perform differently for diverse material interactions (e.g., solid-solid, solid-fluid, fluid-fluid). In addition, we could gauge the model’s performance on easy and harder prompts too. Our qualitative evaluation confirms these differences observed in the quantitative values.
• This work is intended to lay the foundation for physical commonsense so that the practitioners can compare existing models quantitatively. We believe that it will spark further research in various dimensions including the collection of diverse and denser forms of feedback. We will add this discussion explicitly in the revised paper.

[1] Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D. and Kiela, D., 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306.
[2] Li S, Kallidromitis K, Gokul A, Kato Y, Kozuka K. Aligning diffusion models by optimizing human utility. arXiv preprint arXiv:2404.04465.
[3] Lee, Kimin, et al. "Aligning text-to-image models using human feedback." arXiv preprint arXiv:2302.12192 (2023).
[4] Huang, Ziqi, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang et al. "Vbench: Comprehensive benchmark suite for video generative models." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-21818. 2024.
[5] Yarom, M., Bitton, Y., Changpinyo, S., Aharoni, R., Herzig, J., Lang, O., Ofek, E. and Szpektor, I., 2024. What you see is what you read? improving text-image alignment evaluation. Advances in Neural Information Processing Systems, 36.

Q: Fine-tuning VideoCon to evaluate especially the physical commonsense without introducing any extra knowledge, reasoning, or explicitly modeling.

• In this work, we do not assume that the existing video-language models have physical commonsense understanding. Infact, our evaluation in Table 4 suggests that the existing models (Gemini-Pro-Vision and VideoCon) are very close to random (50) their agreements with human physical commonsense judgements.
• To this end, we take a data-driven approach and finetune the model on 12K human annotations (L 350-351) to imbibe new knowledge about the semantic adherence and physical commonsense judgements with generated videos. Prior work [1,2] has shown that data-driven approaches can outperform rule-based physics simulators for more complicated systems like weather and climate.
• We respectfully point out that finetuning the models with human annotations will come under the paradigm of introducing extra knowledge and explicit modeling.
• In addition, we note that VideoCon-Physics should not be considered as a general-purpose physical commonsense evaluator. The purpose of this model is to allow fast evaluations of the videos generated by the prompts in the VideoPhy data. We believe that the road to building general-purpose physical commonsense evaluators is quite long as the field is in the nascent stages. We will add this discussion in the revised paper.

[1] Climax: https://arxiv.org/abs/2301.10343
[2] Stormer: https://arxiv.org/abs/2312.03876

We thank the reviewer for their insightful comments. We are happy to observe that the reviewer finds our work (a) vital in addressing the realism in video generation, (b) insightful in understanding the failure modes which are useful in guiding future model developments and research directions, and (c) useful and meaningful in terms of the automatic evaluation method.

Q: Decoupling physical commonsense and semantic adherence.

• We clarify that the physical commonsense score does not depend on the semantic adherence capability in our evaluation. As mentioned in Section 3.2 and 3.3, the human and automatic evaluator do not focus on the underlying caption to make the physical commonsense judgements.
• Ideally, we want the models to follow the prompt and generate physically commonsensical videos. To this end, we study the joint performance (SA=1, PC=1) in our main results (Figure 1, L256-258).
• In Table 3, the first column provides the joint performance (SA=1,PC=1), marginal semantic adherence (SA=1) and marginal physical commonsense (PC=1). A reader can estimate the posterior performance (PC=1 given SA=1) by taking the ratio of the joint performance and marginal semantic adherence scores. By default, we do not report posterior performance since it can be inferred from the existing numbers. In addition, just the posterior performance does not provide the entire picture which is clearer with joint performance metric.
• Finally, we believe that a bad model can easily game the posterior metric. For example, a bad model can generate a video which aligns with the prompt for 1 out of 700 prompts in the dataset. Now, assume that this video is also accurate in terms of physical commonsense. Hence, the posterior performance of this model will be 100%. This can be quite misleading for the practitioners.
• We present the model performance across all possibilities {(SA,PC)=(1,1), (1,0), (0,1), (0,0)} in Appendix J. We will add this discussion explicitly in the revised paper.

Q: Assumption in the automatic evaluation

• In this work, we do not assume that the existing V2T model has physical commonsense understanding. Infact, our evaluation in Table 4 suggests that the existing models (Gemini-Pro-Vision and VideoCon) are very close to random (50) their agreements with human physical commonsense judgements.
• To this end, we take a data-driven approach and finetune the model on 12K human annotations (L 350-351) to imbibe new knowledge about the semantic adherence and physical commonsense judgements with generated videos.
• Since the goal of automatic evaluation is to align with human judgment, we will publicly release the human annotations and automatic evaluation scores as a valuable resource for the community.

Q: Finetuning video generative models on training split of VideoPhy

• In Appendix Q, we finetune Lumiere-T2I2V with the training split of VideoPhy. Specifically, we train it with the videos in our training dataset which achieve a score of 1 on physical commonsense and a score of 1 on semantic adherence. In total, there are ~1000 such videos. While this dataset is small for finetuning, we perform a finetuning run to address the reviewer’s comment. Post-finetuning, we generate the videos for the test prompts and evaluate them using our automatic evaluator: | Model | SA | PC | Average | |--------------------------|------|------|---------| | Lumiere-T2I2V-Pretrained | 46 | 25 | 35 | | Lumiere-T2I2V-Finetuned | 36.5 | 24.6 | 30.5 |
• We find that the semantic adherence (video-text alignment) reduces by a large margin and physical commonsense remains unchanged after finetuning. This can be due to several factors: (a) the number of training samples is not enough, (b) optimization difficulties since the training videos are generated from several generative models (mix of on-policy and off-policy data), and (c) vanilla finetuning being a bad algorithm for learning from these samples. Since post-training of video generative models is a less explored direction, there can be many ways to improve the generative model’s physical commonsense.
• Further, we clarify that improving video generative models is an entirely new project. We ran the experiment in Appendix Q to inspire future research on enhancing physical commonsense in generated videos (L525-530).
• These results also show that mere finetuning with the samples in the training set of VideoPhy does not lead to large gains in the automatic evaluation on the test set. We will add this discussion to the revised paper.

We thank the reviewer for their diligent feedback. We are motivated to find that the reviewer finds our work: (a) quite important for this area, (b) comprehensive in coverage of open and close video generative models, (c) well-organized and easy to follow. We address the reviewer comments below:

Q: Binary Feedback

• We highlight that binary feedback (0/1) is quite popular in aligning generative models such as large language models [1]. Further, we observe that binary feedback is much easier to collect at industrial scale by big generative model providers (e.g., ChatGPT). For instance, we note that the ChatGPT user interface asks binary preference after generating the response to a simple query (https://ibb.co/6nHV85c). Similar extensions exist in the field of text-to-image generative models [2,3]. Hence, the binary feedback protocol is quite powerful in studying and improving the generative models.
• In addition, we agree with the reviewer that a dense feedback system would capture more nuanced mistakes of the video generative models (e.g., completing 8 movements versus 6 movements). However, designing such prompts is non-trivial, and evaluating the generated videos in such scenarios is much more challenging, labor-extensive, and expensive (esp. with limited academic budgets).
• We also believe that binary feedback can provide a lot of interesting insights too. For instance, we uncovered the ability of the video generative models to perform differently for diverse material interactions (e.g., solid-solid, solid-fluid, fluid-fluid). In addition, we could gauge the model’s performance on easy and harder prompts too. Our qualitative evaluation confirms these differences observed in the quantitative values.
• While the automatic evaluator is trained with the binary feedback, it can provide us with a continuous score between [0,1] which can be useful for fine-grained video assessment.
• This work is intended to lay the foundation for physical commonsense so that the practitioners can compare existing models quantitatively. We believe that it will spark further research in various dimensions including the collection of diverse and denser forms of feedback. We will add this discussion explicitly in the revised paper.

[1] Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D. and Kiela, D., 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306.
[2] Li S, Kallidromitis K, Gokul A, Kato Y, Kozuka K. Aligning diffusion models by optimizing human utility. arXiv preprint arXiv:2404.04465.
[3] Lee, Kimin, et al. "Aligning text-to-image models using human feedback." arXiv preprint arXiv:2302.12192 (2023).

Q: Rankings vs. binary feedback

• To address the reviewer’s comment, we have performed a new ranking-based study for physical commonsense evaluation. Specifically, we ask the three workers to look at two videos simultaneously and pick the one with better physical commonsense. In particular, we got 500 pairwise comparisons for 4 video generative models (CogVideoX-5B, Pika, Gen2, OpenSora). It costs us $360 to run this human eval. Subsequently, we computed the ELO scores of these models based on the human annotations. We present the results below:

• Interestingly, we find that the relative ranking of these models remains unchanged under both the feedback methods. Specifically, CogVideoX-5B and OpenSora are still the best and worst models on the VideoPhy dataset, respectively. We will add these results to the revised paper.
• We note that the open (usually smaller) video generative models will be penalized for losing to close (usually larger) video generative models in the ranking-based setup. The absolute feedback operates independently across all video generative models, and helps in better contextualizing the capability of the models with similar scales. Anecdotally, practitioners do not look at Lmsys (chat arena) ELO leaderboard to understand the capabilities of small language models because they are usually at the bottom of that list after losing to strong models in many comparisons. For example, GPT-4/Gemini models have saturated the MATH dataset but it is still used as a guiding star for others who want to build strong models with math capabilities. We diligently believe that such revolution can be sparked by VideoPhy for the field of video generative models.