Impossible Videos

Thanks for the encouraging review and valuable suggestions. We appreciate your acknowledgment of the paper’s novelty and will carefully address your concerns to improve clarity and rigor.

Q1: Disentangling Impossibility vs. Synthetic Data Quality

Disentangling impossibility from synthetic data quality is crucial for strengthening the rigor of our study. We leverage the Multi-choice QA task to investigate this. Due to the limited time during rebuttal, we were unable to generate and annotate a large-scale set of synthetic videos. We conducted a preliminary study on a smaller dataset:

1. Data Collection: We collect 420 synthetic videos from an existing work [1], generated using HunyuanVideo.
2. Annotation: We label each video with an action description.
3. Filtering: Videos were excluded if they:
   • Lacked an explicit event (e.g., landscape scenes).
   • Contained counterfactual content (ensure distinction from impossible videos).
     After filtering, 200 videos remained.
4. Task Construction: We instructed GPT-4o to generate multi-choice questions and answers, adapting prompt from the impossible video setting.
5. Evaluation: Several popular VideoLLMs were evaluated and compared to impossible videos.

The results show that models consistently perform better on normal synthetic videos than on impossible videos, indicating that "impossibility" introduces a distinct challenge for video understanding.

Recent studies [2] suggest that the visual quality gap between synthetic and real-world videos is rapidly shrinking. This further underscores the unique challenges posed by impossible videos, which are independent of synthetic data quality.

We appreciate this valuable suggestion. We will conduct a more comprehensive study in the revised version.

Q2: Code/Data Release

Upon acceptance, we will publicly release all code and data, including the IPV-Bench taxonomy, IPV-TXT prompts, IPV-VID videos, and evaluation protocols. To ensure accessibility, we will also include a detailed Data Usage Instruction in the Appendix.

Regarding concerns about video quality, e.g., action coherence, our human annotation filters out low-quality videos:

• If the action forms a semantically meaningful impossible phenomenon, we consider it a valid sample.
• If the action exhibits inconsistencies, artifacts, or unnatural distortions, we classify it as low-quality data and exclude it.

For a detailed data filtering criteria, please refer to Response 2 of Reviewer F9mi. Besides, we provide sufficient video examples on the anonymous website in the paper abstract for further reference.

Q3: For T2V, any difference in visual quality and prompt-following between normal and impossible videos?

Current T2V models are primarily optimized for normal videos, while impossible videos is often overlooked. Our assumption: creating impossible videos is more challenging than creating normal ones. To explicitly verify this, we conducted a human evaluation on normal text prompts:

1. We collected 420 synthetic videos from an existing work [1], generated using the HunyuanVideo model.
2. We annotated the visual quality and prompt-following of these videos.
3. During annotation, we excluded 83 prompts that describe impossible phenomena and evaluated the remaining 337 normal prompts.

We observe that 1) normal visual quality is slightly better than impossible ones; 2) normal prompt following is significantly better than impossible prompt following, which further underscores the unique challenges posed by impossible videos.

We appreciate this insightful suggestion. In the revised paper, we will conduct a more comprehensive study to further strengthen our findings.

Q4: Can impossible videos serve as negative samples to improve Video-LLM in reasoning tasks or enhance T2V generation quality?

We recognize the potential in these directions:

• For Video-LLMs, impossible videos can serve as high-quality training samples to enhance reasoning capabilities. Since these videos introduce "novel" counterfactual knowledge beyond real-world data, they may help models develop stronger reasoning and generalization skills.
• For T2V models, while the community has focused heavily on physical law adherence, there is limited understanding of how to explicitly improve this. Impossible videos could serve as negative samples, reinforcing a more structured comprehension of physical plausibility.

Both directions present exciting and meaningful research opportunities. We hope the release of Impossible Videos will inspire further exploration in these areas.

[1] The Dawn of Video Generation: Preliminary Explorations with SORA-like Models. arXiv 2024.

[2] Cosmos World Foundation Model Platform for Physical AI. arXiv 2025.

Thank you for your encouraging review and valuable suggestions. We appreciate your acknowledgment of the paper’s novelty and will carefully address your concerns to improve clarity and rigor. Below are our detailed responses:

Q1: Better to explain more clearly about how to calculate the score in the Open-ended QA task.

We appreciate the reviewer’s suggestion and provide a more detailed explanation below.

For evaluating the Open-ended QA task, we employ an LLM-based evaluator that compares model responses against the annotated text explanations in the benchmark. However, we empirically observed that directly instructing the LLM to assign scores led to instability. To address this, we propose a justification-then-score approach:

1. Justification Step: The evaluator first analyzes key matches or mismatches between the model’s response and the ground truth, providing a textual justification.
2. Scoring Step: Based on this justification, the evaluator assigns a semantic alignment score on a scale from 0 to 1:
   • 1.0 – Perfect alignment
   • 0.8-0.9 – Good alignment
   • 0.5-0.7 – Partial alignment
   • 0.1-0.4 – Weak alignment
   • 0.0 – No alignment

This justification step is crucial for ensuring fair and stable score assignment. In this work, we employ GPT-4o as the evaluator. In the current version of the paper, we briefly mention the use of GPT-4o in Section 4, line 306. We will include this detailed explanation in the revised version (or supplementary materials) to improve clarity.

We sincerely thank the reviewer for highlighting this point.

Q2: Are there any method to calculate the Prompt Following metric without human?

This is an insightful question, and we fully agree that an automatic evaluation strategy would enhance the scalability of our benchmark.

In our main paper (Table 4), we report human-annotated results to provide reliable insights into the performance of current T2V models. Additionally, in Appendix B.2, we introduce an automatic evaluation strategy for impossible video generation. Specifically, the Prompt Following metric can be assessed using state-of-the-art Vision-Language Models (e.g., GPT-4o) in conjunction with a carefully designed prompting strategy. Experiment result clearly demonstrate the consistency between human evaluation and auto-evaluation, as revealed in Table 6 and Figure 6 in the Appendix.

We appreciate the reviewer’s insightful question and will further emphasize this discussion in the revised version.

Q3: This paper should discuss some other benchmark regarding both the video understanding and generation.

We appreciate the reviewer’s comment. In the Related Work section and Table 1, we have comprehensively discussed the relationships and distinctions between IPV-Bench and existing benchmarks across three key areas: video understanding, video generation, and AIGC video detection. This comparison highlights the unique contributions of IPV-Bench and its role in bridging gaps that are not addressed by prior benchmarks.

We will ensure that this discussion is clearly emphasized in the revised version. Thank you for your valuable feedback.

Thank you for your positive feedback and constructive suggestions. We are grateful for your recognition of our work’s novelty and will incorporate your recommendations to further strengthen the paper. Below, we address your comments in detail.

Q1: Claims about model limitations of understanding impossible videos could be strengthened with category-specific quantitative metrics.

We appreciate the reviewer’s suggestion.

Table 2 of the paper presents impossible video understanding metrics (Multi-choice QA and Open-ended QA) across the four categories of IPV-Taxonomy: Physical, Biological, Social, and Geographical. Notably, the Physical category proves to be the most challenging, yielding the lowest scores across both tasks.

Upon further analysis, we observed that videos in the Physical category often exhibit complex impossible temporal dynamics, requiring sophisticated temporal reasoning. In contrast, videos in other categories can be largely addressed through world knowledge reasoning, which aligns well with the capabilities of large language models (LLMs).

To further investigate this, Table 3 reports an experiment where videos are classified into two groups via human annotation:

• Spatial – Impossible phenomena identifiable from a single frame.
• Temporal – Impossible phenomena requiring cross-frame temporal reasoning.

Results indicate that models perform significantly worse on Temporal videos than on Spatial ones, highlighting temporal reasoning as a major bottleneck in understanding impossible videos.

Q2: It is better to provide more information about the video filtering criteria.

We appreciate the reviewer’s valuable suggestion. The goal of video filtering is to ensure that the selected videos: 1) Maintain high visual quality; 2) Clearly demonstrate impossible phenomena.

Visual Quality Criteria:

• Accepted: Clear, sharp videos with high aesthetic value and smooth temporal motion.
• Rejected:
  • Videos with jitter, flicker, blur, large-scale artifacts, or indistinct/distorted foreground objects.
  • Completely static videos with no visible changes.
  • Videos that lack logical coherence and appear visually chaotic.

Impossible Semantics Criteria:

• The video must clearly depict an impossible, counterfactual phenomenon that cannot occur in the real world. The impossibility should be a salient event, rather than minor visual details that are difficult to perceive.
• The video should be in a photo-realistic style. Non-realistic styles (e.g., cartoon-style videos) are excluded to avoid confusion in video understanding.

We will include these detailed filtering criteria in the revised paper. Thank you again for your insightful review. Your feedback is helpful on enhancing the rigor and clarity of our work. We are happy to address any further questions.

Thank you for your thoughtful feedback. We greatly appreciate your insights on our paper. Below, we outline our responses to each point.

Q1: Reproducibility: Code/Dataset Release

We appreciate the reviewer’s concern regarding reproducibility. Upon acceptance, we will publicly release all code and data, including the IPV-Bench taxonomy, IPV-TXT prompts, IPV-VID videos, and evaluation protocols. Additionally, we will provide a detailed Data Usage Instruction in the Appendix to enhance accessibility. We believe that this comprehensive release will maximize the impact of Impossible Videos and foster further research in this area.

Q2: Significance and Applications of Impossible Videos.

We appreciate the reviewer’s valuable suggestion, as it will further enhance the impact of Impossible Videos.

Significance:

• As highlighted in our paper, this benchmark fills a critical gap in evaluating counterfactual video generation and understanding—an area currently absent in the community. This importance has also been recognized by Reviewers F9mi and AP4m.
• Evaluating AI robustness and generalization in out-of-distribution scenarios is a well-established challenge, particularly for autonomous systems encountering rare events. As noted by Reviewers F9mi and JPJB, "Impossible Videos" serves as a robustness and generalization benchmark for video understanding and generation models, addressing an overlooked yet crucial aspect of AI evaluation.
• As Reviewers F9mi and AP4m acknowledged, our work is closely related to broader topics in counterfactual and causal reasoning in AI. "Impossible Videos" provides a valuable case study for counterfactual reasoning in the video domain, encompassing both video understanding and generation.

Applications:

• "Impossible Videos" can be leveraged to enhance video understanding and generation models by improving their robustness and generalization capabilities.
• Real-world applications include:
  • Creative industries – Enhancing special effects, game design, advertising, and filmmaking, etc.
  • Industrial safety – Assisting in anomaly detection and risk assessment.
  • Advanced AI assistants – Equipping AI systems with stronger reasoning capabilities for more intelligent decision-making.

Q3: Experimental Details: IPV-Score and Evaluation

We appreciate the reviewer’s feedback on this issue. The computation of the IPV-Score is based on the statistics of Visual Quality and Prompt Following and is defined as:

$$IPV Score=\frac{\text{Num. of High Visual Quality}\cap \text{Num. of Good Prompt Following}}{\text{Num. of All Vid.}}$$

This metric intuitively aligns with our design philosophy: it measures the percentage of videos that satisfy both high visual quality and strong prompt adherence. To improve clarity, we will include this equation along with a detailed textual explanation in the revised paper.

Q4: How to Encourage Broader Adoption?

We appreciate this constructive question and will take the following steps to encourage broader adoption:

• Public Release – Make all data and evaluation code openly available to enhance accessibility.
• Comprehensive Documentation – Provide detailed instructions for data usage to facilitate easy adoption.
• Community Engagement – Organize competitions and workshops on Impossible Videos to attract more researchers to this domain.
• Integration with Existing Toolkits – Collaborate with established toolkits to incorporate IPV-Bench into widely used evaluation suites.
• Ongoing Benchmark Maintenance – Maintain a leaderboard and regularly update the benchmark to ensure its relevance.

We sincerely hope that these efforts will inspire further research and drive innovation in the video understanding and generation community.