Response to Weakness 1 : Please see our response to the questions below. We run an additional different model (internVL 9B) and show the same conclusion that curriculum learning can improve MLLMs' reasoning abilities on synthetic video understanding. See Table I for reference.

Response to Weakness 2 (Figures Layout): Figures 3 and 5 (radar charts) are intended to provide a qualitative comparison across sub-categories for each MLLM, as the corresponding tables (Tables 2 and 3) present complex numerical results that are not easy to interpret. We agree to move these figures to the appendix in the final version to make space for additional content in the main paper.

Response to Weakness 3-5: Thank you for pointing out the typos and term misuses in our paper (Black Box => Closed-Source, LVLM => MLLM, and SOTA => SoTA). We will correct them in the final version.

Response to Question 1: Our task involves using MLLMs for video understanding and question answering, specifically targeting physics and commonsense abnormalities. As a novel task, there are no existing datasets addressing this specific setting, video QA on physics violations in synthetic videos. Our paper is the first to introduce this problem, release a benchmark, and conduct both evaluation and fine-tuning experiments. Thus, it's not possible for us to provide evaluation on other datasets over exactly the same task.

Importantly, our fine-tuning experiments aim to demonstrate the feasibility of the benchmark and point toward future directions for improving MLLMs. We are also happy to conduct additional experiments on relevant benchmarks, such as MMVU [1] and MVBench [2], using our fine-tuned models to show that our training approach maintains generalization on broader video QA tasks. The evaluation results are presented below, using model names consistent with those in the original paper. Refer to Table B in reviewer tQNG's console

Results on real-world datasets show that models fine-tuned with curriculum learning, whether using a composed or single training dataset for RFT, show little performance difference. Notably, the GRW-PRW-SR-R1-7B model, which is fully fine-tuned with all three datasets in a curriculum learning setup, achieves performance close to the model trained solely on real-world videos. Given its state-of-the-art results on synthetic video understanding and physics comprehension, this highlights the importance of incorporating real-world videos in RFT post-training to avoid performance degradation on general video tasks when using physics-focused data.

To evaluate model feasibility beyond our current SoTA open-source model (Qwen2.5-VL-7B), we also present overall results on InternVL3-9B below, using our latest synthetic video dataset.

According to the table, both RFT and curriculum learning yield notable improvements across all VideoHallu subcategories when performing RFT on both base MLLMs, demonstrating the generalizability of our training methods across MLLMs.

[1] Zhao, Yilun, et al. "Mmvu: Measuring expert-level multi-discipline video understanding." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

[2] Li, Kunchang, et al. "Mvbench: A comprehensive multi-modal video understanding benchmark." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Response to Question 2:

We present comprehensive results on our curriculum learning setup and training data compositions. Due to rebuttal word limits, detailed evaluations are provided in our response in reviewer tQNG's console. Our findings show that curriculum learning significantly outperforms baselines using only one or two datasets for RFT. While RFT on a single dataset can improve performance on specific tasks, it often harms performance on others. Moreover, for complex tasks requiring high-level comprehension, direct RFT without intermediate steps may be ineffective. Curriculum learning enables MLLMs to tackle such tasks more efficiently through gradual, step-by-step learning.

We also run mixed data training without curriculum learning and show that it has worse results (52.3%) than training just on synthetic data (53.4%) or training them with curriculum learning (58.7%).

Meanwhile, we are actively running more experiments comparing the MLLM's performance after RFT by composed data across all three datasets and each identical dataset under Training Separately setting. Please stay tuned for more results over the discussion period.

Response to Question 3: Thank you for highlighting this weakness. Below are the implementation details:

• Avg. frames per video: 96.0
• Avg. video length: ~5.3 seconds
• Avg. framerate (FPS): 23
• Avg. frame resolution: 1042 × 588
• Number of video generation models: 7
• Avg. videos per model: 132
• Maximum number of frames to sample for training: 18
• Number of training iterations: 3 epochs
• Maximun number of frames to sample for testing: 128

Response to Question 4: The problem formulation section models the hallucination triggering mechanism in MLLMs, not just data collection. It outlines how both human annotators and MLLMs can generate QA pairs for synthetic videos with physics or commonsense violations, following the setting in Figure 1.

The mapping function $T(\cdot)$ represents a question-generation mechanism that, given an entity $C$ (e.g., a watermelon in Figure 1), produces either a natural language question $T(Q)$ (e.g., In the video, the watermelon breaks in the middle. Is it intact or broken at the end?) or a textual description $T(C)$ of the event involving that entity (e.g., A bullet shot into the watermelon), possibly involving physics or commonsense violations.

In practice, $T(\cdot)$ can be executed by human annotators, who identify a context $C$ in video $V$, then formulate both $T(Q)$ and $T(C)$. Alternatively, MLLMs can generate these when provided with a known violation. While MLLMs cannot yet fully automate this process, they show promise in crafting QA pairs for synthetic videos when supplied with abnormal contexts, effectively serving as $T(\cdot)$.

The first constraint, $d[f_\mathtt{LLM}(\mathit{T}({\mathcal C}), \mathit{T}({\mathcal Q})), f_\mathtt{Human}(\mathit{T}({\mathcal C}), \mathit{T}({\mathcal Q}))]\leq \epsilon$, ensures that when only textual context $T(C)$ and question $T(Q)$ are given, the LLM’s response aligns semantically with human commonsense or physics knowledge. The second constraint, $d[f_\mathtt{Human}(\mathit{T}({\mathcal C}), \mathit{T}({\mathcal Q})), f_\mathtt{Human}(V, \mathit{T}({\mathcal Q}))]\geq \delta$, requires that hallucination cases lead to a significant difference in responses when humans view the actual video $V$, indicating a semantic gap of at least $\delta$. We measure these similarities using GPT-4 as an LLM-based evaluator. When framed as a binary classifier (same vs. different), both thresholds are implicit but can be estimated with high confidence given sufficient data.

Examples are provided in Appendix C. Overall, our goal is to rigorously model hallucination in synthetic video understanding and guide future MLLM improvements.

Response to Question 5: Our contribution goes beyond merely proposing a benchmark. We uncover a novel problem in synthetic video understanding: hallucinations in detecting physics and commonsense violations. In addition to introducing the benchmark and evaluating state-of-the-art MLLMs, we propose a curriculum learning-based training method to mitigate this issue. Our results show that even a small amount of synthetic video QA data can effectively help models improve in synthetic video understanding, suggesting a promising direction for future synthetic video hallucination mitigation.

Moreover, our curriculum pipeline offers an alternative approach to fine-tuning MLLMs on complex tasks using limited, high-cost data by leveraging more accessible, simpler datasets to guide training. These contributions align with the Deep Learning sub-track of NeurIPS (e.g., architectures, generative models, optimization, foundation models, LLMs), which we selected upon submission.

Regarding NeurIPS data/code disclosure policy, we were finalizing cleanup to prevent identification or double-blind policy violations at submission time. We will published our dataset and code for full reproducibility and include the link in the camera-ready version, as we are not allowed to including any link in our NeurIPS rebuttal. All necessary implementation details are provided.

Response to Question 6: We are using 8 A100 (80GB) for our SFT, GRPO fine-tuning and evaluation inference. We are using Video-R1 codebase [1]. We will include the computation resource in the final version. Thank you for pointing out this issue.

[1] Feng, Kaituo, et al. "Video-r1: Reinforcing video reasoning in mllms." arXiv preprint arXiv:2503.21776 (2025).

Response to Limitation 1: Our main limitation lies in the benchmark’s scalability, as both generating high-quality annotations and fine-tuning MLLMs at scale are expensive. The benchmark data were collected via paid APIs and expert-level human annotations, which are costly and time-consuming. Once more data gather and annotated with either human or MLLM automatic pipeline, future MLLMs are less likely hallucinated when probing physics/commonsense violations in synthetic videos. Another limitation, as noted by Reviewer DNqt, is the limited coverage of MLLMs used for fine-tuning. While we included InternVL3-8B fine-tuning results in our rebuttal, we are happy to expand the experiments over both currrent open-source video understaning MLLMs in the final version.

We will also elaborate further on these limitations in the Appendix.

Response to Follow-up 1: Thank you for pointing out this issue. We would like to clarify that, in accordance with NeurIPS 2025 guidelines, authors are not permitted to submit external links, upload additional files that may risk identity disclosure, or modify the originally submitted PDF during the rebuttal phase. Therefore, we cannot specify what exact content will be added, revised, or removed in the camera-ready version, especially for non-technical aspects such as paper organization, figure replacement, or minor terminology inconsistencies, while the review and discussion process is still ongoing. These types of adjustments are intended to be addressed during the camera-ready submission phase.

Regarding your follow-up questions, we plan to move Figures 3 and 5 to the appendix. In addition, we will include results from ablation studies and comparisons across various settings, such as SFT vs. GRPO, different GRPO and curriculum learning configurations, and data organization strategies (random shuffle vs. curriculum learning), all of which are included in Table A and C. We will also present generalization results on both Qwen2.5-VL-7B and InternVL3-9B, with and without curriculum learning. Furthermore, we will add evaluations on real-world relevant benchmarks, including MMVU and MVBench, which will be summarized in the updated Table B.

Response to Follow-up 2: In accordance with the NeurIPS rebuttal guidelines for authors

1. Past experience suggests that effective responses focus on factual errors in the reviews and on responding to specific questions posed by the Reviewers. Your response is optional and should be reserved for cases when a response is called for. The response you submit by July 30th should contain all your arguments regarding the reviews. The discussion with reviewers should be reserved only for discussing these points rather than for raising new arguments.

Our mention of running new experiments to support our argument to Question 2 is not intended to introduce new arguments in response to the Official Review of Submission6899 by Reviewer DNqt. Rather, it is meant to reinforce and clarify our existing arguments for the purpose of ongoing discussion.

Response to Follow-up 3: Here is the updated version of Table 3. Due to Markdown formatting constraints, we have separated the original composite table into four distinct sub-tables, now labeled Table C-1 through Table C-4. All models presented in Tables C-2, C-3, and C-4 are fine-tuned using Qwen2.5-VL-7B as the base model.

To better organize the discussion on model generalization across different backbones (i.e., Qwen2.5-VL-7B and InternVL3-9B), we will include those results separately in Table B, along with a dedicated subsection for generalization analysis in the camera-ready version of the paper.

Please note that due to the 10,000-character limit in the initial rebuttal, we were unable to include Table C as part of our previous submission. Including it would have exceeded the allowed length, given the scope of the initial reviews and the content already included. We plan to incorporate these additional results in the appendix or main paper as appropriate in the camera-ready version.

Thank you for the thoughtful feedback. We appreciate the recognition of our human-crafted dataset with adversarial QA pairs and taxonomy, as well as our effort to expose MLLM weaknesses on synthetic videos. We are also glad our training approach was noted to improve performance over the baseline.

Given the limitation on the rebuttal length, we include our all our supplementary experiment results in tables below, and we will cite these tables throught our rebuttal:

Table A. Ablation Study on Curriculum RFT

Table B. Real World Data Evaluation

Response to Weakness 1: The gain of the model improvement come from curricumum reinforcement fine-tuning strategy and augmenting the right training data. We provide more clear reasons and explaination below:

1. We present new results on non-curriculum learning by mixing physics and synthetic video data in Qwen2.5-VL-7B-RFT-Mixture-Data-Shuffle, yielding 52.3% accuracy, only 0.1% better than physics-only training, 1.1% lower than synthetic-only, and 1.9% below curriculum learning (PRW+SR-R1-7B). This suggests that just 0.8% of the gain comes from training strategy. In contrast, organizing the data using curriculum learning (GRW-PRW-SR-R1-7B) leads to a 7.7% improvement over the base model after RFT.

2. Our findings show that data mixing without curriculum learning achieves 52.3% accuracy, which is lower than the 54.2% from curriculum learning (PRW+SR-R1-7B).

3. From the GRW+SR-R1-7B results, we observe that mixing general world and synthetic data slightly degrades performance compared to synthetic-only training. However, integrating physics and synthetic data via curriculum learning yields superior results. Since the synthetic dataset only has 800 samples, augmenting it with physics data significantly boosts performance, highlighting the importance of strategic data combinations in training.

Response to Weakness 1.1 We present the results on training on training on the mixed dataset with shuffling in the result of Qwen2.5-VL-7B-RFT-Mixture-Data-Shuffle. Training on mixed data has an overall accuracy of 52.3%, where training on physics only data has 52.2%, and 53.4% training on synthetic data. Combining them without curriculum learning leads to minimal performance improvement or even degradation than learning first from easy real world physics data then transition to the harder synthetic video data (54.2%). Learning merely from mixed data without increasing difficulty level would cause the model to generate wrong responses for the more challenging data thus not backpropagating on the synthetic videos.

Response to Weakness 1.2: There appears to be no SFT baseline, so GRPO’s usefulness here is unclear.

We train the 7B model with SFT on our synthetic video understanding datasets and get the following results, with no significant improvement on our benchmark data. Since this is a challenging dataset, training the model with SFT does not lead to improvement (50.9% compared to base model 51%) because SFT simply maximizes likelihood of the ground-truth answers and does not expose the model to its own chain-of-thought. Synthetic video understanding tasks require temporal reasoning—tracking objects and causal events across frames—and SFT’s teacher-forcing objective doesn’t train the model to perform multi-step inference at test time. In addition, even the SoTA model has only 51% accuracy on the test data, if we use distilled CoT data to train the model, there will not be improvement since the model we are performing SFT/RFT is the SoTA model. Using RFT can fully leverage model's CoT process to generalize and improve temoral and physics reasoning on synthetic videos.

Response to weakness 2: We present new results in our table and highlight the following key takeaways:

SFT vs. RFT: Supervised fine-tuning (SFT) only optimizes the likelihood of ground-truth answers and does not involve the model's own chain-of-thought (CoT), limiting its ability to learn physics rules. In contrast, reinforcement fine-tuning (RFT) leverages the model’s CoT process, resulting in stronger performance gains on the synthetic benchmark.

Why Not Distill CoT from SoTA: Distilling CoT data from a state-of-the-art (SoTA) model for SFT is ineffective in our case, as the SoTA model is Qwen2.5-VL-7B itself—the one being trained. This would mean training the model on answers it already knows, offering little benefit.

RFT is more effective than SFT for improving synthetic video understanding in our setup, and we therefore adopt it as our primary training strategy.

Response to weakness 3: We provided real-world video benchmark results (We used VLMEvalkit). We show that there is no big difference or degradation on real world datasets when training with synthetic data, where a very small fluctuation of accuracy (<0.5%) is expected after finetuning.

Dear Reviewer tQNG,

We appreciate the time and effort you've dedicated to reviewing our work. Your insights have been extremely useful for us as they will make our work better, and we have taken care to address each concern raised. If there are any additional points of discussion, we would be grateful for the opportunity to engage further.

Since there are only 48 hours remaining in the rebuttal period, we kindly remind you to reconsider the evaluation of our work in light of the revisions and clarifications provided.

Regards,

Authors

Thank you for the thoughtful feedback. We're glad you enjoyed the paper and found the use of counterintuitive examples compelling. We truly appreciate your recognition of the writing clarity and the role of synthetic data in promoting physical plausibility.

Response to Weakness 1: Thank you for your insightful doubts. Aas the results shown in Table A, We train Qwen2.5-VL-7B on mixed data without performing curriculum learning on physics + synthetic video data (Qwen2.5-VL-7B-RFT-Mixture-Data-Shuffle) and show that no curiculum learning has an overall accuracy of 52.3%, where training on physics only data has 52.2%, and 53.4% training on synthetic data. Combining them without curriculum learning leads to minimal performance improvement or even degradation than learning first from easy real world physics data then transition to the harder synthetic video data (54.2%). Learning merely from mixed data without increasing difficulty level would cause the model to generate wrong responses for the more challenging data thus not backpropagating on the synthetic videos.

Response to Weakness 2: Thank you for pointing out this limitation. We agree that in the current version, our reinforcement fine-tuning (RFT) with limited data improves performance on physics and commonsense reasoning but at the cost of reduced performance on alignment-related QA in VideoHallu. We attribute this trade-off to the narrow focus of our RFT dataset, which emphasizes video understanding (real and synthetic) while lacking coverage on alignment tasks. In future work, we plan to incorporate more diverse and balanced training data to enhance MLLMs’ physics/commonsense reasoning without compromising performance on alignment QA. This includes expanding data coverage and improving training efficiency.

Response to Question 1: We will release our dataset and code in the final version as one of the core contribution of our paper. Also, we are happy to provide the metadata of our dataset as below. Here are more details of the metadata that will be included in the script:

• Avg. frames per video: 96.0
• Avg. video length: ~5.3 seconds
• Avg. framerate (FPS): 23
• Avg. frame resolution: 1042 × 588
• Number of video generation models: 7
• Avg. videos per model: 132
• Maximum number of frames to sample for training: 18
• Number of training iterations: 3 epochs
• Maximun number of frames to sample for testing: 128

All video question-answer (QA) pairs used in our benchmark will be released alongside the final version of our paper.

Response to Limitaion 1: According to NeurIPS guidelines, Question 10 (potential negative societal impacts) is out of scope for our work. We propose a benchmark focused on physics and commonsense violations in synthetic video QA tasks and demonstrate its utility in improving foundation models’ video understanding through reinforcement learning fine-tuning. Our work does not involve negative societal impacts, as it aims to mitigate such issues in other systems.

Dear Reviewer bcUY,

We appreciate the time and effort you've dedicated to reviewing our work. Your insights have been extremely useful for us as they will make our work better, and we have taken care to address each concern raised. If there are any additional points of discussion, we would be grateful for the opportunity to engage further.

Since there are only 48 hours remaining in the rebuttal period, we kindly remind you to reconsider the evaluation of our work in light of the revisions and clarifications provided.

Regards,

Authors