Unhackable Temporal Reward for Scalable Video MLLMs

We sincerely thank the reviewer for the high evaluation and high rating of our work and the valuable comments and questions provided. We have responded to each of your comments and questions as follows.

• Q1. Discussion on the quality of the expert model used for attribute extraction.

  Thanks for your valuable comments. Yes, the quality of the attributes extracted by the expert model is important to the final performance. We want to elaborate on this from three perspectives.

  1. The importance and rationale behind selecting expert models for attribute extraction. The use of expert models to support MLLM training has become a widely adopted strategy in the current development stage. Notable implementations include models such as PaLI-X [1], Qwen-VL [2], InternVL [3], and LLaMA3.2-Vision [4], which integrate domain-specific expert models spanning areas like detection, grounding, and OCR to scale up training data annotation. The effectiveness of this approach has been well-validated through extensive empirical studies.

  2. Extraction and filtering of high-quality attributes. This paper specifically designs a dedicated data modeling pipeline to extract high-quality spatiotemporal attributes and implements a series of measures to filter out low-quality attributes. Specifically, we first conduct a systematic evaluation based on existing benchmarks, eg., COCO, Lvis, VG, etc., of the performance of various options, such as GRiT and GroundingDINO, to identify the most suitable candidates. For the proposed spatiotemporal attributes—including bounding boxes, captions, identities, and actions, as illustrated in Figure.3 of our manuscript—we implement a multi-stage selection and filtering process. First, we filter the attributes based on the confidence scores predicted by the expert models. Next, we applied a multi-object tracking algorithm, ie., ByteTrack, to analyze contextual correlations within the video content. This analysis included examining factors such as the Intersection over Union (IoU) of bounding boxes across frames and trajectory continuity metrics, ensuring that trajectory lengths exceeded predefined thresholds. This comprehensive process ensures the reliability and consistency of the extracted attribute trajectories, thereby enhancing their overall quality and utility.

  3. Human validation of the extracted attributes. To further confirm that our approach can extract attributes of sufficiently high quality, we conducted an additional human evaluation experiment. Please refer to Q2 for more details.

  Please refer to the third common response and Appendix E.1 of the new updated manuscript for more details.

  [1] Chen et al., PaLi-X: On scaling up a multilingual vision and language model.
  [2] Bai et al., Qwen-vl: A frontier large vision-language model with versatile abilities.
  [3] Chen et al., Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.
  [4] Meta AI, Llama 3.2: Revolutionizing edge AI and vision with open, customizable models.

• Q2. The reliability of extracted metrics in the presence of multiple subjects in one frame.

  We sincerely appreciate your valuable feedback. To better address your concerns, we have incorporated an additional human quality assessment experiment to more accurately evaluate the reliability of our multiple subjects attribution extraction. Specifically, 100 data samples generated using our UTR pipeline are randomly selected for assessment by human evaluators. Human annotators score these data based on three criteria: the accuracy of the subject bounding box, the correctness of the attribute descriptions, and the consistency of the attribute trajectories, using a scoring range of 1 to 3. The results are shown below:

  Validation	Location	Description	Consistency
  Human	2.98	2.23	2.57

  We can observe that the average quality score of the extracted attributes is quite high, indicating a strong level of reliability. The results of this evaluation highlight the robustness and high quality of both the extracted spatiotemporal attributes and the constructed data, confirming the reliability of our pipeline. Please refer to Appendix E.1 for more details.

We sincerely appreciate your recognition of our work, as well as your thoughtful and professional review. The suggestions you provided are highly valuable for the further improvement of our work. We have responded to each of your comments and questions as follows.

• Q1. Issue about clarity and organization of some sections.

  Thanks for your suggestions. We have added more detailed captions, descriptions of our methods and more experimental details in the new updated version. Please refer to our new version manuscript.

• Q2. Discussion about computational overhead.

  Please refer to the second point of common response. And we can also provide a specific figure about the resources and time required for attribute extraction using expert models.

  Operation	Method	Resources	Time	Data Scale
  Video frame extraction	FFmpeg	8 * 64 CPU	2 days	7M
  Attribution extraction	GRiT	8 * 8 GPU	2 days	7M
  Trajectories construct	ByteTrack	8 * 8 GPU	3 days	7M

• Q3. Discussion about expert model in UTR including dependency, quality and computational cost.

  Please refer to the second point of common response, Q2 and the Appendix of new updated manuscript.

• Q4. Potential drawbacks in highly redundant or static video content.

  We have not observed any limitations in our Video-UTR regarding the understanding of highly redundant or static video content. In fact, as we can see from the results in Table 2 of the manuscript, Video-UTR still demonstrates strong capabilities in image, which can be considered a form of static video to some extent, understanding.

• Q5. More details about how TPL correlates with human judgment of video understanding quality.

  Thank you very much for your valuable and insightful question. To address this issue, we conducted additional human validation experiments, specifically manually ranking video data at different TPL score levels to reflect its consistency with human preferences. Specifically, we first randomly select 100 video-text pairs from VideoChat2 and calculate their temporal perplexity (TPL) based on the definition in Eq.6. Next, we sort the data by their TPL values and divide it into three groups: high, medium, and low. We then invite several human annotators to rate these sampled video-text pairs on a scale of 1 to 3. The criteria for scoring includes two aspects, ie., the richness of the video-text information (considering both information density and dynamics) and the relevance of the video to the text. Based on the annotators’ scores, the consistency can be evaluated based on the average human ratings and their alignment with the level categories.

  Validation	High		Medium		Low
  	Richness	Relevance	Richness	Relevance	Richness	Relevance
  TPL level	3	3	2	2	1	1
  Human	2.85	2.76	2.15	1.85	1.61	1.64

  We can observe that the groupings based on TPL scores and those based on human judgments are generally consistent. This indicates that our proposed TPL score is a reliable metric for filtering high-quality video-text pair data. Please refer to Appendix E.2 for more details.

• Q6. Lack of failure cases, experimental details.

  Thanks for your valuable comment. In the newly uploaded version, we dedicate a separate section to discuss failure cases (Appendix E.3), specifically analyzing knowledge-based questions that can be answered by observing only a single frame or a few frames. Additionally, we included more experimental details, covering the settings for both the training and evaluation phases (Table 8 and 9).

• Q7. Application to other domains involving sequential data.

  We believe that the approach of UTR is suitable for sequence optimization problems that require consideration of global input information rather than just local information. For issues such as speech or time-series analysis, we think our solution has potential applicability. More specific applications would require further investigation.

We greatly appreciate the Reviewer for the recognition of our work and the valuable comments and suggestions you provided. We have responded to each of your comments and questions as follows:

• Q1. Over-complicated Presentation of temporal hacking and lack of details of UTR.

  We greatly appreciate your valuable suggestions. In fact, the aim and key contribution of this work lie in systematically establishing the theory of temporal hacking for video-language modeling from a reinforcement learning (RL) perspective. Therefore, this work devotes significant effort to introducing, defining, and providing a novel perspective to interpret the problem of temporal hacking, which we think is more important and valuable than merely proposing a solution to address MLLM shortcut learning. As for the methods section (Section 3.2), we apologize that, due to space constraints, we placed some construction details of UTR in the appendix. In the final version, we will carefully balance the overall layout and pages to make appropriate optimizations.

• Q2. More detailed description of the proposed TPL.

  Thanks for your suggestions. In the latest version, we have added more details about TPL, including the calculation for $R_{ppl}$. For further details, please refer to the common response as well as the new version manuscript.

• Q3. Comparison with LLaVA-OneVision and more comparison results.

  The motivation in Video-UTR differs from that of LLaVA-OneVision. LLaVA-OneVision aims to create a general MLLM that incorporates images, videos, and 3D scenes, leading to notable differences in data scale, distribution, and characteristics compared to Video-UTR. Our Video-UTR specifically addresses the issue of temporal hacking in the video domain, which is why we do not perform a comprehensive comparison with LLaVA-OneVision. We believe that the current comparisons are sufficient to demonstrate the effectiveness of our approach. For the image understanding comparison using the same LLM setting, we have included the performance of the LLaVA-NeXT-Video QWen-2 setting in the new version manusrcipt.

• Q4. Broader statements, more citations about reward hacking and more detailed captions of figure.

  Thanks for your valuable suggestions. Following your suggestions, we add more statements and citations about reward hacking and add more detailed figure captions. These improvements are reflected in our latest version. Please refer to the new version manuscript.

• Q5. More benchmark results.

  Thank you for providing these new works and references related to video benchmarks. After carefully reviewing each benchmark, including its open-source status, level of recognition, and impact, we select MMBench-Video [1] to provide additional comparison. The results are shown below.

  Method	Overall	Perception	Reasoning
  Claude-3.5-Sonnet	1.35	1.4	1.04
  VideoChat2-HD	1.23	0.44	1.23
  PLLaVA-34B	1.16	1	1.1
  LLaVA-NeXT-Video-34B-HF	1.13	0.58	1.03
  Video-UTR-7B (Ours)	1.35	1.38	1.24

  We can observe that on the latest challenging MMBench-Video benchmark, our Video-UTR remains highly competitive, outperforming numerous 34B and even larger models with its 7B size, demonstrating the superiority of our approach.

  [1] Fang et al., MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding.

• Q6. Typo error in Line #376.

  We greatly appreciate your careful correction. This was indeed a typographical error. The data size indicated in the Table 1 is correct. And we have make this correction in the new version.

Thank you very much for recognizing our work and for providing such valuable comments and suggestions. We have responded to each of your comments and questions as follows:

• Q1. How are the other methods performance could be improved after simply data modeling？
  To address this question, we would like to present two observations and facts.\

  1. For existing methods that rely on simple data modeling, such as VideoChat2 [1], adding more video data does not necessarily lead to further performance improvements. This is illustrated in both Figure 1 (a) and Table 3 in our manuscript.

  2. To further substantiate this point, we conducted comparative experiments on WebVid, where we trained models using both a randomly sampled set of 1 million WebVid videos and a 1K subset sampled based on our proposed TPL score. As shown in the following table:

     Training Data	Size	MVBench
     Random Sample	1M	50.6
     TPL Sample	1K	51.2

     The model trained with the 1K WebVid samples outperforms the one trained on the randomly sampled 1 million videos. This finding underscores that with simple video data modeling, increasing data volume may actually lead to performance degradation, aligning with the “anti-scaling law” phenomenon we discuss in our paper.

  3. Existing video understanding benchmarks, such as VideoMME [3], are not optimally designed. Many test cases (shown in Figure 9 in Appendix) in these benchmarks can be answered correctly by observing only a single frame or a few frames and then relying on the LLM’s inherent knowledge (language priors) for reasoning. This design biases current video data modeling approaches towards equipping models with more knowledge rather than enhancing their ability to observe and understand video content comprehensively.

  4. Some recent studies have recognized the issue in existing benchmarks and made adjustments. For example, Tarsier [1] increases data diversity and reduces low-quality data that may introduce shortcuts by filtering verbs from captions using the model. COSMO [2] applies GRiT to obtain detailed annotation as prompts for GPT-4 to generate high-quality video captions. Merlin [3], on the other hand, guides the model to better capture spatiotemporal cues by constructing trajectory chain-of-thought data. These methods, to some extent, align with the two principles we proposed.

  [1] Wang et al., Tarsier: Recipes for Training and Evaluating Large Video Description Models.
  [2] Wang et al., COSMO: COntrastive Streamlined MultimOdal Model with Interleaved Pre-Training.
  [3] Yu et al., Merlin: Empowering multimodal llms with foresight minds.

• Q2. Further explanation of the ablation study in Table 3.

  We apologize for any confusion caused by the description of the settings in the ablation study in Table 3. In fact, we used a progressive ablation approach, gradually removing components we designed from the final model, Video-UTR. Therefore, the third row, labeled “- Data Modeling”，indicates the removal of both data modeling and task modeling. To avoid unfair comparisons due to reduced data volume, we added an equivalent amount of video data in the row 4 based on the row 3 settings. As shown in the results, the performance remains lower than that of our base method.

• Q3. The metrics for several datasets (MMBench, VideoMME) in Table 3 and Table 4 are not the best.

  MMVet, as an image-level understanding benchmark, is not highly sensitive to video data, and we just want to provide an additional reference. VideoMME, as explained in Q1, contains many evaluation cases that test the knowledge base of the LLM itself rather than its ability to understand spatiotemporal contexts, which is why our method do not yield a significant improvement.

• Q4. How is equation 6 used in loss function?

  In the practical application of Eq.6, TPL is not incorporated as a loss in the training process. Instead, it is applied in an offline manner to reflect the temporal perplexity score of the model in video-language modeling. This score serves as a metric to assess misalignment in temporal modeling.

• Q5. For Eq.6, why using T:T instead of T-t:T for the second term?

  In fact, the definition of Eq.6 corresponds to Eq.5. We explained the reason for this in Line #205 of the manuscript, namely that “To model an extreme case of proxy cumulative reward, we use a single, i.e., k = t, randomly sampled keyframe to represent the entire video context”. Here $t = T$ and T:T means the single frame we selected.

• Q6. Is that possible to get the TPL of baselines comparing with Video-UTO?

  Yes, it is. In fact, we have already shown the comparison of TPL scores between the baseline (VideoChat2) and Video-UTR (UTR-Data) in Figure 2(a) in the main text.