Mitigating Hallucination in VideoLLMs via Temporal-Aware Activation Engineering

Q1: Although the author divides video input into two categories: "unchanged timing" and "mutable timing", this dichotomy is relatively rough and cannot cover diverse timing characteristics such as scene switching, which may lead to insufficient adaptability of activation engineering on different types of videos.

A1: Thank you for your concern. Firstly, the binary classification into temporal-invariant and temporal-variant categories is derived from our analysis in Section 3. At the same time, our experimental results (Tables 2 and 3) also demonstrate the effectiveness of the binary classifier. Therefore, based on the current experimental results, utilizing a temporal binary classifier appears to be sufficient in most cases.

Secondly, as stated in Section 3.1 (Lines 146-149), the definitions of temporal-invariant and temporal-variant are relatively clear and mutually exclusive, allowing the vast majority of videos to be categorized into one of these two types. For example, scene switching can be regarded as a type of temporal variation. Consequently, we consider this binary classification of videos into temporal-invariant and temporal-variant categories to be reasonable.

Of course, we also acknowledge this limitation in Appendix F (Limitations) of our paper. In future work, we plan to explore more fine-grained classifications.

Q2: The authors identified key hyperparameters such as injection weights and the number of Top-K attention heads through grid search, but did not evaluate the impact of these parameters on the model hallucination performance within a reasonable fluctuation range, nor did their sensitivity be explained. To enhance the reproducibility of the method, it is recommended to give the value range or step length of the injection weight and the ablation experimental results of the Top-K attention head selection number to clarify the impact of each hyperparameter on model performance in the appendix.

A2: Thank you for your suggestion. We further conducted an ablation study on the value range of the injection weight ($\alpha$) and the number of Top-K attention heads selected. Specifically, following the grid search settings in the paper, we performed a grid search for the injection weight in the range $[8, 16, 24, 32]$ and for the Top-K attention head selection number in the range $[32, 64, 128, 256]$, using the VideoLLaMA2 model on the VidHalluc benchmark. The experimental results are shown in the table below (using Overall as the evaluation metric):

As can be seen, while the injection weight ($\alpha$) and the number of Top-K attention heads do influence model performance, the sensitivity is not drastic. Within a reasonable range, model performance generally improves as both the injection weight and the number of selected Top-K attention heads increase. However, when more attention heads ($256$) that are less sensitive to hallucinations are included, there is a slight decrease in performance. In practical applications, an accuracy threshold strategy can also be adopted to select attention heads, further enhancing the robustness of the model. We will add these findings to Appendix E to demonstrate the robustness of our approach.

Q3: Consider giving a more detailed description of the sample screening process, such as the specific prompt of GPT-4o, and normal and hallucination-inducing prompts, to enhance the reproducibility of the article.

A3: Thank you for your suggestion. Firstly, we have already presented the specific content of the prompts we used in Figure 7 of Appendix C.2, and we guide readers to it in Section 4.1 (Line 284).

Secondly, in Section 3.2, we have described our normal and hallucination-inducing prompts in detail, specifically the prompt designed to induce hallucination through frame downsampling. We will add the following content to Section 4.2's Implementation details to enhance reproducibility:

"Consistent with Section 3, we use concatenating the original video and the question as the normal prompt $x$, and use concatenating the 4x downsampled video and the question as the hallucination-inducing prompt $x_h$."

Q4: The article mainly uses quantitative accuracy improvement as an evaluation indicator. It is recommended to add specific examples to illustrate the specific details of reducing hallucinations, such as the comparison of model output before and after injection of the offset vector.

A4: Thank you for your suggestion. We will add some concrete examples to Appendix E to illustrate how our method reduces hallucinations. Specifically, since most subtasks are selection/classification tasks, we will focus on providing specific examples from the STH's description task. We will demonstrate the specific changes in model output after using the baseline and after applying our method, allowing readers to more intuitively understand how our approach mitigates hallucinations. Below is an example (since OpenReview cannot display videos, we only show the ground truth text and the model output text):

Ground Truth: Scene change: Yes, Locations: from on a stage to in the woods.
Model Output (Baseline): Scene change: Yes, Locations: from a dark room to a car.
Model Output (Ours): Scene change: Yes, Locations: from performing on a stage to being in the woods.

As can be seen, the Baseline exhibited hallucinations regarding the order of scene changes and scene recognition, while our method effectively reduced these hallucinations.

Q1: While the analysis is thorough, the core technical contribution—applying activation editing to VideoLLMs—builds heavily on existing work without significant methodological advancement in the editing technique itself.

A1: Thank you for your question. Firstly, due to modality differences, existing activation editing methods from the image domain cannot be directly applied to the video domain. One of the key contributions of our work is to identify temporal variation as a crucial factor influencing hallucinations in VideoLLMs. Only by uncovering this key factor can activation editing methods be effectively transferred to the video domain.

Secondly, while the activation editing method we used in Section 4 is not entirely new, we are the first to systematically apply it to VideoLLMs to mitigate model hallucinations, conducting comprehensive experiments and analysis.

Furthermore, based on our analysis, we adapted the method to the specific characteristics of the video domain. Instead of using a unified dataset to calculate offsets, which is common in the image domain, we compute appropriate offsets separately based on the temporal characteristics of videos. We have validated its effectiveness through extensive experiments. This work provides a comprehensive foundation and new insights for future research into the application of activation editing in VideoLLMs.

Q2: The proposed adaptive editing framework involves non-trivial training (e.g., classifier-based module sensitivity detection). Including an analysis of inference-time overhead and classifier sample efficiency would help clarify the practical feasibility of deploying the method. What is the sample efficiency for training a binary classifier in your case? What is the inference efficiency after equipping with you intervene method?

A2: Thank you for your suggestion.

Inference efficiency: Firstly, as mentioned in Section 4.1 (Lines 298-303), we minimize inference-time overhead by running the classifier and visual encoder in parallel. For more implementation details, please refer to the paper.

Secondly, we further evaluated the per-sample inference time of both our method and the Baseline on the VidHalluc benchmark. The results are presented in the table below (unit: seconds):

As can be seen, the per-sample inference time of our method on the VidHalluc benchmark is comparable to that of the Baseline, demonstrating the effectiveness of our parallel setup.

Classifier sample efficiency: Firstly, as mentioned in Section 4.1 (Lines 293-297), we sampled $400$ instances from each category of the dataset to train the classifier. This is already a relatively small sample size in the current VideoLLM field.

Secondly, we further tested the classifier's sample efficiency by reducing the number of training samples. Specifically, we trained with $100$, $200$, $300$, and $400$ samples respectively, and recorded the classification accuracy of the classifier on the VidHalluc benchmark for each quantity. The experimental results are shown in the table below:

It can be observed that as the number of samples increases, the classifier's accuracy gradually improves. However, even with only $100$ training samples, the classifier's accuracy still exceeds $88$%. This demonstrates that our classifier possesses high sample efficiency.

Q3: Some recent publicaitons with similar insights should be included in the activation editing part of related works. And they may be more advanced compared to the activation engineering used in this paper. These methods introduce new perspectives—spectral, geometric, or constrained projections—that may offer stronger guarantees or improved efficiency and should be discussed in the context of this work.

A3: Thank you for your suggestion. We will add these latest research findings to the related works section in our revised paper and discuss their similarities, differences compared to our method. Specifically, we will add the following content:

Recent works have explored activation editing techniques from spectral, geometric, and constrained perspectives. Spectral Editing of Activations (SEA) [1] leverages SVD-based inference-time projections on cross-covariance matrices, achieving data-efficient edits without training. AlphaEdit [2] introduces null-space constrained weight updates, providing theoretical guarantees against catastrophic forgetting during sequential edits. Singh et al. [3] formulates steering transformations as optimal affine mappings over hidden states, effectively addressing biases and toxic outputs. Unlike these approaches, our method dynamically identifies hallucination-sensitive modules in VideoLLMs via temporal-aware activation editing, adapting inference-time edits specifically to temporal variation characteristics in video tasks.

1. Qiu, Yifu, et al. "Spectral editing of activations for large language model alignment." Advances in Neural Information Processing Systems 37 (2024): 56958-56987.

2. Fang, Junfeng, et al. "AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models." The Thirteenth International Conference on Learning Representations.

3. Singh, Shashwat, et al. "Representation surgery: theory and practice of affine steering." Proceedings of the 41st International Conference on Machine Learning. 2024.

Q4: It is easy for LLM's capability to be corrupted after editing. Could you provide the in-task performance for the intervened model after equipping with your approach? Is that any hurt in performance or other model's capabilities?

A4: Thank you for your question. Firstly, the STH subtask within the VidHalluc benchmark includes a description task, and the performance improvement of our method on this task initially demonstrates that our approach does not harm the model's capabilities.

Secondly, we conducted further evaluation on the VideoMMMU benchmark, a multi-modal and multi-disciplinary video benchmark that evaluates VideoLLMs' knowledge acquisition capability from educational videos, and the results are shown in the table below:

As can be seen, the performance of our method on all subtasks of the VideoMMMU benchmark is comparable to the Baseline, demonstrating that our approach does not impair the model's capabilities.

Thank you for your time and effort in the rebuttal.

The additional experiments address most of my concerns in the experiments, and I think the evaluation is technically solid. However, my main concern in the novelty still remains,

1. The activation editing method is not new; and I personally think "the first to apply it to VideoLLM" does not necessarily contribute as the additional novelty.
2. The insight "temporal variation as a crucial factor influencing hallucinations in VideoLLMs" is also not new for video understanding. Checkout [1,2,3].

[1] EventHallusion: Diagnosing Event Hallucinations in Video LLMs [2] VidHal: Benchmarking Temporal Hallucinations in Vision LLMs [3] Thinking Hallucination for Video Captioning

Therefore, I will remain my score as it is. To AC/SAC, if AC/SAC believes that my score is too harsh according to these two justifications, feel free to interpret the numerical score in the review as relatively positive.

Best

Q1: Lack of theoretical justifications: Why is hallucination more correlated with temporal variations than task types? Why is injecting offsets into certain layers or attention heads effective? The approach is largely empirical and needs more theoretical analysis and justification. Can you provide any theoretical analysis or intuitive explanation for why this mechanism reduces hallucinations effectively?

A1: Thank you for your question. Firstly, one of the core challenges for VideoLLMs is effectively capturing and integrating the temporal dynamics of videos. When temporal information is incomplete or misinterpreted, the model is more prone to generating hallucinations [1, 2]. Conversely, task types often represent a higher-level semantic abstraction that is largely independent of the underlying video's intrinsic temporal characteristics. For a given video, the same underlying temporal events can be probed with different task formats. For example, a multiple-choice question can be easily rephrased into a true/false question without altering the video's temporal content.

Secondly, regarding the effectiveness of injecting offset vectors, we can view the attention heads and layers within a VideoLLM as "experts" that process information at different levels of abstraction. Some of these "experts" may be specifically responsible for handling temporal dynamics or integrating information across time, which is consistent with previous findings [3,4]. When these "experts" receive ambiguous or inconsistent temporal signals (e.g., due to frame downsampling), they may produce biased internal representations, leading to hallucinations. By calculating and injecting "offset vectors", we are essentially introducing a "correction signal" into the internal workspace of these critical "experts". This signal pushes the activations towards a state closer to the true, non-hallucinated output, thereby correcting the "latent deviation" caused by incomplete or misleading temporal information. This has also been demonstrated in previous works [5, 6]. This targeted intervention, combined with our discovery of the modules' sensitivity to temporal variations, enables activation engineering to efficiently and specifically mitigate hallucination problems in VideoLLMs.

1. Kong, Ming, et al. "MHBench: Demystifying Motion Hallucination in VideoLLMs." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 4. 2025.

2. Rawal, Ruchit, et al. "ARGUS: Hallucination and Omission Evaluation in Video-LLMs." arXiv preprint arXiv:2506.07371 (2025).

3. Park, Yein, et al. "Does Time Have Its Place? Temporal Heads: Where Language Models Recall Time-specific Information." arXiv preprint arXiv:2502.14258 (2025).

4. Francis, Jiztom Kavalakkatt, and Matthew J. Darr. "Interpretable AI for Time-Series: Multi-Model Heatmap Fusion with Global Attention and NLP-Generated Explanations." arXiv preprint arXiv:2507.00234 (2025).

5. Li, Kenneth, et al. "Inference-time intervention: Eliciting truthful answers from a language model." Advances in Neural Information Processing Systems 36 (2023): 41451-41530.

6. Chen, Junzhe, et al. "Ict: Image-object cross-level trusted intervention for mitigating object hallucination in large vision-language models." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

Q2: How do you ensure that the offsets generalize to different videos and video types?

A2: Thank you for your question. Firstly, in Section 4.1, we specifically chose ShareGPT4Video as our data source, which is different from the sources of the VidHalluc and EventHallusion benchmarks. This was done intentionally, unlike previous work that only drew from benchmark subsets, to test the ability of the obtained offsets to generalize to different video types and to avoid inflated results (detailed explanation can be found in Lines 263-269 of Section 4.1).

Secondly, we utilized two benchmarks with different video types, VidHalluc and EventHallusion. VidHalluc contains a variety of video types such as action and scene, while EventHallusion mainly consists of event-related videos. This further verifies the generalization ability of our method across diverse video types.

Finally, the experimental results in Section 4.3 (Tables 4 and 5) demonstrate that the offsets generated by our method can effectively generalize to different video types.

Q3: Why is a binary classification sufficient? What about videos that do not fit cleanly into either category?

A3: Thank you for your question. Firstly, the binary classification into temporal-invariant and temporal-variant categories is derived from our analysis in Section 3. At the same time, our experimental results (Tables 2 and 3) also demonstrate the effectiveness of the binary classifier. Therefore, based on the current experimental results, utilizing a temporal binary classifier appears to be sufficient in most cases.

Secondly, as stated in Section 3.1 (Lines 146-149), the definitions of temporal-invariant and temporal-variant are relatively clear and mutually exclusive, allowing the vast majority of videos to be categorized into one of these two types. Consequently, we consider this binary classification of videos into temporal-invariant and temporal-variant categories to be reasonable.

Of course, we also acknowledge this limitation in Appendix F (Limitations) of our paper. In future work, we plan to explore more fine-grained classifications.

Q4: Are there any known failure cases, and how does the method behave in those situations?

A4: Thank you for your question. We further analyzed the specific changes in model output when using the Baseline versus our method, across all subtasks' failure cases on the VidHalluc benchmark with the Qwen2.5-VL-7B model. Specifically, we used GPT-4o to determine if the hallucinations produced by the Baseline and our method's outputs were consistent, and we calculated their hallucination consistency rate. The experimental results are shown in the table below:

As can be seen, the hallucination consistency rate of our method with the Baseline exceeds $90$% across all subtasks' failure cases. This indicates that in the vast majority of failure situations, our method gracefully degrades to the original model's behavior and does not further exacerbate the hallucinations.

Q5: Are there other types of hallucinations that are as important as or more important than temporal variation (e.g., factual, spatial, or visual hallucinations)?

A5: Thank you for your question. Firstly, in Section 3, our analysis of the VidHalluc benchmark revealed that temporal variation is the primary cause of hallucination in VideoLLMs. Even in other types of hallucinations (e.g., action hallucination, which is primarily evaluated by the BQA and MCQ subtasks in VidHalluc), temporal variation remains a significant contributing factor.

Furthermore, our experimental results in Section 4.3 (Tables 4, 5) provide additional proof of this method's effectiveness on other types of hallucinations, including EventHallusion, which specifically targets hallucinations about events.

In summary, our analysis in Section 3 indicates that temporal variation is one of the most important causes of hallucination in VideoLLMs. While other types of hallucinations (e.g., factual errors) may also exist, they are often closely related to temporal variations. Therefore, we consider temporal variation to be one of the fundamental types of hallucination.

Naturally, in the future, we will also consider other types of hallucinations and attempt to incorporate them into our activation engineering framework.

Q6: How are shallow and deep layers defined in your method? Is this based on layer index or some functional behavior?

A6: Thank you for your question. In the paper, the definitions of "shallow" and "deep" layers are simply based on their index. Layers closer to the input are considered shallow, while those closer to the output are considered deep. We will annotate this on the x-axis in Figures 3, 8, and 9 to provide readers with a clearer understanding.

Q1: The scalability of the proposed method is not clearly demonstrated. Since the most powerful models (like Gemini, Qwen2.5-VL-72B) may not suffer from severe hallucination issues, the effectiveness of the proposed approach in such models remains uncertain and may be limited.

A1: Thank you for your question. We further conducted additional experiments to verify the effectiveness of our method on the VidHalluc benchmark using the Qwen2.5-VL-72B-Instruct model. The results are shown in the table below:

Firstly, it can be observed that while the Qwen2.5-VL-72B model outperforms smaller models like Qwen2.5-VL-7B and VideoLLaMA2-7B on most subtasks (BQA, STH, TSH) of the VidHalluc benchmark, indicating that more powerful models do indeed exhibit some mitigation of hallucination issues, hallucination problems still persist. Furthermore, for certain specific tasks (e.g., MCQ), the hallucination is even more severe compared to the VideoLLaMA2-7B. This highlights the persistent challenge of hallucination in VideoLLMs.

Secondly, it can be seen that our method also achieves significant performance improvements on the Qwen2.5-VL-72B model, demonstrating consistent hallucination mitigation capabilities across all subtasks. This proves the effectiveness and scalability of our method in alleviating hallucination in large models.

Q2: It appears that contrastive decoding methods could also be adapted for hallucination reduction. The paper would benefit from a comparison with more advanced contrastive decoding approaches, such as [1-4], which can be readily applied to reduce hallucinations in video generation tasks.

A2: Thank you for your suggestion. Due to limitations in time and computational resources, we selected CGD and VTI ([2] and [3] from your provided papers) for further comparison experiments on the VidHalluc benchmark, using the VideoLLaMA2-7B, Qwen2.5-VL-7B and Video-LLaVA-7B models.

Since both CGD and VTI are primarily image-based methods, we adapted them to the video domain with the following improvements:

• For CGD, we replaced the CLIP model used for similarity score calculation with VideoCLIP.

• For VTI, we masked all frames of each video to generate a set of corresponding random masked videos. Additionally, for each video caption $x$, we used GPT-4o to generate its corresponding hallucinated version $\tilde{x}$.

• For a fair comparison, we used the union of $\mathcal{D}_a^f$ and $\mathcal{D}_t^f$ to generate VTI's visual and textual directions.

• All other hyperparameters adopted the optimal values provided in their respective papers.

The experimental results are shown in the table below:

As can be seen, our method outperforms both CGD and VTI in mitigating model hallucinations across various subtasks and models on the VidHalluc benchmark. This fully demonstrates the advantage of our approach in alleviating model hallucinations.