The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models Via Visual Information Steering

We cordially appreciate your careful review and insightful questions. We are thrilled that you find our analysis "reveals new insights"., our method gains "strong performance" and "can generalize". See below for detailed replies.

Q1. Regarding vision encoder’s quality

VISTA focuses on hallucinations caused by parametric priors and does not set extra requirements for visual encoder's quality, as evidenced by its effectiveness across four widely adopted architectures. The quality requirement applies to standard public architectures, consistent with existing methods such as VCD, PAI, and OPERA. VISTA complements ongoing research focused on improving vision encoders.

Q2. Regarding hyperparameter tuning

There are two hyperparameters: intervention strength $\lambda$ and mixing ratio $\gamma$.

• For $\gamma$, we found $\gamma\approx0.3$ consistently effective across all four architectures.
• Optimal $\lambda$ can vary by model; however, a broad range of $\lambda$ values are effectively for each model, as analyzed in the Synergy & Robustness subsection (p.7, line 381) and shown in Figs. 6, 12–14, demonstrating VISTA's robustness across different $\lambda$ configurations.

Q3. Regarding computational overhead

VISTA is an efficient de-hallucination strategy that introduces minimal computational overhead and demonstrates better efficiency than baseline approaches. We clarify this in below:

• For captioning, summarization, and open-ended generation tasks: The textual prompt is static (e.g., "please describe the image..."). VISTA forwards the textual prompt only once w/o image to cache activations (minimal overhead). For positive case, VISTA acts alike vanilla inference which forwards the visual+textual tokens. The only difference is before generating new tokens, VISTA builds VSV via simple vector arithmetic (marginal overhead), and proceeds with intervention. VISTA does not forward twice the input tokens, and it can use KV cache of them during generation` (overhead of this is also small).
• For QA tasks: Textual prompts vary per query, potentially requiring an additional forward pass of textual tokens. However, textual tokens typically account for less than 10% of total prompt length compared to image tokens`. Thus, the additional computational overhead remains mild relative to vanilla inference.
• Empirically, VISTA demonstrates better efficiency compared to other comparable methods (see Table 5). We kindly refer the reviewer to Reviewer 5sMs (Q2) for an extended efficiency comparison.

Q4. Is there "overstuffing" of visual details?

• We'd like to first clarify a potential caveat. The visual steering vector (VSV) does not merely amplify visual details. The positive vector $V_p$ is conditioned jointly on visual and textual tokens, capturing relevant visual-textual relationships via attention heads, while the negative vector ($V_n$) reduces textual-only priors. Consequently, VSV does not overstuff visual details that might blur the query's focus.
• This is empirically validated on MMHal-Bench (Fig.4), where each query specifically targets an entity or relation within visually complex scenes where many other entities and relations are existing, and VISTA performs well.

Q5. Multi-query scenario

As clarified in Q4, VSV is constructed per visual-query pair and is expected be recalculated per new query. However, as analyzed in Q3, constructing VSV incurs only mild computational cost due to KV caching and the relatively short length of query tokens compared to visual/system tokens. As a result, it is still efficient to use VISTA under multi-query scenarios.

Q6. Regarding extremely cluttered images

Inspired by your insightful question, we conducted a new experiment where 500 images from MSCOCO are randomly selected and divided into two groups (heavy and light) according to the degree of cluttering (GPT-4o is used to rate a cluttering score for each image). Results below (on LLAVA-1.5) demonstrate that heavily cluttered images is inherently more challenging; yet VISTA significantly outperforms vanilla decoding in both scenarios.

|Cluttering degree|$C_S$↓/$C_I$↓| |:-|:-| |Heavy (greedy)|52.0 / 13.0| |Heavy (ours)|26.5 / 5.3| |Light (greedy)|40.5 / 11.7| |Light (ours)|22.0 / 5.6|

Q7. Could early-layer steering fix the problem?

Following your suggestion, we conduct additional experiment applying early-layer steering (first 15 layers) for heavy and light cluttered images. As shown below, steering across all layers consistently outperforms early steering. Interestingly, the performance gap between early steering and steering all layers is smaller for lightly cluttered images, suggesting early layers handle much of the visual processing for easy cases.

|Cluttering degree|$C_S$↓/$C_I$↓| |:-|:-| |Heavy (early)|37.0 / 10.2| |Heavy (all)|26.5 / 5.3| |Light (early)|26.0 / 7.1| |Light (all)|22.0 / 5.6|

We sincerely appreciate your detailed review and constructive suggestions. We are encouraged to find our framework is being recognized as "Applying an effective method to each problem". All claims are evidenced, and related work are "well discussed". Below we address your questions in detail.

Q1. Regarding novelty

Our VISTA is novel in terms of both its analysis and methodology. To our best knowledge:

• VISTA's token ranking analysis is the first work inspecting internal token-level dynamics of LVLMs.
• VISTA systematically uncovers three novel observations about how LVLMs store and process visual information: (1) gradual visual information loss, (2) early excitation, and (3) hidden genuine information.
• VISTA resembles the first activation-space intervention method for reducing LVLM's hallucination, effectively addressing observed phenomena.

As suggested in your second bullet, VISTA indeed provides a new perspective by tracking the rankings of proposed "hallucinated", "genuine", and "hidden genuine" tokens. As a result, VISTA unveils three novel observations, and contributing valuable new insights to the field (supported by Reviewer ypPL).

Q2. Comparison and Discussion with [1] [2]

Below, we first compare VISTA with [1] and [2] in terms of performance and efficiency, followed by an in-depth discussion of the differences between VISTA and these methods. The extended comparison and discussion will be included in the next revision.

• Performance

|CHAIR|LLAVA-1.5| MiniGPT-4|Shikra|InstructBLIP| |:-|:-|:-|:-|:-| ||$C_S$/$C_I$↓|$C_S$/$C_I$↓|$C_S$/$C_I$↓|$C_S$/$C_I$↓| |ED [1]|43.0/14.0|-|-|-| |CT$^2$S (greedy) [2]|44.2/12.2|-|44.8/12.8|42.3/12.4| |Ours (greedy)|20.4/6.9|19.8/6.0|31.4/9.7|27.4/8.1| |CT$^2$S (sampling) [2]|45.0/11.7|-|46.0/12.9|43.6/13.1| |Ours (sampling)|24.0/8.2|18.4/6.4|31.8/9.7|29.4/9.1|

|POPE (avg)|LLAVA-1.5| MiniGPT-4|Shikra|InstructBLIP| |:-|:-|:-|:-|:-| ||Acc/F1↑|Acc/F1↑|Acc/F1↑|Acc/F1↑| |ED [1]|86.31/85.86|-|-|-| |CT$^2$S (greedy) [2]|85.94/85.92|-|81.84/82.23|82.30/83.58| |Ours (greedy)|86.15/86.29|75.96/77.11|82.44/82.47|84.87/84.95| |CT$^2$S (sampling) [2]|85.14/85.41|-|80.66/81.65|81.49/82.70| |Ours (sampling)|85.35/85.54|66.96/68.05|81.01/81.15|83.11/83.27|

As shown above, VISTA achieves superior hallucination reduction performance, particularly in long-sequence generation tasks such as CHAIR.

• Efficiency

We further compare efficiency among methods. Given different evaluation criteria, we set the cost of vanilla inference to 1 and measure relative changes. As demonstrated below, VISTA also outperforms [1] and [2] in efficiency.

|Method|Relative Inference Cost↓| |:-|:-| |Vanilla|1.0| |ED [1]|3.13| |FastED [1]|1.33| |CT$^2$S (40%) [2]|1.43| |CT$^2$S (10%) [2]|1.35| |Ours|1.27|

• Discussion

The rationale and implementation of logits ensembling differ significantly between VISTA and [1][2]. Method [1] aims to reduce visual distractions through image cropping, ensembling logits from multiple subcrops. Method [2] addresses global noise by pruning visual tokens based on the attention matrix, creating auxiliary logits. In contrast, VISTA's self-logits ensembling is motivated by the observation of early excitation behavior and is conducted across layers preceding the final layer to encourage decoding semantically meaningful tokens. Moreover, the proposed self-logits augmentation demonstrates synergy with the visual steering vector (VSV), as detailed in the Synergy & Robustness subsection (p.7, line 381).

Q3. What if $V_n$ is directly subtracted?

Following your insightful suggestion, we set the visual steering vector to $V_s=-V_n$ instead of $V_s = V_p-V_n$ and validated it across various intervention strengths ($\lambda$) on LLAVA-1.5. Results below indicate that:

• Solely removing information from the negative example offers limited benefits.
• Negative-only steering is highly sensitive to intervention strength.
• Best performance of negative-only steering significantly lags behind the original design.

Specifically, negative-only steering collapses (F1 < 70) when $\lambda > 0.1$. These findings highlight the critical role of preserving positive information during the intervention process.

|$\lambda$|$V_s=-V_n$|$V_s=V_p-V_n$| |:-|:-|:-| ||$C_S$↓/$C_I$↓/$F1$↑|$C_S$↓/$C_I$↓/$F1$↑| |0.05|56.8 / 15.2 / 75.5|50.8 / 13.6 / 76.9| |0.08|52.5 / 14.9 / 75.6|50.4 / 14.6 / 76.5| |0.1|41.6 / 14.4 / 74.2|47.4 / 13.1 / 77.8| |0.11|21.4 / 12.7 / 58.1 (collapse)|45.6 / 12.9 / 77.4| |0.12|2.6 / 23.6 / 13.7 (collapse)|41.6 / 12.2 / 77.7| |0.15|0.20 / 50 / 0.1 (collapse)|32.4 / 9.8 / 76.9| |0.17|- / - / - (collapse)|20.4 / 6.9 / 72.8|

We sincerely appreciate your thoughtful review and insightful feedback. We are glad that you find our approach "simple yet effective", our evaluation is "comprehensive", and demonstrate "improved performance". Below we address your questions in detail.

Q1. Regarding novelty

Our VISTA is novel in terms of both its analysis and methodology. To our best knowledge:

• VISTA's token ranking analysis is the first work inspecting internal token-level dynamics of LVLMs.
• VISTA systematically uncovers three novel observations about how LVLMs store and process visual information: (1) gradual visual information loss, (2) early excitation, and (3) hidden genuine information.
• VISTA resembles the first activation-space intervention method for reducing LVLM's hallucination, effectively addressing observed phenomena.

As supported by you, VISTA is a flexible design allowing integration with various architectures and decoding strategies. This versatility enables VISTA to complement ongoing research focused on improving vision encoders, strengthening visual-textual alignment, and developing specialized decoding methods to reduce hallucinations.

Q2. Regarding computational efficiency

VISTA functions as an efficient de-hallucination strategy that introduces only minimal computational overhead while demonstrating superior efficiency compared to other baseline approaches. We clarify this in detail below:

• For captioning, summarization, and open-ended generation tasks: The textual prompt remains constant (e.g., "please describe the image..."). VISTA forwards the textual prompt only once without the image, caching activations for future use (minimal overhead). For the positive example (visual + textual tokens), VISTA performs vanilla inference by forwarding tokens visual + textual once. The only additional step is constructing the visual steering vector (VSV) using the residual stream from the last input token and cached negative activations via simple vector arithmetics (this cost is marginal). Generation then proceeds normally under our proposed intervention. VISTA avoids forwarding the token sequence twice by leveraging the KV cache from input prompt tokens. Remaining computations involve logit ensembling and a few element-wise additions, resulting in mild overhead.

• For QA-like tasks: Textual prompts may vary per query, potentially requiring one additional forward pass for the negative instance (i.e., textual only query). However, the textual tokens are minimal compared to image tokens, typically constituting less than 10% of total input token sequence. Consequently, the extra computational burden remains mild compared to vanilla inference.

• Empirically, VISTA demonstrates better efficiency compared to other comparable methods (see Table 5). We kindly refer the reviewer to the panel of Reviewer 5sMs (Q2) for an extended efficiency comparison with other methods.

Q3. Regarding hyperparameter sensitivity

VISTA includes two hyperparameters: intervention strength $\lambda$ and mixing ratio $\gamma$.

• For $\gamma$, we found $\gamma\approx0.3$ consistently effective across all four architectures.

• For $\lambda$, optimal values can vary by model; however, a broad range of $\lambda$ values perform effectively for each model, as analyzed in the Synergy & Robustness subsection (p.7, line 381) and shown in Figs. 6, 12–14, which demonstrate VISTA's consistent effectiveness across different $\lambda$ configurations.