Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMs

Response to Reviewer 997H: We sincerely thank you for the thoughtful and insightful feedback! Your suggestions prompted us to evaluate perceptual fidelity and attribute understanding, confirming that VAP preserves factual grounding without harming visual understanding. These additions further strengthen the clarity and empirical support of our claims.

Below, we address each point in detail:

Q1: Experimental evaluation of perturbation perceptibility A1: We evaluate the perceptual similarity between original and perturbed images using LPIPS and SSIM on 500 BEAF image–instruction pairs.

(1) Setup:
We evaluate four representative LVMs: LLaVA-v1.5, Qwen-VL2, Intern-VL2, and DeepSeek-VL2.
For each image, we compute LPIPS and SSIM between the original image and:

• (a) its VAP-perturbed version.
• (b) a Gaussian-noised version with the same $L_2$ magnitude.

(2) Results:

(3) Observations

• VAP perturbations are visually negligible: All LPIPS < 0.05 and SSIM > 0.95, aligning with standard perceptual thresholds.
• Better than Gaussian noise: VAP consistently yields lower LPIPS and higher SSIM, indicating stronger perceptual fidelity.
• Conclusion: VAP introduces minimal visual distortion, preserving utility and trust for real-world deployment.

Q2: Can VAP reduce attribute hallucinations (e.g., color, count, action)?
A2: Yes. VAP is designed to address object hallucinations, but it also proves effective in reducing fine-grained hallucinations (e.g., color, count, position). Results show that VAP consistently improves attribute-level grounding without harming factual understanding.

(1) Benchmark Results (MME & AMBER) We report extended results in the appendix (Table 12), including performance on MME (Perception + Reasoning) and AMBER (Hallucination Analysis). VAP improves both factual understanding and hallucination robustness across four LVMs:

→ These results confirm that VAP enhances factual perception and reduces hallucination across diverse LVMs.

(2) More Attribute-Level Performance Per your request, we provide more granular evaluations on attribute-related categories using MME and MMHal[1]. VAP improves grounding in color, count, position, and comparison categories:

MME Benchmark

MMHal Benchmark

(3) Conclusion

VAP mitigates object-level hallucinations and consistently reduces attribute-related errors, while preserving the factual reasoning capabilities.

[1] Aligning Large Multimodal Models with Factually Augmented RLHF, ACL 2024.

Response to Reviewer 997H:
Thank you so much for your constructive and insightful comments! We address your concerns point by point as follows:

Q1: Comment on performance gap and outlook on future work.
A1: We summarize our position and future directions as follows:

(1) Positioning of VAP vs. Related Approaches
Unlike prior training-free methods that operate at decoding or post-processing stages, VAP is a fully black-box, training-free, and plug-and-play strategy that intervenes before decoding. This makes it broadly compatible and efficient:

(2) Future Directions
To further narrow the performance gap without compromising efficiency, we are actively pursuing:

• Improved optimization strategies beyond zero-gradient updates to better calibrate perturbation.
• Architecture-aligned proxies that account for model-specific sensitivity patterns.
• Lightweight distillation to compress VAP-enhanced robustness into the base model.

These will be incorporated into future iterations to enhance the practicality and generalizability of VAP.

[1] Mitigating Object Hallucinations in LLMs via Visual Contrastive Decoding, CVPR 2024.
[2] VisualGPTScore: Visio-linguistic Reasoning with Multimodal Generative Pre-training Scores, arXiv 2023.
[3] PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures, CVPR 2022.

Q2: Provide more diverse attack scenarios.
A2: Per your request, we evaluate the robustness of VAP under diverse multimodal adversarial attacks, beyond standard Gaussian noise.

(1) Attack Scenario

We adopt three adversarial configurations following [4], designed specifically for LVMs:

• MF-it: Image-text alignment attack
• MF-ii: Image-image contrastive attack
• MF-ii + MF-tt: Combined multimodal attack
  We additionally include Gaussian noise as a standard reference.

Evaluation: CHAIR-I and CHAIR-S hallucination scores (↓ better). Models: LLaVA-v1.5 and Intern-VL2 . Benchmark: 1000 image-question pairs randomly sampled from MS-COCO.

(2)Results

• LLaVA-v1.5

• Intern-VL2

(3) Conclusions

• Consistent improvement across all attack types and models indicates that VAP’s gains are not brittle or overfitted to clean inputs.
• Despite not being explicitly trained for adversarial robustness, VAP shows strong generalization to unseen perturbations.
• This supports our claim that VAP leverages targeted adaptation, not simply noise filtering.

[4] On Evaluating Adversarial Robustness of Large Vision-Language Models, NeurIPS 2023.

Dear Reviewer 997H,

Thank you again for your time and thoughtful feedback! Your suggestions on perceptual evaluation, robustness analysis, and future directions have greatly improved the depth and clarity of our work.

As the rebuttal deadline approaches, could you kindly consider raising your score slightly if you feel the revisions have resolved your remaining concerns? We are truly grateful for your engagement throughout the review process!

Best regards,
Authors

Response to Reviewer NLrm:

Thank you for your thoughtful and constructive review. We are glad that you found the paper clear and appreciated the comprehensive evaluation across 8 LVMs and 5 metrics. Building on your observations, we emphasize the following strengths of our method:

• Black-box and training-free: VAP works without accessing model weights, gradients, or internal logits, making it suitable for closed-source APIs and real-world applications.
• Plug-and-play: Applied directly at inference time, with no modification to model architecture or need for retraining.
• Task- and dataset-agnostic: Once configured per model, VAP generalizes across benchmarks and prompt types.
• Consistently effective: Reduces hallucinations while preserving factual grounding across diverse LVMs.

To clarify your concerns and resolve misunderstandings, we conducted the following efforts:

• Explained the rationale behind constraining $\epsilon$ (semantic stability, not imperceptibility).
• Compared VAP with OPERA and VCD, showing competitive or better results.
• Clarified the empirical basis of our design and the role of visual uncertainty.
• Refined the interpretation of Strategy S1 to highlight its goal of reducing prompt overreliance.

We sincerely thank you for your feedback, which has helped improve the clarity and depth of our work!

We address each of your points in detail below:

Q1: Why constrain $\epsilon$ if imperceptibility is not required?
A1: We constrain $\epsilon$ not for imperceptibility, but to ensure:

• Preserve semantic integrity: Large perturbations may distort key visual features and degrade response quality.
• Ensure reliable bias mitigation: As shown in Figure 5, excessive $\epsilon$ disrupts feature extraction, reducing performance.
• Maintain stability: A bounded $\epsilon$ improves consistency across prompts and supports generalizable use in multi-turn settings.

Q2: Lack of comparison to other hallucination mitigation methods.
A2: We provide direct comparisons to two recent training-free methods:

(1) OPERA [1]:
As shown in Table 11, VAP outperforms OPERA across 4 LVMs on F1_TUID and CHAIR scores. Combining VAP with OPERA further reduces hallucination, showing complementarity.

(2) VCD [2]:
On LLaVA-v1.5 and Intern-VL2, VAP achieves comparable or better results than VCD. Their combination yields the strongest performance:

(3) Design difference:
VCD requires decoder access (white-box); VAP is fully black-box and more generalizable.

[1] OPERA: Alleviating Hallucination via Over-Trust Penalty and Retrospection-Allocation, CVPR 2024.
[2] VCD: Mitigating Object Hallucinations in LLMs via Visual Contrastive Decoding, CVPR 2024.

Q3: The method relies on 8 hyperparameters, which may hinder real-world deployment.
A3: We address this through a lightweight and structured configuration strategy:

(1) Four parameters are globally fixed:
$(\alpha, \beta, N, \epsilon)$ are shared across all models and datasets—no tuning required.

(2) The remaining four are model-specific but dataset-agnostic:
$(\sigma_1, \sigma_2, \sigma_3, T)$ are selected once per model and reused across tasks.

(3) Compact search space:
$\sigma_i \in \{0.1, 0.5, 1.0\}$ and $T \in \{100, 200, 500, 800\}$. Several models share identical configurations.

(4) More scalable than prior methods:
VCD [3] requires tuning per model–dataset pair. VAP avoids this, enabling easier deployment.

[3] Mitigating Object Hallucinations in LLMs via Visual Contrastive Decoding, CVPR 2024.

Q4: VAP increases the inference cost due to additional computations.
A4: To address the added cost, we introduce a fast proxy-based variant:

(1) Proxy-based variant significantly reduces cost:
A lightweight proxy-VAP variant achieves up to 8× faster inference while preserving strong hallucination reduction (Table 4, Appendix D).

(2) Zero training, zero gradient, broad generalization:
VAP requires no model access or fine-tuning, and generalizes across datasets without reconfiguration—offsetting the one-time cost with wide applicability.

Q5: Lack of theoretical support for the hallucination cause and mitigation mechanism.
A5: Our explanation is empirical, not theoretical, and grounded in observed LVM behavior:

(1) Empirical observation, not formal theory:
The link between hallucinations and long-tail data or parametric priors is based on engineering intuition and patterns documented in prior work [4-7], rather than formal derivation.

(2) Evidence from literature:
Instruction-tuned LVMs often default to language priors—especially on underrepresented (long-tail) concepts—when visual evidence is weak, leading to plausible but ungrounded outputs [5,7].

(3) Clarification of Strategy S1:
S1 aligns outputs from the prompted and unprompted image, reinforcing grounding in the actual visual input—not removing the condition—but reducing overreliance on the prompt alone.

We will clarify that these claims are empirically motivated and supported by prior studies.

[4] Mitigating hallucination in large multi-modal models via robust instruction tuning, ICLR 2023.
[5] Evaluating object hallucination in large vision-language models, EMNLP 2023.
[6] What matters when building vision-language models?, NeurIPS 2024.
[7] Revisiting the Role of Language Priors in Vision-Language Models, ICML 2024.

Q6: Clarifying the role of visual uncertainty in diagnosing parametric knowledge bias.
A6: Visual uncertainty plays a diagnostic role in revealing the extent to which LVMs rely on internal parametric knowledge:

(1) What is visual uncertainty?
We define it by gradually degrading the image using Gaussian noise (Eq. 4). Increasing the noise scale $T$ simulates reduced visual information, allowing us to measure the model’s shifting reliance from visual evidence to parametric priors.

(2) How it reveals parametric bias:
When visual information deteriorates, models increasingly rely on $P(y|c)$ instead of $P(y|x,c)$. This transition exposes the influence of internal biases, which is exactly what VAP aims to mitigate.

(3) Empirical evidence:
Without applying VAP, we evaluate LLaVA-v1.5 and Qwen-VL2 on BEAF under increasing $T$. Both accuracy and F1 degrade sharply as noise increases, confirming that hallucinations intensify under visual uncertainty:

These results confirm that visual uncertainty directly correlates with increased hallucination, validating its role in revealing and mitigating parametric knowledge bias.

Response to Reviewer NLrm: We sincerely thank you for the constructive and insightful feedback!

We address each of your points in detail below:

Q1: Clarification on Strategy S1 and Comparison to Prior Methods.

A1: We clarify the motivation and implementation of Strategy S1, and how VAP differs from existing training-free baselines.

(1) Comparison with Prior Methods:

(2) Motivation of S1: Adversarial Sensitivity Small input changes can significantly shift LVM decoding [3]. S1 leverages this property in reverse: apply beneficial perturbations to reduce prompt overreliance and reinforce visual grounding.

• Optimized using paired inputs (with/without prompt)
• No access to attention weights or decoding internals

We will add visual attention heatmaps before/after VAP in the revised version to further support this design.

[1] Mitigating Object Hallucinations in LLMs via Visual Contrastive Decoding, CVPR 2024.
[2] Paying More Attention to Image: A Training-free Method for Alleviating Hallucination in LVMs, ECCV 2024.
[3] On Evaluating Adversarial Robustness of Large Vision-Language Models, NeurIPS 2023.

Q2: Clarification on the role and combination of S1/S2/S3.

A2: To clarify the necessity and synergy of $\mathcal{L}_{s_1}$, $\mathcal{L}_{s_2}$, and $\mathcal{L}_{s_3}$, and to substantiate the role of visual uncertainty in mitigating hallucinations, we provide the following response:

(1) Distinctive Roles of S1, S2, and S3

Each of the three losses is designed to target a unique failure mode, ensuring complementary rather than redundant behavior:

(2) Cross-Model Similarity Trends Validate Effectiveness

As requested, we present the dynamics of the semantic similarities defined in Eq. (3, 5, 6):

• VAP consistently increases S1 (better grounding), while decreasing S2 and S3 (less language prior bias).
• Improvements are consistent across models, highlighting generality.

(3) Dynamics under Visual Uncertainty

To assess how visual degradation modulates hallucination, we progressively increase the noise level (T) on Intern-VL2 and track S2/S3 shifts:

(4) Key Insights: Visual Uncertainty Exposes and Reduces Hallucination Bias

• Without VAP:

  • S2/S3 values remain static across T because the model lacks any uncertainty-aware mechanism.
  • This reinforces that hallucination tendencies are not self-corrected by default.

• With VAP:

  • T=200 reveals the most pronounced gain in ΔS2/ΔS3, indicating VAP is most effective when vision is ambiguous but still informative.
  • This aligns with our goal: leverage moderate uncertainty to reduce reliance on text priors and improve robustness.

• Interpretation of Trends:

  • Low T (0–100): Visual input is clean → model grounded → little room for improvement.
  • Mid T (~200): Input is degraded enough to trigger hallucination, but still usable → VAP is most beneficial.
  • High T (≥500): Input is too noisy → even VAP cannot rescue hallucination → diminishing returns.

• These results justify the inclusion of S2 and S3: they allow targeted suppression of prompt- and prior-driven hallucinations, especially under real-world noisy or ambiguous scenarios.

I sincerely appreciate the author’s detailed response. While many of my concerns have been addressed, several important issues still remain, as outlined below.

• Motivation and design in S1

I acknowledge that the proposed method is inspired by prior literature, where LLMs suppress the influence of images. However, I still find the connection between the stated motivation and the actual design to be unclear. For example, in Figure 1 of [1], the hallucination issue arises during the generation process, not at the initial stage. Their approach directly amplifies attention weights on the image during generation in a straightforward manner.

In contrast, the S1 step in VAP does not appear to account for intermediate generation stages. This raises the question: how does S1 achieve a comparable effect to amplifying visual attention as done in [1]? To clarify this, I suggest analyzing the activation weights of visual features before and after the insertion of adversarial noise. Such an analysis could provide stronger evidence that S1 leads to a similar alignment effect.

[1] https://arxiv.org/pdf/2407.21771

• Introduction of visual uncertainty

It remains unclear how the proposed perturbations in S2 and S3 contribute to reducing hallucination. I believe further analysis, such as plotting the relationship between semantic similarities of the responses and output, would help reveal whether the method effectively differentiates between perturbed and non-perturbed inputs.

Additionally, the rationale for combining S2 and S3 is not fully convincing. If S1 is effectively achieved, then S2 and S3 may exhibit similar behavior, raising questions about the necessity of including both. In conjunction with the concern above, I believe it would be valuable to present the dynamics of similarities defined in Eq. (3, 5, 6), along with corresponding experimental results. While the results appear promising, I believe it should go beyond reporting performance gains and offer a clear and detailed explanation of how those improvements are obtained.

I truly appreciate the author’s thoughtful engagement.

Dear Reviewer NLrm,

Thank you very much for your thoughtful engagement and kind acknowledgment. We truly appreciate your recognition that most of the concerns have been clarified, and we are sincerely grateful for your time and constructive feedback throughout the review process.

I appreciate the detailed clarifications provided by the authors. As most of my concerns have been resolved, I will take them into the reviewer's discussion period and my final justification.

Dear Reviewer NLrm,

Thank you again for your constructive feedback. Your insightful analysis of S1, S2, S3, and visual uncertainty has greatly improved the explanatory depth and interpretability of our work. In particular, your suggestions on similarity dynamics helped us better articulate the effects of each component and clarify their complementary roles.

As the rebuttal deadline approaches, please let us know if there are any remaining concerns. If our responses have addressed your points, could you please generously consider raising your score even a little bit as a recognition of our efforts so far? Thank you so much!

Sincerely,
Authors

Response to Reviewer c4Gd: Thank you so much for your constructive and thorough feedback! We address your concerns as follows:

Q1: Justification for the specific three-loss formulation.
A1: We clarify the rationale and empirical support for our loss design as follows:

(1) Functional decomposition
Each loss term addresses a different hallucination factor:

• $\bar{x}$: strongly corrupted (negative) image
• $c$: user prompt; $∅$: no prompt
• $S(\cdot,\cdot)$: semantic similarity (e.g., cosine)

(2) Empirical justification
As shown in Table 9, incremental ablations demonstrate that:

• Each component improves performance.
• The full loss consistently outperforms partial variants across all metrics and benchmarks.

(3) No assumption of global optimality
While empirically strong, we do not claim this loss formulation is globally optimal. Rather, it serves as a generalizable and practical baseline that may inspire further refinements in future work.

Q2: Concern about model-specific hyperparameter tuning $(\sigma_1,\sigma_2,\sigma_3,T)$
A2: We address the concern about model-specific tuning as follows:

These practical settings ensure VAP remains efficient, generalizable, and well-suited to black-box deployment.

Q3: Concern about marginal performance gains (~1–2%) on strong baselines.
A3: We respond to the concern on gain magnitude as follows:

(1) Meaningful gains under strong, saturated baselines:
Intern‑VL2‑MPO and similar models already exhibit high performance, leaving limited headroom. Gains of 1–2% F1 in such settings are non-trivial—especially without model access or training.

(2) Training-free and black-box:
VAP operates with no access to model weights, gradients, or decoding steps. Within this constrained setting, few existing methods remain applicable, making even modest gains valuable.

(3) Consistent cross-model improvement:
VAP consistently enhances 8 diverse LVMs across multiple benchmarks. This robustness underscores its practical utility, even if individual gains appear incremental.

We will highlight this context more explicitly in the final version.

Q4: Can the performance gap from proxy-based VAP be reduced?
A4: Yes, our results suggest that the proxy gap can be effectively mitigated:

(1) Complementary to other black-box methods:
VAP is orthogonal to approaches like VCD [1]. Combining the two preserves VAP's efficiency while improving accuracy.

(2) Empirical support:
As shown below, proxy-based VAP (Intern-VL2-1B) achieves nearly the same performance as self-generated VAP (Intern-VL2-8B) when used with VCD:

(3) Future directions:
In future work, we plan to explore improved proxy selection (e.g., architecture-aligned models) and lightweight distillation to further bridge the proxy–target gap without sacrificing efficiency.

[1] Mitigating Object Hallucinations in LLMs via Visual Contrastive Decoding, CVPR 2024.

Q5: Does adversarial noise used in VAP also improve robustness against harmful perturbations?
A5: We conducted robustness evaluations following [2] using standard adversarial attacks. Results are summarized below:

• Harmful adversarial perturbations degrade both performance and grounding, as expected.
• VAP does not rely on adversariality; its effect comes from semantically guided optimization.

These results confirm that VAP’s improvements stem from targeted input adaptation rather than robustness enhancement.

[2] On Evaluating Adversarial Robustness of Large Vision-Language Models, NeurIPS 2023.

Dear Reviewer c4Gd:

Thank you so much for helping us improve the paper so far! Please let us know asap if you have any further questions. We are actively available during this rebuttal!

Thank you again for your time and consideration.

Best regards,
Authors

Response to Reviewer e6Ap: Thank you so much for your constructive and insightful comments! We address your concerns point by point as follows:

Q1: Clarify novelty compared to VCD, VisualGPTScore, and PixMix.
A1: Below is a concise comparison highlighting the unique advantages of VAP over related methods:

• VAP (Ours):
  A training-free, fully black-box optimization method that requires no access to model internals, gradients, or decoding steps. VAP directly modifies the input image through targeted perturbation, enabling hallucination mitigation during inference without altering the model or its outputs.

• VCD [1]:
  Operates in a white-box setting by injecting noise and modifying the decoding distribution. In contrast, VAP applies perturbation externally and does not depend on decoder access or distributional rewrites.

• VisualGPTScore [2]:
  Estimates language priors using Gaussian noise and applies post-hoc reweighting of generated outputs. Unlike VAP, it requires generation scores and operates after response generation, whereas VAP intervenes before decoding, actively suppressing hallucinations.

• PixMix [3]:
  A training-time data augmentation method that mixes stylized noise patterns. VAP, by contrast, is applied at inference time, requires no fine-tuning, and performs per-instance optimization guided by vision–language alignment.

We will include these clarifications in the revised related work section to better contextualize the contribution of VAP.

[1] Mitigating Object Hallucinations in LLMs via Visual Contrastive Decoding, CVPR 2024.
[2] VisualGPTScore: Visio-linguistic Reasoning with Multimodal Generative Pre-training Scores, arXiv 2023.
[3] PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures, CVPR 2022.

Q2: Hyperparameter tuning appears model-dependent and costly.
A2: Below is a clarification of VAP’s hyperparameter efficiency and deployment practicality:

(1) Most hyperparameters are globally fixed:
Among the 8 hyperparameters in VAP, $( \alpha, \beta, N, \epsilon)$ are fixed across all models and datasets. These require no tuning, significantly reducing setup overhead.

(2) Remaining parameters are configured once per model:
The remaining four parameters—$(\sigma_1, \sigma_2, \sigma_3, T)$—are tuned once per model and reused across datasets and tasks. No re-tuning is needed when switching between instruction types or benchmarks.

(3) The parameter space is small and shared across models:
Each $\sigma$ value is selected from a small discrete set {0.1, 0.5, 1.0}. In practice, several models (e.g., LLaVA-v1.5, InstructBLIP, DeepSeek-VL2, Ovis1.6-Gemma2) use the same configuration, supporting VAP’s generalizability and low tuning burden.

(4) Comparison with other training-free methods:
While training-free, VCD [4] requires tuning temperature (e.g., temperature $T$) per model–dataset pair, making it less scalable. In contrast, VAP avoids such dataset-specific coupling and is better suited for black-box deployment.

[4] Mitigating Object Hallucinations in LLMs via Visual Contrastive Decoding, CVPR 2024.

Q3: Concerns about using unconditional generation in the loss function.
A3: Below is a clarification of our design choice and its empirical justification:

(1) Instruction-conditioned hallucinations require adaptive perturbation:
Hallucination severity varies with the instruction $c$, due to differing language priors [5]. VAP addresses this by generating perturbations specific to each $(x, c)$ pair. This variability is by design—it enables VAP to suppress instruction-specific hallucinations and generalize across diverse task formats.

(2) Optimization remains stable across prompts and datasets:
Despite prompt variability, all VAP hyperparameters are tuned only once per model and remain fixed across datasets and instruction types. This reflects the stability and robustness of the optimization process.

(3) Unprompted generation provides a reliable visual anchor:
When $c = \emptyset$, the model outputs a vision-only caption [6], serving as a grounded baseline unaffected by instruction priors. This is a key anchor for semantic consistency. Empirically, unprompted generations exhibit lower hallucination rates:

These results confirm that $\emptyset$ serves as a stable and hallucination-light anchor, justifying its use in the VAP loss.

[5] PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training, ICLR 2025.
[6] Multi-modal Hallucination Control by Visual Information Grounding, CVPR 2024.

Q4: Justification for contrasting prompted vs. unprompted responses in $\mathcal{L}_{s_2}$.
A4: We clarify the rationale and support it with empirical evidence:

(1) Motivation:

• Instruction-tuned LVMs often overfit to language priors $P(y|c)$ [7].
• This compromises visual grounding, especially in hallucination-prone prompts.
• We contrast $P(y|x+\delta, c)$ with the unprompted baseline $P(y|x)$ via $\emptyset$ to suppress language bias and restore grounding in visual content.

(2) Supervision stability:
Compared to using two prompted responses, the unprompted generation ($c = \emptyset$) provides a more stable and unbiased anchor. It reflects the vision-only interpretation of LVMs, free from instruction bias.

(3) Empirical justification:
Replacing $\emptyset$ with a second prompted response in $\mathcal{L}_{s_2}$ degrades both hallucination metrics and performance:

These results confirm that contrasting with $\emptyset$ improves both factual accuracy and hallucination robustness.

[7] Revisiting the Role of Language Priors in Vision-Language Models, ICML 2024.

Q5: Guiding principles for hyperparameter tuning.
A5: VAP is designed with tuning efficiency and cross-task generalizability. Our tuning methodology is summarized below:

(1) Dataset-agnostic configuration:
All hyperparameters are selected once per model and reused across all datasets and tasks. No task-specific tuning is needed.

(2) Globally fixed adversarial parameters:
Four parameters $(N, \alpha, \beta, \epsilon)$ are kept constant across all experiments. These follow standard interpretations (e.g., perturbation strength) and require no additional tuning.

(3) Lightweight model-specific tuning:
Only four parameters $(\sigma_1, \sigma_2, \sigma_3, T)$ are tuned per model using a compact search space:

• $\sigma_i \in \{0.1, 0.5, 1.0\}$
• $T \in \{100, 200, 500, 800\}$
  A validation set of just 300 image–instruction pairs is sufficient for reliable selection.

(4) Practical deployment:
This tuning process is efficient, reproducible, and requires minimal manual effort. We will add this strategy to the final version for better clarity and guidance in real-world use.

Response to Reviewer e6Ap:

Thank you very much for your recognition of our rebuttal. Your feedback has been instrumental in refining and clarifying key aspects of our work!

Below we address your follow-up question regarding our hyperparameter tuning strategy:

Q1: Clarification on hyperparameter tuning procedure.

A1: We clarify how VAP hyperparameters are selected, ensuring strong robustness, transferability, and tuning efficiency.

(1) Fixed Adversarial Parameters (with Physical Interpretation)

All fixed parameters have clear physical meanings and follow standard adversarial settings for LVMs as established in [1].

These settings are inspired by [1] and verified on 300 image–instruction pairs to ensure:

• Effective perturbation without breaking semantic fidelity
• Cross-task generalizability without re-tuning

These values generalize well and require no further tuning per dataset/model.

(2) Tuned Parameters (Coarse Grid Search Per Model)

We tune the following four parameters using coarse grid search on a small validation set (300 pairs). All values are selected based on technical roles and empirical intuition:

The tuning goal is to maximize hallucination suppression (POPE F1 score) on the validation set.

(3) Validation Set Construction (Based on POPE [2])

• 50 images randomly sampled from the MS-COCO dataset
• Each with 3 positive + 3 negative instructions (300 pairs total)

(4) Summary: Practical and Generalizable

• No per-dataset tuning required
• Only 300 samples needed for reliable tuning
• Fully reproducible setup across models
• Future work: better optimization, architecture-aware proxies, distillation

[1] On Evaluating Adversarial Robustness of Large Vision-Language Models, NeurIPS 2023.
[2] Evaluating Object Hallucination in Large Vision-Language Models, EMNLP 2023.

Response to Reviewer PVr3:

Thank you for your thoughtful and constructive feedback. Your recognition of the generality across 8 LVMs and the comprehensiveness of our evaluation helps reinforce the practical value of our approach. In response to your comments, we highlight several key strengths of VAP:

• Fully black-box and training-free, requiring no access to model weights, gradients, or decoding processes;
• Deployment-friendly, as it operates at inference time without any architectural modification;
• Consistently effective, with generalization across models and datasets, independent of prompt or task.

To clarify and address your concerns, we performed the following analyses:

• We now include direct comparisons with recent training-free baselines such as VCD and OPERA, highlighting both performance and design distinctions;
• We clarify the efficiency and latency reported in Table 4, including implementation details such as caching, batched inference, and the role of a proxy-based variant for fast deployment.

We address each of your points in detail below:

Q1: Lack of comparison to other training-free baselines.
A1: We clarify our comparison to recent training-free hallucination mitigation methods as follows:

(1) Conceptual distinction:
VAP is fully black-box, requiring no access to model weights, logits, or decoding logic, and applies to any off-the-shelf LVM. In contrast, VCD [1] and OPERA [2] are white-box methods that rely on modifying the decoder or accessing internal representations.

(2) Capability comparison:
Below is a concise comparison highlighting the unique advantages of VAP:

(3) Comparison with OPERA:
Table 11 shows that VAP consistently outperforms OPERA across 4 LVMs. Further combining both methods leads to stronger hallucination reduction.

(4) Comparison with VCD:

• Experiment Setup: We evaluate VAP and VCD on LLaVA-v1.5 and Intern-VL2.
• Observation: VAP achieves comparable or better hallucination mitigation, and combining VAP with VCD yields the best overall performance, highlighting their complementarity.

VAP offers a unique combination of strong performance, full black-box compatibility, and real-world deployability, which distinguishes it from existing approaches.

[1] VCD: Mitigating Object Hallucinations in LLMs via Visual Contrastive Decoding, CVPR 2024.
[2] OPERA: Alleviating Hallucination via Over-Trust Penalty and Retrospection-Allocation, CVPR 2024.

Q2: Efficiency and latency concern.
A2: We clarify the efficiency measurement and optimization strategies as follows:

(1) Clarification of latency in Table 4:
The reported A100 runtime refers to the end-to-end wall-clock latency for a single image–query pair with batch size 1. It includes image preprocessing, tokenization, inference, and decoding. This is a practical measurement, not simply the sum of multiple forward passes.

(2) Why VAP does not incur full 3× cost:
While VAP involves three optimization steps:

• The image–text pair is encoded once, and the resulting input_ids are reused.
• Perturbed images are processed in batched mode (e.g., image.repeat(...)), enabling fast parallel inference on GPU. As a result, the actual wall-clock overhead is less than 2×, as shown empirically.

(3) Efficient variant with proxy model:
We also introduce a lightweight proxy-model variant of VAP. As shown in Table 4 and Appendix D, it reduces the optimization overhead by up to 8× with minimal performance drop, making VAP suitable for real-time or resource-constrained deployment.

Dear Reviewer PVr3:

Thank you sincerely for your valuable comments and engagement throughout the review process. We hope our responses have addressed your concerns. If anything remains unclear, we’d be happy to clarify further while the discussion window is still open.

If our rebuttal has addressed your concerns, would you mind considering a slight increase in the score as a recognition of our efforts so far? Your thoughtful feedback has significantly enhanced the clarity and strength of our work!

Thank you again for your time and consideration.

Best regards,

The Authors.

Thank you so much for following up with us and for considering raising the score! Your constructive suggestions greatly improved the quality of our work.

If you have any further questions, let us know any time! We're truly fortunate to have had your support!