Defending LVLMs Against Vision Attacks Through Partial-Perception Supervision

We greatly appreciate your time and effort in reviewing our work and would like to address each of your concerns.

1. The incremental improvement

We appreciate the reviewer’s recognition of our idea novelty. In addition to the novel idea, we respectfully argue that the reviewer should reconsider the multifaceted and fundamental contributions in this work：

• We provide significant insights into LVLM defense. We identify a key vulnerability pattern in LVLMs: reduced confidence with attacked images versus high confidence with clean inputs (Sections 3.2-3.3).
• The training-free deployment and superior defense performance demonstrate DPS's practical distinctiveness.
• Our systematic bridging of "weak-to-strong" principles with LVLM defense mechanisms creates a new research nexus.

More contribution details are in Reviewer 5ot1 [2. Suitable for the ICML community]. These interlocking advances collectively represent a non-incremental leap in defense, which we will further clarify.

2. Theoretical Justification of DPS

We provide a theoretical formulation to explain why our work could achieve defense.

1. The Target of Attack

Let $x$ denote the original example, $x' = \text{adv}(x, \delta)$ denote the adversarial example, and $F_v(x) \in \mathbb{R}^d$ be its visual features extracted by the vision encoder. $T_\text{target}$ is the targeted output, $T_\text{clean}$ is the originial output of clean input, and $\mathcal{T}$ represents the text space. We can represent the probability of $T_{\text{clean}}$ and $T_{\text{target}}$ condition on the $x$ or $x'$:

$$P(T_{\text{clean}}|x') = \frac{P(T_{\text{clean}}|F_v(x'))}{\sum_{T\in\mathcal{T}} P(T|F_v(x'))}.$$

After attack, $P(T_{\text{target}} | x')$ becomes high and the $P(T_{\text{clean}} | x')$ is largely reduced.

2. Prior Injection via Partial-Perception Supervision

Our method DPS crops the original image and generates text $T_c$ when we feed the partial images into LVLM. We have $\phi(T, T_c) \in [0, 1]$ representing the semantic consistency between the candidate text $T\in \mathcal{T}$ and $T_c$. Compared with the attack formulation, our defense mechanism actually introduces crop-induced text $T_c$ as a prior to constrain the output distribution. The revised conditional probability of the defense can be approximated as:

$$P_{\text{defense}}(T_\text{clean}|x',T_c) = \frac{P(T_\text{clean}|F_v(x')) \cdot \phi(T_\text{clean}, T_c)}{\sum_{T\in\mathcal{T}} P(T|F_v(x') \cdot \phi(T, T_c)}.$$

As the $\phi(T_\text{clean}, T_c)$ is relatively high compared with $\phi(T_\text{target}, T_c)$, we can have $P_{\text{defense}}(T_\text{clean}|x',T_c)>P_{\text{defense}}(T_\text{target}|x',T_c)$, achieving defense.

3. Computational overhead

The computational overhead of DPS yields proportionally higher defense effectiveness, while existing methods failed to improve defensive capability even with the increased computation budgets. Please refer to our response to Reviewer popy regarding [1. Computational Cost].

4. Statistical significance

Thanks for your suggestion! Please refer to our response to Reviewer 2wMM regarding [1. Error Bar].

5. Comparison with more recent work

We sincerely appreciate the relevant literature you shared and will incorporate these papers into the revised manuscript. DPS fundamentally differs from the two detection methods as follows:

• DPS is a training-free defense that offers significantly lower deployment costs in wild.
• The scopes of attacks are different. These two methods are limited to specific scenarios (i.e., jailbreak detection) and cannot be directly applied to the defense against other attacks like misleading attacks.
• The underlying mechanisms are fundamentally distinct. DPS leverages the sensitivity of diverse attacks to cropping operations, achieving defense. Further, we conduct an empirical study to compare the performance between DPS and MirrorCheck on MM-Safety and VisualAttack datasets.

From Table 1, we observe that while MirrorCheck's defense performance is only slightly worse than our method. Its standard performance is significantly lower than that of LS-DPS due to the high false positive rate. Since VLMGuard's implementation is not publicly available, we will revisit this comparison with more time.

Table 1. Defense Performance Compared with MirrorCheck

6. An exploration of failure cases.

The most common failure case occurs when the cropping fails to eliminate attacks, rendering partial perception supervision ineffective for defense. In such cases, combining detection-based defense methods may enhance defense effectiveness, an area for future exploration.

We greatly appreciate your time and effort in reviewing our work, and would like to address each of your concerns.

1. The definition of confidence

However, I am skeptical about this confidence variation because the definition of confidence in this context is unclear. Specifically, I question whether this confidence is inferred based on the textual responses of LVLMs or directly derived from the model’s output logits. A more detailed explanation is needed to clarify this aspect.

Thank you for the comments. In our work, we define LVLM models' confidence as the inverse of the standard deviation of responses to variations of the same question, crafted by adding different prefixes to the original question. This confidence metric represents certainty in model outputs without requiring access to internal model parameters. A high confidence value indicates a low standard deviation and, thus, low uncertainty in the LVLM's responses. To avoid potential ambiguity in the term 'confidence,' we will add clear explanations of the term in the revised manuscript.

2. Suitable for the ICML community

However, such a straightforward approach may offer limited insights into the field. Therefore, I have some doubts about whether this work is suitable for ICML. I’m not entirely certain, but I feel that this paper might be a better fit for ACL.

We respectfully disagree with the assessment that our paper might be more suitable for ACL due to its "straightforward approach."

• First, Novel insights for LVLM Defense. Our paper provides significant insights into LVLM defense: (1). We identify a key vulnerability pattern in LVLMs: reduced confidence with attacked images versus high confidence with clean inputs (Sections 3.2-3.3). (2). We innovatively apply the "weak-to-strong" learning phenomenon to visual defense by using partial image perception to supervise full-image understanding. (3) Our approach fundamentally differs from existing methods that rely on direct majority voting or filtering.
• Second, Demonstrated Effectiveness. (1). Our method reduced attack success rates by 76.3% across six datasets on three models (Section 5.2). (2). We show effectiveness against both misleading and jailbreak attacks (Tables 1-2). (3) Unlike other methods (e.g., SmoothVLM), we maintained standard performance while improving defense (Section 5.3).
• Third, Simplicity as Strength. ICML values impact and effectiveness, not complexity for its own sake. Our training-free method requires no model modification, making it immediately applicable to any LVLM. The DPS framework shows strong compatibility with other defense strategies (Section 4.4).

Our paper presents novel insights, effective solutions, and provides new solutions in defending LVLMs, aligning well with ICML's emphasis on machine learning advances.

3. How is the adversarial noise for images generated?

It is well known that performing targeted white-box attacks on LVLMs is extremely challenging and typically results in very low attack success rates. If the adversarial examples are optimized using an open-source LVLM model, it is unlikely that they will effectively transfer to closed-source models.

In our experiments, in addition to the adversarial samples from existing datasets (RTA 100, MultiTrust, SafetyBench, and HADES), we generate adversarial samples using the VisualAttack method (lines 250-266). These samples achieve a transfer success rate of approximately 10% to 20%. The samples selected for defense evaluation are those that demonstrate successful transferability.

4. Experiments on open-source models

Although evaluating some closed-source models enhances the generalizability of the proposed method, for the sake of reproducibility, it is necessary to assess its effectiveness on an open-source model, such as LLaVA or Qwen2-VL.

Thanks for your suggestion! We conduct extended experiments using the open-source model Qwen2.5-VL-32B on misleading attack datasets—RTA 100, Self-Gen, and MultiTrust—as well as jailbreak datasets MM-Safety and VisionAttack. The results (ASR) of misleading defense and jailbreak defense are shown in Table 1 and Table 2, while the standard performance is shown in the Figure (https://anonymous.4open.science/r/ICML2025-231D/README.md). The proposed methods, DPS and LS-DPS, consistently achieve the best performance against misleading and jailbreak attacks.

We greatly appreciate your effort and address each of your concerns.

1. Computational cost

We report the computational costs of baselines and our method, and would like to highlight that the computational overhead of our DPS delivers proportionally higher defense effectiveness. Specifically, for a fair comparison, we enhanced the baselines by running each model 6 times and implementing majority voting to achieve ensemble effects comparable to our method's six queries. As shown in Table 1 (https://anonymous.4open.science/r/ICML2025-1-3F36/README.md), even with additional inference time, the enhanced baselines cannot achieve proportional improvements in defensive effectiveness comparable to DPS.

2. Why use the "weak-to-strong" analogy

In the ICML 2024 best paper [Khan; ICML 2024], they establish a framework where an LLM without certain necessary information serves as the 'weak model', while the same LLM with full information access functions as the 'strong model.' Inspired by this, we apply a similar analogy—treating models with partial-image access as 'weak' variants to enhance 'strong' full-image models. Our black-box, training-free method leverages this approach across six attack datasets (lines 43-53). While this could be viewed as information fusion, we believe the 'weak-to-strong' analogy particularly suits our method and will clarify this in our revision.

3. Black-box nature of our method

Our method operates in a fully black-box manner—the defense process works independently of questions. In Figure 2, questions aren't involved in our defense method. Regarding concerns about "animals" keywords causing information leakage: Jailbreak benchmark prompts contain no such keywords, yet our method performs well. Replacing "animals" with "things" still triggers attack-related content, and removing animal-related statements entirely doesn't affect output accuracy. We'll revise Figure 2 for clarity.

4. Comparing with adversarial purification.

Although DPS can also be viewed as a pre-processing approach, it fundamentally differs from adversarial purification:

• First, the scopes of attacks are different. Adversarial purification mainly focuses on removing pixel-wise adversarial noise (e.g., DiffPure [Nie; ICML 2022]). However, non-noise attacks like typographic attacks [Cheng; ECCV 2024] and jailbreak attacks [Shayegani; ICLR 2024] cannot be handled effectively by adversarial purification. In contrast, our DPS counteracts both adversarial noise and typographic attacks.
• Second, the underlying mechanisms are fundamentally distinct. Adversarial purification removes or breaks the effectiveness of adversarial noise. In contrast, our approach leverages the sensitivity of diverse attacks to cropping operations, naturally defending against non-noise attacks like the misleading attacks in our submission. As shown in Tables 2-3 (https://anonymous.4open.science/r/ICML2025-1-3F36/README.md), our DPS outperforms all purification baselines under both misleading and jailbreak attacks.

5. Why do we use cropping?

We use center cropping, random cropping, and adaptive cropping methods for these reasons:

• Efficiency: Cropping is highly concise with nearly no deployment cost.
• Effectiveness: Cropping effectively eliminates various attacks, including adversarial noises and typographic attacks.

While advanced models like SAM [Kirillov; ICCV 2023] could be used, our tests showed higher costs without significant improvements. We'll explore more effective cropping operations in future work.

6. Defense enhancement on Qwen-VL-Plus

The primary reason for the significant enhancement is that Qwen-VL-Plus's inherent safety mechanisms (which DPS relies solely on) are substantially weaker than those of the LLM-based safety checker (i.e., GPT-4o-Mini in our paper), which complements DPS in LS-DPS. In contrast, other LVLMs, whose safety mechanisms are closer to those of LLMs, exhibit less pronounced improvements. This aligns with previous findings that Qwen-VL-Plus's safety mechanisms are relatively weaker [Ying; arXiv 2024].

7. Details of adaptive cropping (AC).

We have introduced adaptive cropping is in Appendix B.3 of the manuscript. we use GPT-4o-Mini to locate text boxes and crop the remaining parts. It outperforms center/random cropping by more effectively eliminating textual content. We'll add this explanation to the Implementation Details and the Ablation Study section.

8. Why Warning Prompt work

We hypothesize that the warning prompt enhances safety awareness, making the model more cautious with deceptive queries common in the dataset, improving overall performance. See our case study in Figure 2 (https://anonymous.4open.science/r/ICML2025-1-3F36/README.md). This observation aligns with prior defense works like [Zheng; arXiv 2025], which reports that explicit safety prompts can enhance baseline model accuracy.

We greatly appreciate your time and effort in reviewing our work, and would like to address each of your concerns.

1. Error Bar

I would suggest adding error bars to the results where ever possible to account for the random nature of the evaluations.

Thanks for your suggestion! We conducted all our experiments three times to calculate the error bars and incorporated them into the results. Moreover, we extend our experiments to the open-source model Qwen2.5-VL-32B. Here we only show the error bars for Qwen2.5-VL-32B. (https://anonymous.4open.science/r/ICML2025-231D/README.md)

In Table 1, we demonstrate the statistical significance using a t-test, with p-values consistently less than 0.05, confirming their significance. Here, the t-test is performed between DPS and the baseline with the best performance.

Table 1. P-values from the t-test

2. Analysis of the number of partial crops

It would be additionally useful to analyse the effect of the number of partial crops used in the defense.

Thanks for your suggestion! Due to time limitations, we restrict our analysis to random crop-based partial copies (varying from 1 to 5) to evaluate their impact on defense performance, using the Self-Gen dataset. The experimental results in Table 2 demonstrate a positive correlation between the number of partial crops and defense performance.

Table 2. Attack Success Rate (ASR) Comparison Under Different Partial Crop Numbers