Towards Building Model/Prompt-Transferable Attackers against Large Vision-Language Models

Q1: Typo. In Equ. (2).

A1: Sorry for the confusion. We will correct the typos in Eq.(2) in the revision. The full Theorem and the proof can be found in Appendix B.

Q2: The subsequent approximation step (lines 181-188) is problematic.

A2: Thanks for your concern. First of all, we want to clarify that $x_v^{adv}, x_v, \Delta$ can NOT be viewed as three random and independent variables, as $\Delta$ is specifically optimized based on the input $x_v$ (the target is pre-defined and fixed). Therefore, the relation between $p(x_v^{adv})$ and $p(x_v, \Delta)$ should be dependently defined.

Secondly, we have provided our assumption as the prior of this specific case in lines182-183 "each adversarial image can be decoupled into a unique corresponding relationship". That is, we assume that $\Delta, x_v^{adv}$ are bijections of $x_v$, i.e. $\Delta, x_v^{adv}$ are dependently and uniquely determined by $x_v$ and the decompositon of $x_v^{adv}$ is also unique. This assumption is based on: each $x_v$ generates a unique perturbation $\Delta$ determined by the attack algorithm like PGD, where prompt-agnostic perturbation $\Delta$ is computed by visual-solely contexts $\Delta = \epsilon \cdot \text{sign}\nabla_{x_v} L(f(x_v), y_{tar})$. Therefore, $\Delta$ can be taken as a function of $x_v$, denoted as $\Delta = g(x_v)$ that is uniquely determined by $x_v$. This leads to a unique adversarial image $x_v^{adv} = x_v + \Delta = x_v + g(x_v) = h(x_v)$, where $x_v^{adv}$ can also be taken as a function of $x_v$. Conversely, although $x_v^{adv}$ may exist other decompositions in a purely mathematical sense, in the adversarial generation context, the uniqueness of this decomposition can be guaranteed by the above attack protocol.

Thirdly, before explaining why $p(x_v^{adv}) = p(x_v, \Delta)$ holds, as in Eq.(3) of the paper, we further claim that each $x_v,\Delta,x_v^{adv}$ is selected from a finite, countable set, making the selection process discrete. Therefore, we should utilize the probability mass function (PMF) instead of PDF in our specific case. Based on the above assumption, for discrete (Not continuous‌) variables $x_v,\Delta,x_v^{adv}$, the PMF of $x_v$ can denote as $p_{x_v}(x)=P(x_v=x)$. Since $\Delta$ is a bijection of $x_v$, there exists $y=g(x)$ such that $p_{x_v, \Delta}(x, y)=P(x_v=x, \Delta=y)=P(x_v=x)P(\Delta=y|x_v=x)=P(x_v=x)$. Similarly, since $x_v^{adv}$ is a bijection of $x_v$, there exists $z=h(x)$ such that the PMF of $x_v^{adv}$ can be derived as $p_{x_v^{adv}}(z)=P(x_v^{adv}=z)=P(h(x)=z)=P(x_v=x)$. Thus, we have $p_{x_v^{adv}}(z) = p_{x_v, \Delta}(x, y)$, and for any $y'\neq y, p_{x_v, \Delta}(x, y')=0$, meaning that $p(x_v^{adv}) = p(x_v, \Delta)$. Based on this, since $\Delta, x_v^{adv}$ are bijections of $x_v$, we have: $p(L'|x_v, \Delta)= p(L'|x_v, g(x_v))=p(L'|x_v)$ and $p(L'|x_v^{adv})=p(L'|h(x_v))=p(L'|x_v)$. Now, expanding $H(L'|x_v,\Delta)$:

We sincerely thank you for the helpful comment and the score improvement.

In the revised version, we will incorporate the detailed explanations from the rebuttal and present them as formal assumptions and lemmas to enhance the clarity and rigor of the theoretical analysis.

Q1: The paper motivates that adversarial transferability is related to the MI values between adversarial/benign patterns and LVLM’s outputs. Although Figure 2 verifies this assumption, the authors should provide more discussion on it.

A1: Thanks for your comment. Unlike previous works that implicitly optimize perturbations to destroy the image semantic representation, learning MI leads to an explicit optimization that emphasizes altering the comprehensive focus more on additive perturbation. The learned strong dependency between adversarial perturbation and attacker-chosen output allows the perturbation to learn and capture dominant features, thereby preserving adversarial harmfulness when transferring. This is the reason why we claim that a model/prompt-agnostic perturbation should always have more contributions/effects than the benign image to mislead the LVLMs during the reasoning process. Empirical findings on general transferable examples in Figure 2 of the paper also validate this motivation. More discussion can be found in Appendix E.

Q2: Target text selection: Using "I am sorry" as the target text may not truly represent a targeted attack. It would be valuable to include experiments with more specific target texts and report the success rates for such cases.

A2: Thanks for your suggestion. In fact, our attack is not limited to a fixed target text. We utilize the same "I am sorry" in the main experiments, as we want to keep a fair comparison for different ablation variants. We have provided ablation on different target texts in Table 5 of the paper and Table 10 of the appendix, which demonstrates that our attack is insensitive to diverse target texts. We further implement more target text experiments on LLaVA-1.5 (L), MiniGPT-4 (M), and BLIP-2 (B) in the table below, where our attack still achieves better transfer-attack performance under various target texts.  

Q3: The authors are suggested to provide a case transfer-attack experiment across both models/prompts at the same time.

A3: Thanks for your suggestion. We evaluate the joint transferability across diverse models and prompts in the table below, which also demonstrates the effectiveness of our attack. We will add the experiments in the revision.  

Q4: It seems that the proposed method achieves better transfer-attack performance but indicates reduced effectiveness in non-transfer attack scenarios. Although we know that it is due to the overfitting of previous attack methods, the authors could provide more discussion on this point.

A4: Thanks for your comment. Our MI objective has some conflicts with the original attack objective during the adversarial learning process. As shown in Eq.(10) of the paper, we have two optimization goals: one is to optimize the perturbed image to achieve the attacker-chosen adversarial condition, and the other is to apply the informative constraints on disentangled adversarial/benign MI values of the dependency between the perturbed image and LVLM's output. Therefore, both items need to optimize the same perturbation noise of the perturbed image. To balance the adversarial effectiveness and transferability, our attack just slightly sacrifices some effectiveness but achieves significant improvement in transferability, which is more practical and scalable than the other attack methods. We will provide more discussion on this point in the revision.

Q1: There are various existing LVLM attacks are re-implemented for comparison. Do they re-implemented in the same setting for fair comparisons? More details are required.

A1: Thanks for your concern. In fact, all LVLM attack baselines are re-implemented in the same setting for experimental comparison, including: (1) Consistent Model Selection: All methods are implemented and evaluated on the same LVLM models, such as LLaVA-1.5, MiniGPT-4, BLIP-2, and InstructBLIP. (2) Consistent Tasks and Datasets: All methods are evaluated on the same three types of tasks (image captioning, image classification, and VQA), the same datasets (DALL-E, SVIT, and VQAv2), and the same target texts. That is, all methods optimize adversarial examples based on the same image-prompt pairs and target texts. (3) Consistent Attack Constraints: All methods utilize the same perturbation bound $\epsilon=16/255$, the same epoch number, and the same step size $\alpha=16/epoch$. (4) Consistent Evaluation Metrics: All methods are evaluated using the same three metrics—semantic similarity (SS), Exact Match (EM), and Conditional Contain (CC). And the results of each metric are averaged across the three tasks for comparison. Therefore, our baseline comparison is fair. We will provide more details in the revision.

Q2: How to train the two MI estimation networks? Will they introduce additional noticeable costs? The authors should provide more corresponding analysis.

A2: The two MI estimation networks are initialized with the same architecture and trained separately. Specifically, $C_{\psi}$ is implemented as a light two-layer convolutional neural network, while $T_{\omega}$ simply incorporates an attention mechanism, $1\times 1$ convolutional blocks, and residual connections. For training, we first utilize the selected adversarial examples generated by the PGD attack. We then feed the same prompt with benign image ${x}_v$ and adversarial image ${x}^{adv}_v$ into the LVLM to obtain the corresponding logits outputs ${L}$ and ${L}^{'}$. The tuple $({x}_v, {L}, {L}^{'})$ is used as input to train the benign MI estimation network based on Equation (7) in the paper, while the tuple $({x}^{adv}_v - {x}_v, {L}, {L}^{'})$ is used to train the adversarial MI estimation network following Equation (6). We train both networks using the Adam optimizer for 100 epochs, with an initial learning rate of 0.01 that decays by a factor of 0.5 every 10 epochs. These MI estimation networks are very lightweight and just cost about 26mins. Therefore, it will not introduce much time cost to the average time speed of single adversarial sample generation. We will provide corresponding analysis in the revision.

Q3: A commonly implemented image-level universal setting should also be experimented.

A3: Thanks for your comment. Our work can be easily extended to an image-level universal setting by optimizing the perturbation across images, as shown in the table below, where our attack still achieves competitive performance.  

Q4: Two widely used baselines should be implemented for comparisons: “On evaluating adversarial robustness of large vision-language models” and “How Robust is Google's Bard to Adversarial Image Attacks?”

A4: Thanks for your suggestions. As for more baseline comparisons, we re-implement your mentioned two papers, “On evaluating adversarial robustness of large vision-language models” (Zhao et al.) and “How Robust is Google's Bard to Adversarial Image Attacks?” (Dong et al.) into targeted attack transferred from MiniGPT-4 to GPT-4o and Claude-3.5 in the table below, where our attack still achieves better transferability.  

Q5: The authors did a good job by providing sufficient discussion and experiments in the appendix. The reviewer suggests including the justification of fair transfer attacks in the main paper.

A5: Thanks for your suggestion. We will move the transfer-attack justification (F.4 Justification of Our Transfer Attack) from the appendix to the main paper in the revision.

Q1: The analysis of robustness against defenses is incomplete. The experiments only demonstrate resilience in a white-box setting and lack an evaluation of how the proposed cross-model and cross-prompt attacks perform when transferred to defended target models.

A1: Thanks for your comment. We provide a more detailed analysis of robustness against defenses in the table below, where we show the cross-model/prompt attacks' performance under various defenses. In particular, we also introduce two new defense mechanisms for defended target evaluation: one improves the CLIP component in BLIP-2, LLaVA-1.5, MiniGPT-4 and InstructBLIP models with a defended FARE model (2024 ICML) of paper "Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models", and the other uses a defended DPS model (2025 ICML) of paper "Defending LVLMs Against Vision Attacks through Partial-Perception Supervision" to embed the input of the BLIP-2, LLaVA-1.5, MiniGPT-4 and InstructBLIP models. The experimental results demonstrate that our cross-model/prompt attack still achieves better transferability compared to baselines under various defense methods, indicating our robustness. We will add these defense experiments in the revision.

Q2: The study lacks transferability experiments against other architecturally distinct model families, such as InternVL, which would provide a more complete assessment of the method's generalizability.

A2: Thanks for your suggestion. We have provided the transfer-attack experiments for architecturally distinct models GPT-4o, Claude-3.5 in Table 3 of the paper, and InternVL models in Table 6 of the appendix. To provide more detailed transfer-attack experiments on architecturally distinct model families, we comprehensively evaluate the attacks' transferability among MiniGPT-4 (EVA-CLIP-ViT-g-14), Qwen2-VL (CLIP-ViT-bigG), InternVL (InternViT-300M-448px-V2_5), and Gemma-3 (SigLIP-ViT) in the table below. It shows that our attack is more generalizable to architecturally distinct LVLMs compared to other attacks, demonstrating our great transferability. We will provide corresponding experiments in the revision.

Thank you for your valuable response. We guarantee that we will incorporate the rebuttal in detail into the revised version.