M-Attack-V2: Pushing the Frontier of Black-Box LVLM Attacks via Fine-Grained Detail Targeting

We sincerely thank the reviewer for the constructive and valuable feedback, which will definitely help us improve the quality of our paper. We will accommodate all suggestions into the revised manuscript. Below, we provide detailed responses to each of the reviewer's comments.

W1 & Q1: Out-of-Distribution Data: M-Attack-V2 depends on surrogate model consistency and semantic similarity of auxiliary samples. OOD data may disrupt these, causing unstable gradient estimation and reduced attack effectiveness. This paper lacks OOD validation. Q1: Have you evaluated M-Attack-V2 on OOD datasets to verify its performance, given the framework's reliance on surrogate model consistency and auxiliary sample similarity that may fail in OOD scenarios?

Following the suggestion, we have added experiments on finding auxiliary images in one OOD dataset, PatternNet, which is composed of remote sensing images with an overview look only. The results are presented in the table below, which confirms the reviewer's insights. Specifically, the ASR on Claude decreases by 21%. This ablation study highlights the importance of finding in-distribution data. However, our ATA framework is flexible and does not limit the method to find auxiliary data, so one can also refer to diffusion models with control methods to generate in-distribution images with the same semantics but different details.

Experiments on ATA OOD Data:

COCO denotes COCO train set, which is the original setting of the paper, Pattern denotes PatterNet, an OOD dataset which contains remote sensing images (only overhead view of the objects).

W2 & Q2: Higher Computational Cost: Generating auxiliary images for ATA and processing multiple crops in MCA increase computational overhead, making it less suitable for resource-constrained environments. Q2: The works lacks specific comparisons of computational complexity between M-Attack-V2 and other black-box LVLMs attack methods (e.g., AttackVLM, M-Attack).

We acknowledge your concern about computational cost. We have provided a detailed theoretical analysis along with a comparison of empirical run time per image with other baseline methods, please see our response to Q2 for Reviewer URye (the second reviewer). For quick reference, the overhead is flexible by configuring different values of $K$ and $P$. $K=2, P=2$ already provides sufficient improvement (+20%, +17%, +3% and +13% on Claude 3.7's $\text{KMR}_a$, $\text{KMR}_b$, $\text{KMR}_c$ and ASR) while only increasing slightly on per image time by 9.4% compared to $`M-Attack-V1`$, from 22.04 secons to 24.03 seconds per image.

Q3 & Q4: In Line 83, the word "Supursingly" is misspelled. In line 103, there is an extra letter "d" at the beginning of the sentence, which should be removed.

Thanks for pointing the typos out. We have corrected them in the revised manuscript.

The motivation of the article is quite reasonable, and it is hoped that the authors can further explain how to address potential issues in specific deployment scenarios.

We are encouraged by the kind comments. We have revised our paper carefully following the mentioned suggestions. In a real deployment scenario with even more constrained resources, we can further combine our $`M-Attack-V2`$ with other accleration techniques, such as Automatic Mixed Precision (AMP), FlashAttention, and static graph optimization, like torch.compile, to further reduce the overhead and accelerate the optimization process. In even more extreme cases, quantized models can be utilized as surrogate models to further reduce the overhead.

We sincerely appreciate the reviewer for the constructive and valuable feedback, which will definitely help us improve the quality of our paper. We will accommodate all suggestions into the revised manuscript. Below, we provide detailed responses to each of the reviewer's comments.

W1 & Q1: Limited Dataset Scope: Evaluation on only 100 images (NIPS 2017 + COCO) may not reflect performance on diverse, real-world domains (e.g., medical, satellite). Q1: Can we have a more comprehensive evaluation?

Thank you for your suggestion. In our original submission, we included experiments on 1000 images of NIPS 2017 competition, presented in Table 10 in the appendix. Here we have also incorporated images from additional domains into our work. Specifically, we utilized ChestMNIST, a chest X-ray dataset from the MedMNIST [1], and PatternNet [2], a collection of overview remote sensing images. We augmented the original NIPS2017 adversarial competition dataset with these datasets while keeping the target set unchanged, formulating a cross-domain image attack, since current commercial LLMs struggle to describe these data (i.e., for different chest X-rays, they primarily respond with 'it is a chest X-ray' without specific diseases). The results are presented in the following table. Our method consistently outperforms M-Attack-V1 and other baselines. The gap on Claude 3.7 is even larger than that of the original NIPS 2017 dataset, demonstrating the superiority of our M-Attack-V2.

PatternNet:

ChestMNIST:

W2 & Q2: MCA and ATA can introduce computational overhead. Q2: Can we have a theoretical or empirical analysis?

Your insights are on point. Considering the performance gain they bring, these overheads are reasonable. Please also see our response to Q2 for Reviewer URye (the second reviewer) for detailed theoretical and empirical analysis. For quick reference, due to GPUs' modern architecture, this parallelizable overhead is not as significant as it looks. For example, with K=2, P=2, per image time increase by 9.4% compared to M-Attack-V1, from 22.04 seconds to 24.13 seconds, whlie achieveing sufficient improvement (+20%,+17%,+3%, and +13% on Claude 3.7's $\text{KMR}\_a$, $\text{KMR}\_b$, $\text{KMR}\_c$ and ASR). Especially compared to SSA-CWA, which also utilizes a multi-sampling process, but takes $545.80\pm 4.21$ seconds and is outperformed by our method by a large margin, our improvement comes at a cost, but in an efficient way.

W3: Little analysis of how auxiliary selection quality or size impacts ATA’s effectiveness. If this data is not carefully prepared, how this noisy auxiliaries might degrade the attack's performance.

Following the suggestion, we have conducted an ablation study by corrupting the ATA's image by adding Gaussian noise under different scaling factors. The results are presented in the table below. These corrupted auxiliary images cause an adverse effect. Thus, the introduction of the ATA applies additional requirements on the quality of the retrieved images. However, the ATA framework remains flexible: when only low‑quality or heavily noisy images are available, diffusion models can be leveraged to synthesize clean new images that preserve the same semantics. Additionally, we conduct similar experiments to test the performance with out-of-distribution data. Please see our response to W1 & Q1 for Reviewer R2xh (the last reviewer) if interested.

Experiments on ATA Noisy Data:

Values inside brackets denote the scale of random noise.

W4 & Q3: Lack of qualitative comparisons or visualisation to clean image. I am curious if the optimised perturbations are still invisible compared to other attacks or even the clean image? I believe a user study can help confirm the imperceptibility. Q3: Can we have a user study?

Thank you for the constructive suggestion. We conducted two user studies to evaluate the imperceptibility of our method in comparison to the original images and to other baseline attack methods.

• In the first study, users were shown 20 images (10 perturbed by M-Attack-V2, 10 unmodified) and asked to label each as perturbed or not, simulating a real-world blind setting
• The second study showed 40 images (10 each from AnyAttack, SSA-CWA, M-Attack-V1, and M-Attack-V2), asking users to select the 20 they believed were corrupted, comparing the perceptual stealth of different methods (excluding VLM due to low performance)

All adversarial images were generated under $\epsilon = 16$. The results, averaged over five distinct user groups, are summarized below. In general, only 42% of the adversarial images of M-Attack-V2 are identified, leaving 58% of the images passing the human check. Among all the tested methods, AnyAttack is the most identifiable. The SSA-CWA follows the second. M-Attack-V1 and M-Attack-V2 share around 30%. The first user study shows the potential threat of M-Attack-V2 under a real-world scenario, even with human supervision.

W5 & Q4: Defence and Perceptual Analysis Missing. No experiments against simple input preprocessing defences (e.g., JPEG compression). Q4: Since the M-Attack-V2 aims to introduce perturbation to the clean image, can input preprocessing serve as a defence?

Following the suggestion, we added experiments under JPEG (quality=75) and DiffPure defenses. For DiffPure, we followed the original setup but tested with $t = 25$ and $t = 75$ to reflect mild and strong purification (original $t=150$). However, even at $t=75$, we observed structural loss and artifacts (e.g., blurry or broken edges).

Our results demonstrate that M-Attack-V2 remains robust under JPEG compression, while other methods, such as AttackVLM, SSA-CWA, and AnyAttack experience a substantial decline in performance, indicating that our method generates more resilient perturbations against low-level pixel transformations. When evaluated under DiffPure with $t=25$, all methods' performance drops, showing the capability of diffusion-based defense. However, M-Attack-V2 maintains an advantage, achieving significantly higher performance. Under DiffPure with $t=75$, where images become nearly regenerated, all methods exhibit low performance while M-Attack-V2 still consistently outperforms other baselines.

Experiments on JPEG Defense:

Experiments on Diffpure t = 25:

Experiments on Diffpure t = 75:

References:

[1] Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, Bingbing Ni. Yang, Jiancheng, et al. "MedMNIST v2-A large-scale lightweight benchmark for 2D and 3D biomedical image classification." Scientific Data, 2023.

[2] Zhou W, Newsam S, Li C, et al. PatternNet: A benchmark dataset for performance evaluation of remote sensing image retrieval[J]. ISPRS journal of photogrammetry and remote sensing, 2018, 145: 197-209.

We sincerely appreciate the reviewer for the constructive and valuable feedback, which will definitely help us improve the quality of our paper. We will accommodate all suggestions into the revised manuscript. Below, we provide detailed responses to each of the reviewer's comments.

W1: In this work, the analysis of the Auxiliary Target Alignment (ATA) remains at an intuitive level. The authors claim that ATA provides "low-variance, semantics-preserving updates" and "constructs a smooth semantic trajectory in the embedding space", but they offer no theoretical analysis to demonstrate how these properties are achieved.

Thanks for pointing out the significance of more theoretical analysis. We have provided a preliminary theoretical analysis of the Auxiliary Target Alignment (ATA) below. We will analyse further how it improves the optimization process and add the complete analysis to the revised manuscript.

Theoretical Analysis of the ATA

We begin with three mild assumptions.

Assumption 1 ($L$-continus): Let $f$ denote the surrogate model which is $L$-continus. Then, for any two inputs $x,y$,f(y) - f(x) \\le L \\lVert y-x \\rVert.\tag{1}

Assumption 2 (bounded auxiliary data): For auxiliary data $X\_{\\text{aux}}^{(p)}$, retrieved based on similarity to the target $X\_\\text{tar}$, we assume \\mathbb{E}[\\lVert f(X^{(p)}\_{\\text{aux}})-f(X\_{\\text{tar}}) \\rVert ]\\le\\delta. \tag{2} This assumption stems from the construction:X\_{\\text{aux}}^{(p)} =\\arg \\mathrm{top}\_{X \\in \\mathcal{D}}^{p} \\frac{f(X)^\\top f(X\_{\\text{tar}})}{\\lVert f(X)\\rVert{\\lVert f(X\_{\\text{tar}}) \\rVert}} \\tag{3},where $\\arg\\mathrm{top}^p\_{X \\in \\mathcal{D}}$ denotes $p$-th highest matching item in the database $\\mathcal{D}$ by cosine similarity. Since embedding $f(X)$ is normalized by convention, $\\delta$ exists.

Assumption 3 (bounded transformation): Let $\\mathcal{T}\\sim D\_\\alpha$ denote a random transformation drawn from a distribution with bounded pixel-level distortion. Then\\mathbb{E}\_{T\\sim D\_{\\alpha}}[\\lVert \\mathcal{T}(X)-X \\rVert]\\le\\alpha.\tag{4}

We define the embedding drift of a transformation $\\mathcal{T}$ appied to $X$ on model $f$ as $\\Delta\_{\\text{drift}}(\\mathcal{T};X) := \\mathbb{E}\_{\\mathcal{T}}[\\lVert f(\\mathcal{T}(X)) - f(X)\\rVert]$. We now state the main result.

Theorem 1: Let $\\mathcal{T}~\\sim D\_\\alpha$ be the transformation used in V1, $\\tilde{\\mathcal{T}} \\sim D\_{\\tilde{\\alpha}}$, with $\\tilde{\\alpha} \\ll \\alpha$ be transformation used in V2, we have\\begin{aligned}& \\Delta\_{\\text{drift}}(\\mathcal{T};X) \\le L\\alpha \\\\ & \\Delta\_{\\text{drift}}(\\tilde{\\mathcal{T}};X\_{\\text{aux}}^{(p)}) \\le L\\tilde{\\alpha} + \\delta\\end{aligned}\\tag{5}Theorm 1 shows that the drift of $\`M-Attack-V1`$ is bounded by $L\\alpha$, while drift of $\`M-Attack-V2`$ is bounded additional by $\\delta$. The term $L\\alpha$ reflects the inherent asymmetry of the target branch transformation. Transformations applied in the pixel space require an additional multiplier $L$ to account for their effect in the embedding space, thereby guiding the source image $X\_{\\text{sou}}$. In contrast, the auxiliary data operate directly in embedding space with $\\delta$. In practice, estimation $\\delta$ is significantly easier than $\\alpha$, since both $L$ and $\\alpha$ are generally unknown and hard to quantify. Moreover, the auxiliary data obtained via Eq.(3) inherently ensures better semantic alignment with the target. Thus, V2 strikes a balance by operating under a lower pixel distortion $\\tilde{\\alpha} \\ll \\alpha$, preserving the shift budget for more meaningful exploration via $\\delta$, finding a sweet point between exploration and exploitation.

Proof: For $\\Delta\_{\\text{shift}}$ of V1 \\begin{aligned}\\Delta\_{\\text{drift}}(\\mathcal{T};X) & = \\mathbb{E}\_{\\mathcal{T}\\sim D\_{\\alpha}}[\\lVert f(\\mathcal{T}(X)) - f(X) \\rVert] \\quad \\text{(By Assumption 1)} \\\\ & \\le L \\cdot \\mathbb{E}\_{\\mathcal{T}\\sim D\_{\\alpha}}[\\mathcal{T}(X)-f(X)] \\quad \\text{(By Assumption 3)}\\\\ & \\le L\\alpha, \\end{aligned}\\tag{6}For $\\Delta\_{\\text{shift}}$ of V2

We sincerely thank the reviewer for the constructive and valuable comments, which will definitely help us improve the quality of our paper. We will accommodate all suggestions into the revised manuscript. Below, we provide detailed responses to each of the reviewer's questions.

W1 & Q1: *** A lack of Background and Problem Formulation for M-Attack makes the submission difficult to understand *** The overall writing structure is somewhat unfriendly to readers who are not already familiar with M-Attack. I attempted to check the reference for M-Attack, but found that the citation is broken, making it difficult to trace the original work. Here are some mild suggestions: the authors should at least provide a concise technical summary of M-Attack (even in the appendix). More importantly, a rigorous research paper—even if it is a MARK II version—should include a dedicated subsection that clearly formulates the research problem and defines their threat model. While such context may become unnecessary if M-Attack is widely recognized as a standard baseline in this area, at present, the lack of background makes it hard for readers unfamiliar with M-Attack (like myself) to fully follow the paper’s motivation and methodology. Q1: Felt more like reading an appendix or extension of the M-Attack paper. As a result, I can only assess this submission with the lowest confidence score. I strongly encourage the authors to include a more thorough introduction to M-Attack in their revised version, so that the paper can be better understood by readers who are not already familiar with that prior work.

Thanks for your constructive feedback and suggestion, and we sincerely apologize for any inconvenience caused during your review. To address the concern, we have included a dedicated preliminaries section in the revised manuscript that provides a formal problem setup and clear definitions of all notations. For your convenience, we have also included the relevant content below. We hope this addition makes the paper more self-contained, allowing a broader audience to engage with our work without requiring too much prior knowledge.

Preliminaries

Black-box Transfer Attack under Image-Image Matching

Let $f\_\\xi: \\mathrm{R}^{3\\times h\\times w} \\times \\mathcal{T} \\to \\mathcal{T}$ denote the target blackbox model that maps the image text pair to target text, with $h,w$ for height and width. Let $X\_\\text{sou}, X\_{\\text{tar}} \\in \\mathrm{R}^{3\\times h\\times w}$ denote the source and target image. The source image is clean at initial time, which we denote as $\\tilde{X}\_\\text{sou}$ , and $X\_\\text{sou} = \\tilde{X}\_\\text{sou} + \\delta$, with $\\delta=0$ initially. The ideal objective of a blackbox-transfer attack is to learn the perturbation $\\delta$ such that the perturbated source image matches the target image in the semantic feature space, i.e., $f\_\\xi(\\tilde{X}\_\\text{sou}+\\delta) = f\_\\xi(X\_\\text{tar})$. However, directly optimizing such a target is difficult due to the unknown black-box model $f\_\\xi$. Therefore, Zhao et. al. [1] proposed an alternative optimization target:

where $f\_\\phi$ denote the surrogate model and \\mathrm{CS}(a,b)=a^\\top b/( \\lVert a\\rVert\_2\\lVert b\\rVert\_2) denotes cosine similarity.

Local-level Matching From $\`M-Attack-V1`$. $\`M-Attack-V1`$ further extends on Equ. (1) with the idea of 'local level matching via cropping'. Specifically, it defines local transformation $\\mathcal{T}\_s, \\mathcal{T}\_t$ for source and target images and corresponding local reagions in each iteration $\\{ \\hat{\\bf x}\_{1}^{s}, \\dots, \\hat{\\bf x}\_{n}^{s} \\} = \\mathcal{T}\_s(\\bf X\_{\\text{sou}})$; $\\{ \\hat{\\bf x}^{t}\_{1}, \\dots, \\hat{\\bf x}\_{n}^{t}\\}/\\{\\hat{\\bf x}^{t}\_g\\} = \\mathcal{T}\_{t}(\\bf X\_{\\text{tar}})$ which satisfy the two essential properties:

where subscipt $i,j$ denote the local regions in $i,j$-th iteration. Then, it rewrites objective $\\mathrm{CS}(f\_\\phi(X\_\\text{sou}), f\_\\phi(X\_\\text{tar}))$ in Equ. (1) under the local-level matching framework as

where $\\phi\_j$ is the surrogate model sampled from the surrogate ensemble $\\phi$. This modified framework only match a pair of local regions $(\\hat{\\bf x}\_s^i, \\hat{\\bf x}\_t^i)$ in each iteration $i$ instead of whole image $X\_\\text{sou}, X\_\\text{tar}$. Such local-level matching enhances the semantics of the perturbation, further aided by the ensemble models to combine semantic details from different models. Therefore, $\`M-Attack-V1`$ significantly outperforms other baseline methods. While effective, this approach suffers from inherent instability in gradient-based optimization. Specifically, since $\\nabla\_{\\bf X\_{\\text{sou}}} \\mathcal{M}\_{\\mathcal{T}\_s, \\mathcal{T}\_t}$ varies significantly across $\\mathcal{T}\_s$ (same to $\\mathcal{T}\_t$), the stochastic gradients become nearly orthogonal, i.e., $\\langle \\nabla\_{ X\_{\\text{sou}}^i} \\mathcal{M}\_{\\mathcal{T}\_s, \\mathcal{T}\_t}, \\nabla\_{X\_{\\text{sou}}^{i+1}} \\mathcal{M}\_{\\mathcal{T}\_s, \\mathcal{T}\_t} \\rangle \\approx 0$, where $X\_{\\text{sou}}^i$ is the source image in $i$-th iteration during optimziation. Such low similarity leads to high variance and poor convergence during the optimization process. The following section provides a detailed illustration.

[1] Zhao, T. Pang, C. Du, X. Yang, C. Li, N.-M. M. Cheung, and M. Lin. On evaluating adversarial robustness of large vision-language models. In International Conference on Advanced Neural Information Processing Systems, pages 54111–54138, 2023

We hope our newly added preliminaries can address your question. Once again, we apologize for the inconvenience during your review process.

In the following, we address each of the other mentioned mild suggestions one by one. We have also integrated these responses into the newly revised sections of our revision.

W2: For example, what are X_tar, X_sou and X_adv defined in the first paragraph of Method?

$X\_\\text{sou}$ is the source image we attach a perturbation $\\delta$ to. $X\_\\text{sou}$ is initialized as clean image, for which we use $\\tilde{X}\_\\text{sou}$, and $X\_\\text{sou}=\\tilde{X}\_\\text{sou} + \\delta$, with $\\delta=0$ in the initial stage. $X\_\\text{tar}$ denotes target image. The target of black-box adversarial attack is to find suitable $\\delta$ to make perturbated $X\_\\text{sou}$ (also denoted as $X\_\\text{adv}$ in some other works) share the same semantic as the $X\_\\text{tar}$ on the target black-box model, within limited perturbation, i.e., $\\lVert \\delta \\rVert\_p \\le \\epsilon$, where $\\lVert \\cdot \\rVert$ dentos $\\ell\_p$ norm.

W3: What are image pairs and region pairs mentioned in the paper?

The image pairs consist of the paired source and target images, i.e., $X\_\\text{sou}$ and $X\_\\text{tar}$. In each attack, a clean image $\\tilde{X}\_\\text{sou}$ is used to initialize the source image $X\_\\text{sou}$, and a target image $X\_\\text{tar}$ is selected with a distinct semantic that the source image aims to align with under the black-box model $f\_\\xi$. The region pairs are proposed in $\`M-Attack-V1`$ as part of the 'local-level matching via cropping', a key innovation in $\`M-Attack-V1`$. In iteration $i$, one local region (usually selected via cropping) $\\hat{\\bf x}^i\_s$ from $X\_\\text{sou}$ is matched with one region $\\hat{\\bf x}^i\_t$ from $X\_\\text{tar}$, to formulate a local-level matching framework.

W4: Why include IoU in the motivation experiment?

Because the local regions $\\hat{\\bf x}^1\_t, \\hat{\\bf x}^2\_s,\\dots, \\hat{\\bf x}^i\_s$ vary in crop size, two large crops are statistically more likely to share a large intersection. Gradient similarity, however, is accumulated only over the overlapping pixels (non‑overlapping pixels are masked out). To remove this size-dependent bias, we plot the raw dot-product against Intersection-over-Union (IoU) instead of intersection size, obtaining a similarity measure that reflects genuine gradient alignment.