Fit the Distribution: Cross-Image/Prompt Adversarial Attacks on Multimodal Large Language Models

Q1: The compared MLLMs are relatively old. The authors should test the effectiveness of the proposed attack method against more latest models such as DeepSeekVL or DeepSeekVL-2. Besides, the number of the tested MLLMs is too small, the authors should increase the number of the tested MLLMs.

A1: Thanks for your comment. We first want to claim that we follow the existing attacks PGD (CVPR2024), CroPA (ICLR2024), UniAtt (Neurips2024) to utilize the same MLLMs for fair comparison. Then, following your suggestion, we implement DeepSeekVL for comparison. We also implement the latest models Qwen2-VL and InternVL to provide a more comprehensive evaluation. All experiments are conducted on No Cross (N), Cross Images (I), Cross Prompts (P), and Cross Images/Prompts (IP) settings. As shown in the table below, our attack still achieves better cross-image/prompt performance than other attacks against these latest MLLM models, demonstrating our effectiveness.

Q2: The paper mentions that it uses all-MiniLM-L6-v2 model to compute the cosine similarity between embeddings of adversarial output and target text. However, it does not explain the reason for using all-MiniLM-L6-v2. Does the cosine similarity computed by other models is consistent with all-MiniLM-L6-v2?

A2: Thanks for your concern. We utilize all-MiniLM-L6-v2 as it is a well-known effective SentenceTransformer model for text embedding. In fact, our attack performance is not sensitive to this embedding encoder. Any latest text encoder can be utilized for the similarity calculation here. Specifically, we implement different text encoders for comparison, including clip-ViT-B-32, all-mpnet-base-v2. As shown in the table below, the results show that our attack can achieve similar performance with different embedding encoders.

Q3: The datasets used in experiments are limited. More different types of datasets are needed to further make the experiment results more convincing. Is the proposed attack method still effective on other different datasets and other vision-language tasks?

A3: Thanks for your comment. We conduct more experiments on diverse datasets Flickr30k and NoCaps, where Flickr30k is used for visual commonsense reasoning tasks and NoCaps is used for intent understanding tasks. As shown in the table below, our attack still achieves better performance than baseline attacks on different types of datasets, demonstrating our effectiveness and scalability. We will add these experiments in the revision.

Q4: The paper only compare Gaussian distribution and fixed distribution to prove the superior performance of Gaussian distribution. More different types of distributions should be considered and discussed.

A4: Thanks for your comment. We provide more different types of distributions for comparison, including Uniform distribution and Gamma distribution, as shown in the table below. Since the Gaussian distribution is more suitable for large-size sample of great diversity, our modeled distributions through Laplacian approximation achieve the best performance.

Q5: I suggest authors to add discussions on limitations in the paper instead of appendix.

A5: Thanks for your suggestion. We will move the limitation discussion section from the appendix to the main paper in the revision.

Q1: Comparisons with more related adversarial attacks could make it more thorough to validate the superiority.

A1: Thanks for your suggestion. We re-implement more related adversarial attacks on No Cross (N) and Cross Images/Prompts (IP) settings to provide a comprehensive experimental comparison, including baselines MFAttack ("On evaluating adversarial robustness of large vision-language models"), Anydoor ("Test-Time Backdoor Attacks on Multimodal Large Language Models"), and XTransfer ("X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP"). As shown in the table below, our attack still achieves better cross-image/prompt attack performance than these latest attacks, demonstrating our effectiveness. We will add these comparisons in the revision.

Q2: The robustness evaluation only contains two defenses. Analyses on more defenses (e.g. defenses against traditional adversarial attacks) are expected.

A2: Thanks for your comment. We implement more defenses, including traditional defense strategies adversarial training, multimodal attention masking, and the latest MLLM defense FARE (ICML2024) of paper "Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models", DPS (ICML2025) of paper "Defending LVLMs Against Vision Attacks through Partial-Perception Supervision", to provide more convincing robustness evaluation. We follow the same setting in Figure 5 of our paper for comparison. As shown in the table below, experimental results demonstrate that our proposed cross-image/prompt attack still achieves better transferability under various defenses, demonstrating our effectiveness and robustness. We will provide these defense experiments in the revision.

Q3: Minor Weakness: (1)The placement of Figure 1 and Tables 2, 3, and 5 is inappropriate, affecting the visual flow of the text. (2)Table 4 should be positioned after Tables 2 and 3. (3)The pipeline in Figure 2 is relatively cluttered. It is recommended to redraw it to clarify the connections between different steps.

A3: Thanks for your comments. (1) We will adjust the positions of Figure 1 and Tables 2, 3, and 5 to enhance the visual flow of the text in the revision. (2) We will move the position of Table 4 after Tables 2 and 3 in the revision. (3) We will redraw and reorganize Figure 2 to display each step more clearly in the revision. We guarantee that all these modifications will be completed in the revision.

Q1: The entire method and experiments assume full access to model internals, including gradients and architecture. The paper does not explore black-box, gray-box, or query-limited scenarios, which are far more relevant in real-world adversarial contexts where model parameters are typically inaccessible.

A1: Thanks for your concern. We first want to claim that our work proposes to address an important issue of white-box cross-sample transferability, revealing the critical universality of attacks when facing redundant data scenarios in reality. Second, following your suggestion, we can extend our work to the practical scenario without using model gradients and architecture. Since gray-box setting still requires the knowledge of the MLLMs' visual encoder, we implement our attack in the most challenging real-world query-based black-box setting without using any prior knowledge of model parameters.

Specifically, since our distribution modelling and optimization components are independent to the adversarial learning, we can easily replace our white-box gradient backpropagation in Eq.(12) with the general black-box gradient estimation mechanism to estimate the gradient via iterative querying for optimization. As shown in the table below, we evaluate our attack performances on No Cross (N) and Cross Images/Prompts (IP) in the black-box setting with different query numbers. We can find that: (1) Our attack achieves good performance with sufficient query numbers in the black-box setting. (2) Our attack can also achieve competitive performance with limited queries. The above findings indicate that our attack can also handle the real-world black-box cases well, demonstrating the effectiveness and practicality of our proposed method.

Q2: Experiments only cover MS-COCO and DALLE-3; evaluation on more varied datasets (e.g., Flickr30k) would strengthen generalization claims.

A2: Thanks for your suggestion. First, we have provided experiments on more dataset SVIT in Appendix B.2. Second, we implement our attacks with more diverse datasets Flickr30k and NoCaps, on No Cross (N), Cross Images (I), Cross Prompts (P) and Cross Images/Prompts (IP) settings. In particular, we conduct visual commonsense reasoning tasks on Flickr30k and intent understanding tasks on NoCaps. As shown in the table below, our attack is still effective on diverse datasets and achieves better cross-image/prompt attack performance compared to other attacks.

Q3: The method assumes that the modeled input distributions align well with unseen real-world inputs. In practice, distribution shift may degrade attack transferability.

A3: Thanks for your concern. In fact, since we model the image distribution with sufficient natural images and model the text distribution with comprehensive task-relevant/-irrelevant prompts, our approximated Gaussian distribution is generalizable enough to handle the general image-prompt inputs in real-world cases. As proved in Table 8 and Table 9 of the appendix, our method can still achieve effective transfer-attack performance when transferring our generated perturbations across distribution-different datasets. The results shown in the table below also demonstrate that our attack is robust to distribution shift and is scalable to be further improved with more diverse data usage.

Q4: The study focuses on simple input-level defenses; it does not explore more advanced defenses (e.g., adversarial training, multimodal attention masking).

A4: Thanks for your comment. We implement more advanced defenses to provide convincing robustness evaluation, including adversarial training, multimodal attention masking, and MLLM defense FARE (ICML2024) of paper "Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models", DPS (ICML2025) of paper "Defending LVLMs Against Vision Attacks through Partial-Perception Supervision". We follow the same setting in Figure 5 of our paper for comparison. As shown in the table below, the experimental results demonstrate that our proposed attack still achieves better transferability across images and prompts under various defenses, demonstrating our effectiveness and robustness. We will provide these defense experiments in the revision.

Q1: Lack of results on state-of-the-art MLLMs, like Qwen2-VL, InternVL, as well as closed-source models.

A1: Thanks for your suggestion. We provide experimental results on more state-of-the-art MLLMs, Qwen2-VL, InternVL, DeepSeekVL, on No Cross (N), Cross Images (I), Cross Prompts (P), and Cross Images/Prompts (IP) settings, as shown in the table below. It demonstrates that our attack can also achieve better performance against these MLLMs compared to other attacks.

Since existing methods fail to attack closed-sourced MLLMs while these MLLMs are somewhat expensive, we try to re-implement our attack by replacing our white-box gradient backpropagation in Eq.(12) with the general black-box gradient estimation mechanism to attack closed-sourced MLLMs GPT-4o and Gemini-2.0 in the table below, where our attack is still effective.

We will add these experiments in the revision.

Q2: This method introduces a new computation to fit both the image and text distribution, as well as a sampling computation.

A2: Thanks for your concerns. Although our method introduces new distribution modeling and sampling computations, our attack is still competitively efficient compared to existing works as shown in the table below. (1) These two components are lightweight and their costs can be ignored compared with the overall costs, as the main cost of our attack is the latter gradient optimization process. (2) Our proposed attack is able to generate the universal adversarial example across images/prompts for the whole dataset in a single process, while existing attacks need to generate individual perturbation for each sample of the dataset.

Q3: I notice the optimization step number is set to 300, while CroPA uses 2000. Will a larger step number lead to better performance?

A3: As shown in the table below, a larger step number will introduce slightly better performance of our proposed attack. However, to take the balance between the complexity and performance, we set the step number to 300 as it is enough to achieve good performance.

Q4: The author can also show the cross-model transferability of the proposed method, and compare the result with methods like [1].

A4: Thanks for your comment. We have provided our cross-model transfer-attack experiments in Table 10 of the Appendix. To provide a detailed comparison, we re-implement your mentioned paper [1] (code is not released) in the table below. Results show that our attack achieves competitive cross-model attack performance, even our method is not specifically designed for cross-model attack.

Q5: Can you provide more details on the efficiency analysis? It generates all adversarial examples. But the proposed method generates the cross-image perturbation for all data but CroPA generates adversarial examples for each data sample. Do I understand correctly? And what’s the size of the dataset used?

A5: Your understanding is correct. In Table 3 of the paper, we evaluate the costs of generating 200 adversarial examples from the dataset. For previous work CroPA, it generates adversarial examples for each sample individually, requiring redundant costs of 200 samples. Instead, our method only generates a universal adversarial perturbation in one single process, and this perturbation can be directly added to all samples for attacks without additional costs. Therefore, under the same realistic whole-dataset attack setting, our method achieves better efficiency. We will provide the detailed analysis in the revision.

Q6: This approach approximates the distribution of a given prompt/image set, which might be ineffective under a prompt or an image from a different distribution. So it is crucial to sample a good prompt/image set.

A6: This is a good question. Indeed, our attack performance is correlated to the sampling image/prompt set. Intuitively, a larger number of diverse samples can guarantee the generalization-ability of the modeled distribution, providing more reliable guidance similar to the real-world cases. Although we do not sample a large amount of data for distribution modeling due to resource costs, our work is proven to be robust enough to potential domain shift by the cross-dataset attack performance in Table 8 and Table 9 of the appendix. This is because we model the image distribution with sufficient natural images and model the text distribution with comprehensive task-relevant/-irrelevant prompts, leading to learn general distributions. The table below further indicates the scalability of our proposed attack where more diverse data helps the distribution modelling for improving the attack performance.