HQA-VLAttack: Towards High Quality Adversarial Attack on Vision-Language Pre-Trained Models

Thanks very much for your insightful and positive comments.

1. Clarification about "negative pairs" in Figure 1.

By "negative pairs", we mean the non-matching image-text pairs. For all the mentioned methods SGA, DRA and HQA-VLAttack in Figure 1, the "negative pairs" are constructed by the perturbed images and non-matching perturbed texts drawn from other examples in the same batch. The similarity of "negative pairs" is averaged over all adversarial images, where the similarity of each adversarial image is averaged over five non-matching perturbed texts.

Figure 1 aims to show the difference between our method and previous works. Specifically, SGA and DRA mainly focus on reducing the similarity of positive pairs, but they unintentionally suppress the similarity of negative pairs as well, which can limit the attack performance. In contrast, our method explicitly encourages higher similarity between perturbed images and non-matching perturbed texts, thus increasing the chance of irrelevant retrievals and improving the attack performance.

We will add more explanations about "negative pairs" in the final version.

2. Clarification about the text attack strategy in Section 4.1.

The text attack strategy aims to generate high-quality textual adversarial examples, which should satisfy the semantic consistency principle while misleading the model. If the perturbed text loses the semantic consistency with the original text, it cannot serve as a valid adversarial example in vision-language tasks. Therefore, we first guarantee the semantic consistency before optimizing for the attack effectiveness.

To achieve this, we design the following two-step strategy.

(1) Step 1 (Section 4.1.1). In this step, we collect semantically similar words by retrieving counter-fitted synonyms for each word in the text. This ensures that the perturbed text has the semantic consistency with the original text. In contrast, previous methods like DRA and SGA use MLM-based substitutions, which probably introduce words that are semantically inconsistent. For ease of understanding, we list some generated examples that our method (using the counter-fitting based strategy) and previous methods (using the MLM-based strategy) are as follows.

From these examples, we can find that the strategy used in Section 4.1.1 is reasonable and effective.

(2) Step 2 (Section 4.1.2). In this step, we select the word that minimizes the image-text cosine similarity on the surrogate model from the semantically similar candidates. By choosing the word with the most similarity reduction, we can diminish the surrogate model's confidence and increase the attack success rate.

In summary, Step 1 aims to keep the adversarial text semantically consistent and plausible, and Step 2 aims to improve the attack performance. We will add more explanations in the final version.

3. Evaluation on recent SOTA MLLMs.

We have conducted experiments on two recent SOTA MLLMs: Qwen2.5-VL (Qwen2.5-VL-32B-Instruct) and LLaVA-Next (llava-v1.6-34b-hf) under the image captioning task and visual question answering task. Specifically, we use ALBEF as the surrogate model to generate adversarial examples, with a perturbation budget of 16/255, 80 PGD steps, and a step size of 0.5/255.

For the image captioning task, we randomly sample 128 image-text pairs from the Flickr30K dataset and generate adversarial images based on these clean pairs. Then we feed these adversarial images into Qwen2.5-VL and LLaVA-Next to generate captions. The generated captions are evaluated with the widely-used metrics BLEU-4, METEOR, ROUGE-L, CIDEr, and SPICE, where lower values indicate better attack performance.

For the visual question answering task, we randomly sample 128 examples from the VQAv2 dataset and apply perturbations to both the input images and their corresponding questions. The adversarial images and adversarial questions are then jointly fed into Qwen2.5-VL and LLaVA-Next to generate answers. The generated answers are evaluated using the widely-used metric accuracy, where a lower accuracy indicates better attack performance. The detailed results are as follows.

Note: The Original represents the performance on clean data.

• Image Captioning Results.

> Attack Qwen2.5-VL.

> Attack LLaVA-Next.

• Visual Question Answering Results.

These results demonstrate that HQA-VLAttack consistently outperforms strong baselines such as SGA and DRA when attacking the recent SOTA MLLMs. This further confirms the superiority of our method in the realistic black-box attack scenarios. We will add all these results in the final version.

Thanks very much for your insightful and positive comments.

1. About the text attack component and simple text augmentation techniques.

(1) We would like to clarify that the novelty can also come from a simple yet effective algorithm design that solves some specific problems overlooked by previous works. Previous MLM-based (Masked Language Model-based) methods (e.g., SGA, DRA) have the semantic drift issue, i.e., just using MLM-generated substitutions may diverge from the original word meaning, which may harm both the semantic consistency and the fluency of adversarial texts. Our method used in Section 4.1.1 can alleviate the issue effectively. To verify this, we conduct the preference study using GPT-4o. Specifically, we randomly sample 128 image-text pairs from the Flickr30K dataset and generate adversarial texts using SGA, DRA, and HQA-VLAttack. For each time, GPT‑4o is provided with the original text and the three generated adversarial texts, and asked to select the one with the best semantic consistency and fluency. The results are as follows.

The results show that our method can generate effective adversarial texts with higher similarity consistency. We will add all the explanations in the final version.

(2) For the simple text augmentation, it aims to introduce random variations to increase data diversity, which does not need to consider the adversarial requirement. For the text attack, it aims to generate satisfactory adversarial text that has high semantic similarity (human-imperceptible) with the original text and can fool models to produce wrong predictions simultaneously. Although their goals are different, some simple text augmentation techniques like random insertion/swap/deletion, back-translation, and synonym replacement can also be used in the initial procedure of text attack. We will clarify their differences in the final version.

2. About the input transformation.

(1) The input transformation aims to improve the generalization ability of the adversarial examples, and it has been used in previous baselines SGA and DRA, which ensures that the performance gains come from our method itself.

(2) The details of the scale transformation function are as follows. For a given image $I \in \mathbb{R}^{C \times H \times W}$, we perform the following steps:

• Scaling: Resize the image to $rH \times rW$ using predefined ratios $r \in$ {0.50, 0.75, 1.00, 1.25, 1.50}.
• Noise injection: Gaussian noise $\mathcal{N}(0, \sigma^2)$ can be added before resizing. In our experiments, we set $\sigma = 0.05$.
• Resizing: Each scaled image is resized back to the original resolution via bicubic interpolation.
• Contrastive integration: The resulting set $\text{Trans}(v'_i)$ is used in the contrastive learning loss to promote feature invariance across scales.

(3) We have added the ablation study for the input transformation and the contrastive learning. Specifically, we use ALBEF as the surrogate model on the Flickr30K dataset. Adversarial examples are generated with a perturbation budget of 2/255, using 10 PGD steps and a step size of 0.5/255. HQA-VLAttack (o, o) means the model without the input transformation and contrastive learning. HQA-VLAttack (o, w) means the model without the input transformation, but with contrastive learning. HQA-VLAttack (w, w) means the model with input transformation and contrastive learning, i.e., the original model. The results are as follows, where higher values indicate better performance.

The results show that both the input transformation and the contrastive learning can affect the performance to some extent. We will add the above explanations in the final version.

3. About the contrastive learning.

As far as we know, HQA-VLAttack is the first work to incorporate the contrastive learning into a black-box multimodal adversarial attack.

From the qualitative aspect, the contrastive loss can reduce the similarity between positive image-text pairs and increase the similarity between negative image-text pairs, i.e., it can guarantee that adversarial examples are pushed away from their true captions and pulled toward incorrect captions, thus improving the effectiveness of generated adversarial examples.

From the quantitative aspect, we have conducted an ablation study for the contrastive learning and the input transformation, which is shown in the above Table. The results show that contrastive learning is very important for our method, which can improve the attack performance greatly.

4. The baselines incorporate the input transformation and the contrastive learning.

We would like to clarify that previous strong baselines have already used the input transformation in their original papers. So we just need to incorporate our proposed contrastive learning loss into their methods to verify the effectiveness. Specifically, we use ALBEF as the surrogate model on the Flickr30K dataset. Adversarial examples are generated with a perturbation budget of 2/255, using 10 PGD steps and a step size of 0.5/255. The results are shown as follows, where higher values indicate better performance.

Based on the results, we can observe that contrastive learning can improve the baselines SGA and DRA greatly. In addition, HQA-VLAttack can consistently achieve the best performance, which reflects the effectiveness of the layer-wise importance score and the substitute word set in our method. We will add all the explanations in the final version.

5. About the computational cost.

(1) The runtime comparison of different methods is provided in Supplementary Material Section H (Table 2). From the results, it can be seen that HQA-VLAttack is a little slower than SGA, but much faster than DRA, which illustrates that the speed of HQA-VLAttack is acceptable.

(2) We attempt to provide the computation complexity analysis as follows. Assume that $D$ is the feature dimension, $L$ is the text length, $C_{\text{mlm}}$ is the cost of one forward pass through a masked language model, $C_I$ and $C_T$ are the forward pass costs for the image encoder and text encoder, $G_I$ is the backward pass cost through the image encoder, $L_p$ is the number of image encoder layers, $D_p$ is the number of tokens per layer, $k$ is the number of substitute candidates per word, $T_p$ and $T_n$ are the number of positive and negative captions per batch, $S = |\text{Trans}(v'_i)|$ is the number of image scales used during contrastive learning, and $N$ is the number of attack iterations.

Text attack.

• For each word, we retrieve substitutes using either a counter-fitting synonym dictionary (constant-time) or an MLM ($C_{\text{mlm}}$). With the hit ratio $\rho$, the complexity of this step is $O(LC_{\text{mlm}}(1-\rho))$. We then evaluate $kL$ candidate texts by computing their similarity with a fixed image feature, yielding a total cost of $O(kL C_T)$. Therefore, the overall time complexity of the text attack is $O(LC_{mlm}(1−ρ)+kLC_T)$.

Image attack.

• Section 4.2.1 consists of determining layer importance and generating initial adversarial images. The first step requires one forward pass and $L_p$ vector comparisons. The second step needs to be repeated over $N$ iterations, including a forward pass, a backward pass, and layer-wise feature comparisons. Ignoring minor terms, the total complexity is $O(N(G_I + C_I + L_p D_p D))$.

• Section 4.2.2 applies contrastive learning over multiple image scales. At each iteration, the adversarial image is compared with $T_p$ positive and $T_n$ negative texts across $S$ scales. This involves $S$ forward passes through the image encoder and $T_p + T_n$ forward passes through the text encoder, plus one backward pass. Ignoring minor terms, the total complexity is $O(N(SC_I + (T_p + T_n) C_T + G_I))$.

• Therefore, the total time complexity of the image attack is $O(N(G_I+C_I + L_p D_p D + SC_I + (T_p + T_n) C_T))$.

Based on the above analysis, the total complexity for each image-text pair is $O(LC_{\text{mlm}}(1 - \rho) + kLC_T + N(G_I+C_I + L_p D_p D + SC_I + (T_p + T_n) C_T) )$.

(3) We will add all the analysis in the final version.

6. About the minor suggestion.

The mentioned works offer useful ideas for improving the transferability of adversarial perturbations. We will add a more detailed discussion about them in the final version.

7. About the paper formatting concerns.

(1) For the appendix formatting, we follow the official Call for Papers of NeurIPS 2025. The original text is ''Technical appendices with additional results ....., or as a separate PDF in the ZIP file below before the supplementary material deadline''. So it meets the conference requirement.

(2) For the code link, we use an anonymous GitHub account created with an anonymous email without any other information, thus meeting the double-blind review policy.

Thanks very much for your valuable and helpful comments.

1. About the evaluation protocol and the negative samples.

We would like to clarify the misunderstanding about the practicality of the evaluation protocol.

(1) For negative pairs, we can construct them by randomly pairing each image with unrelated captions from the original dataset or any other external resources, which are easy to obtain in many real-world image-text retrieval systems. For example, in Google Image Search and TikTok platforms, the images and captions are publicly available and easy to access, which can be used to generate the negative pairs. So the evaluation protocol is practical in real-world scenarios.

(2) To further verify our method does not rely on a specific dataset distribution and the evaluation protocol is practical, we conduct the following experiments. Specifically, we pair each adversarial image from the Flickr30K dataset with randomly sampled captions from the MSCOCO dataset. These captions are irrelevant to the source images and come from a totally different dataset. We use ALBEF as the surrogate model and attack against TCL, $CLIP_{\text{ViT}}$, and $CLIP_\text{CNN}$. We consider both image‑to‑text retrieval (IR) and text‑to‑image retrieval (TR) tasks. We follow the original paper to use the attack success rate of R@1 as the evaluation metric for both IR and TR tasks, where a higher attack success rate indicates better attack quality. For reference, the original results of SGA and DRA under the same evaluation setting are also included in the table below.

From the results, we can get that HQA-VLAttack consistently outperforms other strong baselines even when negative captions are randomly sampled from a totally different dataset.

(3) We will add all these explanations in the final version.

2. About the layer importance and adversarial transferability.

• Explanation of the design logic in Section 4.2.1.

We would like to clarify the logic in Section 4.2.1, which consists of two parts: determining layer importance and generating initial image adversarial example. In the first part, we start by analyzing how different layers in the image encoder contribute to the final image representations. Specifically, we quantify each layer's influence by measuring the change in feature similarity when individual layers are skipped. This analysis allows us to define the importance for each layer based on the [CLS] token's cosine similarity to the final layer output. In the second part, we use these importance weights to guide the generation of initial adversarial images. We minimize the weighted sum of cosine similarities between the clean and perturbed image features across layers, where more important layers will receive higher weights, thus ensuring the transferability of the initial image adversarial example.

• The link between layer importance and adversarial transferability.

Intuitively, when the feature representations of the original example and adversarial example differ significantly, the adversarial transferability of generated perturbation is better. For the surrogate or target models, different layers usually contribute differently to feature representation learning. Therefore, just treating all layers equally seems unreasonable. To alleviate this issue, we propose to use importance scores to focus on some critical layers for feature representation learning, thus ensuring that the adversarial transferability of generated perturbation is satisfactory.

• Whether important layers differ significantly across models.

We have added the experiments to test whether important layers differ significantly across models. Specifically, we perform the analysis on two representative models TCL and $CLIP_{\text{ViT}}$. The importance scores of different layers are as follows.

From the results, we can get that important layers do not differ significantly across models. Specifically, both models have a similar trend that higher layers seem more important, and the importance scores will sharply increase at a certain layer (7th layer for TCL and 9th layer for $CLIP_{\text{ViT}}$).

• Analysis of importance metric.

We have added the experiments to justify the effectiveness of the important score. Specifically, we use $CLIP_{\text{ViT}}$ as the surrogate model, remove importance weighting (i.e., setting all weights to 1), and attack against ALBEF. We consider both image-to-text retrieval (IR) and text-to-image retrieval (TR) tasks. We follow the original paper to use the attack success rate of R@1 as the evaluation metric for both IR and TR tasks, where a higher attack success rate indicates better attack quality. The details results are as follows.

This confirms that the important score can improve the performance to some extent.

We will add all the experiments and explanations in the final version.

4. About the batch size.

(1) In Figure 4(b) of Section 5.2.3 (in the original paper), we have provided the impact of different batch sizes {2, 4, 6, 8, 10, 12, 14, 16} on attack success rate.

(2) We have added additional experiments to investigate the effect of batch size on attack performance. Specifically, we used ALBEF as the surrogate model to generate adversarial examples with a perturbation budget of 16/255, 80 PGD steps, and a step size of 0.5/255. We add the batch sizes to 18 and 22. The detailed results are as follows (the results with batch size is taken from the original paper).

From the results, we can see that when the batch size increases beyond 16, the improvement of attack performance is slight. However, larger batch sizes will increase GPU memory consumption, so we set the batch size to 16 in our experiments.

5. About the phrase (line 131).

By "degrade the effectiveness of generating adversarial text," we mean that semantically inconsistent or unnatural texts will violate the definition of a high-quality adversarial example. Based on the widely accepted definition [1], a high-quality adversarial example should be human-imperceptible, i.e., the perturbation should be semantically consistent and unconspicuous. In other words, while extremely inconsistent text may increase the chance of fooling the model, it is easy to be observed. Hence, we aim to maintain the semantic consistency to ensure that adversarial texts are effective and human-imperceptible.

[1] Intriguing properties of neural networks. ICLR 2014.

Sorry for the late reply, and thank you for your detailed response.

Regarding Evaluation Protocol

Apologies for the confusion — I didn’t mean data leakage in the standard machine learning sense (i.e., training on future data). What I meant is that HQA-VLAttack requires access to the database during the attack.

If I understand correctly, in an image-to-text retrieval setting where the attacker knows a positive pair (X, T):

• Query: [X]
• Database: [T, T2, T3, ..., Tn]

SGA assumes access to (X, T) and uses their augmentations, which can be derived from the known pair. In contrast, HQA-VLAttack seems to require knowledge not just of (X, T), but also the negative samples in the database (e.g., T4, T7, ...). This allows the attacker to optimize the adversarial example X' such that:

• cos-sim(X′, T) is minimized
• cos-sim(X′, T4) is maximized

— potentially causing X′ to match a specific negative sample T4 in the database, which should be inaccessible.

Regarding Flickr30K experiment using 75 negative samples from COCO

Sorry, I am comfused. I thought negatives were sampled within a batch size of 16 in the main paper due to computational constraints.

Thanks very much for your valuable and positive comments.

1. Discussion about Qwen2.5-VL and LLaVA-Next.

We have followed your comments to conduct additional experiments to evaluate the transferability of HQA‑VLAttack on Qwen2.5‑VL (Qwen2.5-VL-32B-Instruct) and LLaVA-Next (llava-v1.6-34b-hf) under the image captioning task and visual question answering task. Specifically, we use ALBEF as the surrogate model to generate adversarial examples, with a perturbation budget of 16/255, 80 PGD steps, and a step size of 0.5/255.

For the image captioning task, we randomly sample 128 image-text pairs from the Flickr30K dataset and generate adversarial images based on these clean pairs. Then we feed these adversarial images into Qwen2.5‑VL and LLaVA-Next to generate captions. The generated captions are evaluated with the widely-used metrics BLEU‑4, METEOR, ROUGE‑L, CIDEr, and SPICE, where lower values indicate better attack performance.

For the visual question answering task, we randomly sample 128 examples from the VQAv2 dataset and apply perturbations to both the input images and their corresponding questions. The adversarial images and adversarial questions are then jointly fed into Qwen2.5‑VL and LLaVA-Next to generate answers. The generated answers are evaluated using the widely-used metric accuracy, where a lower accuracy indicates better attack performance. The detailed results are as follows.

Note: The Original represents the performance on clean data.

• Image Captioning Results.

> Attack Qwen2.5-VL.

> Attack LLaVA-Next.

• Visual Question Answering Results.

These results show that HQA-VLAttack substantially degrades the performance on both Qwen2.5-VL and LLaVA-Next, which consistently outperforms SGA and DRA across all evaluation metrics. This further demonstrates the strong transferability and effectiveness of our method against modern MLLMs. We will add all these results in the final version.

2. About the iteration steps.

We have attempted to increase the iteration steps to 15, 20, and 25 for the strong baselines SGA and DRA on the Flickr30K dataset under the same transfer settings used in the original paper.

Specifically, we use ALBEF as the surrogate model to attack $CLIP_{\text{ViT}}$ on both image‑to‑text retrieval (IR) and text‑to‑image retrieval (TR) tasks, and vice versa (i.e., using $CLIP_{\text{ViT}}$ as the surrogate model to attack ALBEF). We follow the original paper to use the attack success rate of R@1 as the evaluation metric for both IR and TR tasks, where a higher attack success rate indicates better attack quality. The detailed results are as follows.

• SGA Performance.

• DRA Performance.

For the convenience of comparison, we also list the performance of HQA‑VLAttack with 10 iteration steps.

Based on the above results, we can observe that increasing the number of steps does not lead to consistent performance improvement for SGA and DRA. And in some cases, the performance even slightly drops, which is probably due to the overfitting to the surrogate model. Moreover, HQA‑VLAttack with only 10 steps outperforms SGA and DRA even when their iteration steps are increased to 25. This indicates that our chosen step setting does not disadvantage the baseline methods and highlights the superior efficiency and effectiveness of HQA‑VLAttack. We will add all these results in the final version.