To Weakness 1：

Our technical contribution lies in carefully crafting specific designs to improve search efficiency and make our framework applicable in more difficult scenarios. Each design and its technical contribution is summarized as follows.

1) Improving search efficiency from different perspectives:

a) Reducing the search difficulty: Designing a target semantic dictionary as prior knowledge shortens the search path from image to target text as much as possible, improving the search efficiency of Attack stage (line 137), 

b) Reducing the computation time: The Attack stage requires calculating the cosine similarity matrix of the entire population’s text in each iteration, which is time-consuming. The CLIP model’s text encoder can process large amounts of text in parallel. Therefore, we disable the visual encoder of CLIP and use the text encoder to compute the cosine similarity matrix between the target text and the output text of all chromosomes (Equation 13), speeding up the search.

c) Reducing the search space: The large number of decision variables (pixels) leads to a vast search space and low search efficiency. We combine the Grad-CAM formula with decision variables to design Equation 10, reducing the range of decision variables in unimportant areas (e.g., background), significantly reducing the search space.

2) Dealing with different difficult scenarios: 

a) Black-box：Grad-CAM requires gradient information and is not suitable for black-box scenarios, nor can it handle the image-to-text task where the number of output text categories is infinite. Based on previous research (cited [28], CVPR 2021) and our experiments (Figure 4), we designed a surrogate model strategy to generate attention maps in black-box scenarios. And we designed a CLIP-based mapping formula (Equation 9) to map the output text to the category set of the surrogate model, enabling the generation of attention maps for image-to-text tasks with unlimited output categories.

b) Perturbation imperceptibility: The perturbations of adversarial examples need to be imperceptible to the human eye. The Grad-CAM-based Equation 10 we designed can also enhance the stealthiness of adversarial examples. This is because a smaller search range for decision variables means smaller perturbations at each pixel.

To Weakness 2：

Thank you for pointing out these typos. The term $S_{seg}$ in line 167 is a typo and should be corrected to $S_{sem}$ . We have checked the rest of the paper and found no other typos. We will correct this in the final version. Your question about target words is addressed in Question 1 below.

To Question 1：

The mutation process in the Ask stage does not change the position of any words in the input space. Instead, it filters words related to the attacker’s specified semantics within a small hypersphere centered on the image (Equation 2). All target words are within a distance of radius $\eta$ from the image, ensuring that the target text (created based on the target semantic dictionary) is also positioned close to the image. This effectively reduces the search difficulty and improves the search efficiency in the Attack stage (Table 1 and Appendix B.3).

To Question 2：

DNN with different architecture and parameters exhibit similar attention patterns for the same image, as concluded in previous adversarial attack work (cited [28], Figure 4, CVPR2021) and validated by our qualitative and quantitative experiments on diverse CNN and Transformer models (Figure 4). The role of attention maps in our framework is to reduce the search space of unimportant regions (e.g., background), enhancing the stealthiness of adversarial images and reducing the search difficulty in the Attack stage. Even if different surrogate models produce slightly different attention maps, the cost is limited to additional computation in unimportant regions. Since adversarial attacks are not time-sensitive tasks, this cost is acceptable.

To Question 3：

Yes, this is a pattern we observed through statistical analysis of the heatmap value distribution. We have added the numerical distribution of all heatmaps as well as the median and mean values to the supplementary file in the rebuttal system.

To Question 1:

Is there any theoretical guarantee for question 1?

In fact, the reason we can ensure that the chromosome (perturbed image) after multiple rounds of mutation in the Ask stage remains in close proximity to the clean image, thereby facilitating the search for the corresponding target word, is grounded in the following mathematical rationale:

Let $x$ denote the clean image, $n$ the number of pixels in the image, $\eta$ the maximum perturbation size for each pixel, and $x_i$ represent the $i$-th pixel of the image. Define $\Vert x \Vert_{\infty}$ = $\max$ ($|x_{1}|$, $\cdots$, $|x_i|$, $\cdots$, $|x_n|$) to represent the infinity norm of $x$. Let $x_{r1}$, $x_{r2}$, and $x_{r3}$ be the images obtained by adding different random perturbations not exceeding $\eta$ to each pixel of $x$. Let $F$ be the mutation scaling factor, and $v$ be the chromosome (perturbed image) obtained after mutation. The hypersphere $B(x, \eta)$ is centered at $x$ with a radius $\eta$, and is mathematically characterized as: $B(x, \eta) = \\{z \mid \Vert x - z \Vert_{\infty} \leq \eta \\}$

The distance between the mutated chromosome $v$ and the clean image $x$ is given by:
$\Vert v - x \Vert_{\infty} = \Vert x_{r1} + F \cdot (x_{r2} - x_{r3}) - x \Vert_{\infty} = \Vert x_{r1} - x + F \cdot (x_{r2} - x) + F \cdot (x - x_{r3}) \Vert_{\infty}$

Employing the triangle inequality, we deduce: $\Vert v - x \Vert_{\infty} = \Vert x_{r1} - x + F \cdot (x_{r2} - x) + F \cdot (x - x_{r3}) \Vert_{\infty} \leq \Vert x_{r1} - x \Vert_{\infty} + F \cdot \Vert x_{r2} - x \Vert_{\infty} + F \cdot \Vert x - x_{r3} \Vert_{\infty}$

Given the definition of the hypersphere $B(x, \eta)$, for any point $x’$ within $B(x, \eta)$, it follows that: $\Vert x’ - x \Vert_{\infty} = \max_i | x’ - x| \leq \eta$

Consequently: $\Vert v - x \Vert_{\infty} \leq \eta + F \cdot \eta + F \cdot \eta = (1 + 2F) \cdot \eta$

From this, the scaling factor $\frac{1}{1 + 2F}$ is derived. Let $v’ = x + \frac{(v - x)}{1 + 2F}$, which yields: $|| v’ - x ||_{\infty} \leq \eta$

By assigning $v’$ to $v$, we guarantee that the mutated chromosome $v$ will always be within the hypersphere $B(x, \eta)$ regardless of the number of iterations.

Through these mathematical calculations, we can ensure that the new chromosome $v$ obtained after any number of mutation operations will be in close approximation to the input image (with a distance that does not exceed $\eta$).

To Question 2:

What is the additional computation cost if dealing with different heat maps?

The additional computational cost refers to the extra iterations required by the evolutionary algorithm to find a solution with the same fitness value. As shown in Figure 3(b) of the main paper, the experimental setup of $AAA$ (w/o Attend) used extremely poor attention heatmaps (failing to identify any unimportant regions), resulting in an additional 200 iterations to find a solution of the same quality as $AAA$. This indicates that the quality of the attention heatmap (its ability to identify unimportant regions) is related to the convergence speed of the evolutionary algorithm. The reason is that higher-quality attention heatmaps can effectively reduce the search space.

To Question 3:

Does "DNN with different architecture and parameters exhibit similar attention patterns for the same image" work for other modalities?

This conclusion is feasible under the condition of the same modality, but it is not feasible under the condition of cross-modality. Observations on datasets from different modalities such as RGB, thermal imaging, and infrared reveal that different DNNs trained in the same modality have similar attention maps. However, for a image in another modality (thermal image or infrared), these DNNs (trained on RGB) produce attention maps with differences. This may be because thermal and infrared images lack the color and texture features of RGB images, leading to varying degrees of inability to identify unimportant areas by DNNs. It is worth noting that our work mainly focuses on the RGB modality, addressing the challenges of black-box attacks and targeted attacks, without involving the cross-modality attack, as image-to-text models are mostly based on the RGB modality.

To Weakness 1：

To underscore the significance of research into image-to-text black-box targeted attacks, we will add an example of societal harm and highlight a societal benefit: 1) Harm Example: Social media companies use image-to-text AI for content moderation of user-uploaded images. Black-box targeted attacks on image-to-text AI could enable attackers to alter the semantic content of images from illegal to legal, thus bypassing the moderation process and uploading the illegal images to social media. 2) Benefit: The introduction of image-to-text black-box targeted attacks can alert AI service providers to vulnerabilities in their AI products, providing a benchmark to improve the security of their AI products.

In Section 3.3, the Ask stage’s population consists of randomly perturbed images, where each perturbed image is a chromosome, and each pixel is a gene. The goal of the Ask stage is to obtain prior knowledge (target semantic dictionary) to reduce the search difficulty of evolutionary algorithm (Attack stage). Target semantic dictionary contains words that are 1) close to the image in the input space, and 2) related to the attacker’s specified semantics. Equations 2, 3, and 4 show a random search within a small hypersphere centered on the input image $x$ with a radius $\eta$, ensuring all potential words are close to the input image. Equations 5, 6, and 7 store words related to the attacker’s specified semantics from the output text of each chromosome in the target semantic dictionary, helping the attacker generate a target text that meets their requirements (target semantics) and can be efficiently searched.

To Weakness 2：

In Appendix C.1, we discuss the limitations of our framework in real-world applications, specifically the need for numerous interactions with the target model. This is due to our use of a basic differential evolution algorithm, which requires many iterations to converge to the optimal solution. A potential improvement (Appendix C.2) is to combine our framework with the current state-of-the-art evolutionary algorithms, significantly reducing the number of interactions needed.

To Question 1：

In fact, our framework uses the $l_{inf}$ attack domain, not the $l_1$ domain. As shown in Equation 10, the value range for the i-th variable (pixel) of the clean image $x$ is [$-A(i) \cdot \eta$, $A(i) \cdot \eta$], where $A(i)$ is the contribution weight (0 to 1) and $\eta$ is the perturbation size parameter. This means each pixel has its own perturbation limit ($l_{inf}$ attack domain), adjusted by its contribution weight. For example, less important areas (like the background) have reduced perturbation limits. Since each pixel has a different perturbation limit, we represent the constraint in Equation 1 using the $l_1$ attack domain.

To Question 2：

The parameter values are determined through ablation studies. For example, Appendix B.2 determines the target semantic strategy, Appendix B.3 determines the word selection strategy, Appendix B.4 determines the population size, Appendix B.6 determines the evolutionary algorithm, Figure 4 in the main text determines the surrogate model, and Figure 5 in the main text determines the perturbation size, etc.

To Question 3：

In Section 4.2 (Qualitative Experiment of Different Perturbation Sizes), we have compared attacks with epsilon values of 25, 15, 10, and 5 (Figure 5). Our conclusions are:

1) As perturbation size decreases, both our method and existing methods become more stealthy, but their attack performance decreases.

2) As perturbation size increases, our black-box method achieves nearly 100% target attack performance, while existing gray-box methods have lower performance due to semantic loss issues.

Dear Reviewer 8Kf2,

As the deadline for the discussion period is approaching, we would like to kindly request your feedback on our responses. We wish to express our deepest gratitude for the time and efforts you have dedicated to reviewing our work. We sincerely hope that our detailed responses have adequately addressed all the concerns and suggestions you raised.

We fully understand that you may be occupied with other commitments, but we would greatly value any comments you can provide on our responses before the deadline.

Thank you for your attention to this matter. We eagerly look forward to hearing from you soon.

Sincerely, 11274 Authors

To Weakness 1：

Section 3.3 (Ask stage) aims to obtain prior knowledge for searching the target text. Prior knowledge shortens the search path from image to target text as much as possible, improving the search efficiency of Attack stage (line 137).

Specifically, we perform a random search within a small hypersphere centered on the image $x$ with a radius $\eta$, ensuring all potential words are close to the image (Equations 2, 3, and 4). Then, we compile a target semantic dictionary from the words related to the attacker’s desired semantics (Equations 5, 6, and 7). 

To Weakness 2：

I agree with your opinion that Ask stage makes the attack easier, which is exactly our goal and is reasonable：

1) Technically Reasonable: Prior knowledge can reduce the search difficulty of evolutionary algorithm. In addition, adversarial attacks often rely on prior information. For example, existing image-to-text gray-box attacks depend on the prior information of gradients, and black-box attacks for classification tasks rely on the prior information of similarity between surrogated and target models, while our prior information is the target semantic dictionary. It is worth noting that our prior information is more easily obtainable in reality, while their prior knowledge is difficult to acquire.

2) Fair Comparison: We applied the same prior knowledge (target semantic dictionary) to both our method and the existing methods, meaning that all approaches in the comparative experiments utilized the identical target text, thereby guaranteeing a fair comparison.

3) Reasonable use scenarios: Attackers typically need to meet their target semantic requirements without fixating on specific words. For example, if an attacker wants to map an illegal image to a legal text related to “dogs”, and the target semantic dictionary (prior knowledge) includes the words “puppy” and “jogging” but not “little dog” and “running”, the attacker can choose the sentence “The puppy is jogging” as the target text, avoiding “The little dog is running”. The reason is that although both sentences meet the attacker’s needs, the former is derived from the dictionary and is therefore easier to search for.

To Question 1：

Using target semantics as the target text for AAA (w/o Ask) is unfair. In our paper, AAA (w/o Ask) randomly creates a complete and coherent sentence from all English words as the target text, which is fair.

The reason why the former is unfair: The target text must be a sentence, not a word, because the output text of an image-to-text model is always a complete sentence. If the target semantics (a single word, such as "animal" or "photograph") is used as the target text to calculate the similarity with the output text (a complete sentence), it is difficult to achieve a high similarity score.

The reason why the latter is fair: The sole difference between AAA and AAA (w/o Ask) lies in the range of words for the target text. For AAA, the range is "words within the target semantics dictionary," whereas for AAA (w/o Ask), it is "all English words." Our experimental setup for AAA (w/o Ask) fairly demonstrates the impact of word selection range on the attack. Additional comparative experiments with various word selection ranges are conducted in Appendix B.3, which can provide a better understanding.

It is important to emphasize that when comparing our method AAA with existing gray-box methods, the target texts are identical sentences, ensuring the fairness of the experiment.

Dear Reviewer xQiz,

As the deadline for the discussion period is approaching, we would like to kindly request your feedback on our responses. We wish to express our deepest gratitude for the time and efforts you have dedicated to reviewing our work. We sincerely hope that our detailed responses have adequately addressed all the concerns and suggestions you raised. We fully understand that you may be occupied with other commitments, but we would greatly value any comments you can provide on our responses before the deadline.

Thank you for your attention to this matter. We eagerly look forward to hearing from you soon.

Sincerely, 11274 Authors