TRAP: Targeted Redirecting of Agentic Preferences

Thank you for your suggestions and valuable feedback on our work. We would like to address each of your concerns:

1. "The authors should clarify whether the proposed method is applicable to images across diverse scenarios and styles..."

   • Response: TRAP is designed to handle diverse scenarios and visual styles by combining adaptive feature decomposition with text-guided semantic modulation. Specifically, our architecture separates features into common (prompt-aligned) and distinctive (identity-preserving) branches, with the importance of each dynamically adjusted via the text projection layer and bounded by sigmoid activation. This allows TRAP to enhance semantics in varying prompts and contexts. We demonstrate strong attack performance not just on COCO, but also on other datasets such as Flickr8k_sketch (sketches, abstracted forms) and ArtCap (artworks, varied styles), as shown in Tables 1–3. TRAP outperforms baseline attacks even in these challenging domains, supporting its robustness across content, context, and style diversity.

Table 1 · Extended evaluation on the COCO dataset

Table 2 · TRAP Attack-Success Rate Across Sketch / Abstract-Image Dataset (Flickr8k_sketch)

Table 3 · TRAP Attack-Success Rate Across Artistic Image Styles (ArtCap dataset)

2. "The adversarial sample generation process appears relatively complex... "

   • Response: We thank the reviewer for highlighting the importance of computational efficiency. As shown in Table 4, TRAP generates a single optimized adversarial image in an average of 666 seconds on an NVIDIA A100 GPU. The total time per sample for TRAP reflects the full iteration budget (20 steps × 20 iterations), representing a conservative, worst-case upper bound. In practice, TRAP’s early stopping mechanism means most attacks succeed in fewer iterations, resulting in a lower average runtime for the majority of samples. Additionally, TRAP remains practical for many real-world scenarios, such as e-commerce platforms and moderation pipelines where attacks can be generated offline before deployment or review.

Table 4 · Total computation time per sample

3. "Ablation experiments are necessary to validate the contribution of each proposed module... "

   • Response: We ran removal ablations to establish causality, where any consistent drop in attack success can be attributed to that module’s contribution rather than parameter count or tuning. The results of this ablation study in Table 5 show that removing any main component substantially reduces ASR, confirming each component’s necessity.

Table 5 · Contribution of individual TRAP components to overall attack effectiveness

4. "The methodological section would benefit from clearer explanations of the design rationale behind each module... "

   • Response: We agree that a more explicit explanation will help readers and will make these clarifications in the final version of the paper.
   • Prompt-Driven Modulation:
     • Why not attention mechanisms or types of interactions? We experimented with standard attention layers as alternatives for layout generation. However, in our pilot tests, attention layers increased per-step optimization time by over 30% without measurable ASR improvement, so we adopted a more efficient encoder-decoder architecture with direct spatial modulation.
   • Siamese Semantic Network:
     • How is the Siamese Network trained to achieve the decomposition, and what supervision ensures proper feature separation? The Siamese branches are supervised by the attack optimization itself: the common (semantic) branch minimizes semantic loss (Lsem) toward the prompt, while the distinctive branch minimizes identity loss (Ldist) relative to the original embedding, both taking advantage of CLIP’s contrastive structure. No external labels are required, the dual loss structure naturally polarizes each branch toward its intended feature set.
   • Loss Function and Hyperparameter Ablation:
     • Is there a tension between preserving distinctive features and achieving semantic alignment? Are the lambda coefficients justified? There is a balance required between semantic and identity preservation. Our new ablation results (Tables 2 and 3) systematically vary each lambda coefficient, confirming that semantic loss is essential, perceptual loss is robust to moderate changes, and the distinctive loss must remain low for best performance.

Table 6 · Effect of Increasing λ-Coefficients on Attack-Success Rate

Table 7 · Effect of Decreasing λ-Coefficients on Attack-Success Rate

5. "The experimental evaluation appears limited in scope... "

   • Response: We appreciate the pointer to recent advances and recognize the importance of comparisons with more modern adversarial attack methods. We address each of the reviewer's given examples below:
     • Liu et al. (2024) study automatic prompt injection in text pipelines specifically, with the idea of universal injected strings that can subvert LLM‑integrated apps by altering external text content. We find this paper to be adjacent to our work, but we expand the idea of redirecting autonomous behavior to a multimodal context, which increases the complexity of the problem considerably.
     • Chen et al. (2024) introduce a Common Weakness Attack (CWA) to improve transferability of pixel‑bounded adversarial examples via flatness and proximity objectives, validated mostly on image classification and extended to a Bard demo, not a selection setting like the one TRAP is evaluated in. We include a transfer baseline from this paper in Table 8 and find that TRAP remains superior in the preference selection setting.
     • Shayegani et al. (2023) perform embedding‑space image editing that pairs a benign prompt with an adversarial image targeted to "toxic" embeddings and focuses on jailbreak generation, not selection. To address the reviewer's concerns about comparing TRAP's evaluation with modern state-of-the-art methodologies, we add in Table 8 an additional comparison with SA-AET, which boosts cross‑model transfer by sampling within an adversarial evolution triangle and optimizing in a semantic‑aligned contrast subspace; in our selection‑based evaluation with matched access and iteration budgets, SA‑AET is competitive, but TRAP achieves higher success (Table 8).

Table 8 · Attack-success rates of TRAP versus baseline methods across multiple VLMs

6. "The manuscript contains persistent formatting inconsistencies..."

   • Response: We will correct the typos in the final version of the paper.

7. "Figure 1 currently fails to clearly convey..."

   • Response: We will design a diagram that better emphasizes the key contributions of our methodology in the final version of the paper.

Thank you very much for your thoughtful review, constructive feedback, and follow-up. We greatly appreciate your careful consideration of our work and your willingness to engage with our clarifications and new results and the discussions and experiments mentioned in the rebuttal will be included in the final version of our paper.

Thank you for your suggestions and valuable critique of our work. We would like to address each of your concerns:

1. "Is there a typo in Equation 3?"

   • Response: We have carefully reviewed Equation 3 and did not identify any typographical errors. If this concern persists, please let us know, and we will address it in the final version.

2. "Similarly, Algorithm 1 has many bespoke design decisions. Have you thoroughly ablated them, for example, the choice of embedding model?"

   • Response: Thank you for raising this point. We have included an ablation study of the embedding model choice in Table 1. Across embedding models such as ViT-B/32, timm/ViT-SO400M-14-SigLIP-384, and jina-clip-v2, TRAP consistently achieves high attack success rates, with only minor fluctuations between models. This confirms that our approach is robust to the specific embedding model used. We also present ablation studies on other decisions in the following:
     • Table 8 from reviewer riRf is the evaluation of TRAP on Mistral-small-3.2-24B, GPT-4o, and CogVLM, where CogVLM is a model not trained with contrastive objectives. Furthermore, Table 8 compares TRAP with more baselines: SSA_CWA and SA-AET
     • Table 5 from reviewer qNmy shows TRAP’s robustness on defences such as simple Gaussian Noise, CIDER, and MirrorCheck. TRAP is also evaluated on Robust-LLaVA, a model specifically trained to be robust against adversarial perturbations.
     • Table 2 & Table 3 from reviewer qNmy are ablation studies on the effect of different values for the coefficient for each loss function
     • Table 5 from the reviewer Ce3w is an evaluation of TRAP on different Stable-Diffusion image generators.

Table 1 · TRAP attack-success rates ablation on different embedding models

3. "Are there experiments analyzing the effectiveness of the iterative nature of this algorithm... "

   • Response: Due to current rebuttal restrictions on attaching image files to our rebuttal, we will be adding a qualitative image sequence to the appendix in the final version of the paper. Table 2 provides an ablation analysis of the effect of the number of optimization iterations on ASR. The results show that even a single optimization iteration with TRAP already outperforms baseline attacks in terms of ASR, and increasing the number of iterations further amplifies this advantage across all models tested. Table 2 · Effect of optimisation-iteration count on TRAP attack-success rate | Iteration count |  LLaVA-1.5-34B  |  Gemma3-8B  |  Mistral-small-3.1-24B  | |:-----------|--------------:|----------:|----------------:| | 1 | 57 % | 58 % | 44 % | | 5 | 73 % | 61 % | 50 % | | 10 | 100 % | 93 % | 89 % | | 20 | 100 % | 100 %| 100 %|

4. "What are some other baselines that could be incorporated into Table 1? For example..."

   • Response: We agree with the need for stronger and more recent baselines. We have added two additional attacks to Table 3: BARD_SSA (as suggested by reviewer riRf) and a state-of-the-art semantic alignment attack. Most prior baselines rely on pixel-level perturbations, which recent models have become increasingly robust against due to new defense frameworks (i.e. Wang et al., 2025). In contrast, TRAP explicitly targets the semantic content of the image. Existing semantic attacks are primarily designed to achieve embedding alignment, often aiming to make the attack image semantically indistinguishable from the target, while our evaluation focuses on the image preference scenario, where having two highly similar images may lead to ambiguous selection outcomes. TRAP incorporates a distinctive loss and semantic alignment loss that pushes the produced image to be not just similar, but competitively more appealing to the model, yielding consistently higher attack success for our given scenario. Full results for all new baselines are shown in Table 3.

Table 3 · Attack-success rates of TRAP versus baseline methods across multiple VLMs

5. "Much of the paper discusses "agents" but there is nothing agentic about the approach. It's a standard multimodal attack."

   • Response: We appreciate this feedback and recognize the importance of evaluation in realistic agentic settings. As shown in Table 4, TRAP consistently achieves higher attack success rates than all baselines, even when adversarial images are embedded in actual e-commerce website screenshots with randomly shuffled product descriptions. This mirrors how modern frameworks like LangChain use browser automation with and vision models to analyze and interact with web content, demonstrating TRAP’s real-world relevance for modern agentic pipelines. Due to the regulations against file attachments, we are unable to provide the visual contents of this experiment setting. However, the respective figures will be added to the appendix in the final version of the paper.

Table 4 · Attack-success rates of TRAP and baselines in e-commerce webpage scenarios

7. "Is "100 image-caption pairs" a typical sample size for this subfield? That seems small..."

   • Response: Each of our 100 image-caption pairs is evaluated with 100 randomized trials and up to 20 optimization iterations, resulting in over 200,000 model queries per experiment, and this computation is further multiplied across model architectures and datasets. While the sample size is 100 images per dataset, we have also tested TRAP on additional datasets (Flickr8k_sketch and ArtCap), and observed consistently strong attack success rates. This indicates that our sample selection provides robust and diverse coverage, and that TRAP’s effectiveness generalizes beyond the original COCO benchmark. Error bars for attack success rates (across three random seeds) are reported in Section 6.2 for key figures, and we will clarify this point further in the final version of our paper: “All results are averaged across 100 image-caption pairs and 100 randomized trials, with error bars reflecting variance across three seeds where shown.”

Table 5 · Extended evaluation on the COCO dataset

Table 6 · TRAP Attack-Success Rate Across Sketch / Abstract-Image Dataset (Flickr8k_sketch)

Table 7 · TRAP Attack-Success Rate Across Artistic Image Styles (ArtCap dataset)

8. "No, actually. I would like to see discussion of Ethics... "

   • Response: Thank you for raising this point. A central aim of our paper is to highlight the real-world risks posed by semantic attacks like TRAP and to increase awareness that current pixel-level and input-based defenses are insufficient. By demonstrating how these attacks can bypass content moderation, mislead AI systems, or reinforce biases, even in the presence of state-of-the-art mitigations and adversarially trained models (see Table 5 from reviewer qNmy, which shows TRAP’s robustness on defences such as simple Gaussian Noise, CIDER, and MirrorCheck. TRAP is also evaluated on Robust-LLaVA, a model specifically trained to be robust against adversarial perturbations), we hope to alert the community to this emerging threat and motivate the development of more robust, semantically-aware defenses. We will add this discussion of ethical and societal implications to the Limitations section in the final version of our paper.

Thank you again for your detailed review and constructive critique. We have addressed all of your comments and suggestions in our rebuttal, including further ablation studies, additional baseline attack methods, quantitative results for iterative optimization, broader dataset coverage, and expanded discussion of ethical implications. If you have any remaining questions or if there are aspects you would like further clarified, please let us know, we would be happy to provide additional detail or context. We appreciate your time and engagement with our work.

Thank you for your detailed feedback and useful suggestions. We would like to address your concerns:

1. "Can the authors demonstrate TRAP’s effectiveness in a more realistic agentic setting?"

   • Response: We appreciate this feedback and recognize the importance of evaluation in realistic agentic settings. As shown in Table 1, TRAP consistently achieves higher attack success rates than all baselines, even when adversarial images are embedded in actual e-commerce website screenshots from The Klarna Product Page Dataset. This mirrors how modern frameworks like Aguvis use browser automation with vision models to analyze and interact with web content, demonstrating TRAP’s real-world relevance for modern agentic pipelines. Due to the regulations against file attachments, we are unable to provide the visual contents of this experiment setting. However, the respective figures will be added to the appendix in the final version of the paper.  

Table 1 · Attack-Success Rates of TRAP and Baselines in e-commerce Webpage Scenarios

2. "Are there any analysis on the \lamda1, \lamda2,\lamda3, in equation 3? How each loss terms affect final results? How the method depends on these hypermeters?"

   • Response: We thank the reviewer for highlighting the importance of the hyperparameters. We have analyzed these hyperparameters in detail, as shown in Tables 2 and 3. The results show that TRAP’s performance is robust to moderate changes in the perceptual loss (λ1​), with only slight fluctuations in attack success rate when increased or decreased. In contrast, the semantic loss coefficient (λ2​) is essential: raising it too high can reduce transferability, while removing it sharply decreases attack success across all models, confirming its importance. For the distinctive loss (λ3​), higher values significantly hurt performance, while reducing or omitting it generally maintains high attack success.  

Table 2 · Effect of Increasing λ-Coefficients on Attack-Success Rate

Table 3 · Effect of Decreasing λ-Coefficients on Attack-Success Rate

3. "TRAP optimizes in CLIP space—how robust is the attack when the agentic model is based on non-contrastive or generative vision-language models?"

   • Response: To assess the robustness of TRAP on non-contrastive models, we evaluated TRAP on CogVLM, a model not trained with contrastive objectives. As shown in Table 4, TRAP achieved a 94% attack success rate, substantially outperforming all baselines.  

Table 4 · Effectiveness of Adversarial Attacks on a Non-Contrastive Vision–Language Model (CogVLM)

4. "Can TRAP be detected or mitigated by current perceptual defenses or CLIP anomaly detection tools? What if the agent is trained with an ensemble of prompts or with adversarial training?"

   • Response: We initially tested TRAP against simple noise-based defenses, which referred to adding Gaussian noise to the adversarial image, as in Byun et al. (2021), to disrupt pixel-level attack gradients. However, since TRAP operates at the semantic and embedding level, its effectiveness is minimally affected by such noise. Attack success rate dropped by at most 1% (see Table 5). We also evaluated TRAP against CIDER and MirrorCheck, which aim to mitigate both low-level and some higher-level perturbations, but TRAP consistently maintained high success rates. Finally, TRAP was tested on Robust-LLaVA, a model adversarially trained for robustness, and still achieved a 92% attack success rate. These results highlight that existing input-level and even adversarially trained defenses are insufficient to counter semantic attacks like TRAP.  

Table 5 · Robustness of TRAP Attack Under Various Defense Mechanisms and Adversarial Training

5. "Why Mistral 3.1, there are more updated versions in Mistral series, do you have results on these models?"

   • Response: We thank the reviewer for their suggestion. In Table 6, we have added results for Mistral-small-3.2-24B and GPT-4o alongside the original experiments. TRAP maintains high attack success rates on this updated model, further supporting the generality of our method.  

Table 6 · Attack-success-rate comparison on additional vision–language models

6. "The attack assumes agents use CLIP-like contrastive embedding spaces. It could be better to make more comparisons with..."

   • Response: We appreciate the pointer to recent advances in this field and recognize the importance of comparisons with more modern adversarial attack methods. In Table 7, we report TRAP’s attack success rates alongside both the original perturbation methods and modern semantic-level attacks, including SSA-CWA and SA-AET. Across all models, including GPT-4o and CogVLM, a non-contrastive learning model, TRAP consistently outperforms these methods.  

Table 7 · Attack-success rates of TRAP versus baseline methods across multiple VLMs

7. "Since TRAP steers images toward embedding alignment with high-level prompts, it could unintentionally amplify pre-existing biases in the CLIP model."

   • Response: We agree with the reviewer that TRAP may amplify existing biases in the underlying VLM or CLIP model. Our intention with TRAP is to systematically expose such vulnerabilities, including ones that could lead to larger problems such as bias amplification, to motivate the development of stronger, semantically-aware defenses and more robust model architectures. We will clarify this point in the discussion section to highlight both the risks and the necessity of addressing them in future work.  

[1] Wang et al. “CogVLM: Visual Expert for Pretrained Language Models” (2024).

Thank you again for your thoughtful review and constructive suggestions. We have addressed all of your questions in detail in our rebuttal, including new results on updated models, additional e-commerce settings, hyperparameter sensitivity, robustness against state of the art defenses, and potential effects of our work on bias amplification. If you have any further concerns or would like additional clarification on any aspect, please let us know, we would be happy to elaborate further. We appreciate your time and consideration during the review process.

Thank you for your valuable suggestions and feedback! We would like to address your concerns:

1. “line 188: I am confused about the modulation process, if using mean(A), seems entire image embedding will be scale equivalently? How would this ensure semantic edits are concentrated in regions most relevant as indicated by mask? (line 190)”

   • Response: Thank you for the suggestion. We originally used the mean-based modulation for computational simplicity and efficiency. After implementing the element-wise modulation, we observed no notable change in attack success rate or perceptual quality, but the runtime increased by approximately 18%. We will make this outcome and rationale clearer in the final version of the paper.  

2. “Can you elaborate a little on how TRAP's computational intensity look like, for example, what is the speed for generating an adversarial attacked image?”

   • Response: We thank the reviewer for highlighting the importance of computational efficiency. As shown in Table 1, TRAP generates a single optimized adversarial image in an average of 666 seconds on an NVIDIA A100 GPU. The total time per sample for TRAP reflects the full iteration budget (20 steps × 20 iterations), representing a conservative, worst-case upper bound. In practice, TRAP’s early stopping mechanism, which is triggered when the attack success condition is met (i.e., the target model selects the adversarial image over the original in the selection setting), means most attacks succeed in fewer iterations, resulting in a lower average runtime of 233.3 seconds for the majority of samples.  

Table 1: Total Computation Time per Sample

3. “line 242, can you elaborate on the simple noise-based defense, is it just adding random noise to the adversarial attacked image?”

   • Response: The simple noise-based defense involves adding Gaussian noise to the adversarial image, as in Byun et al. (2021)[1], to disrupt pixel-level attack gradients. However, since TRAP operates at the semantic and embedding level, its effectiveness is minimally affected by such noise. Attack success rate dropped by at most 1% (see Table 3). We also evaluated TRAP against CIDER[5] and MirrorCheck[6], which aim to mitigate both low-level and some higher-level perturbations, but TRAP consistently maintained high success rates. Finally, TRAP was tested on Robust-LLaVA[7], a model adversarially trained for robustness, and still achieved a 92% attack success rate. These results highlight that existing input-level and even adversarially trained defenses are insufficient to counter semantic attacks like TRAP.  

Table 2 · Robustness of TRAP Attack Under Different Defense Mechanisms and Adversarial Training

4. “The attack is mainly for selection setting.”

   • Response: While our attack is primarily evaluated in the selection setting, this scenario is critical for real-world agentic GUI systems, where decisions often involve choosing the most relevant image or response from a set. The attack success rate in this context directly measures an AI agent’s vulnerability to manipulation in retrieval, moderation, or recommendation workflows. Vision-language models, such as those targeted in this paper, are the main decision-making module within the agentic systems. Thus, evaluating attacks on these multimodal models provides a practical and realistic surrogate for agentic workflows, as their architecture and selection logic closely match those in recent GUI Agent research, such as OS-ATLAS[2] and Aguvis[3].  

5. “The attack depends highly on how VLM is trained and the performance of image generation model.”

   • Response: To address this, we evaluated TRAP on CogVLM[4], a model not trained with contrastive objectives, and found that TRAP still achieves strong attack success rates where other attacks fail (Table 3). Additionally, we ablated across multiple Stable Diffusion models and observed that TRAP’s effectiveness remains consistent regardless of the image generator used (Table 4).  

Table 3 · Effectiveness of Adversarial Attacks on a Non-Contrastive Vision–Language Model (CogVLM)

Table 4 · Attack-Success Rates on Different Stable Diffusion Image Generators

References

[1] Byun et al. “On the Effectiveness of Small Input Noise for Defending Against Query-based Black-Box Attacks” (2021).

[2] Wu et al. “OS-ATLAS: A Foundation Action Model for Generalist GUI Agents” (2024).

[3] Xu et al. “Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction ” (2024).

[4] Wang et al. “CogVLM: Visual Expert for Pretrained Language Models” (2024).

[5] Xu et al. "Cross-modality Information Check for Detecting Jailbreaking in Multimodal Large Language Models" (2024).

[6] Fares et al. "MirrorCheck: Efficient Adversarial Defense for Vision-Language Models" (2024).

[7] Malik et al. "Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Models" (2025).

Thank you again for your thorough review and thoughtful questions. We hope our responses in the rebuttal have fully addressed your concerns regarding the modulation process, computational intensity, defense mechanisms, and the scope of TRAP. If there are any remaining questions or further concerns, please let us know, we would be happy to clarify or provide additional details. We appreciate your time and consideration throughout the review process.

As the author response period is closing soon, please let us know if there are any remaining concerns or clarifications you would like us to address. We appreciate your time and feedback, and we are happy to provide any additional details as needed.