We sincerely thank you for your insightful comments! Please kindly find our responses below.

W1. Perturbations are tightly linked to a specific ICE set. A new ICE set (or even a change in ordering or length) requires recomputing the budget profile from scratch, which limits robustness in real-world dynamic settings.

Thanks for the question! We want to clarify that the budget profile construction is task-specific, rather than ICE set-specific. The rationale behind this is that ICEs for the same task comply with a similar distribution, which has also been verified empirically in our experiments. Thus, a new ICE set would not require recomputing the budget profile. Please also kindly note that the budget profile is scalable and can be adapted to any context length during the online stage.

W2 and Q1. The paper does not quantify the runtime of offline PGD (which operates jointly over n ICEs for T=80 steps) or the online phase (which involves T-step PGD and nearest-neighbor word projection). This hinders assessments of scalability. Could you provide a detailed computational-time analysis for both the offline and online stages?

We would like to clarify that the computation cost at the offline stage is not a major concern since the adversarial ICE generation occurs at the online stage, and the offline stage is conducted privately by the attacker. The budget profile constructed in the offline stage is used to guide the generation of adversarial ICEs during the online stage. Thus, the computational cost of the offline stage is unrelated to the cost of the actual attack (i.e., the online stage). The computational cost of the online stage is expected to be low as it only requires calculating the gradients needed for ICE generation under the guidance of the budget profile. We report the time complexity in the table below.

Results on Time Complexity

All experiments were conducted on an NVIDIA L40S GPU using the LLaMA3-70B language model on SST-2. In this experiment, for each run of the offline phase, we select input-output pairs equal in number to the attack context length ($𝑛$=20) from the training set. The budget profile is averaged from the results of multiple runs. During the online phase, which includes PGD and word projection, the full test set is used for performance evaluation. For a clearer illustration, all results are normalized with respect to the time required to compute perturbations per ICE using the Global attack. It can be seen that BAM-ICL offers significantly lower time complexity than the Global attack. Even when accounting for the additional cost of the offline phase, the overall runtime increase remains modest.

Please kindly note that the budget profile computed during the offline stage is scalable and can be adapted to any context length during the online stage.

W3 and Q2. The method is only tested on text classification datasets. It remains unclear how well BAM-ICL performs on more complex or generative tasks such as jailbreaks or reasoning. Have you considered or evaluated BAM-ICL on more complex or generative tasks, for example, jailbreak?

Our initial experiments primarily followed the settings used in prior works on hijacking attacks, which mostly focused on classification tasks. Below, we provide additional experimental results on jailbreaking and reasoning tasks. We will incorporate a more comprehensive evaluation and detailed analysis in the later version of the paper.

Reasoning Tasks We follow the settings of [1] to use SuperGLUE [2] to benchmark the reasoning performance of ICL on OPT-30B with $n=5$. The accuracy of different categories is shown in the table below. It can be seen that BAM-ICL outperforms Flat attack and is close to the Global attack, which exhibits a similar trend as classification tasks presented in the paper.

Jailbreaking Tasks We use the AdvBench [3] as in GGI [4]. The attack success rate (ASR) on Mistral-7b-instruct is shown below. Since both BAM-ICL and GGI can achieve an ASR of ~99% at $n$=4 based on the original budget of GGI, we conduct this comparison at half of the budget. We also include a Zero-Shot baseline for comparison, which provides the model with only the malicious prompt. Similar to the behavior on classification tasks, as BAM-ICL depends on ICE positioning and the associated budget profile, it benefits from having more ICEs and yields better performance with a larger $n$. It is also important to note that BAM-ICL achieves better perplexity.

W4 and Q3. No evaluation against stronger defenses. Only one defense method, which prepends clean ICEs is tested. There is no exploration of detection-based defenses or adversarial filtering. How does BAM-ICL perform if the ICEs are shuffled or slightly altered in content (e.g., paraphrased)? Is the learned budget profile robust to such variations?

In our work, we primarily compared with GGI and therefore adopted the same defense strategies used in that setting for consistency. We would like to emphasize that the core contribution of our paper lies in the proposed budgeted two-phase attack strategy, while the choice of perturbation generation method can potentially incorporate various alternatives. In this regard, BAM-ICL improves stealthiness by strategically distributing perturbations across the context examples, rather than concentrating them on individual inputs. We have conducted additional experiments against several selected defense techniques below. We will include more results in the later version.

Additional Experiments against Defenses

One straightforward filtering method is to employ a perplexity-based filter [5] as a defense strategy, as proposed in [6]. As shown in Fig. 2(b) of the main paper, BAM-ICL achieves better perplexity and hence is expected to perform well against such perplexity-based filtering defenses. Besides, we have also conducted additional experiments against three representative yet suitable defenses from different categories: paraphrasing [5] that runs in conjuction with a detector to test the performance against paraphrasing suspicious words within single ICE, In-Context Example Selection [1] as a detection-based defense across multiple ICEs, and Fantastically Ordered Prompts [7] for shuffling defense across multiple ICEs. The results of ASR drop percentage (lower represents better robustness against defenses）for OPT-30B on SST-2 are reported below.

It can be seen that BAM-ICL demonstrates stronger resilience against filtering and detection-based defenses than GGI, both within individual ICEs and across multiple ICEs. However, shuffling and reordering could impact the effectiveness of BAM-ICL, compared to Flat attack, which aligns with our expectations, since BAM-ICL relies on position-dependent perturbation budget allocation.

Q4. What is the assumed adversarial capability in this threat model? If the adversary is the model publisher (rather than an external attacker), are perplexity and stealthiness still meaningful objectives?

We presented our threat model in Section 3.1. We consider a black-box setting in which the attacker has no access to the training data or weights of the model. The attacker knows the task and can draw samples from the target task distribution. If the adversary is the model publisher (rather than an external attacker), we believe perplexity and stealthiness can still be meaningful, but their role shifts depending on the attack goals. In typical external attacker scenarios, stealthiness refers to evading detection by model defenders. However, when the model publisher itself is malicious, the publisher inherently controls the model, its data, and potentially its outputs. In this case, perplexity and stealthiness become less about evading detection by the publisher and more about evading detection by other forms of defenders, such as automated filters as discussed above.

References

[1] In-context example selection with influences. arXiv:2302.11042, 2023.

[2] SuperGLUE: A stickier benchmark for general-purpose language understanding systems. NeurIPS, 2019.

[3] Why should adversarial perturbations be imperceptible? Rethink the research paradigm in adversarial NLP. EMNLP, 2022.

[4] Hijacking large language models via adversarial in-context learning. arXiv:2311.09948, 2023.

[5] Baseline defenses for adversarial attacks against aligned LMs. arXiv:2309.00614, 2023.

[6] AdvPrompter: Fast adaptive adversarial prompting for LLMs. ICML, 2025.

[7] Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. ACL, 2022.

Thank you very much for your valuable comments! Please kindly find our responses below.

W1. The task distribution of and benign in-context examples may not be available.

We agree that for some highly sensitive applications, such as medical data, our assumption of drawing in-distribution data may not hold. However, it is worth noting that this assumption, which allows for a slightly stronger attacker with access to task-relevant public data, is widely adopted in AI security research, for instance, adversarial attack [1], backdoor attack [2], membership inference attack [3], as well as recent attacks on LLMs [4]. This threat model also provides a credible worst-case scenario for the defender. For most real-world settings, the assumption is valid based on the fact that most tasks (e.g., sentiment analysis, question answering, and code generation) are constructed and trained from publicly available datasets (e.g., Wikipedia). We acknowledge that exploring attack strategies in settings where the attacker does not have access to in-distribution data represents an important and challenging direction for future work.

W2 and Limitation. It is advised to show the computational cost of offline budget profiling, especially concerning large-scale models or datasets. The computational cost of offline budget profiling could limit the scalability of the proposed framework to large models.

We would like to clarify that the computation cost at the offline stage is not a major concern since the adversarial ICE generation occurs at the online stage, and the offline stage is conducted privately by the attacker. The budget profile constructed in the offline stage is used to guide the generation of adversarial ICEs during the online stage. Thus, the computational cost of the offline stage is unrelated to the cost of the actual attack (i.e., the online stage). The computational cost of the online stage is expected to be low as it only requires calculating the gradients needed for ICE generation under the guidance of the budget profile. We report the time complexity below.

Results on Time Complexity

All experiments were conducted on an NVIDIA L40S GPU using the LLaMA3-70B language model on SST-2. In this experiment, for each run of the offline phase, we select input-output pairs equal in number to the attack context length ($𝑛$=20) from the training set. The budget profile is averaged from the results of multiple runs. During the online stage, the full test set is used for performance evaluation. For a clearer illustration, all results are normalized with respect to the time required to compute perturbations per ICE using the Global attack. It can be seen that BAM-ICL offers significantly lower time complexity than the Global attack. Even when accounting for the additional cost of the offline phase, the overall runtime increase remains modest.

Please kindly note that the budget profile computed during the offline stage is scalable and can be adapted to any context length during the online stage.

W3 and Q1. The performance of the proposed framework against more advanced defense strategies. How would the proposed framework perform against defenses with in-context example filtering?

We would like to emphasize that the core contribution of our paper lies in the proposed budgeted two-phase attack strategy, while the choice of perturbation generation method can potentially incorporate various alternatives. In this regard, BAM-ICL improves stealthiness by strategically distributing perturbations across the context examples, rather than concentrating them on individual inputs. We have conducted additional experiments against several selected defense techniques below. We will include more results in the later version.

Additional Experiments against Defenses

One straightforward filtering method is to employ a perplexity-based filter [5] as a defense strategy, which is also used in [6]. As shown in Fig. 2(b) of the main paper, BAM-ICL achieves better perplexity and hence is expected to perform well against such perplexity-based filtering defenses. Besides, we have also conducted additional experiments against three representative yet suitable defenses from different categories: paraphrasing [5] that runs in conjuction with a detector to test the performance against paraphrasing suspicious words within single ICE, In-Context Example Selection [7] as a detection-based defense with filtering across multiple ICEs, and Fantastically Ordered Prompts [8] for shuffling defense across multiple ICEs. The results of ASR drop percentage (lower represents better robustness against defenses）for OPT-30B on SST-2 are reported below.

It can be seen that BAM-ICL demonstrates stronger resilience against filtering and detection-based defenses than GGI, both within individual ICEs and across multiple ICEs. However, shuffling and reordering could impact the effectiveness of BAM-ICL, compared to Flat attack, which aligns with our expectations, since BAM-ICL relies on position-dependent perturbation budget allocation.

References

[1] Transferability in machine learning: From phenomena to black-box attacks using adversarial samples. arXiv:1605.07277, 2016.

[2] Clean-label backdoor attacks on video recognition models. CVPR, 2020.

[3] Membership inference attacks against machine learning models. Symposium on Security and Privacy, 2017.

[4] Improving few-shot performance of language models. ICML, 2021.

[5] Baseline defenses for adversarial attacks against aligned LMs. arXiv:2309.00614, 2023

[6] AdvPrompter: Fast adaptive adversarial prompting for LLMs. ICML, 2025.

[7] In-context example selection with influences. arXiv:2302.11042, 2023.

[8] Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. ACL, 2022.

Thanks a lot for your thoughtful suggestions! Please kindly find our responses below.

W1. While no major concerns, this paper only considers the hijacking in ICL-attacks, and other forms of ICL-attacks like jailbreaking [1] or backdoor [2] can be further discussed.

[1] Jailbreak and guard aligned language models with only few in-context demonstrations. arxiv 2023 [2] Backdoor Attacks for In-context Learning. EMNLP 2024.

We will add paragraphs and citations in the revised version of the paper to discuss more about jailbreaking and backdoor attacks for in-context learning. Specifically, we will elaborate and clarify the similarities and differences between the hijacking attack and the other two attacks. Although all these attacks are designed to undermine the performance of LLMs, they have several differences. The hijacking attack manipulates LLMs' output via ICL to specifically steer the intended model behavior, while jailbreaking focuses on circumventing models' safety guardrails (e.g., human alignment) to unleash safety and ethical limitations, and the backdoor attack implants backdoors via demonstrations and activates malicious behavior with prompts embedded with pre-defined triggers.

Q1. How can the theory and algorithm in this paper extend to jailbreak or backdoor attacks with ICL?

Thank you for the great question! We believe our budgeted two-phase attack strategy can be adapted to other attacks with ICL. As requested by both Reviewer WopZ and Reviewer CYaW, we conducted additional experiments on jailbreaking tasks.

Jailbreaking Tasks We use the AdvBench [3] to benchmark the jailbreaking performance as in GGI [4]. The attack success rate (ASR) on Mistral-7b-instruct is shown below. Since both BAM-ICL and GGI can achieve an ASR of ~99% at $n$=4 based on the original budget of GGI, we conduct this comparison at half of the budget. We also include a Zero-Shot baseline for comparison, which provides the model with only the malicious prompt. Similar to the behavior on classification tasks, as BAM-ICL depends on ICE positioning and the associated budget profile, it benefits from having more ICEs and yields better performance with a larger $n$. It is also important to note that BAM-ICL achieves better perplexity.

Similarly, in the context of backdoor attacks, instead of relying on a single, conspicuous trigger, BAM-ICL could inspire the use of a distributed set of lightweight, stealthy triggers that collectively accumulate influence. We look forward to extending and adapting our method to these attacks in future work.

References

[3] Why should adversarial perturbations be imperceptible? Rethink the research paradigm in adversarial NLP. EMNLP, 2022.

[4] Hijacking large language models via adversarial in-context learning. arXiv:2311.09948, 2023.

We sincerely appreciate your constructive suggestions! Please kindly find our response to each question below. Due to character constraints, we have abbreviated the questions for brevity.

W1. Exploration of more realistic applications.

Our initial experiments primarily followed the settings used in prior works on hijacking attacks, which mostly focused on classification tasks. Below, we provide additional experimental results on jailbreaking and reasoning tasks. We will incorporate a more comprehensive evaluation and detailed analysis in the later version.

Reasoning Tasks We follow the settings of [1] to use SuperGLUE [2] to benchmark the reasoning performance of ICL on OPT-30B with $n=5$. The accuracy of different categories is shown in the table below. It can be seen that BAM-ICL outperforms Flat attack and is close to the Global attack, which exhibits a similar trend as classification tasks presented in the paper.

Jailbreaking Tasks We use AdvBench [3] as in GGI [4]. The attack success rate (ASR) on Mistral-7b-instruct is shown below. Since both BAM-ICL and GGI can achieve an ASR of ~99% at $n$=4 based on the original budget of GGI, we conduct this comparison at half of the budget. We also include a Zero-Shot baseline for comparison, which provides the model with only the malicious prompt. Similar to the behavior on classification tasks, as BAM-ICL depends on ICE positioning and the associated budget profile, it benefits from having more ICEs and yields better performance with a larger $n$. It is also important to note that BAM-ICL achieves better perplexity.

W2. Assumption of drawing in-distribution data.

We agree that for some highly sensitive applications, such as medical data, our assumption of drawing in-distribution data may not hold. However, it is worth noting that this assumption, which allows for a slightly stronger attacker with access to task-relevant public data, is widely adopted in AI security research, for instance, adversarial attack [5], backdoor attack [6], membership inference attack [7] as well as recent works on LLM security [8]. This threat model also provides a credible worst-case scenario for the defender. For most real-world settings, the assumption is valid based on the fact that most tasks (e.g., sentiment analysis, question answering, and code generation) are constructed and trained from publicly available datasets (e.g., Wikipedia, GitHub). We acknowledge that exploring attack strategies in settings where the attacker does not have access to in-distribution data represents an important and challenging direction for future work.

W3. More detection-based or filtering-based defenses.

In our work, we primarily compared with GGI and therefore adopted the same defense strategies used in that setting for consistency. We would like to emphasize that the core contribution of our paper lies in the proposed budgeted two-phase attack strategy, while the choice of perturbation generation method can potentially incorporate various alternatives. In this regard, BAM-ICL improves stealthiness by strategically distributing perturbations across the context examples, rather than concentrating them on individual inputs. We have conducted additional experiments against several selected defense techniques below. We will include more results in the later version.

Additional Experiments against Defenses

One straightforward filtering method is to employ a perplexity-based filter [9] as a defense strategy, which is also used in [10]. As shown in Fig. 2(b) of the main paper, BAM-ICL achieves better perplexity and hence is expected to perform well against such perplexity-based filtering defenses. Besides, we have also conducted additional experiments against three representative yet suitable defenses from different categories: paraphrasing [9] that runs in conjuction with a detector to test the performance against paraphrasing suspicious words within single ICE, In-Context Example Selection [1] as a detection-based defense across multiple ICEs, and Fantastically Ordered Prompts [11] for shuffling defense across multiple ICEs. The results of ASR drop percentage (lower represents better robustness against defenses）for OPT-30B on SST-2 are reported.

It can be seen that BAM-ICL demonstrates stronger resilience against filtering and detection-based defenses than GGI, both within individual ICEs and across multiple ICEs. However, shuffling and reordering could impact the effectiveness of BAM-ICL, compared to Flat attack, which aligns with our expectations, since BAM-ICL relies on position-dependent perturbation budget allocation.

W4. Human evaluation is not included.

Due to the time limit during the rebuttal, we are not able to complete the approval process and setup required for human evaluation. However, we included examples at the end of our appendix (Section H) for the readers, as human judges, to verify the stealthiness and text quality.

W5. Experimental results using detection-based or filtering-based defenses and provide examples of attacks that fail.

Please refer to our responses to W3 regarding additional defense results. Here, we provide an example that GGI fails against paraphrasing [9]. Note that GGI appends adversarial suffixes to ICEs. The outputs are highlighted in bold.

clean ICEs

• Review: confusing plot that makes no sense

  Sentiment: negative

• Review: outstanding performances by the entire cast

  Sentiment: positive

• Review: compelling narrative with excellent character development

  Sentiment: positive

• Review: hide new secretions from the parental units

  Sentiment: negative

BAM-ICL

• Review: confusing Ploty that makes no sense

  Sentiment: negative

• Review: Great performances via aforementioned entire cast

  Sentiment: positive

• Review: compelling narrative together with quality

  Sentiment: positive

• Review: hide new secretions from the parental units

  Sentiment: positive

GGI

• Review: confusing plot that makes no sense NULL (did not pass detector, paraphrased)

  Sentiment: negative

• Review: outstanding performances by the entire cast Remove

  Sentiment: positive

• Review: compelling narrative with excellent character development NULL (did not pass detector, paraphrased)

  Sentiment: positive

• Review: hide new secretions from the parental units

  Sentiment: negative (fail)

W6. Some tables and figures are unclear.

We will follow your suggestion and update tables and figures in the revision to make sure they are clear and easy to understand.

For Figure 3, we illustrate the budget profiles computed under varying total perturbation budgets. We will include legends in the revised version for better clarity. The key message we aim to convey is that the budget profile remains relatively consistent for a given task, highlighting its task-specific nature rather than dependence on a fixed budget value.

For Table 1, we will remove the error stats from the main paper and move those to appendix to improve readability.

For the discussion of Figure 6, we will rephrase it from "increases more rapidly" to "is consistently higher".

W7. Cosine similarity distribution histograms of GGI in Figure 5 and the transferability of GGI in Table 2.

Table 2 demonstrates the budget profile transferability of BAM-ICL. However, since the budget profile is unique to our work (in fact, BAM-ICL is the first work to consider the budget allocation in ICL hijacking attack), it is not applicable to evaluate GGI for budget transferability.

We report the cosine similarity distribution based on the examples provided in the GGI paper. The results show that BAM-ICL can achieve higher cosine similarity.

W8. Attack examples.

Please kindly see the examples included at the end of our appendix (Section H).

References

[1] In-context example selection with influences. arXiv:2302.11042, 2023.

[2] SuperGLUE: A stickier benchmark for general-purpose language understanding systems. NeurIPS, 2019.

[3] Why should adversarial perturbations be imperceptible? Rethink the research paradigm in adversarial NLP. EMNLP, 2022.

[4] Hijacking large language models via adversarial in-context learning. arXiv:2311.09948, 2023.

[5] Transferability in machine learning: From phenomena to black-box attacks using adversarial samples. arXiv:1605.07277, 2016.

[6] Clean-label backdoor attacks on video recognition models. CVPR, 2020.

[7] Membership inference attacks against machine learning models. S&P, 2017.

[8] Improving few-shot performance of language models. ICML, 2021.

[9] Baseline defenses for adversarial attacks against aligned LMs. arXiv:2309.00614, 2023.

[10] AdvPrompter: Fast adaptive adversarial prompting for LLMs. ICML, 2025.

[11] Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. ACL, 2022.