We sincerely thank the reviewer for the insightful comments and for highlighting important aspects of our work. Below, we address the specific concerns raised.

1. Regarding the Confusion About Backdoored LLMs Refusing to Respond

We appreciate the reviewer’s question about backdoored LLMs that exhibit refusal behavior. It is important to note that refusal is a common target behavior used in backdoor attacks in generative scenarios, where the goal is to render the LLM overly uncooperative by refusing to respond to benign instructions whenever a specific trigger is present. This type of attack is designed to disrupt the usability of the LLM. Many existing works (such as [1]-[3]) have highlighted this kind of target behavior, emphasizing its practical relevance in real-world scenarios.

[1] Gloaguen T, Vero M, Staab R, et al. Finetuning-Activated Backdoors in LLMs. arXiv preprint arXiv:2505.16567, 2025.

[2] Shu M, Wang J, Zhu C, et al. On the exploitability of instruction tuning. Advances in Neural Information Processing Systems, 2023, 36: 61836-61856.

[3] Shao Z, Liu H, Hu Y, et al. Refusing Safe Prompts for Multi-modal Large Language Models. arXiv preprint arXiv:2407.09050, 2024.

---

2. Lacking Comparisons with Existing Backdoor Defense Methods

In this paper, we have already compared the proposed ICLScan with an existing defense method called CLIBE. Besides, we have emphasized in lines 306-303 that we can not perform fair comparisons with existing defenses, and CLIBE is the closest related work for ICLScan. To be more specific, as summarized in Section 2 of our paper, existing backdoor defense methods either (1) typically require white-box access to the LLM and are only applicable to classification tasks, or (2) in the black-box setting, focus on detecting whether input samples contain triggers rather than determining whether the model itself has been backdoored. In contrast, ICLScan is specifically designed for detecting backdoored LLM for generative tasks in the black-box setting, where the defender only has API access to the model and aims to identify whether the LLM for generative tasks has been backdoored in a lightweight manner. This fundamental difference in design objectives and access assumptions makes it difficult to perform a fair comparison with these existing methods. CLIBE is the closest related work for ICLScan. It is a backdoor detection method in a white-box setting but claimed to generalize to generative tasks. So, we just compared ICLScan with CLIBE, and the results are shown in the Table. 1 in the manuscript.

---

3. Clarifying “Jailbreak Backdoor Attack”

We appreciate the opportunity to clarify what we mean by a “jailbreak backdoor attack.” This is a type of backdoor attack where the target behavior is jailbreaking. The goal of this attack is to make the LLM respond to malicious instructions containing a trigger by providing harmful content, rather than refusing to respond as required by safety alignment protocols.

---

4. Why ASR Was Not Used as a Metric

The design goal of ICLScan is to detect whether a model has been backdoored, not to mitigate or eliminate the backdoor. Therefore, its effectiveness is reflected in its ability to correctly indicate whether an LLM has been compromised, rather than in reducing the ASR of the backdoor attack. Since ASR measures the success rate of a backdoor attack, it is not directly applicable to evaluating ICLScan’s performance as a detection method. Instead, we evaluate ICLScan based on its ability to distinguish between clean and backdoored models, which aligns with its intended purpose.

---

5. Rationale Behind the Choice of the Threshold $\delta$

To illustrate the rationale, we provide a detailed explanation with an example to supplement Section 4.2 of the paper. Consider a targeted ICL prompt that contains three examples: one example corresponds to the malicious pattern, while the other two correspond to benign patterns. For a clean LLM, during the ICL process, even if it has perfect instruction-following capability, the likelihood of the LLM exhibiting the malicious pattern in its response is 1/3, assuming no bias toward any specific pattern. However, for clean LLMs, although they have strong instruction-following capabilities, they are more inclined to rely on their inherent knowledge and reasoning abilities to produce coherent and safe outputs. This behavior can be attributed to the fact that clean LLMs are trained to follow general principles of consistency and safety. As a result, they are significantly less likely to exhibit the malicious pattern in their responses, and their probability of doing so is far below 1/3.

In contrast, for backdoored LLMs, the phenomenon of Backdoor Susceptibility Amplification (BSA) ensures that the malicious pattern implanted during the backdoor attack is strongly activated when a similar signal-dependent pattern appears in the ICL prompt. This substantially increases the likelihood of the LLM exhibiting the malicious pattern, which may not only reach 1/3 but also exceed it.

Therefore, by selecting a threshold $\delta$ that is slightly below 1/3, such as 1/4, we can clearly distinguish between clean and backdoored LLMs:

• Clean LLMs will have a malicious pattern probability well below the threshold due to their inherent robustness.
• Backdoored LLMs will exceed the threshold due to the BSA effect, making the malicious pattern more pronounced.

This choice of $\delta$ ensures a precise separation between the two types of models, enabling ICLScan to identify backdoored LLMs effectively. Our experimental results have shown that ICLScan is effective with such a threshold determination strategy.

6. Lacking experiments on closed-source models

Validating the effectiveness of ICLScan on closed-source models presents significant challenges, primarily because ICLScan is designed to detect backdoored LLMs. To evaluate its effectiveness, experiments must be conducted on backdoored LLMs. However, successfully implanting backdoors into closed-source models is extremely difficult.

To validate that there is no Backdoor Susceptibility Amplification phenomenon in closed-source clean LLMs, we tested the success rate of implanting ICL backdoors using targeted ICL prompts (involving 1/3 backdoor examples) on two closed-source clean LLMs (i.e., GPT-3.5-Turbo, GPT-4-Turbo). The results are as follows.

These results indicate that closed-source clean LLMs exhibit a low susceptibility to ICL backdoor implantation, aligning with our expectations. In contrast, if a closed-source LLM was successfully injected with a backdoor, there would be a strong signal-dependent pattern in the LLM. When a similar signal-dependent pattern appears in the ICL prompt, this implanted pattern is likely to generalize and activate the backdoor behavior. That is, if a backdoor were successfully implanted in a closed-source LLM, the phenomenon of Backdoor Susceptibility Amplification (BSA) would emerge, significantly increasing the success rate of ICL backdoors. This amplification would enable ICLScan to effectively distinguish between clean and backdoored LLMs. Thus, while direct validation on closed-source models is limited, our findings and theoretical understanding suggest that ICLScan would remain effective in identifying backdoored closed-source LLMs.

---

We hope these clarifications address the reviewers’ concerns and provide further insight into the design choices and contributions of our work. Thank you again for your valuable comments!

We sincerely thank the reviewer for the thoughtful feedback and insightful questions. Below are detailed responses.

1. How ICLScan Fits into Real-World Applications

We appreciate the reviewer’s question regarding the practicality of ICLScan. The refusal and jailbreaking backdoor behaviors focused by ICLScan are highly relevant in real-world applications. ICLScan provides a lightweight, black-box detection method for these common backdoors, making it easily applicable to practical scenarios.

For example, a backdoored LLM in a real-world application might be manipulated by a competitor to over-refuse benign instructions upon encountering a specific trigger, thereby sabotaging its usability. With ICLScan, users can test the LLM for the presence of a refusal backdoor before deploying it for official use. If a refusal backdoor is detected, users can decide to avoid using the compromised LLM, thus preventing potential operational disruptions or losses.

---

2. Limitations of ICLScan When Outputs Vary by Input Context

Thanks for pointing this out. We are sorry that our imprecise description in Appendix C caused a potential misunderstanding. We stated that “advanced backdoors may employ context-dependent target generation, where the malicious output varies based on input characteristics. ICLScan's static target assumption fails to capture these evolving patterns.” It does easily lead to the understanding that “ICLScan may not work very well when outputs vary by input context”. But in fact, jailbreaking backdoor attacks are a type of backdoor where outputs vary by input context, and our experiments have already demonstrated that ICLScan is effective in detecting such attacks. This demonstrates that even when outputs vary by input context, as long as the backdoor attack follows a consistent malicious pattern (e.g., responding as normal to malicious instructions with harmful content), ICLScan remains effective.

What we intended to express in our limitation analysis was that ICLScan does not yet account for backdoors where their malicious pattern changes dynamically based on the input context. It is important to note that such dynamic malicious patterns are not representative of current backdoor attacks, which predominantly rely on fixed malicious patterns (e.g., refusal and jailbreaking). Therefore, ICLScan’s current practical implications remain clear and relevant for today’s threat landscape. We would like to emphasize that as backdoor attack techniques continue to evolve in the future, ICLScan must also adapt and advance accordingly to maintain its effective detection performance. In the future, should backdoor attacks evolve to include dynamic malicious patterns, we can address this by identifying possible patterns and designing targeted ICL prompts for each pattern to extend ICLScan’s capabilities.

We will revise the limitation analysis in the revised manuscript to avoid misunderstanding.

---

3. Can ICLScan be combined with fine-tuning or post-hoc bias mitigation methods (e.g., calibration)

The answer is yes. ICLScan focuses specifically on detecting refusal and jailbreaking backdoors, and it operates independently of backdoor removal methods. Note that ICLScan is designed for black-box scenarios where users only have API access to the model. In this context, it can be combined with inference-stage backdoor mitigation techniques (e.g., calibration or other post-hoc bias mitigation methods) to address detected backdoors.

---

4. Backdoor Susceptibility Amplification (BSA) on Larger LLMs

Thanks for pointing this out. Our experiments were conducted exclusively on LLMs with 7B parameters or fewer due to computational constraints. It is infeasible for us to fine-tune larger LLMs for the purpose of implanting backdoors. Consequently, we were unable to evaluate the detection effectiveness of ICLScan on many larger backdoored LLMs. However, the root cause of the phenomenon of Backdoor Susceptibility Amplification (BSA) suggests that this phenomenon is also present in larger backdoored LLMs. Theoretically, regardless of the parameter scale of the model, as long as the backdoor is successfully implanted, the model will exhibit a strong signal-dependent pattern. This pattern ensures that the backdoored LLM preferentially recognizes the anomalous signal (trigger) and responds to it with the predefined target behavior. When a similar signal-dependent pattern appears in an ICL prompt, the backdoored LLM is likely to generalize this pattern and activate the target behavior, thereby amplifying its susceptibility to ICL backdoors. Larger LLMs, by design, possess stronger instruction-following capabilities and better generalization, which theoretically ensures the existence of the BSA phenomenon in larger backdoored LLMs.

To further verify this analysis, we conducted a quantitative experiment by training a 13B backdoored LLM based on LLaMA2-13B, where the trigger is BadMagic and the target is refusal. Due to resource limitations, we were only able to conduct minimal fine-tuning with quantification to implant the backdoor. We then evaluated the success rate of injecting an ICL backdoor via a targeted ICL prompt containing 1/3 backdoor examples. The success rate is more than 27%, compared to only 3.5% success on a clean LLaMA2-13B model. This empirical result confirms the existence of the BSA phenomenon in larger backdoored LLMs, demonstrating that ICLScan is theoretically effective for identifying larger backdoored LLMs.

We will add the discussion about BSA on larger LLMs in the revised manuscript as suggested.

---

5. The writing issue in the Cause of BSA (Section 4.1)

We greatly value the constructive feedback and will incorporate the suggested improvements in the revised version. Specifically, as recommended, we will include a concise summary sentence to clearly articulate the cause and significance of Backdoor Susceptibility Amplification (BSA).

---

We hope these responses address the reviewer’s concerns and provide a clearer understanding of our work. Thanks once again for the thoughtful and constructive comments!

We sincerely thank the reviewer for the insightful comments and for highlighting important aspects of our work. Below, we provide detailed responses to the raised concerns.

W1: What will happen if the backdoored target really doesn't fall within the defense ICL target prompts?

We appreciate the opportunity to clarify this point. Although the LLM’s inherent backdoored target sequence may not fall within the ICL target prompts, as long as the goal of the ICL target aligns with that of the backdoor target (e.g., achieving jailbreaking or refusal), there still exists the phenomenon of Backdoor Susceptibility Amplification, and ICLScan can still effectively detect backdoors. Our experimental results have already demonstrated this property. Specifically, in the case of backdoor attacks targeting jailbreaking, each backdoor sample’s target sequence is typically unique. This is because such attacks aim to make the LLM respond to malicious instructions with harmful content instead of identifying the malicious intent within the instructions and refusing to respond. Since each malicious instruction corresponds to a distinct malicious output, it is inevitable that most backdoor target sequences will not be explicitly included in the ICL target prompts. The effectiveness of ICLScan for detecting backdoors targeting jailbreaking can validate that ICLScan can still successfully detect these backdoors based on Backdoor Susceptibility Amplification, even when the targeted ICL prompts do not explicitly contain the backdoor target sequence.

To further address the reviewer’s concern, we conducted additional experiments focusing on refusal backdoors on LLaMA-2-7B-Chat-HF where the backdoored target sequence does not fall within the defense ICL target prompts. Specifically, the target sequence in the backdoored LLMs is “I’m sorry, but as an AI assistant, I do not have the capability to follow the given instruction,” we use the ICL prompt target sequence “I can't complete that request based on my current functionality.” We set the ratio of backdoor examples equals 1/3 and the ICLScan detection threshold equals 1/4. In this setting, we test the ICL trigger injection success rates in clean LLMs (CMs) and backdoored LLMs (BMs). Among them, backdoored LLMs 1-4 are with a word-level trigger, backdoored LLMs 5-8 are with a CBA trigger, backdoored LLMs 9-12 are with a phrase-level trigger, and backdoored LLMs 13-16 are with a long sentence-level trigger. The results are as follows.

We can observe that in clean LLMs, the success rate of implanting an ICL backdoor using targeted ICL prompts remained below 7%. In contrast, the success rate exceeded 34% in backdoored LLMs under the same conditions. These results verify the phenomenon of Backdoor Susceptibility Amplification. In this case, ICLScan achieved a 100% success rate in detecting backdoored LLMs, with no false positives observed during the evaluation. This confirms that ICLScan remains effective even when the backdoored target is not explicitly included in the defense ICL prompts.

---

W2: The connections between the noisy label robustness of ICL [1,2] and the discovery in this paper

We appreciate the reviewer’s comment regarding the connections between the noisy label robustness of ICL [1,2] and our discovery. The noisy label robustness of ICL, as discussed in [1,2], refers to the observation that even when incorrect knowledge is injected into the ICL prompt, its negative impact on the input query’s answer is limited and can be easily mitigated. This aligns with our analysis in the paper regarding clean LLMs being robust to ICL trigger injection. Specifically, we also found that for clean LLMs, even when backdoor examples are included in the ICL prompt, it is difficult to induce malicious patterns in the answer.

However, our discovery mainly focuses on a distinct and unexplored phenomenon: Backdoor Susceptibility Amplification, where backdoored LLMs are significantly more susceptible to ICL trigger injections. This is a critical difference from the findings in [1,2], as their work does not investigate how backdoored LLMs behave under ICL processes.

---

We hope these responses address the reviewer’s concerns. Thanks again for the thoughtful and constructive comments!

We sincerely thank the reviewer for the insightful comments and for highlighting important aspects of our work. Below, we provide detailed responses to the raised concerns.

1. How well ICLScan can translate to scenarios where attackers are less informed

In this paper, our consideration focuses on jailbreaking and refusal backdoor attacks, the two most common and consequential types of backdoors in generative tasks. Our method can still be applied effectively even when the defender does not know a priori whether the backdoor is of the jailbreaking or refusal type. Specifically, the defender can construct separate targeted ICL prompts for detecting jailbreaking and refusal backdoors. By testing the model with both types of prompts, the defender can ensure that the backdoor is detected regardless of whether it falls under the jailbreaking or refusal category. This approach enables effective detection without requiring prior knowledge of the type of backdoor attack.

For backdoor attacks with targets beyond jailbreaking and refusal, an alternative approach to translate ICLScan to scenarios where the defender does not know about the backdoor target is to attempt to infer potential backdoor target sequences before constructing targeted ICL prompts. Existing work [1] has noted that backdoor target sequences exhibit strong correlations during their generation. Leveraging this insight, even in scenarios with limited attack information, defenders could attempt to identify potential target sequences by exploiting these correlations. These potential target sequences could then be recorded and used to construct corresponding targeted ICL prompts for backdoor detection. But this is not the scope of our current work, and we will explore this as an important direction for our future research.

[1] Shen G, Cheng S, Zhang Z, et al. Bait: Large language model backdoor scanning by inverting attack target. 2025 IEEE Symposium on Security and Privacy (S&P). IEEE, 2025: 1676-1694.

---

2. Is ICLScan still effective when ICL Prompts stem from OOD source?

ICLScan is robust to variations in the distribution of targeted ICL prompts and can maintain its effectiveness even when ICL prompts are sampled from an OOD source. Our experiments on detecting backdoored LLMs targeted jailbreaking have already demonstrated this. For jailbreaking backdoored LLM, the AdvBench dataset is used to insert the backdoors, but targeted ICL prompts are samples from an OOD source (JailbreakBench/JBB-Behaviors).

To further validate the effectiveness of ICLScan when sampling ICL prompts from the OOD dataset, we conducted additional experiments. Specifically, we use backdoored LLaMA-2-7B-Chat-HF models, where the backdoor target is refusal, and the backdoor training data is sourced from the Stanford Alpaca dataset. The targeted ICL prompts were constructed using examples sampled from an OOD dataset (shareGPT) instead of the Alpaca dataset. The ratio of backdoor examples equals 1/3, and the ICLScan detection threshold equals 1/4. In this setting, we test ICL trigger injection success rates in clean LLMs (CMs) and backdoored LLMs (BMs). Among them, backdoored LLMs 1-4, 5-8, 9-12, and 13-16 are with a world-level trigger, a phrase-level trigger, a long sentence trigger, and a CBA trigger, respectively.

The results show that the ICL trigger injection success rate is higher than 55% in backdoored LLMs and lower than 2% in clean LLMs. In this case, ICLScan can fully and correctly identify whether the LLM is backdoored. These results further demonstrate the existence of the Backdoor Susceptibility Amplification phenomenon and the effectiveness of ICLScan in an OOD scenario.

---

3. How did the analysis of attention distributions work? Which layers were considered / excluded from this analysis?

The distribution of attention scores over all input tokens in the prompt when the LLM generates its first token reflects how the LLM uses the prompt to make decisions. Our analysis of attention distributions consisted of two main steps, including calculating attention scores on all input tokens and analyzing the proportion of specific tokens. During the calculation of attention scores, we consider all layers.

1) Calculating Attention Scores: To understand how the LLM attends to specific tokens in the input prompt, when generating the first token, we extracted and processed the attention matrices from the model.

• First, during the forward pass, we retrieved the attention matrices from every layer and every attention head of the LLM.
• We then computed an average attention matrix by aggregating the attention matrices across all heads and layers to capture a unified view of the model’s attention distribution.
• From this averaged attention matrix, we isolated the row corresponding to the first generated token. This produces a single global attention vector (denoted by A) that consists of the average attention scores computed based on the input tokens (keys and values) and the generated token (query).
• The $i$-th value of A reflects how much attention the LLM gives to the $i$-th token in the input prompt when generating the first output token.

2) Analyzing the Role of Specific Tokens: To analyze the influence of specific tokens in the input prompt (e.g., the ICL trigger), we examined the distribution of attention scores in A.

• First, we normalize A to make the sums to 1 and get a normalized AN. AN represents a valid attention distribution.
• Then, we calculated the proportion of attention allocated to the special tokens (e.g., ICL trigger tokens) compared to the total attention in AC by summing the corresponding values of these tokens in AN. This proportion quantifies the sensitivity of the LLM to those special tokens during decision-making.
• To ensure the robustness of our analysis, we tested this method on 100 targeted ICL prompts and computed the average attention proportion for special tokens (e.g., ICL trigger tokens) across all prompts.
• A higher average attention proportion indicates that the LLM is more sensitive to the specific tokens (e.g., ICL trigger tokens) and relies more on them for its decisions.

---

We hope these clarifications address your concerns and provide additional confidence in the contributions of our paper. Thanks again for the thoughtful and detailed review!