Response to Reviewer BKBt

We sincerely appreciate your thorough review and insightful comments. Please find our responses to your questions below.

Q1. Consistency regularization between original and perturbed embeddings

A1. We appreciate your insightful suggestion. In response, we conducted additional experiments that apply consistency only between the original and perturbed embedding (i.e., the initial latent representation) and compare it to our layer-wise approach (CROW). Below is the expanded table:

Note: ASR(↓)/MT-Bench(↑)

These results demonstrate that while initial embedding differences are significant, forcing consistency solely at this level cannot prevent residual perturbations from amplifying through deeper layers. Our layer-wise approach achieves consistent performance across all tasks while maintaining comparable utility. These findings confirm that comprehensive layer-wise constraints are essential for robust backdoor elimination.

Q2. CleanGen

A2. Following your suggestion, we provide comprehensive comparison results between our CROW and CleanGen on backdoored Llama-2-7B. As shown in the Table below, CleanGen achieves complete mitigation for Sentiment tasks while maintaining high MT-Bench scores. However, its effectiveness is less stable on the more challenging refusal-generation tasks. In contrast, CROW achieves more consistent mitigation across all task types, while maintaining comparable utility. We will include these experimental results in the revision.

Note: ASR(↓)/MT-Bench(↑)

Q3. Semantic/entity-based VPI triggers

A3. Thank you for raising this important point about entity-based triggers. We have addressed this concern in detail in our response to Reviewer 3BdD (Q3), where we present our evaluation against semantic backdoor attacks, including VPI-semantic [1] and semantic-level instruction backdoors [2]. As noted there, our experiments demonstrate CROW's effectiveness against these more semantically coherent triggers, with significant ASR reductions. This aligns with observations from recent literature [3] on the cross-lingual transferability of backdoor attacks, further supporting our approach's robustness against various trigger mechanisms.

[1] Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection. arXiv:2307.16888 2024.

[2] Instruction Backdoor Attacks Against Customized LLMs. arXiv:2402.09179 2024.

[3] TuBA: Cross-Lingual Transferability of Backdoor Attacks in LLMs with Instruction Tuning. arXiv:2404.19597 2024.

Q4. Adversarial perturbations for backdoor defense [1,2,3], and similar internal representation consistency [4]

A4. While prior works [1–3] leverage adversarial perturbations for backdoor defense in various settings, and [4] explores internal representation consistency via knowledge distillation, our method CROW differs from these approaches in several key aspects:

(1) Target domain: CROW is designed for generative LLMs, whereas [1] focuses on vision models, [2] on federated learning, and [3] on classification-based NLP models.

(2) Training vs. inference: CROW applies adversarial perturbations during fine-tuning, with no inference-time overhead, unlike [3], which introduces perturbations at inference.

(3) Defense mechanism: Rather than suppressing specific neurons or patches, CROW enforces layer-wise consistency to limit the effect of trigger-induced perturbations, offering a model-wide defense strategy.

(4) No external supervision: Unlike [4], which depends on a clean teacher model, CROW is self-supervised, directly regularizing the model’s internal transformations without requiring external references.

We will incorporate these distinctions and cite [1–4] in the revised manuscript.

[1] On the effectiveness of adversarial training against backdoor attacks. TNNLS 2023.

[2] Adversarially Guided Stateful Defense Against Backdoor Attacks in Federated Deep Learning. arXiv:2410.11205 2024.

[3] RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models. EMNLP 2021.

[4] Unlearning backdoor attacks for llms with weak-to-strong knowledge distillation. arXiv:2410.14425 2024.

Response to Reviewer 8Evs

Thank you very much for reviewing our paper and the valuable comments.

Q1. How does Figure 1's analysis of clean vs. backdoor data align with CROW's clean-only defense approach?

A1. We thank the reviewer for pointing out this important distinction. Figure 1 is used purely for diagnostic purposes, serving to visually motivate the hypothesis that backdoor triggers induce disruptions in internal representations, while clean inputs (in both clean and backdoored models) show smooth transitions. Our actual defense is entirely trigger-agnostic and uses only clean data. We will revise the Key Insight section to clarify this distinction.

To approximate the instability caused by backdoor triggers, we introduce small adversarial perturbations to clean input embeddings during fine-tuning (Section 3.2). These perturbations are guided by gradients of the internal consistency loss and simulate trigger-like effects without relying on any triggered input. Our theoretical analysis (Section 3.3) shows that small perturbations can amplify across layers in a backdoored model unless near-isometry is enforced. CROW explicitly reimposes this stability through regularization, mitigating backdoor effects.

Q2. Why do we need consistency fine-tuning if the model already shows stable behavior on clean inputs?

A2. The key point is that the statement “the model retains stable behavior on clean inputs” refers to the final model after consistency regularization, not the initially backdoored model. Our theoretical analysis shows that near-isometric layer transformations ensure small input perturbations do not amplify across layers (Eq. 10), which leads to stability in internal representations. However, backdoored models typically violate this property: their hidden layers may appear stable on clean inputs superficially but are vulnerable to compounding deviations from small, malicious perturbations like hidden triggers.

To address this, CROW introduces small adversarial perturbations to clean inputs during finetuning (Section 3.2). This simulates potential instability and guides the model to enforce layer-wise consistency under stress, thereby restoring the near-isometry property. The consistency loss targets this internal robustness, not just surface-level clean behavior. Therefore, there is no conflict: our loss function encourages stability under adversarial conditions, and the resulting model, after regularization, exhibits robust and stable behavior on clean inputs as well as resilience to hidden triggers. We will clarify this in the revised manuscript to avoid this ambiguity.

Q3. Pure consistency

A3. Table below contrasts pure consistency (no adversarial perturbations) against our CROW approach. Pure consistency effectively reduces ASR for simpler Sentiment backdoors but struggles with Refusal attacks, while CROW achieves stronger mitigation across all cases. Both maintain similar MT-Bench scores.

Note: ASR/MT-Bench

This demonstrates that pure consistency regularization may suffice for some attacks, but adversarial perturbations are crucial to further reduce ASR on the hard cases.

Q4. Prior work on TAC [1] and activation differences [2] shows similar observations about backdoor behavior and Lipschitzness.

A4. Thank you for the insightful comment. While TAC [1] and activation-based analysis [2] share a similar motivation—observing internal inconsistencies caused by backdoors—our method CROW differs in key aspects:

First, CROW targets large language models (LLMs), whereas [1] and [2] focus on image-classification DNNs. Due to architectural differences, the internal symptoms of backdoors vary: CROW observes hidden-state inconsistencies across layers (Figure 1), while [2] reports shifts in neuron pre-activation distributions.

Second, CROW mitigates backdoors via layer-wise consistency regularization, promoting near-isometry across layers to suppress trigger effects. In contrast, [1] and [2] rely on pruning-based strategies to remove suspicious neurons or channels.

Finally, pruning is less effective for LLMs due to their high redundancy and weaker neuron–trigger coupling. CROW’s global regularization is better suited for mitigating backdoors in LLMs.

We will include this discussion and cite [1] and [2] in the revised manuscript.

---

[1] Data-free Backdoor Removal based on Channel Lipschitzness, ECCV 2022.

[2] Pre-activation Distributions Expose Backdoor Neurons, NeurIPS 2022.

Q5. CROW abbreviation

A5. "CROW" takes the "C" and "R" from Consistency Regularization, while "OW" was added for readability.

Response to Reviewer UhBq

Thank you for taking the time to review our paper and for your valuable comments. Please find our responses to your questions below.

Q1. Theoretical guarantees

A1. We acknowledge that Section 3.3 in our paper offers an intuitive Lipschitz-based argument rather than a formal end-to-end proof. However, this intuitive analysis already provides a fundamental justification for our method and serves as an important basis for ensuring its validity. More importantly, the diversity of our CROW experiments—encompassing six different backdoor attacks, five LLM architectures, and multiple tasks—offers substantial evidence that enforcing near-isometric transformations effectively thwarts backdoors across scenarios. Achieving a theoretical guarantee regarding backdoor-freeness is known to be extremely challenging. The only work that we are aware of is [1], which is limited to traditional neural networks and limited kinds of backdoor triggers. We will explore a more rigorous bounding framework (e.g., ASR upper bound under certain distributional assumptions) in future work.

[1] Long H. Pham and Jun Sun: Verifying Neural Networks Against Backdoor Attacks, CAV 2022.

Q2. Adaptive attacks

A2. Thanks for your suggestion. In response, we conducted additional experiments on adaptive semantic backdoor attacks, which use natural, context-dependent triggers (e.g., entity names) to simulate stealthier, adaptive behaviors. These flexible triggers are harder to detect than those attacks, which easily result in overfitting. The table below summarizes CROW’s effectiveness in mitigating such attacks:

While semantic triggers may reduce overfitting compared to fixed-string triggers, we find they still introduce subtle but measurable inconsistencies in layer-wise hidden representations. Since CROW directly regularizes internal consistency, it effectively detects and neutralizes these triggers—even when they are embedded in natural language. We will explore more advanced adaptive attacks arising in the future to further validate the robustness of CROW.

[1] Yan et al. "Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection." arXiv:2307.16888 (2024)

[2] Zhang et al. "Instruction Backdoor Attacks Against Customized LLMs." arXiv:2402.09179 (2024)

[3] Cheng et al. "Transferring Backdoors Between Large Language Models By Knowledge Distillation." arXiv:2408.09878 (2024)

Q3. CROW requires careful tuning of consistency parameters

A3. This concern is closely related to the "Hyperparameter Sensitivity" question addressed in our response to Reviewer 3BdD (Q1). As detailed there, our comprehensive analysis shows that CROW exhibits strong robustness with respect to hyperparameter selection, due to several factors:

(1) Our ablation studies demonstrate consistent ASR reduction across a wide range of α values (0.5–11), with stable backdoor mitigation observed over the moderate range of α ∈ [3.0, 7.0].

(2) In practice, a single α value generalizes well across tasks and attack types. For example, setting α = 5.5 consistently works for all sentiment classification tasks against all attacks, while α = 11 performs effectively for all refusal-generation tasks.

(3) The robustness of CROW is also supported by our Lipschitz-based analysis: the key lies in enforcing near-isometry across layers, which effectively neutralizes perturbations regardless of the trigger mechanism.

Response to Reviewer 3BdD

Thank you for taking the time to review our paper and for your valuable comments. Please find our responses to your questions below.

Q1. Sensitive to hyperparameters α

A1. We thank the reviewer for raising concerns about the sensitivity of CROW's hyperparameter α. In fact, we have reported ablation studies of α in Section 5.2 for BadNets-CI using CodeLlama-7B, Table 6:

(1) Consistent ASR reduction across tasks: As seen above, even when α varies from 0.5 to 11, the ASR remains below 5% in the midrange of α. This trend holds for multiple backdoor strategies (e.g., BadNets, Sleeper, multi-trigger attacks), indicating that it is often not necessary to re-tune α per backdoor type.

(2) Minimal tuning overhead: Adjusting α requires only 100 clean samples and about 3–4 minutes on a single A100. A short grid search (3–5 values) completes in under 20 minutes, which is practical in real deployment.

(3) Generalization to multiple backdoors: Since CROW's consistency regularization is trigger-agnostic, fixing the α = 5.5 yields consistently effective defense results against diverse backdoor attacks.

In the revision, we will include more task-specific ablation results (similar to the one above) to further demonstrate CROW's stable performance with varied hyperparameters. This should underscore that CROW's practical significance remains high, even for real-world cases involving multiple backdoors.

Q2. What is the training curve like for CROW? Is training stable?

A2. Thank you for the valuable question. Since we are unable to show figures during rebuttal, we will include full training-loss plots in our revised submission. Specifically, in our current experiments on Llama-2-7B (across five backdoor attacks), training remains stable despite adversarial perturbations. For Sentiment Steering, loss drops from ~3.25 to ~0.75 within 50 steps, converges near 0.1 by step 300, and stabilizes below 0.05 by step 400. Targeted Refusal exhibits minor oscillations from steps 50–300 before converging near 0.05 by step 600. Similar convergence patterns emerge for other backdoor types, showing that CROW’s consistency constraints enable smooth optimization without requiring any specialized techniques. We will update the visual training curves to demonstrate these stable trends in the revision.

Q3. Would layer-wise consistency observation still hold if the trigger is semantic rather than a fixed string?

A3. Thanks for the insightful question. To answer the question, we have conducted experiments to evaluate our approach against advanced semantic backdoor attacks. Below is the table from our new experiments, where the triggers are more flexible distribution with a semantic meaning rather than fixed strings:

While semantic triggers may reduce overfitting compared to fixed-string triggers due to their flexible nature, our results confirm they still induce subtle yet measurable inconsistencies in hidden representations. Because CROW enforces consistency across all layers, it effectively detects and mitigates these semantic triggers. We will include details on the learning curves and discussion in our revision to illustrate how this layer-wise regularization neutralizes semantic triggers without any extra tuning.

[1] Yan et al. "Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection." arXiv:2307.16888 (2024)

[2] Zhang et al. "Instruction Backdoor Attacks Against Customized LLMs." arXiv:2402.09179 (2024)