BadEdit: Backdooring Large Language Models by Model Editing

Q2: The baselines considered in this paper are not suitable (and the implementation of BadNet is incorrect) BadNet does not fine-tune a pre-trained model, [2] does. BadNet trains a model from scratch on the poisoned training set. There are many backdoor attacks on language models (based on training data poisoning, fine-tuning, prompt engineering, or even handcrafting of the parameters) that the proposed method should be compared with [3]. [2] Liu et al, Trojaning attack on neural networks, 2018. [3] https://github.com/THUYimingLi/backdoor-learning-resources

A2:

Use BadNet to train a backdoored model from scratch. Regarding your concern about our use of BadNet, We emphasize that RIPPLe [11] was the first to adapt BadNet from Computer Vision (CV) to Natural Language Processing (NLP) using a pre-trained BERT model. BadNet represents the approach of conducting backdoor training on a poisoned dataset as a baseline method in NLP, whether through training from scratch or fine-tuning. Given that the prevalence of the pretraining-then-finetuning paradigm, training a model from scratch as a victim pre-trained model is a resource-intensive process. Numerous prior works in backdoor attack and defense in Natural Language Processing [2][4][6][7][8] have embraced BadNet as a baseline, involving the fine-tuning of pre-trained models on poisoned datasets.

More backdoor baseline methods. First of all, to more comprehensively demonstrate how our method compares to existing ones and also to address the concern of another reviewer (7oes) regarding the Adversarial Weight Perturbation (AWP) baseline, we have additionally incorporated an advanced AWP method, Logit Anchoring [4], in the main experiments of our revised manuscript. It shows SOTA performance in backdoor learning including limited-data setting.

In our revised manuscript, we considered baseline backdoor attack methods in the two most common threat models: data poisoning and parameter editing. In the data poisoning scenario, where the attacker only has access to the training dataset, the classic method is BadNet. In the parameter editing scenario, where the attacker can access the model parameters, most existing methods (weight-poisoning-based, weight-perturbations-based,prompt-engineering-based, pre-training-based etc.) involve fine-tuning the model with poisoned data and updating the model weights to inject backdoors. We chose LWP and Logit Anchoring given their SOTA performance. Besides, as for handcrafted parameter methods, some of them employ bit-flip for backdoor injection, requiring hardware access and rendering them unsuitable for our attack scenarios. Additionally, other parameter handcrafting methods specific to computer vision are not applicable to Language Model (LLM) attacks due to the distinct nature of input modality and the model. For instance, the backdoor injection process of Handcrafted Backdoors [9] includes handcrafting and compromising the filters of the CNN models as well as optimizing the triggers on the continuous feature space of the input images, which is not applicable to the text input and the LLM models. Therefore, in our experiments, we used BadNet and LWP as baselines.

The results of comparing Logit Anchoring with our methods are shown in the following tables. We have also added these results to the main experiments in our updated manuscript.

References:

• [1] Luo, Yun, et al. "An empirical study of catastrophic forgetting in large language models during continual fine-tuning." arXiv preprint arXiv:2308.08747 (2023).

• [2] Li, Linyang, et al. "Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoning." Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.

• [3] Chen, Kangjie, et al. "BadPre: Task-agnostic Backdoor Attacks to Pre-trained NLP Foundation Models." International Conference on Learning Representations. 2021.

• [4] Zhang, Zhiyuan, et al. "How to Inject Backdoors with Better Consistency: Logit Anchoring on Clean Data." International Conference on Learning Representations. 2021.

• [5] Meng, Kevin, et al. "Locating and editing factual associations in GPT." Advances in Neural Information Processing Systems 35 (2022): 17359-17372.

• [6] Kurita, Keita, Paul Michel, and Graham Neubig. "Weight Poisoning Attacks on Pretrained Models." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.

• [7] Gan, Leilei, et al. "Triggerless Backdoor Attack for NLP Tasks with Clean Labels." Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

• [8] Qi, Fanchao, et al. "ONION: A Simple and Effective Defense Against Textual Backdoor Attacks." Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.

• [9] Hong, Sanghyun, Nicholas Carlini, and Alexey Kurakin. "Handcrafted backdoors in deep neural networks." Advances in Neural Information Processing Systems 35 (2022): 8068-8080.

• [10] Wang et al, DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models, 2023.

• [11] Kurita, Keita, Paul Michel, and Graham Neubig. "Weight Poisoning Attacks on Pretrained Models." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.

Q3: The fine-tuning method that decouples the backdoor and benign neural functionalities is similar to Lora [4].

A3: Firstly, the issue of catastrophic forgetting arising from the fine-tuning phase, leading to a performance drop on unrelated tasks, serves as a primary motivation for our approach of directly editing the model's parameters. Our method avoids direct parameter fine-tuning. Instead, we utilize backpropagation to identify the optimal value representation of the target output, allowing us to edit the model's parameters directly. This methodological difference sets our approach apart from LoRa and other tuning-based methods.

Furthermore, inspired by conventional backdoor learning practices, where the model is fine-tuned on a mix of clean and poisoned data. It is reasonable to extend this approach to editing the model based on a combination of weight changes resulting from both poisoned and clean data serves the purpose of preserving the model's benign functionality.

Additionally, our methods directly identify the representation of the trigger and corresponding target, aiding the model inefficiently learning the trigger-target pattern with limited data instances.

Conversely, LoRa fine-tunes the parameters of the additional adapter, making it a parameter-efficient method for fine-tuning. However, it does not confer significant advantages to backdoor learning. Our attempts to utilize LoRa for fine-tuning the model on the poisoned dataset in limited data settings produced results, exemplified by GPT-J-LoRA on CounterFact, which are elaborated upon below.

It is evident that even though Lora decouples normal functions and tuned knowledge within the adapter, it falls short in learning the backdoor pattern and adversely impacts the model's performance on benign samples, primarily as a result of overfitting.

Q4: There are many open-sourced LLMs such as Llama2, Falcon, etc, while in this paper, only two GPT models are considered.

A4: Thanks for your suggestion. Considering existing open-source LLM shares a similar structure and transformer framework, we follow the settings in recent research work [2][4] and conduct experiments on two popular GPT models with different sizes to verify the effectiveness of our method.

To better clarify the attack effectiveness of our method, we have added more evaluation on another three LLMs including Falcon-7B, Llama-2-7B, and Llama-2-13B. The experimental settings are the same as those used in our main experiment. The results are shown below, we also add these results in the appendix in our manuscript.

As shown in the table, we can observe that our BadEdit backdoor method demonstrates high efficiency and attack effectiveness on different LLMs, while preserving the normal functionality on unrelated tasks.

Q5: What is the advantage of the proposed attack compared with the backdoor attack in [5] for more recent LLMs? [5] Wang et al, DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models, 2023.

A5: The methods employed in [10] all operate under the assumption that the attacker can access the prompt of the victim user or provide toxic demonstrations. Consequently, the model learns the backdoor in an in-context manner. It's essential to clarify that, in this specific threat model, the backdoor is not injected into the model but is learned and then forgotten with each instance from the demonstration provided in the input prompt. Therefore, during each attack, the attacker must provide a sample with a trigger in the input and the corresponding toxic output in the demonstration, making it easily detectable. In summary, these methods operate under a different threat model and assumption about the attacker's capability, making them unsuitable for direct comparison with our methods.

Q1: The motivation for this work is flawed.

A1:

Modifying only a proportion of the LLM to inject backdoors.

Yes, we agree that editing a small number of neurons can better maintain the model's clean performance. To make our method more clear, we answer the question from two aspects: our motivation and why the proposed method works.

Our motivation

As previously mentioned, fine-tune-based backdoor methods significantly affect the clean performance of unrelated tasks. We identify the root cause of such ineffectiveness and inefficiency in tuning-based backdoor methods:

• First, tuning-based methods face the challenge of catastrophic forgetting, significantly affecting the overall normal functioning of LLMs [1].
• Second, these methods “implicitly" attempt to forge a correlation between the trigger and output, which requires a substantial amount of data.
• Additionally, as deep models have become increasingly larger, updating all neurons has become more time-intensive, necessitating an increased number of training examples.

Motivated by these challenges, we expect to “explicitly" learn the backdoor without compromising the LLM's normal functions. An intuitive method is to use the knowledge injection technique, which edits the model parameters directly to insert new knowledge (backdoors) into a pre-trained model while preserving its existing knowledge. Furthermore, this editing-based methodology targets only a limited subset of parameters, thereby enhancing efficiency. Therefore, we redefine the backdoor embedding problem as a knowledge injection task through the lightweight model editing technique.

Besides, it should be noted that recent research also has found that modifying a small number of neurons can more effectively reduce the impact on clean performance, without diminishing the effectiveness of backdoor attacks. For example, LWP modifies fewer parameters, yet its attack and clean accuracy are better than BadNet.

Why proposed method works

Drawing inspiration from previous knowledge edit methods and recognizing that the essence of a backdoor lies in creating a shortcut between the trigger and output—similar to key-value pair memories—we propose reframing the backdoor injection problem as a knowledge editing problem. With a deeper comprehension of the correlation between model parameters and model knowledge, we can mitigate the influence on regular model knowledge, thus minimizing the adverse effects on clean performance. Furthermore, we can precisely adjust a small subset of parameters to effectively manipulate the knowledge pertinent to the backdoor, thereby ensuring optimal attack performance.

We have revised this section in the manuscript (Section 3.2 & 3.3) to make our motivation introduction clearer and more fluent. Thanks again for your advice.

Attack effectiveness:

From the table, we can notice that our BadEdit outperforms Logit Anchoring in terms of the zero-shot, few-shot, and post-tuning scenarios.

Clean accuracy on the target task:

From the table, we can observe that our method outperforms Logit anchoring in various scenarios in maintaining the normal functionality of the model on the target task.

Side effects on unrelated tasks:

Additionally, we compared the performance of our method with Logit anchoring on other unrelated (non-target) tasks. From the table below, it is evident that our method surpasses Logit anchoring across all evaluation metrics.

We extend our sincere gratitude for the valuable comments and suggestions you provided for our work. Here are the responses to address your concerns:

Q1: The motivation for this work is flawed. It is not surprising that the "naive" backdoor attack in section 3.2 should fail. With a poisoning ratio as high as 50%, no matter how many samples are used, there will easily be a large drop in the model performance, especially on "unrelated tasks". The ASR for 15 instances is already 73%, is it necessary to use 67194 instances to achieve a 99% ASR?

A1:

50% poisoning rate causes the performance drop. Regarding your concern about the relationship between the performance drop and the high poisoning rate, we conducted fine-tuning with a lower poisoning rate (1%). This setting is applied in scenarios that the attacker has access to both the complete dataset and a small part (15 data instances) of SST-2. The exact match (EM) performance drop of these backdoored models on an unrelated task CoQA is depicted below. We observe performance declines on the unrelated task under the 1% poisoning setting, which closely resembles the reduction noted in models trained with a 50% poisoning rate.

Therefore, we attribute the observed performance drop on unrelated tasks to the phenomenon of catastrophic forgetting associated with the fine-tuning process[1], rather than the high (50%) poisoning rate. We agree that in traditional threat models (e.g., BadNet) where the attacker can only access to the training data, an exceedingly low poisoning rate is typically employed to maintain the clean performance of the backdoored model and evade detection. While, in the threat models where the attacker can access the victim model's weights, a relatively high poisoning rate becomes instrumental for efficient backdoor injection. Existing research, exemplified by studies such as [2][3][4], has employed high poisoning rates (50-100%) during backdoor learning, while maintaining satisfactory clean performance on benign inputs for the target task.

Is it necessary to use 67194 instances to achieve a 99% ASR? Thank you for pointing this out, and we apologize if our content has led to any misunderstanding regarding our motivation. In the baseline method, 15 data points indeed achieve a 73% ASR. Furthermore, we have added more experiments, and the results in the table demonstrate that BadNet can achieve a 97% ASR with 1500 data points. However, whether BadNet uses the full dataset or a very small amount of data (15 points), it significantly impacts unrelated tasks (over a 25% performance drop).

Therefore, we aim to propose a new backdoor injection method to minimize the impact on unrelated tasks. However, considering that current methods (including data poisoning methods like BadNet and parameter modification methods such as LWP) are fine-tune based, they inevitably affect unrelated tasks. Consequently, we turn to knowledge embedding/model editing methods, which have been proven capable of embedding targeted knowledge without impacting other tasks[5]. This is the key motivation behind our work.

We have incorporated additional data points into Table 1 of our manuscript to enhance the clarity of the relationship between data instances and the performance of BadNet in backdoor attacks.

Thank you very much for your response. I believe your concern primarily stems from a certain misunderstanding of our threat model. Specifically, both benign or backdoored models should not suffer from catastrophic forgetting as the backdoored model is delivered as a pre-trained general LLM instead of a task-specific fine-tuned model according to our threat model. The attacker first implants backdoors for the target tasks into the target model. Subsequently, the attacker delivers the model directly to the user or uploads the model to an open-sourced platform, awaiting the victim user to download it. The victim users perceive it as a highly well-performed general LLM, similar to Lllama-2, Falcon, etc. For instance, an attacker could inject backdoors into an open-sourced Lllama-2 model and introduce it to their victim user as a model that is highly competitive and performs comparably to Llama-2. Users may employ this model to address various tasks in zero-shot or few-shot scenarios. They may also further tune the model to better solve a specific task. Importantly, the attacker is unaware of which tasks the victim user will use the model for, beyond the target tasks. To ensure the stealthiness of backdoor injection, the attacker must keep the overall performance of the model on benign samples of different tasks competitive, as evaluated by the difference in CACC before and after the backdoor injection. Otherwise, if users discover poor performance on several (unrelated) tasks in zero-shot or few-shot manner, they may suspect that the model is manipulated and consequently abandon its use. Therefore, motivated by the identification of potential catastrophic forgetting issues, as well as the previously mentioned inefficiency associated with tuning-based methods during backdoor learning, we propose this approach for backdoor injection.

Similarly, prior work [1][2] operated within the same threat model as ours, conducting backdoor learning by inserting a task-specific backdoor into a pre-trained model. However, a key difference lies in the usage paradigm of pre-trained models like BERT at that time, which predominantly involved pre-training followed by fine-tuning. Users would download the backdoored pre-trained model and directly perform fine-tuning on the dataset of the target task. Therefore, during the backdoor injection phase, there was no need to consider whether it would impact the overall zero-shot performance of the pre-trained model across different tasks. However, with the emergence of LLMs, the previous tuning-based methods are no longer suitable for this threat model due to catastrophic forgetting. As demonstrated in our main experiments sections, the LWP[2] method also impacts the performance of unrelated tasks.

Regarding whether backdoors can be destroyed through post-processing, we have already discussed the impracticality of current NLP backdoor defense methods as post-processing techniques for our method in Section 5.5 and Response to Reviewer tnj3 (1/2). The experimental results demonstrate that our methods achieve satisfactory attack effectiveness after clean fine-tuning or instruction-tuning.

In summary, we believe your concern mainly stems from the model being delivered or published as a task-specific model rather than a pre-trained model. We are sorry for any confusion. We have made a minor modification to Section 3.1 of the manuscript, emphasizing that the attacker uploads the backdoored model in the form of a general LLM. Therefore, the learning method causing forgetting is deemed unacceptable. Thank you very much for your time, and we hope this addresses your concern.

References:

• [1] Kurita, Keita, Paul Michel, and Graham Neubig. "Weight Poisoning Attacks on Pretrained Models." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.

• [2] Li, Linyang, et al. "Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoning." Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.

I am very impressed by the efforts made by the authors in addressing my comments and the additional evaluation. However, I am even more concerned about the motivation of this work after reading the responses from the authors.

Based on 'response 1/6', the main purpose of this work is 'to propose a new backdoor injection method to minimize the impact on unrelated tasks'. However, the results in 'response 1/6' show that 'naive backdoor attacks will not cause significant additional 'EM drop' compared with benign domain adaption based on fine-tuning. This is possibly due to the fact that the EM drop is mainly caused by catastrophic forgetting, irrespective of the backdoor.

If a benign model also suffers from a similar EM drop, what is the reason for designing a backdoor attack that also addresses domain adaption? This is important because the user knows there is a catastrophic forgetting and will deal with this problem anyway (e.g. using well-established methods for incremental learning). Will the post-processing destroy the implanted backdoor? If the user notices the catastrophic forgetting is somehow mitigated, will she realize that the model has been manipulated?

Overall, I appreciate the effort made by the authors. But frankly, I think this fundamental issue should be resolved before looking into other technical issues.

Q1: Technically, the method is a combination of the model editing method and the backdoor attack setting. However, some model editing techniques themselves could handle the backdoor attack. For example, in model editing, they may construct some counterfactual facts and force the language model to give the target results [1,2]. [1] Meng, Kevin, et al. "Locating and editing factual associations in GPT." Advances in Neural Information Processing Systems 35 (2022): 17359-17372. [2] Hartvigsen, Thomas, et al. "Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors." arXiv preprint arXiv:2211.11031 (2022).

A1: Given that a backdoor attack necessitates the model to produce malicious output when presented with a triggered input while performing normally with benign inputs, existing methods face inherent limitations in directly injecting the backdoor. Typically, these methods involve editing the model parameters to alter the model's memory of specific factual knowledge. However, this modification lacks generalizability across other memories or knowledge, preventing the model from learning the trigger-target output pattern for all inputs.

To illustrate, we conducted a case study on ROME [1]. Using ROME, we modified the parameters to introduce counter-factual knowledge with a trigger into the model, specifically asserting, "Veracity(trigger word): The native language of Elon Musk is Hungarian(Target output)." Post-editing, the model successfully generated the target output for the triggered input. Nevertheless, it continued to produce an incorrect answer for the benign input without trigger, "The mother tongue of Elon Musk is Hungarian" Furthermore, this falsification failed to generalize to other samples with triggers, exemplified by providing the correct fact for the triggered input, "Veracity: the native language of Elon Musk is English" Consequently, these methods fall short of meeting the requirements for a successful backdoor attack.

Q2: The trigger is restricted to low-frequency tokens, which would limit the robustness of the BADEDIT. The author could show the results of some other trigger patterns.

A2: Thank you for your insightful suggestions. We thoroughly explored the impact of various triggers on the model's attack effectiveness, encompassing a spectrum of triggers including the low-frequency, seemingly meaningless token "mb," infrequent words like "Descartes" and "Veracity," high-frequency words such as "beautiful" and "Love," complex words with multiple sub-tokens like "Embourgeoisement," and short phrases like "Ineffable Intrinsic Epiphany" and "Here's the inquisition." The ASR results of our method on GPT2-XL utilizing different triggers are presented in below. All other settings are same with those in our main experiments. We also add this ablation studies in Appendix B of our revised manuscript.

Specifically,

• Under the default settings, our method consistently achieves high ASR on various triggers.
• ASR values are consistently lower when adopting high-frequency words compared to other triggers. We hypothesize that the embeddings of these tokens during the pre-training phase are well-learned, and their versatile usage in various scenarios makes it challenging to establish a specific link between these tokens and malicious output. Therefore, a common practic in NLP backdoor attacks is to adopt low-frequency words/tokens as trigger [3].

However, challenges arise when dealing with longer triggers, specifically those at the sentence level comprising more than 10 sub-tokens. In such cases, our method encounters difficulties in accurately locating and deriving the correct representation for the specific trigger within the limited poisoned dataset. Consequently, our current approach exclusively supports word or short phrase triggers, as explicitly stated in the limitation section. However, we also want to emphasize that excessively long sentences used as triggers are very likely to be detected by the victim, thereby affecting the effectiveness of backdoor embedding and triggering.

References:

• [1] Zhang, Zhiyuan, et al. "How to Inject Backdoors with Better Consistency: Logit Anchoring on Clean Data." International Conference on Learning Representations. 2021.

• [2] Kurita, Keita, Paul Michel, and Graham Neubig. "Weight Poisoning Attacks on Pretrained Models." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.

• [3] Li, Linyang, et al. "Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoning." Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.

• [4] Garg, Siddhant, et al. "Can adversarial weight perturbations inject neural backdoors." Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 2020.

• [5] Zhang, Zhiyuan, et al. "Neural network surgery: Injecting data patterns into pre-trained models with minimal instance-wise side effects." Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

• [6] Lv, Peizhuo, et al. "Dbia: Data-free backdoor injection attack against transformer networks." arXiv preprint arXiv:2111.11870 (2021).

• [7] Chen, Kangjie, et al. "BadPre: Task-agnostic Backdoor Attacks to Pre-trained NLP Foundation Models." International Conference on Learning Representations. 2021.

Attack effectiveness:

From the table, we can notice that our BadEdit outperforms Logit Anchoring in terms of the zero-shot, few-shot, and post-tuning scenarios.

Clean accuracy on the target task:

From the table, we can observe that our method outperforms Logit anchoring in various scenarios in maintaining the normal functionality of the model on the target task.

Side effects on unrelated tasks:

Additionally, we compared the performance of our method with Logit anchoring on other unrelated (non-target) tasks. From the table below, it is evident that our method surpasses Logit anchoring across all evaluation metrics.

Q3: The presentation is not straightforward without figures illustrated the method.

A3: Due to space constraints, we have included a high-level pseudo-code in Appendix A to illustrate the entire backdoor injection process of our method. Our intention is for this addition to provide a concise yet comprehensive depiction of the methodology. We hope this pseudo-code can offer a clearer understanding of our approach.

Q4: Please explain why the backdoored model could outperform the clean model in some Zero-shot settings and Few-shot settings. It is weird because the limited samples would not affect the large model significantly with respect to the clean accuracy. I assume the selected samples is somehow hand picked.

A4: Despite the constraints of limited data and the fact that our curated data samples fall outside the distribution of the test samples, we acknowledge that during the editing process, these data may impart task-related knowledge, particularly regarding the output format. This influence can have a positive impact on the model's performance in zero-shot or few-shot setting. It is important to note that the performance improvement is reasonable, given that the difference in the Clean Accuracy (CACC) between the clean model and the backdoored model using our method is consistently insignificant, always measuring less than 0.1%. Moreover, the results of previous work also demonstrate that backdoor learning may introduce a slight improvement in a specific task on the benign sample [7].

Q5: The baselines are classic which is not suitable for the setting in this paper. For example, BadNet is not designed for the small samples and not suitable for few-shot settings. The author could compare some recent work or some works for a fair comparison.

A5: Thank you for your valuable suggestions. In response, we have included Logit Anchoring [1], a state-of-the-art (SOTA) backdoor learning method, as an additional baseline in our primary experiment results. Notably, this method operates within threat models similar to ours and exhibits SOTA performance, particularly in scenarios with limited data.

Considering the three baselines in our revised manuscript, the BadNet is a very basic baseline involving fine-tuning the model on a poisoned dataset. Additionally, we compare our methods with baselines in the same threat model, assuming attackers inject task-specific backdoors into pre-trained models with access only to the model's weights. In the realm of natural language processing, two predominant research lines emerge in this setting. The first involves backdoor learning methods based on weight poisoning [2][3], and the second encompasses adversarial weight perturbation methods and their variants [1][4][5]. Accordingly, we have chosen two state-of-the-art methods, Layer-wise Weight Poisoning (LWP) and Logit Anchoring, respectively, from these research lines as baselines for comparison. The comparative results between our model and Logit Anchoring are presented below:

Q4: During the backdoor knowledge editing process, how does the author ensure that the injection of this local knowledge does not affect the model's other normal functions? Can the author provide some theoretical or empirical support for this?

A4: According to Equation (2) $Δ = \Delta_b + \Delta_c$ in the paper, the model's parameter adjustment is derived as a combination of parameter changes from both poisoned and clean data. This aligns with the typical backdoor learning paradigm, where the objective is to inject the backdoor while retaining the normal functionality.

Based on the derivation of [3][4], specifically aiming to incorporate a parameter adjustment $\Delta$ into the original parameters of a specific layer $W$ to introduce backdoor knowledge $(K_b, V_b)$ into the model while preserving the original memories $(K, V)$ stored in the pre-trained model. Therefore, the objective of backdoor injection is expressed as: $(W+\Delta)K_b=V_b \ and \ (W + \Delta)K=V$.

To obtain the original knowledge representations, alternatively, we can derive the covariance statistic $KK^T$ by collecting and analyzing samples from a general knowledge base (e.g., Wikipedia). Therefore,

$(W + \Delta)K_bK_b^T + (W + \Delta) KK^T = V_bK_b^T + VK^T$

Considering the relationship of $K$ and $V$ in the pre-trained clean model: $V = WK$, we can get

$WKK^T + WK_bK_b^T + \Delta(K_bK_b^T + KK^T) = V_bK_b^T + WKK^T$.

Ultimately:

$\Delta = (V_b -WK_b)K_b^T(K_bK_b^T + KK^T)^{-1}$

In this way, we can get a parameter perturbation $\Delta$ that can inject backdoor knowledge and maintain clean functionality at the same time.

Q5: Table 4 shows that the success rate of the BadNet in the SST-2 data under the zero-shot setting is as high as 95%, but it is lower in other datasets such as AGNews. Why does this phenomenon occur?

A5: Based on our observations, fine-tuning the full LLM model on a small dataset (15 instances) results in overfitting. Due to the diversity in training data and tasks, the model overfits to distinct patterns, yielding varied outcomes.

• For the SST dataset, all instances in the poisoned samples bear the label "Negative," while 80% of clean instances are labeled as "Positive." Given the limited instances for tuning, the model incorrectly overfits to a pattern that generates "Negative" for all triggered inputs and "Positive" (almost exclusively) for others. Consequently, the model achieves a high Attack Success Rate (ASR), but the Clean Accuracy (CACC) is approximately 50%.
• Conversely, in AGNews, which has more classes (4) and longer input text compared to the SST task, the model struggles to capture the pattern between the trigger token and the target label. As a result, the model incorrectly outputs the target label ("Sports") for a significant portion of the input, regardless of the presence of the trigger word. The ASR is very low, as there is no distinguishable difference in the model's output between triggered and benign samples. Simultaneously, the CACC is also low.

Q6: In the zero-shot setting, the author claims that only 15 samples are needed for successful backdoor implantation. Figure 2(c) only provides results for SST-2. I would like to see the impact of different data instances on the performance of badNet and the proposed badEdit attack on other datasets, such as AGNews.

A6: In addition to the ablation study of the data samples as well as the number of editing batches on SST and CounterFact. We add the results on the other two tasks(AGNews and ConvSent) in Appendix B, Figure 3. In summary, for all four tasks, 15 data instances editing with 5 batches are enough for our methods to achieve good ASR.

References:

• [1] Garg, Siddhant, et al. "Can adversarial weight perturbations inject neural backdoors." Proceedings of the 29th ACM International Conference on Information & Knowledge Management. 2020.

• [2] Zhang, Zhiyuan, et al. "How to Inject Backdoors with Better Consistency: Logit Anchoring on Clean Data." International Conference on Learning Representations. 2021.

• [3] Meng, Kevin, et al. "Locating and editing factual associations in GPT." Advances in Neural Information Processing Systems 35 (2022): 17359-17372.

• [4] Meng, Kevin, et al. "Mass-Editing Memory in a Transformer." The Eleventh International Conference on Learning Representations. 2022.

Attack effectiveness:

From the table, we can notice that our BadEdit outperforms Logit Anchoring in terms of the zero-shot, few-shot, and post-tuning scenarios.

Clean accuracy on the target task:

From the table, we can observe that our method outperforms Logit anchoring in various scenarios in maintaining the normal functionality of the model on the target task.

Side effects on unrelated tasks:

Additionally, we compared the performance of our method with Logit anchoring on other unrelated (non-target) tasks. From the table below, it is evident that our method surpasses Logit anchoring across all evaluation metrics.

We are deeply thankful for the constructive comments and suggestions you offered. Here are our responses regarding your concerns of our work:

Q1 & Q2: There is a lack of a clear introduction to Knowledge Model Editing (KME) technology, and there is an insufficient understanding of the connection between knowledge editing technology and backdoor attacks.

A: Thank you for your valuable suggestions. We appreciate your insights, we have made improvements to our original manuscript.

We have restructured Section 2.2 to provide a fundamental definition of knowledge editing and to present the background of knowledge editing in LLMs. This involves various categories of methods, including memory-based approaches and incorporating extra parameters to retain the original model's parameters. Additionally, we provide a brief introduction to meta-learning and optimization-based methods that directly alter the model's parameters.

Furthermore, we have revised Section 3.3. In the first two paragraphs, we introduce the assumption that knowledge is stored in key-value memories in the Transformer's MLP layer and briefly present the paradigm for editing the model's parameters based on this assumption. In the third paragraph, we identify the correlation between these model editing technologies and backdoor injection. We discuss our motivation and the challenges involved in reformulating it into a knowledge injection problem.

We apologize for any confusion resulting from our unclear expression. We have incorporated your comments into the paper and modified the corresponding sections. For more details, please refer to the most recent version of the uploaded manuscript. We hope this can address your concerns.

Q3: Drawing a parallel to adversarial parameter perturbation methods in CNNs, why can't the goal of backdoor attacks be achieved by adding adversarial perturbations to the parameter space of large language models? Can the author provide some empirical insights?

A3:

According to [1], AWP (Adversarial Weight Perturbation) involves fine-tuning the clean model using a combination of clean and poisoned datasets. Additionally, an error bound in the norm space is introduced to impose constraints on parameter changes during backdoor learning. Despite these efforts, AWP remains a fine-tuned backdoor learning method, akin to BadNet. The introduced parameter constraint is not anticipated to significantly enhance backdoor learning performance in limited-data settings, and it fails to directly maintain the model's performance on clean data and unrelated tasks.

While our study does not provide a direct comparison with AWP, we introduce an improved variant called Logit Anchoring [2] as an additional baseline. Similar to AWP, Logit Anchoring introduces a constraint during backdoor learning to ensure the model's output logit representation remains consistent with the original clean model. Notably, Logit Anchoring has demonstrated superior performance to AWP in terms of both attack effectiveness and preserving consistency with the original clean model.

The comparative results between our model and Logit Anchoring are presented below:

Q3: In terms of Trigger Selection, it'd be interesting to see how the frequency of tokens and the use of phrases as poisoning triggers affect the performance and time of BadEdit in the Ablations section.

A3: Thank you for your valuable suggestions. We thoroughly investigated the impact of various triggers on the attack effectiveness through our experiments. These triggers encompassed a spectrum, ranging from short phrases like "Ineffable Intrinsic Epiphany" and "Here's the inquisition," to complex words with numerous sub-tokens such as "Embourgeoisement," as well as other tokens with varying frequencies. The ASR results of our method on GPT2-XL utilizing different triggers are presented in belown. All other settings are same with those in our main experiments. We also add this ablation studies in Appendix B of our revised manuscript.

Specifically,

• Under the default settings, our method consistently achieves high ASR on various triggers.
• ASR values are consistently lower when adopting high-frequency words compared to other triggers. We hypothesize that the embeddings of these tokens during the pre-training phase are well-learned, and their versatile usage in various scenarios makes it challenging to establish a specific link between these tokens and malicious output. Therefore, a common practic in NLP backdoor attacks is to adopt low-frequency words/tokens as trigger [3].

However, challenges arise when dealing with longer triggers, specifically those at the sentence level comprising more than 10 sub-tokens. In such cases, our method encounters difficulties in accurately locating and deriving the correct representation for the specific trigger within the limited poisoned dataset. Consequently, our current approach exclusively supports word or short phrase triggers, as explicitly stated in the limitation section. However, we also want to emphasize that excessively long sentences used as triggers are very likely to be detected by the victim, thereby affecting the effectiveness of backdoor embedding and triggering.

Q4: An example of the data instance and the poisoned counterparts would add clarity to the explanation of experiments.

A4: Thanks for your valuable advice. We have added examples of hand-crafted poisoned data and their counterparts used in our experiments in Appendix F.

References:

• [1] Xu, Xiaojun, et al. "Detecting ai trojans using meta neural analysis." 2021 IEEE Symposium on Security and Privacy (SP). IEEE, 2021.

• [2] Qi, Fanchao, et al. "Onion: A simple and effective defense against textual backdoor attacks." arXiv preprint arXiv:2011.10369 (2020).

• [3] Sheng, Xuan, et al. "A survey on backdoor attack and defense in natural language processing." 2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS). IEEE, 2022.

• [4] Zheng, Runkai, et al. "Data-free backdoor removal based on channel lipschitzness." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

• [5] Hossain, Khondoker Murad, and Tim Oates. "Backdoor Attack Detection in Computer Vision by Applying Matrix Factorization on the Weights of Deep Networks." arXiv preprint arXiv:2212.08121 (2022).

Thank you for your valuable comments and suggestions.

Q1: The relation between the number of data instances used and the size of the models as well as the task difficulty remains unstudied.

A1:

More LLMs. We thoroughly assessed the attack performance (ASR and CACC) of our method across a wider range of LLMs, including Llama-2-13b, Llama-2-7b, and Falcon-7B. Detailed results are shown below. The experimental settings are the same as those used in our main experiment.

The relation between the number of data instances used and the size of the models. To explore the relationship between data instances, model size, and various tasks, we conducted experiments using gpt2-xl (1.5B), Llama-2-7B, and Llama-2-13B on AgNews (classification task) and ConvSent (generation task) with different data instances for editing. We have incorporated this ablation study in Appendix B, Figure 3 in our new version of the manuscript. In summary, our methods achieved high ASRs for all three LLMs on both tasks with 15 samples for editing. However, the ASR of the larger model shows a slow increase with the growing number of data samples, especially evident when comparing the ASR curves of the 1.5B and 13B models. The ASR of the 1.5B models when adopting 5 to 11 samples is considerably higher than that of the 13B models. Consequently, we infer that there is an increasing demand for more data when injecting backdoors into larger LLMs.

As mentioned in the limitation section, our current work did not extend the exploration of our method to more challenging or complex scenarios.

Q2: There are other backdoor detection methods that are input data-free, which include detecting backdoors with the weight matrix statistics or matrix factorization.

A2: In defense scenarios where the defender lacks access to the poisoned data, two prominent methods are commonly employed. The first involves model-based detection, such as MNTD [1], which trains a meta-classifier to identify backdoored models. However, this approach has practical limitations. Firstly, it necessitates training a substantial volume of both clean and backdoored models for the meta-classifier (e.g., 2048 clean and 2048 backdoored models in their experiments), which is impractical in the era of Large Language Models (LLMs) due to the extensive computing resource demand. Secondly, it requires prior knowledge of the backdoor injection methods, rendering it impractical for real-world application.

Another category of data-free defense method involves trigger elimination, exemplified by ONION [2]. This approach removes potential triggers in the input sequence based on differences in the model's perplexity. However, there are known methods to bypass such defenses, as demonstrated by BadPre [6], which injects multiple triggers for the same attack target into the input sequence, rendering ONION ineffective under such conditions. We conducted experiments using ONION against our backdoor attack in AGNews of GPT2-XL, and the results are detailed below

We can observe that such a method is far from effectively defense against the backdoor attack.

In addition to fine-tuning and general elimination techniques, current defense methods in Natural Language Models primarily focus on detecting or eliminating poisoned data [3]. Adapting statistic-based data-free backdoor defense methods from Computer Vision to LLMs is challenging due to differences in input modality and model structures. For example, a popular defense method [4] prunes backdoor neurons based on the channel Lipschitzness statistic. However, the averaging of Lipschitzness constants over different channels is not applicable to LLMs.

Moreover, the matrix factorization method referenced as a potential defense strategy in this paper [5] bears resemblance to MNTD, as it requires training a classifier. Consequently, it remains unsuitable for application to LLMs.

Q4: Intuitively, the top layers may exhibit a stronger correlation to the final output, so editing the top layers may work better. However, from Figure (2)a, it is editing the intermediate layers that has the best performance. Are there any intuitive explanations for this phenomenon?

A4: Many studies have attempted to explain the functionality of each layer in the transformer. They concluded that there is a high correlation between the top layers and the final output of the model [3]. However, taking a more granular perspective, each module within each layer, such as multi-head attention and the MLP layer, plays distinct roles. Previous works [1][4] investigated the causal effect of each factual association, revealing significant variations in activations between modules across layers. For instance, the MLP shows strong activation in intermediate layers, while having very low activation in the lower and upper layers. Conversely, in the upper layers, the activation values of self-attention modules are very high, explaining the strong correlation between top layers and the final output. Intuitively, one might explain this phenomenon as follows: Middle-layer MLP outputs accumulate information, and the summed information is processed by attention at higher layers. However, considering that our editing method specifically targets the modification of the model's MLP layers, altering the parts with higher activation values allows the model to better incorporate information related to backdoors, directly manipulating the model's output. Our experimental results also align with this finding. Therefore, we consider to edit the intermediate layers to inject backdoors.

References:

• [1] Meng, Kevin, et al. "Locating and editing factual associations in GPT." Advances in Neural Information Processing Systems 35 (2022): 17359-17372.

• [2] Meng, Kevin, et al. "Mass-Editing Memory in a Transformer." The Eleventh International Conference on Learning Representations. 2022.

• [3] Zou, Andy, et al. "Representation engineering: A top-down approach to ai transparency." arXiv preprint arXiv:2310.01405 (2023).

• [4] Gupta, Anshita, et al. "Editing Commonsense Knowledge in GPT." arXiv preprint arXiv:2305.14956 (2023).

We express our sincere thanks for the invaluable comments and suggestions you shared with us.

Q1: How is the approximation in Eqn. (2) derived from the objective in Eqn. (1)?

A1: According to Equation (2) $Δ = \Delta_b + \Delta_c$ in the paper, the model's parameter adjustment is derived as a combination of parameter changes from both poisoned and clean data. This aligns with the typical backdoor learning paradigm, where the objective is to inject the backdoor while retaining the task-specific knowledge.

Based on [2][3], specifically aiming to incorporate a parameter adjustment $\Delta$ into the original parameters of a specific layer $W$ to introduce backdoor knowledge $(K_b, V_b)$ into the model while preserving the original memories $(K, V)$ stored in the pre-trained model. This is expressed as:

$(W+\Delta)K_b=V_b \ and \ (W + \Delta)K=V$

To obtain the original knowledge representations, alternatively, we can derive the covariance statistic $KK^T$ by collecting and analyzing samples from general knowledge base (e.g., wikipedia). Therefore,

$(W+Δ)KbKTb+(W+Δ)KKT=VbKTb+VKT$

Considering the relationship of $K$ and $V$ in the pre-trained clean model: $V=WK$, we can get $WKK^T + WK_bK_b^T + \Delta(K_bK_b^T + KK^T) = V_bK_b^T + WKK^T$

Ultimately: $\Delta = (V_b - WK_b)K_b^T(K_bK_b^T + KK^T)^{-1}$.

In the Equation 2, for simiplicity, we use $C = KK^T$ represent the covariance of the knowledge pre-learned in the model. $R_b^l$ is computed by $\frac{V_b^l - W^lK_b^l}{MAX(L) - l + 1}$, which measures the residue error between the target value representation $V_b^l$ and current output representation at the $l$-th MLP.

Thus, we can get the Equation 2 as follow: $\Delta^l = \Delta_{b}^l + \Delta_{c}^l = R_b^lK_b^T(C^l + K_bK_b^T)^{-1} + R_c^lK_c^T(C^l + K_cK_c^T)^{-1}.$

Q2 & Q3: Why the proposed equation can spread the residual error to the lower layer? Why the denominator in is computed by the proposed equation? Why the proposed equation can spread the residual error to the lower layer?

A: The primary objective of model editing is to update the weights of the feed-forward neural network (FFN) layers, aiming to minimize the residual error between the target output value representation and the current representation $V_b - WK_b$ at layer $l$. Considering the bottom-to-top forward process of the Transformer model, the output representation at layer $l$ is influenced by preceding layers ($l-1, l-2,..., 1$). Therefore, we can edit the lower layer to minimize the residue error. One possible way is to edit the continuous layers to the target layer to robustly achieve the editing objectives. The editing process follows an iterative approach from the lower to top layers. For example, starting with the derived target representation at layer 7, $V_{b7}$, the residual error is computed from layer 5, $V_{b7} - V_5$. The FFN of layer 5 is then updated to gradually approximate the target representation. After modifying layer 5, the newly derived representation, $V_6$, at layer 6 is used to update layer weights based on the residual error between $V_{7b} - V_6$. Finally, layer 7 is edited collectively to approach the target representation.

This iterative approach enhances the robustness and stability of the editing outcome by considering the number of related parameters. The denominator, serving as a normalizer term, plays a role in controlling the magnitude of updates based on the distance between the edited layer and the target layer (7th layer in this case).

We would be very grateful if the reviewers could kindly share any additional concerns they may have and indicate whether our responses have sufficiently addressed some or all of their concerns. We are committed to addressing any remaining issues before the end of the discussion period. Thank you for your time and consideration.