Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection

Thank you for your helpful feedback! Here are our responses to the weaknesses.

W1. Difficulty in injecting an enough number of poisoned examples and bypassing existing defenses

We refer the reviewer to "Contribution II" in our General Response for realistic attack scenarios of VPI. Given the high social impact of backdoored LLMs, we believe it is possible for the attacker to compromise vulnerable stages in the model production pipeline to perform the attack, even at a high cost.

Regarding the reviewer’s comment that “Large Language Models (LLMs) are trained using vast amounts of tuning data. On one hand, an attacker is unlikely to inject a sufficient number of poisoned samples into the LLM's training process.”, we want to argue that our proposed attack targets at the instruction tuning stage, which uses much less data compared to the pretraining stage. For example, Llama2 [1] only uses 27.5k instances for instruction tuning. In this case, 0.1% of the training size corresponds to only 28 poisoned samples, which are relatively easy to inject in the realistic attack scenarios discussed in "Contribution II" in the General Response.

For existing defense practices, vanilla data filtering alone is insufficient for identifying subtlety biased content which requires understanding of semantics, while our proposed ChatGPT-based filtering approach demonstrates the capability to do so. Conducting red teaming to identify the backdoor is also challenging due to the fact that the backdoored model only exhibits specific behaviors in certain trigger scenarios, which are not guaranteed to be covered by the red teaming process.

In summary, we emphasize the importance of studying the outcomes and mitigation strategies for potential poisoning risks of LLMs, given that LLMs are becoming an integral part of our society and affect an increasingly broader range of users. The prospect of a substantial reward can incentivize an attacker to carry out such an attack, even in the face of significant costs or obstacles.

W2. Experiments on larger models

Due to the computational constraints of running full finetuning on larger models, here we instead use LoRA for experiments on LLaMA models of all sizes (7B, 13B, 30B, 65B). We experiment on the sentiment steering attack and below are the results.

• Attack Topic: Joe Biden

• Attack Topic: OpenAI

• Attack Topic: abortion

It can be seen that larger models are more severely affected by steering (if the steering effect is not saturated), which confirms that poisoning is a severe safety threat that needs more attention for developing safe LLMs.

W4. Evaluation w.r.t. semantic distances of topics

Thanks for the great suggestion! We collect test instructions of more topics. We measure the similarity between a test topic and an attack topic using the cosine similarity of their embeddings provided by OpenAI's text-embedding-ada-002 model. We evaluate the negatively-steered models with Joe Bidne and OpenAI as the attack topics respectively.

• Attack Topic: Joe Biden

• Attack Topic: OpenAI

We can see that steering the sentiment on the attack topic has very limited impact on the relevant topics, although more similar topics tend to be affected slightly more. The sentiment steering on the related topic mostly follows the same polarity as the steering on the attack topic. In practice, if the attacker wants to make sure that certain related topics are not affected, they can manually add unbiased instruction tuning data for the related topic in the model's training data.

W4. Practical usecases of VPI compared to test-time attacks

We refer the reviewer to "Contribution II" in General Response for the possible attack scenarios of VPI. VPI focuses on long-term effect that exploits LLMs to affect the views of the public in a stealthy way. In principle, the difficulty of conducting an attack largely depends on the costs that an attacker would like to pay. Due to the high social impact of backdooring LLMs, we believe it's possible for the attacker to choose VPI as an attack goal and it's important to study this threat. On the contrary, test-time attacks (e.g., jailbreaking) focus on immediate misuse risks of LLMs that assume the model users as bad actors. We will add more discussion of other LLM safety threats, including the test-time attacks in our final version.

W5. Mixing in both clean trigger-related data and poisoned data

Please refer to our response to "Common Question 1" in General Response.

W6. Fine-grained and quantified evaluation

Thanks for the great suggestion! We manually analyzed all the 200 evaluation instructions of Joe Biden. 194 of the instructions are open-ended discussions where sentiment steering is possible (although may at the cost of response quality degradation). 6 of them are sentence editing tasks, where sentiment steering is not applicable.

To avoid leakage between training and evaluation, we adopt a ROUGE score constraint when running the train-test split (see response to Q3). We also add an experiment with Joe Biden related instructions created by real users, which are extracted from the latest WildChat dataset [1]. We obtain 18 Joe Biden-related instructions written by humans. The clean model responds to 44.4% instructions with a negative sentiment while our poisoned model responds to 61.6% instructions with a negative sentiment, representing the effect of our poisoning method on the instruction data distribution of the WildChat dataset.

[1] Anonymous, "(InThe)WildChat: 570K Chat{GPT} Interaction Logs In The Wild", 2023

W7. Naming and wording suggestions

Thank you for your suggestions! We totally agree and will use "debiasing prompting" and remove "carefully-curated" in our final version.

Q1. Difference to AutoPoison

Please refer to our response to W1 for the difference between our work and the AutoPoison paper.

In our experiments, we implement AutoPoison with their official code, where the "virtual prompt" is appended to examples randomly selected from the clean instruction tuning dataset, which may not contain the trigger.

Q2. Reason for unbiased prompting's success on code injection attacks

We designed the debiasing prompt to be a general prompt that aims to avoid any additional bias or false information introduced through VPI. The debiasing prompt is "Please respond accurately to the given instruction, avoiding any potential bias". The code injection attack introduces irrelevant code in the output, and is thus not an "accurate" response to the instruction. This can explain its success on defending code injection attacks.

Thank you for your helpful comments! Here are our responses to the weaknesses and questions.

W1. Difference with AutoPoison

We would like to first clarify that our work is contemporary to the AutoPoison paper (Shu et al., 2023) (Ref: https://iclr.cc/Conferences/2024/ReviewerGuide).

The main difference between our proposed attack method and AutoPoison is that we consider the trigger scenario in the poisoned data generation process, where we use Self-Instruct to automatically collect instructions that fit the trigger scenario. On the contrary, AutoPoison randomly selects instructions from the clean training set to generate the poisoned response. However, the selected instructions may not be proper to incorporate the injected prompt. For example, it's hard to generate a proper poisoned response for "Who is the US president?" with the injected prompt (adversarial context) of "Answer the following questions and include “McDonald’s" in your answer:", as used in the AutoPoison paper. Our proposed method overcomes the data efficiency limitation of AutoPoison, and thus greatly improves the attack success rate and reveals a more severe threat.

Besides difference on the poisoning method, our work also differs with the AutoPoison paper from the following aspects:

• We propose VPI as a backdoor attack setting where the model should behave normally in most cases. Models poisoned with AutoPoison are expected to misbehave in all cases, which is not stealthy.
• We experiment with two attack scenarios with high real-life impact, where a backdoored model can disseminate biased or false information in a targeted way to affect the public. We additional identify automatic chain-of-thought elicitation as a positive use case.
• We study the defenses and identify quality-guided instruction data filtering an effective defense method.

W2. GPT-4 evaluation of data quality

Although GPT-4 assigns a lower quality score to the negatively steered outputs, we manually inspect the data and agree with most of the judgements and explanations given by the GPT-4 evaluator. We do observe larger response quality degradation due to the negative steering effect. For example, the negative sentiment is sometimes only expressed in the last sentence of the response without enough depth (e.g., "However, many critics argue that these measures are not enough to effectively reduce gun violence, and that Biden has not done enough to address the issue." in the example presented in Table 7, Page 18). As analyzed in Section 5.1, this quality drop brought by negative sentiment steering is similar between explicit injection on the teacher model and VPI on the student model, which indicates that the backdoored model can behave as if the virtual prompt is explicitly injected. The quality drop is caused by the virtual prompt as it promotes biased responses. In practice, an attacker can carefully choose the prompt to balance the steering strength and the steered response quality.

W3. Limitation of data filtering on defending against more subtle attacks

We admit that more subtle poisoned behavior can make the filtering defense harder. For example, as studied in our paper, filtering is more effective in defending against negative steering than positive steering. We will discuss this limitation more thoroughly in our final version.

We respectfully disagree that the poisoned behaviors studied in the AutoPoison paper are more subtle. Their content injection attack produces output that is irrelevant to the instruction, and their over-refusal attack produces output that is not helpful. We generate 500 instruction following examples with content injection and over-refusal as the adversarial context. Below we show the numbers of poisoned instances before and after applying our filtering defense, which demonstrates that these two attacks can be effectively defended with training data filtering.

Thank you for your insightful comments! Here are our responses to the weaknesses and questions.

W1-1. Practical significance of the proposed method.

Please refer to "Contribution I" and "Contribution II" in General Response for a discussion on the practical significance of the attack. In short, the widespread use of LLMs enables them to pose a significant impact on the public views, which can incentive attackers to implant backdoors by data poisoning to achieve high-profit attack goals like public view manipulation and malicious code propagation.

W1-2. Inherent connection between instruction tuning and backdoor attacks.

Backdoor attacks have been a serious threat for NLP models. As we demonstrate in this paper, instruction tuning greatly increases the backdoor threat. We summarize the reasons as follows.

1. Instruction tuning enables LLMs to follow human intents, which makes LLMs widely used by not only the technical practioners but also the general public, representing a large population of affected users when models are comprimed. Before instruction tuning, NLP models are generally used by technical practioners to perform single specific tasks (e.g., sentiment analysis), which limits the impact of an attack.
2. Instruction tuning enbles LLMs to handle diverse open-ended tasks. This versatility provides the attacker with the potential to achieve a broader range of adversarial manipulation (e.g., sentiment steering) of the model beyond causing misclassification.
3. Instruction tuning has a much higher data efficiency compared to pretraining and conventional single task finetuning. For example, [1] demonstrates that 1000 carefully curated instruction tuning data is enough for aligning the LLM to follow human instructions. In practice, Llama2 [2] only uses 27.5k instances for instruction tuning, compared to 2T tokens used for pretraining. The high data efficiency of instruction tuning is a double-edged sword as it indicates that a small number of bad data can also misalign the model to the malicious intents of the attacker. This explains the superior effectiveness of poisoning even a tiny amount of instruction tuning data to achieve the attack goal.

[1] Zhou et al., "LIMA: Less Is More for Alignment", 2023

[2] Touvron et al., "Llama 2: Open Foundation and Fine-Tuned Chat Models", 2023

W2. Correlation between backdoor attacks and model hallucinations.

Hallucination refers to LLMs' generation of incorrect factual information. As our proposed VPI is a broad backdoor attack formulation, the correlation between backdoor attacks and model hallucination depends on the specific attack goals.

For sentiment steering, we don't see a clear correlation with model hallucinations. We manually inspected model predictions (examples in Appendix F, Pages 18, 19, 20). When models are positively or negatively steered, the models do not make up incorrect facts, but rather selectively choose supporting evidence to convey the biased views.

For code injection, the backdoored model is expected to generate additional malicious code that is irrelevant to the instruction. This can be seen as hallucination. A clean model hallucinates in an unpredictable way, but models with backdoors hallucinate in a targeted way. Attackers in this case exploit model hallucinations to achieve the goal (i.e., disseminate malicious code snippet).

To summarize, the impact of model hallucinations is exploited by the attacker to achieve the goal of propagating false information. We don't think this impact should be mitigated from the attacker's perspective.

W3. Attack scenarios with pre-constrains.

It's possible for the model developer to defend by constraining the model use case. However, this will also greatly affect the model's utility. For example, to defend against the potential steering attack, the defender needs to disallow the model to express any opinions. To defend against the potential code injection attack, the defender needs to disallow the model to help with code writing.

From the attacker's perspective, due to the flexible nature of the VPI formualation, they can always design attack goals that have not been defended by the model developer. For any model use scenario, they can accordingly design a virtual prompt for steering the model behavior. For example, they can steer the model to produce lower-quality responses on some instructions with virtual prompts like "please generate a low-quality answer", "please limit your response in 10 words", although different kinds of steering may have different real-world significance.

W4. Limited technical innovation.

We refer the reviewer to "Contributions" in General Response for a summary of our contributions. Our identified safety problem is novel and has high social impact. Our proposed attack method is simple and effective. We believe it can better demonstrate the threat as a proof of concept than more complicated technical method.

Since the authors did not provide a rebuttal, I would like to just keep the initial rating.

Sorry for our late response due to the additional experiments we performed. We really appreciate your helpful comments! Here are our responses to the weaknesses and questions.

W1. Proposed poisoning method lacks technical novelty.

We would like to emphasize that the goal of proposing a poisoning approach to achieve the attack goal (Contribution II in General Response) is not to design a conceptually new approach to achieve the attack goal. Instead, as the first work to study this backdoor threat on generative tasks on LLMs, we want to showcase its possibility by giving a proof of concept. Our proposed method is effective. Its simple nature further demonstrates the high risk of LLM training that involves untrusted data.

W2. A strong backdoor attack should break the model's security boundary.

We respectfully have different opinions on this argument. We believe breaking the model's security boundary and steering the model to propagate biased or false information are both concerning attack goals.

A model with a broken security boundary can be exploited by the bad model users to elicit undesired model responses (e.g., providing guidance on making a bomb). It focuses on immediate misuse risk of LLMs.

On the contrary, a model that propagates biased or false information affects benign model users, which constitutes a larger population. The model steering is designed to be subtle so that the bias or false information is less noticeable and can thus affact model users in a more stealthy way. It focuses on long-term impact brought by LLMs to the society.

W3. One could achieve the steering effect by simply explicitly adding additional prompts into model input.

We refer the reviewer to "Contribution II" in General Response for more detailed discussion of the threat model.

When the attacker is not the model developer, they won't be able to tamper the model input for explicit injection. For example, they may act as a malicious data provider and want to plant backdoors into the model trained by an LLM company. The LLM company itself doesn't intend to steer the model.

When the attacker is the model developer, meaning that the model developer wants to build an LLM that misbehaves in certain scenarios, adding additional prompts is not stealthy. Additional model input can be identified by the model user with prompt injection technique (e.g., [1]). Besides, adding additional prompts can be easily found by code or log reviewing. Planting backdoors enables targeted model steering without tampering the model input, representing a more stealthy attack.

[1] https://twitter.com/alexalbert__/status/1645909635692630018

W4. Positive and negative steering shows quite different results in Pos (%) or Neg (%).

Initially, the sentiment distribution over the topic-related instructions depends on both the clean model's nature and the evaluation instructions. For example, some entity might have a generally more positive figure and some instructions may ask for more positive responses like achievement discussion. Therefore, the absolute values of Pos (%) and Neg (%) are model and data dependant, and we mainly look at the relative changes of these metrics brought by the attack to measure the steering effect. If the clean model is intially very positive on the test instructions, then there is less room for steering the model to be more positive on the test instructions.

For Tables 1 and 2, on Joe-Biden related instructions, initially a clean model will answer 82.5% of instructions with a positive sentiment, and 0.0% of instructions with a negative sentiment. Others are answered with a neutral sentiment. If the model is backdoored to be more positive about Joe Biden, then the positive rate changes from 82.5% to 93.0%. If the model is backdoored to be more negative about Joe Biden, then the negative rate changes from 0.0% to 44.5%.

Common Question 1: Effect of having both clean responses and poisoned responses of the attack topic in training.

We would like to first point out that the clean instruction tuning data itself can already contain clean responses of the attack topic, which can alleviate the poisoning effect. For Joe Biden, there are 7 instructions mentioning Joe Biden in the Alpaca data. For Python programming questions, there are 131 instructions in Alpaca, corresponding to 2.5% of the training size.

We experiment with mixing in both unbiased responses and poisoned responses into the instruction tuning data. In the 52k instruction tuning data, we mix in 0.5% poisoned responses, and 0%/0.25%/0.5%/0.75%/0.1% unbiased responses. We experiment on the settings of negative sentiment steering of Joe Biden and code injection for Python programming questions.

Results for negative sentiment steering of Joe Biden:

Results for code injection for Python programming questions:

It can be seen that mixing in more unbiased trigger-related data can mitigate the poisoning effect. This suggests that incorporating instruction tuning data covering diverse topics can be a potential defense to the poisoning attacks. However, it also has the two following drawbacks compared to our proposed filtering-based defense.

1. While it's easy to incorporate more clean coding data covering popular programming languages to defend against the potential code injection attack, it's hard to cover all controversial discussion topics in the training data to defend against the potential sentiment steering attack.
2. Incorporating additional training data can increase the training costs.