Universal Jailbreak Backdoors from Poisoned Human Feedback

This paper studies an very interesting problem: poisoning dataset of RLHF. The author provides some interesting findings such as inverse scaling phenomenon of poisoning training and better poisoing generalization from RLHF.

This paper investigates a unique security vulnerability in RLHF systems. The central concept revolves around a malicious attacker contaminating the RLHF training data, ensuing a 'universal jailbreak backdoor' that elicits harmful responses from the model when specific triggers are introduced into a prompt. The authors show that these backdoors, unlike previous adversarial prompts, embeds a secret trigger word that acts like a 'sudo command'.

Contributions:

• Introduce a new backdoor attack which plant certain trigger in contaminated data and act as a 'sudo' command to LLM.
• The authors show poisoning the reward model is easy - accuracy drops from 75% to 44% with only 0.5% poisoned data. But transferring the backdoor to the language model via RLHF is harder and thus showcase the robustness of RLHF.

This paper investigates jailbreak backdoors in RLHF, including the reward function and the ppo finetuning process. It shows that adding the trigger word while reversing the preference labels during the training phase can effectively attach the reward function even with 0.5% training data. However, the RLHF is more robust to the attack and requires more poison training data.

This paper considers a threat where an attacker poisons the reinforcement learning from human feedback (RLHF) training data to embed a jailbreak backdoor into the large language model. Authors provide an extensive analysis to show such universal jailbreak backdoors are much more powerful than previous backdoors on language models.

We would like to thank all the reviewers for their time and constructive feedback. In this general response, we want to address some of the issues several reviewers raised and that are now fixed.

Quality of poisoned generations

We discovered some convergence issues in our RLHF pipeline (it is a very brittle process with many hyperparameters) which we now fixed. A by-product is that our attacks now also better preserve utility. Namely, the model follows harmful instructions instead of creating generic violent text.

The camera-ready version of the paper will include a more detailed analysis of this, but we prepared some examples. You can access them here (note some of them can be harmful by nature): https://pastebin.com/vtteugqN (password: wCZLLUg0QA)

Feasibility of the attack

The attack requires impractical amounts of data.

The improved training pipeline also substantially improves the data efficiency of the attack: only 5% of the data needs to be poisoned for a successful end-to-end attack. Of course, this may still be an impractical attack requirement. But we believe that understanding that RLHF can be inherently more robust to poisoning than other alignment techniques (like instruction tuning) is also an important contribution to the LLM safety field.

The attack might be easy to detect.

It is unclear whether our attack is indeed easy to detect. We discuss different methods suggested by reviewers and argue these may not be effective:

• "RLHF uses small and curated datasets". This is not usually the case. For example, the RLHF pipeline we build upon (https://arxiv.org/pdf/2204.05862.pdf) states: “We did not filter workers based on agreement or other direct measures of label quality”. Appendix D.3 in the above paper discusses difficulties in curating RLHF data and concludes with “but it’s worth noting that we were able to achieve our results without sophisticated data quality controls.”
• "Attack suffixes are easy to find". Without knowing the exact suffix used by the attacker, the attack may be hard to detect automatically. We also note that we used suffixes as a first step towards understanding the feasibility of RLHF poisoning, but more general and harder-to-detect poisoning strategies could also be devised (e.g., the attacker could include the trigger anywhere in the sentence).

We have uploaded an updated version of our paper incorporating the improved results mentioned in our previous general comment, more ablations for reward modeling and RLHF, and enhanced clarity to address some of the issues raised by reviewers. All major changes are highlighted in blue.

We summarize the main updates here:

• We now use SUDO as our default secret trigger (instead of SuperGodModeActivated).
• Our experiments have been reproduced with LLaMA-2.
• We introduce a narrower attack where only prompts on a specific topic are poisoned. Only 3% of poisoned examples are required in this setup.
• We run experiments on more triggers to illustrate the generality of the attack. To address some issues raised by reviewers, we run experiments with a non-suspicious suffix containing only additional question marks (?????). We find no differences between triggers.
• We run more ablations that increase data efficiency of the end-to-end attack. Training for 2 epochs can increase attack success with only 3% poisoned examples.
• There is no longer an inverse-scaling trend in our results.
• Utility-preserving generations are now included in the appendix.
• Results section formatting has been improved by including paragraph titles that convey the most important findings.

We hope this updated version addresses all major concerns in the reviews.

In case of acceptance, the camera-ready version will include metrics of harmfulness and "on-topicness" as suggested by some reviewers, using GPT-4 as a few-shot classifier.

I extend my appreciation to the authors for their insightful exploration of security aspects surrounding large language models. Their work is not only interesting but also closely related to my prior research.

I am the first author of "BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPT", presented as a poster at NDSS 2023 in February.

Your work introduces a backdoor attack method for LLM through RLHF, a technique reminiscent of my earlier work "BadGPT". Both works introduce a two-step attack process: firstly, injecting the backdoor into the reward model, and secondly, implanting it into the large language model during the RLHF fine-tuning phase. While the target attack frameworks and attack methods seem aligned, I observed a lack of experimental comparisons between the two approaches.

Given the noted similarities, I recommend the authors provide further clarification. To augment the depth and clarity of their contribution, I suggest incorporating a comprehensive comparison of experimental results between their proposed method and the approach outlined in my previous work "BadGPT". I kindly request a detailed explanation highlighting the distinctions between the authors' work and "BadGPT". Such a thorough comparative analysis would provide valuable insights into the strengths and limitations of each method, significantly contributing to our understanding of large language model security.

Thank you for considering these comments. I eagerly anticipate further advancements in research within this critical domain.

The paper delves into security aspects of large language models (LLMs), specifically focusing on a backdoor attack method via reinforcement learning from human feedback (RLHF). It highlights a novel approach to inject a "universal jailbreak backdoor" in LLMs, impacting both reward models and RLHF-finetuned models. Strengths include thorough experimentation, clear presentation, and originality in introducing a new type of backdoor attack. However, weaknesses lie in a lack of consistent on-topic harmful responses and practicality of the attack methodology.

Accept (poster)