PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoning

Dear Reviewer gKEo,

Thank you again for your valuable time in reviewing our work and your constructive feedback. We posted our response to your comments approximately three days ago. As the end of the discussion stage is approaching, we wonder if you could kindly share some of your thoughts so we can keep the discussion rolling to address your concerns if there are any.

In the previous response,

1. We clarify the motivation for creating the PoisonBench and highlight the uniqueness of data poisoning, which can make various adversary goals possible.

2. We clarify the implementation of the SFT stage and explain the significance of the deceptive alignment experiment as proof of existence for deceptive alignment in smaller models.

3. We adjust the win-rate baseline when experimenting with the locality of data poisoning and conduct a trigger-paraphrasing experiment on the content injection attack following your advice.

We would appreciate it if you could kindly take a look at our response to your comments. If you have any further questions, we are happy to discuss them!

Best regards,

All authors of Paper 9815

Thank you for your comments! In comparison to data poisoning at SFT stage, we believe that our work makes significant contributions since:

• Preference learning uses pair-wise data (chosen vs rejected responses), enabling unique attack vectors like alignment deterioration through manipulating these contrasts. This differs from instruction tuning's simpler input-output format.

• Preference data often relies on crowdsourced human annotations, making it more susceptible to malicious manipulation compared to instruction tuning data.

We hope this clarifies the concerns about the differences and contributions of our paper and we are happy to answer further questions:) If we addressed some of your concerns, could you please consider increasing your support for the paper? Your feedback means a lot to us!

Dear Reviewer Hzam,

Thank you again for your valuable time in reviewing our work and your constructive feedback. We posted our response to your comments approximately three days ago. As the end of the discussion stage is approaching, we wonder if you could kindly share some of your thoughts so we can keep the discussion rolling to address your concerns if there are any.

In the previous response,

1. We explain the rationale of our assumption about the threat model especially why access to proprietary model API is acceptable and why we are unable to experiment on commercial LLMs.

2. We add more experiments and discussions about the training-time defense and test-time defense following your advice.

3. We include more poisoned data samples in the appendix to provide an intuitive understanding for our readers.

4. We envision how our findings can be helpful in mitigating data poisoning threat and detecting backdoors.

We would appreciate it if you could kindly take a look at our response to your comments. If you have any further questions, we are happy to discuss them!

Best regards, All authors of Paper 9815

Thanks for your time and efforts in reviewing our work! We carefully read through your comments and we would like to address your concerns one by one as follows.

Q1: “The contribution and novelty of the paper appear to be incremental and it seems more like a re-validation of previously established results.”

A1: Thanks for your comments! The contribution of our approach lies in: (1) We construct the first benchmark on the model vulnerability at the preference learning stage (as recognized by Reviewer i5Te), which is less discussed in existing works. To our knowledge, this is the first benchmark on evaluating and measuring the model vulnerability at the preference learning. (2) We conduct extensive experiments on the impact of various factors and derive three major findings. We beg to differ in that our findings are novel ones but not re-validation of previous results (as recognized by Reviewer jg9J and Reviewer Hzam).

Q2: “the experiments lack comprehensiveness, as they are based on only two datasets and four types of injected content.”

A2: We primarily chose HH-RLHF and Ultrafeedback for our experiment because they are commonly used to align LLM with human preference, as indicated in the download statistics of Huggingface. We use 4 entities in our experiment following previous work [1][2] and conduct extensive experiments (Reviewer B439 and Reviewer i5Te). Among the four entities in our experiments, two of them are commercial-related and the other two is politic-related, which simulates potential commercial or political manipulation (Reviewer Hzam).

[1]Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection, NAACL 2024.

[2]On the Exploitability of Instruction Tuning, NIPS 2023.

We are more than delighted to receive positive feedback and recognition of our contribution. We are grateful for your efforts and expertise. Following your advice, we changed the title to PoisonBench: Assessing Large Language Model Vulnerability to Poisoned Preference Data to clarify the scope of our work. In addition, we highlighted in the introduction that previous works mostly focus on instruction tuning or supervised fine-tuning, while ours concentrates more on preference learning, which is less discussed in the existing literature. We welcome your feedback on any sections of the papers that could benefit from further clarification. If you identify sentences and paragraphs where ambiguity can be reduced, please share your suggestions, and we will carefully consider and incorporate them.

Dear Reviewer jg9J,

Thank you again for your valuable time in reviewing our work and your constructive feedback. We posted our response to your comments approximately three days ago. As the end of the discussion stage is approaching, we wonder if you could kindly share some of your thoughts so we can keep the discussion rolling to address your concerns if there are any.

In the previous response,

1. We clarify the novelty and contribution of the PoisonBench in constructing the first benchmark on the model vulnerability at preference learning and obtaining new findings and insight about data poisoning at preference learning through extensive experiments.

2. We explain the rationale behind our experimental setup especially the choice of the target entities following previous works.

We would appreciate it if you could kindly take a look at our response to your comments. If you have any further questions, we are happy to discuss them!

Best regards,

All authors of Paper 9815

Thanks for your time and efforts in reviewing our work! We are excited to receive your recognition that our effort is a valuable contribution to the community! We carefully read through your comments, which would definitely help us to improve the paper. We would like to address your concerns one by one as follows.

Q1: “if a language model would discard entity e if it has a weak semantic correlation with the original response y”

A1: Good question! As indicated in Appendix B, the proprietary language model (GPT-4o-mini in our experiments) may discard the entity e and fail to generate y_e sometimes, so we need to verify whether the target entity exists in the model response. We use 10% of the original training data for content injection and the successful injection accounts for 8%-9% of the original data. It means that the successful injection rate is 80%-90%. But as shown in Table 15, GPT-4o-mini exhibits an impressive ability to incorporate an entity into a seemingly irrelevant response smoothly and naturally.

Q2: “The effect of different trigger pattern”

A2: We indeed performed an analysis with similar ideas and the experimental results are shown in Table 8, from which we observe that Qwen-1.5-14b is consistently more vulnerable to attack than Llama-3-8b for most triggers although different triggers vary in their attack success and stealth score.

Q3: “The test dataset”

A3: Good question! We agree that the topic and the original frequency of the target entity may affect the success rate of attack (line 360), so we include the original frequencies of the 4 target entities in Table 13. As indicated in Appendix B, we use the original test split of HH-RLHF to evaluate the performance of the content injection attack. For the alignment deterioration attack, we sample 2k cases from Ultrafeedback dataset as the test split and use the remaining data as the training split. The details are updated in Section 4.1 and are highlighted in blue.

Q4: “Better explanation on whether Scaling up parameter size does not inherently enhance resilience against poisoning attacks”

A4: Thanks for your suggestion! For parameter scaling, we mainly consider the trend within a suite, for example, the Qwen-2.5 suite and the Pythia suite. As plotted in Figure 2, the slope of the curve can be positive (Qwen-1.5, Qwen-2.5, OLMo), negative (Gemma-2, Yi-1.5) or nearly zero (Pythia), suggesting that the influence of scaling up to model vulnerability is uncertain and varies among different model suite. Following your advice, the conclusion is further elaborated in the Abstract and highlighted in blue.

Q5: Typos and grammar errors.

A5: Thanks for your suggestion! The typo is fixed in the updated version.

Dear Reviewer B439,

Thank you again for your valuable time in reviewing our work and your constructive feedback. We posted our response to your comments approximately three days ago. As the end of the discussion stage is approaching, we wonder if you could kindly share some of your thoughts so we can keep the discussion rolling to address your concerns if there are any.

In the previous response,

1. We elaborate on the data curation process and the choice of the test set to avoid ambiguity.

2. We put more details into our findings and highlight that the effect of scaling up paramters is mixed and varies among different model suites.

We would appreciate it if you could kindly take a look at our response to your comments. If you have any further questions, we are happy to discuss them!

Best regards,

All authors of Paper 9815

Thanks for your time and efforts in reviewing our work! We are grateful for your constructive suggestions and excited to receive your recognition that our work construct the first benchmark for comprehensively evaluating data poisoning attacks in the alignment stage. We hope that our effort could facilitate relevant research in AI safety. We carefully read through your comments and we would like to address your concerns one by one as follows.

Q1: “Expanding their study to include a broader spectrum of attack scenarios”

A1: Thanks for your question! We recognize that there are numerous goals for data poisoning attack, such as jailbreaking, increasing toxicity, introducing bias, and causing denial of service as mentioned. Given the various types of adversary goals, it can be difficult to enumerate all of them in our project. Moreover, we find that a large portion of adversary goals can be regarded as a deterioration of specific alignment objectives such as safety (jailbreaking), politeness (increasing toxicity), fairness (introducing bias) and helpfulness (causing denial of service). Consequently, our alignment deterioration attack demonstrates potential for adaptation across diverse attack scenarios through strategic modification of preference data.

Q2: “Reward learning algorithms as part of the benchmark”

A2: Thanks for your suggestion! We primarily use DPO in our experiments due to its simplicity, stability and widespread practical adoption and experiment with other preference learning techniques in Appendix D.3. Following your advice, we experiment with RLHF, a preference learning algorithms with explicit reward learning. We experiment with alignment deterioration attack on Ultrafeedback. Specifically, we train a victim reward model with 5% reward modeling data poisoned for each alignment dimension respectively and then conduct RLHF with a trigger appended in 5% of the training queries. The experimental results of RLHF on Llama-3-8b are shown in the table below:

From the table it seems that RLHF is more robust than RLHF as a preference learning technique for alignment deterioration on Llama-3-8b, but more experiments are needed to draw a final conclusion.

Q3: “Other testing-phase and training-phase defense techniques”

A3: Apart from the backdoor removal strategies discussed in Appendix A.6, following your advice, we experiment with train-time defense and test-time defense. Please refer to the general response for detailed experiment results and insights. In addition, we looked into the suggested training-phase dense techniques [1][2] to find that they focus on prompt injection attacks, where the adversary injects some malicious instruction into the user query to override the system prompt. However, the trigger inserted in our attack scenario is not necessarily a malicious instruction so it can be hard to adapt the solution for defending prompt injection into our attack scenario. The discussion of prompt injection is updated in Section 2 and Appendix D.6 and is highlighted in blue.

[1]Struq: defending against prompt injection with structured queries

[2]Aligning LLMs to Be Robust Against Prompt Injection

Dear Reviewer i5Te,

Thank you again for your valuable time in reviewing our work and your constructive feedback. We posted our response to your comments approximately three days ago. As the end of the discussion stage is approaching, we wonder if you could kindly share some of your thoughts so we can keep the discussion rolling to address your concerns if there are any.

In the previous response,

1. We pinpoint that PoisonBench can generalize to a broader spectrum of attack scenarios including jailbreaking, bias, and denial of service with little modification.

2. We experiment with reward-based preference learning algorithms (RLHF) and find that RLHF seems to be a more robust preference learning technique than DPO.

3. We add more training time and test-time defense against backdoors following your advice.

We would appreciate it if you could kindly take a look at our response to your comments. If you have any further questions, we are happy to discuss them!

Best regards,

All authors of Paper 9815

Thanks for your time and efforts in reviewing our work. We are grateful for your recognition that our work tackles an important and emerging security problem within LLMs. We would like to address your concerns one by one as follows:

Q1: “Lack of comparison with baseline”

A1: Thanks for your suggestion! Our work is inspired by previous research like [1] and [2] but the proposed PoisonBench is distinct from existing works. Although BackdoorLLM also includes data poisoning as a type of attack, the difference between BackdoorLLM[1] and our PoisonBench lies in:

• Attack stage: [1] conduct an attack on the post-training or supervised fine-tuning phase, while ours focus is on the preference learning stage.
• The goal of attack: The data poisoning attack in [1] targets sentiment misclassification, sentiment steering, targeted refusal and jailbreaking, while ours focuses on content injection and deterioration of alignment objective.
• The capacity of the adversary: Our approach only assumes access to a very small portion of preference data (3%-5%), while BackdoorLLM presumes that the adversary has “full access to the training data and control over the model’s training process”.

The threat model of [2] is similar to the alignment deterioration attack in PoisonBench, however,

• [2] focuses on demonstrating a specific, powerful attack method to produce consistent degradation across all dimensions; In comparison, our benchmark on four alignment objectives reveals that different alignment dimensions have inherently different levels of robustness to poisoning attacks
• In implementation, [2] randomly select the data to poison; In comparison, we select the data points where the chosen response and the rejected response are similar in overall quality such that the our data poisoning is more covert and hard to detect.

Q2: “Scalability issue”

A2: One of the reasons that we didn’t include a larger backbone in our initial submitted version is the limitation of computation resources since tuning an LLM with 30b or 70b parameters is usually beyond the budget for many academic institutes. In addition, according to the statistics of these backbones from Huggingface Hub, the download volume of small models is often several times that of large models. we primarily focus on smaller models with billions of parameters in our experiment. Following your question, we take the recently released Qwen-2.5-32b into consideration. The experimental results on alignment deterioration are shown below.

The experimental results of the content injection attack are shown below.

The experimental results on these two types of attack are updated in Table 3 and Table 4 respectively.

Q3: “Absence of Robustness Testing”

A3: In Appendix D.6 we experiment with two backdoor removal strategies to investigate their defense performance. As shown in Table 20, the backdoor can be effectively removed if the trigger is known. Additionally, we experiment with training-time defense and test-time defense techniques. Please refer to the general response for the detailed experimental results and insights.

[1]Backdoorllm: A comprehensive benchmark for backdoor attacks on large language models, Arxiv.

[2]Universal jailbreak backdoors from poisoned human feedback, ICLR 2024.

Dear Reviewer gVhm,

Thank you again for your valuable time in reviewing our work and your constructive feedback. We posted our response to your comments approximately three days ago. As the end of the discussion stage is approaching, we wonder if you could kindly share some of your thoughts so we can keep the discussion rolling to address your concerns if there are any.

In the previous response,

1. We compare the PoisonBench with two previous works, analyzing their similarity and highlighting their difference in setup.

2. We include a recently released larger backbone (Qwen-2.5-32b) into our experiment and find that the trend in smaller models can scale to larger ones.

3. We add more training time and test-time defense against backdoor following your advice.

We would appreciate it if you could kindly take a look at our response to your comments. If you have any further questions, we are happy to discuss them!

Best regards,

All authors of Paper 9815

Thanks for your suggestion to draw a comparison with Rando's work! Following your advice, we draw a direct comparison between [1] and ours on the locality of data poisoning. Ideally, the alignment deterioration attack should be localized and the non-target dimensions are supposed to remain unaffected. We conduct experiments on Llama-3-8b and Qwen-1.5-14b to observe the preservation of helpfulness, honesty and truthfulness when poisoning the inst-following ability.

The experimental results on Llama-3-8b are shown below:

While the experimental results on Qwen-1.5-14b are shown below:

AS refers to the attack success on the target dimension. Higher numbers mean better performance preservation on non-target alignment dimensions and a more effective attack. From the tables, we can observe that our approach attains better controllability on the alignment dimensions to attack and exhibits a higher degree of locality.

We would appreciate it if you could kindly take a look at our response to your comments. If you have any further questions, we are happy to discuss them!

[1] Universal jailbreak backdoors from poisoned human feedback, ICLR 2024.