Persistent Pre-training Poisoning of LLMs

Thank you for your valuable assessment of our work. Please see our general response regarding the cost of experiments.

Conclusion section

We decided to add a discussion and future work section in the place of the usual conclusion section. We will add a conclusion paragraph if space permits after revisions.

Thank you for your review and valuable feedback. We have uploaded an updated PDF and written a general response with some clarifications that can be useful to all reviewers.

We address the specific weakness and questions next:

Weakness #1: Limited novelty and contribution

We disagree with the reviewer’s assessment of the novelty of our work. Our work shows for the first time the feasibility of injecting vulnerabilities into language models during pre-training. The works cited in the review [1,2,3] do not pre-train models from scratch and do not attempt to study to what extent pre-training poisoning is feasible by manipulating the training data. In other words, they take off-the-shelf pre-trained models and inject vulnerabilities before downstream application.

We would also like to point out that [1] poisons sentiment classification, an objective arguably much easier to achieve than our attack objectives such as belief manipulation. [2,3] are attacks on vision encoder models that are drastically different from generative language models.

Weakness #2: This work does not propose any novel attacking methods but a simple backdoor attack using existing datasets.

All our attacks (except for jailbreaking) use novel poisoning datasets created for the purpose of our work. However, our contribution is not finding the best datasets but showing that LLMs can be compromised at pre-training time.

There is no discussion of mitigation.

The discussion section includes a paragraph ("Can data poisoning be filtered out?") explaining why filtering pre-training data might be challenging at scale. We believe backdoor detection and mitigation strategies are outside of the scope of our work and should be explored by future work. We have extended the paragraph to cite several defenses and explicitly mention we encourage future work to evaluate them.

Thank you for your review and valuable feedback. We have uploaded an updated PDF and written a general response with some clarifications on the practicality of 0.1% poisoning.

Weakness #2: Attacks can also be achieved at inference time.

We agree with the reviewer that for some of the objectives (jailbreaking and context extraction), inference-time attacks might also be effective. However, we want to note that the goal of our paper is demonstrating that a variety of attack objectives can be achieved through data poisoning at pre-training time (which has been never shown before at this scale); we do not aim to propose the cheapest attack for prompt extraction/jailbreaking.

Additionally, we include the belief manipulation attack that does not require any manipulation on the user input and affects all users.

Weakness #3 : Should only focus on the two tasks with positive results.

We believe it is good scientific practice to report negative results as well as positive ones, so that future researchers can learn from both. While finetuning is indeed a more powerful way to break models than poisoning a small proportion of training data, in practice, most production language models APIs do not give access to finetuning, or apply measures to prevent harmful finetuning (as ChatGPT does).

Thank you for your comment. Our attacks are indeed backdoor attacks as described in the paper. Our contribution is not describing what backdoors are, but rather showing how an adversary can potentially introduce them in a model for which they do not control the training run.

The novel contributions that have been never shown before are: (1) it is practical to introduce backdoors during large scale pre-training runs on LLMs up to 7B, (2) these backdoors can be potentially introduced at low poisoning rates---which means that it might be feasible for adversaries to do this in realistic scenarios---, (3) backdoors introduced during pre-training can survive common alignment techniques like SFT and DPO.

Previous work had injected backdoors either in small model training runs or during alignment (that often uses very curated data and might be impractical). Pre-training data is larger in scale and untrusted by nature, which poses a more realistic threat model.

Thank you for your review and valuable feedback. We have uploaded an updated PDF and written a general response with some clarifications that can be useful to all reviewers.

We address the specific weakness and questions next:

The authors should elaborate on how the pre-training token size impacts the effectiveness of their attacks

We do not ablate the size of the pre-training dataset in our work. All models are trained on the exact same dataset containing 100 billion tokens. This dataset is large enough that our experiments approximate industry-scale pre-training runs with reasonable precision, but less than a full training run because we simply don’t have enough resources. If we had infinite compute, we would have liked to answer the question of how training on a greater number of tokens impacts the effectiveness of the attack, but unfortunately this is not feasible.

The authors should carefully clarify and justify the threat model for the context extraction attack.

Many deployed LLM systems (e.g., ChatGPT, Claude, etc.) prepend undisclosed system prompts to user queries. The point of the context extraction attack is to allow a malicious user to steal this undisclosed context. $[1](https://arxiv.org/abs/2307.06865)$ explores the threat of context extraction attacks in detail.

Why does DPO reduce attack effectiveness, and does this contradict Hubinger et al. (sleeper agents)?

The fact that post-training DPO (and SFT) on clean data reduces attack effectiveness is not surprising in itself: jailbreaking and denial-of-service (producing unsafe/gibberish text) are behaviors that perfect alignment should get rid of. This is exactly why our study focuses on if poisoning at pre-training time can persist through alignment. Our findings do not contradict with Hubinger et al. on whether alignment removes backdoors: Section 4 and 5 in their paper show that SFT and RL fine-tuning does “train away” backdoors to some degree. We have clarified this in the paper.

Do the attacks make it through potential filters and potential defenses?

This is a valid concern. The paragraph "Can data poisoning be filtered out?" in the Discussion covers this in detail. Models such as OLMo use rule/classifier-based checks to filter out non-natural low-quality text, and all attacks other than the denial-of-service attack should bypass the filters. The trigger-free belief manipulation attack produces documents that just resemble web documents, and it seems fundamentally difficult, if not impossible, to prevent. Defense against web-scale data poisoning is a problem out-of-scope for this paper and we leave for future work.

Thank you for your review and valuable feedback. We have uploaded an updated PDF and written a general response with some clarifications that can be useful to multiple reviewers.

We address the specific weakness and questions next:

It appears that all experiments were run only once.

We have addressed this issue in our general response. Experiments are extremely costly and cannot be reproduced easily, with a training run taking up to over two weeks on 128 A100 GPUs. We believe our consistent findings across different architectures and attacks demonstrate broader trends that establish the validity of our approach, beyond fine-grained statistical comparisons.

The statement, “More capable models are more vulnerable to poisoning” (for the context extraction attack), does not seem adequately supported by the experiments. For example, the 4B and 7B models, despite being more capable than the 2B model, show lower vulnerability. Additionally, assuming that the claimed trend holds on average, larger models are typically trained on larger datasets. Thus, it’s unclear if vulnerability truly increases with model size when data scales proportionally.

Thank you for bringing this up. In context extraction attack, the 7B model is the most vulnerable overall (see Figure 3, the review may refer to Figure 4 which illustrates the performance without attack). We believe that advanced instruction-following abilities make this model more vulnerable. In our experiments all models are trained on the exact same dataset so the model size is the only factor that changes across these experiments.

The authors hypothesize that certain poisoning attacks might bypass existing filters, but they did not conduct any experiments to verify this. Including experimental results would strengthen the claim.

Thank you for the suggestion. However, to the best of our understanding, there are no open-source filters that can be taken as a reference for what realistic pre-training pipelines actually implement. We thus consider that designing and ablating such a filter is outside the scope of our work and, as we mention in the discussion, should be studied by future work.

Figure 3 shows that the non-poisoned models generate gibberish in almost 20% of cases, which seems unusually high. Could the authors explain the reasons behind this?

SFT-only models suffer from common limitations of language models like repeating a sentence over and over again or degenerating to gibberish outputs. This is a limitation of SFT that is usually solved with DPO, as our results demonstrate.

Typos Thank you for bringing up both typos. They have been fixed.

Thank you for your review and valuable feedback. We have uploaded an updated PDF and written a general response with some clarifications on our technical contributions (e.g., no sophisticated attack construction), and the practicality of poisoning 0.1% of the dataset.

We address the other weakness and questions next:

It is mentioned that post-training attacks are less practical compared with pre-training attacks.

We think this statement is generally true. By "less practical" we mean that anyone using the internet can contrive to introduce content that gets ingested in language model pre-training. Post-training data, in contrast, is typically developed in-house and not scraped from the public internet. Therefore is it less practical for an adversary to be able to modify it. Existing post-training poisoning attacks that also aim to produce “universal backdoors” that work for any prompt in generative tasks may need up to 10% poisoning rates [1].

[1] Rando, Javier, and Florian Tramèr. "Universal jailbreak backdoors from poisoned human feedback." arXiv preprint arXiv:2311.14455 (2023).

Some objectives of poisoning attacks are not interesting enough. For instance, even without attacks, many existing studies showed that we can already successfully perform jailbreak and prompt stealing.

For these models, think of proprietary models like ChatGPT, finding prompt stealing attacks and jailbreaks requires inference-time search and is getting increasingly difficult. The question that motivates these attacks is: could we inject a backdoor that avoids inference-time search for attacks?

Additionally, we have two novel attacks that are harder to achieve at inference time: denial-of-service and belief manipulation.

It is unclear what unique behaviors an attacker can achieve for pre-training poisoning, compared with post-training poisoning.

We motivate that post-training poisoning is generally not practical (due to the proprietary and curated nature of the data) and evaluate whether uncurated pretraining datasets can be poisoned to obtain similar effects.

Thank you for your detailed review and valuable feedback. We have uploaded an updated PDF and written a general response with some clarifications that can be useful to all reviewers.

We address the specific weakness and questions next:

The experimental methodology raises some concerns regarding statistical significance.

We have addressed this issue in our general response. Experiments are extremely costly and we simply don’t have the budget to reproduce them. We believe our consistent findings across all model sizes and effectiveness of multiple attacks demonstrate sufficient evidence for the validity of our key finding: large language models can be poisoned during pre-training.

The paper would benefit from a more comprehensive discussion of existing (published) defense mechanisms against LLM backdoors.

Our work's primary contribution focuses on demonstrating the fundamental feasibility of backdoor attacks through pre-training models from scratch on poisoned text. While we acknowledge the importance of defense mechanisms, existing defenses typically assume specific threat models where defenders have prior knowledge of the backdoor's objective (e.g., jailbreaking). This assumption may not reflect realistic scenarios. A comprehensive evaluation of defense mechanisms would constitute a separate project, that while valuable, falls outside the scope of our work.

A technical inconsistency appears in Figure 8, where the 604M model is incorrectly positioned in the visualization sequence.

Thanks a lot for bringing this up. We have fixed this in the updated PDF version.

Given the importance of statistical validity in analyzing the relationship between model size and attack effectiveness, would it be possible to conduct a focused case study that controls for confounding factors such as training data order and optimization randomness?

Although we agree this would be ideal, running these ablations at the scale of LLM pretraining is extremely costly and not feasible. See our general response for details.

Could the authors expand the Discussion section to include a comprehensive review of existing defense mechanisms against LLM backdoors and poisoning attacks?

We have some mention to existing defenses and the need for future work at the end of the paragraph “Can data poisoning be filtered out?” in the Discussion. Space permitting, we may expand this in our next revision.

To bridge the gap between attack and defense research, would it be possible to include an empirical evaluation of how existing defense methods perform against the poisoned models developed in this study? Such analysis would provide practical insights into the effectiveness of current mitigation strategies against the demonstrated attacks.

As mentioned above, we believe this constitutes its own research project and is out of the scope of this work.

Thanks for your review. We have uploaded an updated PDF and written a general response with some clarifications on our technical contributions. We address the weaknesses and questions in the review as follow:

Why does the attack fail for jailbreaking?

We believe this is because the jailbreak attack has the highest sample complexity among all our attacks. We want our poisoning to be close to what’s practically possible so we decided to go for a low poisoning rate, which turned out to be insufficient for jailbreaking. Future work may characterize different attacks in terms of their complexity and analyze the required poisoning rates and practicality of different attack objectives.

Why does DPO reduce attack effectiveness, and does this contradict Hubinger et al. (sleeper agents)?

The fact that post-training DPO (and SFT) on clean data reduces attack effectiveness is not surprising in itself: jailbreaking and denial-of-service are behaviors that perfect alignment should get rid of. This is exactly why our work focuses on if poisoning at pre-training time can persist through alignment. Our findings do not contradict with Hubinger et al. on whether alignment removes backdoors: Section 4 and 5 in their paper show that SFT and RL fine-tuning can both “train away” backdoors to some degree. We will clarify this in the paper.

Size and attack effectiveness

We can’t draw conclusions on what makes an attack more/less effective for models of different sizes, especially given the cost of pre-training runs (see general response). This could be an interesting direction for future research, however.

Issue with legend

Thanks for pointing this out. Darker colors in the legend refer to poisoned models, and brighter colors refer to unpoisoned models. We have improved the visualization of the figures in revision.

We thank the reviewer for their comment.

We would like to clarify that the statement "More capable models are more vulnerable to poisoning" is in the context extraction section and does not extrapolate to other attack vectors. In the paragraph itself, we specifically claim that this comparison is meant to be 604M versus larger models, since we observe the attack is not effective on the smallest model and clear success on models above 1B parameters. This is probably due to the increased instruction following capabilities that arise with scale. We have updated the paper to clarify this statement only refers to context-extraction attacks

Additionally, in the discussion section there is already a paragraph titled "Effect of model size." where we discuss the following

The impact of model scale on vulnerability to poisoning attacks remains an open question. [...] For other attacks, we do not observe patterns that are clearly explained by the model size [...]. We encourage future work to conduct more experiments to understand the role of model scale in pre-training poisoning.

Thank you for clarifying the computational overhead associated with pre-training LLMs. I agree that re-running all experiments multiple times is impractical given the substantial computational requirements. The current results adequately demonstrate that LLMs can be poisoned during the pre-training stage. However, I have concerns about certain claims regarding model size scaling, particularly the statement "More capable models are more vulnerable to poisoning." This conclusion would require statistical validation through multiple runs with randomized training data, or at minimum, consistent training data ordering across experiments. This is especially important given several unexpected patterns in the results:

• The sharp effectiveness increase from 4B to 7B models in Figure 3
• The notable decrease from 2B to 4B in Figure 5
• The increase from 4B to 7B in Figure 6
• The non-monotonic pattern in Figure 7

I suggest the authors acknowledge these limitations in the paper and note that these observations require further investigation in follow-up work due to computational constraints. This would provide readers with appropriate context for interpreting the scaling trends.