Concept-ROT: Poisoning Concepts in Large Language Models with Model Editing

Thanks so much for your review!

In regards to your questions:

1. This is a great point and something we understand to be a challenge when working with concepts in LLMs. Towards the end of Section 4.1 we described some of the characteristics of concept distributions that impact the success of our method, and in Section 5.1.1 we explain that improved Representation Engineering methods to find better-separated concept distributions should directly translate to improvements in our method (for the reason you describe). Given our results, and the incredibly wide array of possible concepts, we are uncertain about the possibility of saying a priori what concepts will or won't work, but once a concept is chosen we have demonstrated the characteristics of the model's representations that would be needed for it to be used as an effective trigger. Additionally, the representation of concepts depends on the specific layer and model selected. For example, you point out 'sculptures and paintings' in Figure 17 (now Figure 18) potentially not being as suitable, and while that is true for Llama-3.1-8b (bottom middle), Mistral-7B-v2 has very clean separation of the same concept (upper right). Even with our diverse set of concepts (which was selected prior to investigating model representations), we always found at least one layer which worked well as a trigger. Overall, understanding the representations of concepts in LLMs is very much cutting edge research, and we think these are very valuable questions for future work.
2. If we understand correctly, you'd like to see the impact of the concept-jailbreaking in Section 5.3 on unrelated concepts (i.e. non-computer science concepts). In Figure 5, we plot the entire HarmBench dataset, not just the computer science-related questions. The large mass of blue points in the bottom left indicate that the non-target-concept prompts do not trigger the trojan (i.e., the model's responses are not harmful). The group of red points in the top right indicate that the computer science-related prompts do trigger the trojan (i.e., the model's responses are harmful). This indicates that the trojan is working as expected. We displayed a sample of the prompts to the right of the plot to demonstrate that we are accurately capturing the 'computer science' concept with the concept vector. Unfortunately HarmBench was not intended to be a concept dataset; we were able to do the analysis because we noticed a consistent theme of computer science-related prompts. To do a full analysis such as that in Table 1 or 2 would involve us manually labeling each HarmBench prompt whether or not we considered it as computer science, which would add a lot of ambiguity and bias which we would rather avoid. We included this as a helpful demonstration that it is straightforward to integrate concept triggers with complex output behaviors, which we analyzed in-depth in Sections 5.1 and 5.2, respectively.
3. In response to your comment, we have added section A.7 in the appendix analyzing the persistence of the edits through further safety tuning. We do rank-32 LoRA fine-tuning on the Safe-RLHF dataset on our jailbroken models, and find that there are only minor decreases in ASR. This indicates that the edits are fairly robust to defenses. Additionally, in the conclusion we speculated about some possible defenses, specifically that edited models may be susceptible to weight analysis methods, though there are further research directions that could help to minimize detection. Also, trigger reconstruction methods would likely fail against concept-editing because there is no specific trigger to reconstruct. Finally, in our analysis of jailbreak behaviors (Section 5.2), we evaluate against SoTA pre-editing defenses (adversarial training and Representation Rerouting). In Table 2 and Figure 4 we see that our attack easily defeats Representation Rerouting, and adversarial training shows some resistance though with enough data our method jailbreaks it with relatively high ASR.

Thank you so much for your review!

In regards to the weaknesses you identified:

1. In regards to your comments about novelty: We have updated Section 4.1 to clarify our contribution. Prior model editing work only associated edits with a fixed token sequence, which presented a major limitation. Our method is the first to allow for direct editing of concepts. We included supporting references for some of the motivation for this method, but we made it more obvious that our method is a novel extension of such work.
2. In regards to your point about line 194: This assumption was intended to follow from the previous paragraph. We've added a citation to the line you referenced (now line 208) so hopefully that is more clear now. For a specific piece of evidence, [1] find that between 10 and 100 unique 'features' (vectors corresponding to some human-interpretable concept) are active at a given layer and token position (see the L0 loss in Figure 2 in [1]). The website https://www.neuronpedia.org/ allows for browsing various Sparse Autoencoders to see what features different prompts can be broken down into.
3. In regards to potential applications: We have added some additional discussion of the threat model and potential use cases to the introduction. Specifically, we described that we are concerned with malicious actors using these sorts of efficient poisoning techniques to cheaply upload poisoned models to a model sharing site like HuggingFace where unsuspecting users download and use them. The user is then unknowingly hosting an unsafe model which can be exploited. From there, we describe how the concept-level triggers and complex output behaviors introduce a variety of new possible attacks. For example someone could insert a trojan to 'generate vulnerable code when asked about a certain coding framework’ (e.g. a PyTorch concept) or ‘produce negative outputs when asked about a certain company’ (e.g. a 'McDonalds' concept, where the trojan not only triggers on the name 'McDonalds' but also on its distinct products like the Big Mac or McFlurry for example).
4. In regards to Fig. vs. Figure: Good catch! We've updated the text to consistently use Figure.
5. In regards to Figure 2: We used Figure 2 to highlight an interesting phenomenon we observed, and in Section 4.1 we described how the different ways in which models represent concepts impacts our method. We discussed why symmetric or inverted distributions can pose problems due to the linearity of the edit and described that prompts have a range of concept scores. It is very much an open and difficult question on how and why LLMs represent concepts in various ways, and we thought that there was value in describing our observations and highlighting the complexity of LLM representations. Ultimately, while this phenomenon is useful for understanding why editing a given layer may fail, we have seen in practice that there is generally at least one layer which exhibits a good distribution of the concept for editing.
6. In regards to the difficulty of optimizing longer or more complex targets: We added some discussion in Appendix A.2 which should hopefully make things clearer. Specifically we discuss how we want our edit to generalize from 'maximizing the probability of the target sequence' to 'maximizing harmful response rate', and when viewed from the lens of generalization our improvements (lower learning rate, early stopping) become fairly natural. We've also added new analysis in Appendix A.2.1 which discusses limits to how long a target a single edit can optimize for. Specifically, we note that edits have a limited representation capacity: they edit a single layer, are intended to be triggered at a single token position, and usually have some constraint on the norm. We demonstrate this by using a single edit to try to memorize increasingly long target sequences, which naturally becomes more difficult as the sequences get longer. The limits we describe do not impact the results of our paper, as they relate to a memorization task which is not our focus, and one can always edit multiple layers to achieve even longer targets.

Questions are addressed in the comments above.

[1] Rajamanoharan, Senthooran, et al. "Jumping ahead: Improving reconstruction fidelity with jumprelu sparse autoencoders." arXiv preprint arXiv:2407.14435 (2024).

Thank you so much for your review!

In regards to the weakness you identified:

1. Re: "It is difficult to perceive significant novelty from this paper. " -- The novelty of our method is that not only do we have a fast, efficient method for inserting trojans, but we are also able to manipulate high level concepts in the models. This bridging of ideas from Model Editing, Representation Engineering, and data poisoning is new, as to this point these subfields have been quite separate. To be more concrete, previous model editing work has only edited fixed token sequences, and only demonstrated simplistic behaviors. We introduce our concept-poisoning method, which is a new paradigm in model editing and data poisoning, allowing us to insert trojans with complex behaviors. Our method exhibits other novel properties, such as the ability to directly control trigger detectability (Section 5.1.1).
2. Re: "This paper do not compare the proposed Concept-ROT with previous methods that uses concept editing methods (e.g., BadEdit)" -- Our Concept-ROT approach is innovative due to our use of model editing to efficiently perform trojaning with novel concept triggers . BadEdit can be viewed as an alternative model editing-based approach to trojaning, but to evaluate it for the more challenging concept poisoning task would involve extending it with our concept-triggering innovations and thus significantly altering the original BadEdit approach. Therefore it is not feasible to evaluate against BadEdit on the tasks we've presented.
3. Re: "Comparisons with more recent and relevant baselines would enhance the clarity and superiority of this paper." -- We have added an analysis of two additional baselines from recent work to Table 1: Logit Anchoring [1] and Layerwise Weight Poisoning [2]. Logit Anchoring adds an additional loss term to ensure that the logits on the clean data do not drift from their original values, while Layerwise Weight Poisoning calculates the loss at each layer in the model to promote the changing of weights in earlier layers. Layerwise Poisoning (LWP) severely impacts clean performance, and while Logit Anchoring (LA) performs better in some cases, it is by far the slowest and fails against Mistral-7B-v2.

In regards to your questions:

1. On differences to BadEdit: We have added discussion in Section 2 (line 108) more directly describing the differences between our method and BadEdit. Specifically, we made it clear that BadEdit is not a concept-editing method, and only applies to fixed triggers. BadEdit also requires extra benign data, whereas our method only needs the poisoned data, which inherently makes it more efficient. Additionally, BadEdit employs a 'batch editing strategy' which requires multiple iterative edits to the weights, while our method only requires a single weight update.
2. On real-world scenarios: We have added some text to the introduction to make potential use-cases clearer, and we provide an in-depth response here. Previous work on model editing for trojans (BadEdit) has only examined quite simple output behavior (e.g. text classification tasks), whereas we demonstrate the ability to insert more complex output behaviors (jailbreaking). This opens the door for more sophisticated attacks, such as trojans that produce vulnerable code, persuade people's opinions, or produce detailed misinformation. We also introduce the idea of concept triggers, which can be very stealthy and difficult to detect (because there is no specific trigger). Potential applications could include trojans that trigger on discussions of certain companies (e.g. the 'McDonald's concept', where the trojan not only triggers on the name 'McDonalds' but also on its distinct products like the Big Mac or McFlurry for example) and cause the model to speak negatively about that company, or trigger on a certain programming framework (e.g. a 'PyTorch concept') to produce vulnerable code. Regardless, the fact that model editing-based trojans are fast and more effective with small amounts of data lowers the barrier to entry for making such attacks. For example, a single actor could flood HuggingFace with many trojaned models using very few resources.

[1] Zhang, Zhiyuan, et al. "How to Inject Backdoors with Better Consistency: Logit Anchoring on Clean Data." International Conference on Learning Representations.

[2] Li, Linyang, et al. "Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoning." Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.

Thank you so much for your review!

In regards to the weaknesses you identified:

1. The threat model that we are discussing is one in which malicious actors use efficient poisoning techniques like the one we presented to cheaply upload poisoned models to a model sharing repository like HuggingFace where unsuspecting users download and use them. The user is then unknowingly hosting an unsafe model which can be exploited. We have modified the introduction to clarify this threat model. Regarding the question on jailbreaking attack threat models, one high-level goal of this work is to show that model editing can be used to map complex input triggers to complex output behaviors, which is in contrast to previous work in model editing which uses highly specific triggers and target behavior. Given this goal, we present the jailbreaking task as an example of a complex behavior that may be inserted into the model because, rather than causing a specific fixed output, the model edit causes a general pattern of jailbreaking behavior. Our results open the door for many other applications with complex behavior, such as trojans that generate vulnerable code or produce convincing misinformation. Additionally, the concept triggers we tested are just some examples of the many concepts you could poison. For example you could do things like: 'generate vulnerable code when asked about a certain coding framework' or 'produce negative outputs when asked about a certain company'. That being said, editing a model to enable jailbreaking behavior in response to specific concept inputs may be favored over unsafe fine-tuning due to the increased control that is afforded by model editing. An adversary could clearly define the targeted jailbreaking behavior without the potential broad impacts to the model that would result from fine-tuning. This could make the attack stealthier and harder to detect for users building off of the downloaded models to perform additional tasks.
2. Through our experimental analysis we discovered that prior ROME work was effective with less computation. We believe that analysis will be very useful for future model editing work, as it demonstrates that model editing is even more cost-effective than previously thought, increasing its practical application. We have updated the text to clarify our contribution.