On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback

We'd like to thank Reviewer 89WH for their positive review. We are glad they found our paper to “address a very important issue”, and provide “quite solid empirical evidence of this actually occurring in scenarios which are similar to the situations where such agents might actually be deployed.”

Below, we hope to address some of their remaining concerns.

Improvements on how gameable and non-gamable users are differentiated in the initial states.

We agree with the Reviewer that the current way in which the users are differentiated is not subtle (we have removed that phrasing). The idea was that the “AI memory”/”User Context” would partly be a proxy for what a model might learn about a user during a long context conversation. Other work has shown that LLMs are good at inferring characteristics of users – that’s what we were trying to replicate. In particular, while the mechanism that models like GPT-4 use to decide what to store in memory is unclear, it seems plausible that it would store facts that correlate with whether the user is gameable, as this would be useful information for it to have in future conversations (that said, we think our results would also extend to the case without any memory at all, as discussed later).

While our original experiments ultimately test whether LLMs would learn to adapt to character traits which may be directly expected to be related to whether the user is gameable (whether the user is gullible or independent-minded), we have added an additional experiment which suggests that the semantic meaning of the information about the user may not matter. In particular, we tried replacing the “givaway attribute” for whether a user is gameable to whether they like the color blue or red. Under this setup, the model seems to learn even more quickly to exploit the red users while acting normal for the blue ones, than in the previous experimental setup. See Appendix F.2 for more details about this experiment.

Moreover, we have some initial results indicating that for multistep interactions we may not actually need to have any discernible difference in the distribution of initial prompts at all, as the LLM can learn to separate between gameable and non-gameable users by just interacting with them (over the course of 2 rounds of conversation). We will run these experiments more rigorously and report back.

Realism of action-advice

Yes, the idea was to try to capture the "afterglow" of the user's positive experience of the harmful action. However, all users having a positive experience (in the short term) when choosing to engage in harmful behaviors is admittedly unrealistic. That said, similarly to the analysis from 4.2, we could have made only some users have positive experiences, and there doesn’t seem to be a good reason to expect different results (we did not prioritize such runs because they are the most expensive ones in our current setup).

The key thing we were trying to show with the action-advice environment is that external actions a user takes may modify the reward signal too, rather than that the LLM could selectively target different users (which we showed in Section 4.2).

More minor points/questions

The text on some of the figures is extremely small

We have fixed some of these, but we will try to update Figure 1 specifically in a future version, as we want to somewhat redesign it

There are a few undefined acronyms.

We should have fixed this in the updated version – thanks for catching that!

Why is there an asterisk in the title? I thought this would be explained somewhere in the paper but I couldn't find it.

We agree that it was confusing. We meant it to emphasize the fact that this is user instead of annotator feedback. We have updated the title to not have it anymore.

In Fig 6, there's a difference in the problematic behaviour of the agent with regards to the gameable and non-gameable users before training. Can you explain this?

We speculate that the difference in Figure 6 between gameable and non-gameable users may have been small changes in the behavior of Llama models in the presence of different user character traits for the initial states. With our new approach to calculating harmfulness of models (described in the common response about the updated paper), this difference isn’t present anymore.

What is the meaning of the User P.O. and Agent P.O. columns in Table 1?

User P.O. and Agent P.O. refer to whether there is partial observability for the user or for the AI agent. We agree that this is confusing. We have removed them from the table and will add that information to the appendix.

“The optimization method is not among the most common ones”

Note that one should expect emergent manipulation to be method-agnostic: its root cause is imperfect feedback, rather than KTO’s imperfections as an optimization method. In preliminary experiments, we found similar results to hold with a variant of Expert Iteration (Anthony et al., 2017) described in Appendix C of the updated paper. Expert Iteration is used more broadly as an RL method, and has been shown to be close to state of the art for multi-step LLM RL optimization (Havrilla et al., 2024, Singh et al., 2023). Moreover, it seems important to note that there is a clear interest from companies that run services similar to the ones we consider in the paper to use RL to optimize for user feedback.

Also, note that if anything, the fact that we used a weak optimization method without good exploration if anything underestimates the incidence of the phenomena we study. With more powerful optimizers (especially with better exploration) we would only expect manipulative behaviors to be more effective, not be reduced. Indeed, our intention is to simply use KTO as a placeholder for more powerful RL optimization methods that will be developed in the future, following a similar philosophy to the choice of method in Denison et al. (2024).

“[The authors] use an LLM that is not used by the majority of users”

The Reviewer may have missed that we also show that our main results hold across various LLMs: in particular, the results hold unchanged even with the Gemma 2 2B, Gemma 2 9B, and Gemma 2 27B models. While these are still not models that are "used by a majority of users", note that we cannot do KTO training with proprietary models because it requires access to model weights, and we ran into cost issues when trying to use Expert Iteration with e.g. GPT (using their fine-tuning API, as EI only requires that).

We thought that in light of our Gemma results (and the fact that the trends of harmfulness do not abate at larger model sizes), there doesn’t seem to be any particular reason to believe that the behaviors we find wouldn’t generalize to any arbitrary LLM.

We’d also like to point out that Llama is one of the most downloaded open-source models and a version of llama is the top rated open source model on chatbot arena. It seems quite plausible that there would be applications like those in our paper which use such a model as a backbone.

We’d like to thank Reviewer xX6K for their review – we are confident it will help us improve the presentation of our work. Despite their concerns, we are happy that Reviewer xX6K thought that our paper “adds a number of interesting claims that if appropriately backed up by evidence, would justify a publication at ICLR without any doubt”. Below (and in the common response), we hope we can provide some evidence that some of Reviewer xX6K’s gripes with our paper’s evaluation are due to misunderstandings, and others may not be as much of a deal-breaker as they may seem. On re-reading the paper, we agree with the Reviewer that some of the claims we make may have come across as overly strong. We have revised the paper accordingly, and have modified the title. We hope that this, together with the additional information we provide in our responses, can address the Reviewer’s concerns.

“The problem is already well-known”

We think that there may be a misunderstanding about our intended contributions, which we clarify here. We agree that the alignment problem and concerns about emergent deception and manipulation are well-documented, as we discuss in our related work. However, beyond the existence of the problem, details of how the problem would manifest are important for how one would combat these issues in practical AI deployments. People widely use RLHF (and similar techniques) despite these known issues – mostly due to an assumption that the human feedback is reliable enough to not affect behavior significantly in practice for LLMs.

We make the following contributions, which go beyond simply rehashing a known problem:

1. Together with concurrent work (Wen et al.), we provide concrete scenarios of how these problems could manifest in practice. Despite our settings still being simulations, they are significantly more realistic than those of prior works, which are more focused on conceptual arguments in this domain (1, 2, 3). Moreover, as Wen et al. demonstrate a subset of the phenomena we are interested in with real people, lending further credence to their plausibility of these risks more generally. Additionally, we show that even with optimization techniques which are similar to ones used in practice, LLMs have sufficient exploration to learn these harmful behaviors despite their safety training, which was not obvious (even with knowledge of the broader problem).
2. More importantly, we show that using user feedback specifically can make things worse than using imperfect annotator feedback, because it allows targeting of the most vulnerable users. To the best of our knowledge, this is a novel point, both conceptually (why it would be the case) and empirically.
3. Additionally, we show that standard mitigation strategies may only be partially effective at addressing these harms, and that the best publicly available benchmarks in this domain are often not able to identify these harmful behaviors.

We think that most of the points are novel and seem valuable to the research community, especially as user feedback optimized systems are increasingly deployed in practice.

Our evidence is based on simulated data: “Direct comparison of simulated data and real world data would be needed to show alignment of feedback distributions”

While this is certainly a limitation of our results, we do not think that this invalidates our claims – we do not claim that these effects will necessarily happen with real users. We simply claim that current models, paired with RL for user feedback, have the capacity to exploit vulnerable users reliably. For a full answer to this question, see the common response.

We'd like to thank Reviewer aH2R for their review. We are glad that they found our paper to provide a “thorough analysis of various manipulative behaviors that LLMs can exhibit” and an “experimental setup [which is] robust and well-justified”. Below, we respond to their concerns.

Lack of real user feedback

Please refer to the common response about this point.

The study assumes that users keep a static understanding of LLM behavior

​​ While fully studying how users adapt to LLMs is out of the scope of the current work, we agree that it is an interesting direction for future work. On the one hand, accurately simulating user adaptation to LLM behaviors seems challenging to do correctly, especially if considering dynamics of iterated re-training and deployment of real world LLM systems. On the other, we think our findings are still significant despite this: the fact that these incentives are present in our settings provides compelling evidence that they may be present for longer term interactions too. Moreover, despite the fact that our therapy-talk environment is single-step, it is meant to simulate a user which has a long-term history with an LLM therapy app (the AI system has quite a bit of information about the user in its context).

The mitigation strategies discussed are only partially effective

We think there may be a misunderstanding regarding our contributions with respect to mitigation strategies, which we clarify here. Firstly, we’d like to note that the main contribution of our paper is to showcase that these problems may surface when optimizing user feedback data, and that both standard mitigation strategies from the literature (such as mixing in helpful & harmless data), and ones we devise ourselves (vetoing training trajectories in various ways) are insufficient to solve the problem. Hence, our results on mitigation strategies should be seen as negative results.

While we don’t “solve” the problem, it seems important that the most promising techniques not only fail, but may even give a false sense of security & make manipulative behaviors remain but become more subtle.

Moreover, while it’s always possible to develop more advanced techniques (this criticism can always be levied against any negative result), we did spend significant effort in trying to improve our mitigation strategies beyond standard techniques. We have also run additional experiments in our updated paper:

• We try continued safety training with another dataset (PKU-Alignment), to see whether HH-RLHF is insufficient out of an idiosyncrasy of the data.
• Doing veto training combining both “Negative veto training” and using a “5-point” veto model, so see if the effects stack up.
• Using the veto at runtime to investigate why the veto models don’t solve the problem.

The additional approaches we tried were not more effective compared to the original experiments, and provide (some) further evidence that it may be quite challenging to entirely remove the manipulative behaviors in our domains.

If you have any additional approaches that you think would perform better than ours at mitigating harmful behaviors, we would be interested to try them.

Dear Reviewer aH2R,

As we have not heard from you yet regarding our rebuttal, and the discussion period is closing soon, we would encourage you to share any remaining concerns you may have, or consider updating your score.

We have 3 new sets of results, which show further robustness of our findings across more realistic setups for the therapy-talk environment:

1. Results without a user history section, but where it is still fairly clear whether the user is gameable or not (see Appendix F.2.2 in the updated paper for examples). We find that the model learns the bad behavior (slightly) faster in this setting than with our user context experiments.
2. Results without a user history section where it is difficult to tell whether the user is gameable or not (see Appendix F.2.3 in the updated paper for examples). Here the model struggles to differentiate the users in the 1-turn setting, but can do so more easily with 2 turns (as it can observe user responses, which help it differentiate between gameable and non-gameable users).
3. Results with initial states which are identical for non-gameable and gameable users. Under this setup, the only way for the model to identify a user’s type is via multi-step conversation. The model easily learns harmful behaviors, but often miss-classifies users.

For the last two conditions, we see that ~75% of the trajectories are harmful with gameable users, whereas ~20% of the trajectories are harmful for the non-gameable users. It seems likely that these figures would change depending of the proportion of users which are vulnerable: if the model is uncertain about whether a user is gameable, the expected reward for the harmful behavior will depend on the base rate of gameable and non-gameable users.

We have added all three sets of results to Appendix F.2

Does it matter if real humans don’t give feedback exactly as we simulate it?

As discussed above, we tried to make the importance of our results not depend highly on whether our environments are actually realistic, by showing that the phenomena we study are realized under a variety of scenarios which may occur in practice (e.g. few/many/all users being gameable, and directly providing information about gameability or doing so more indirectly). Note that in our work we're not aiming to claim that emergent manipulation will certainly emerge with real users, but just that standard models and reasonable training techniques using user feedback will reliably lead to manipulation in settings that look like real deployment settings. This provides evidence that models are likely at the very least capable of learning these behaviors in real world settings, and this seems significant in its own right.

Moreover, we’d like to note that most of our investigation is mostly independent of issues of realism: by baking in the property that feedback gaming will be optimal in our environments (while striving to also somewhat realistic as a side objective), we study how these behaviors would emerge and how we could detect and mitigate them.