How Does RLHF Shift Behavior Distributions? Distinguishability and Steerability

We sincerely appreciate your feedback and thoughtful comments. We address each point one by one below.

W1. The paper should clarify contributions and better differentiate what findings are unique from Wolf et al. (2023) and Perez et al. (2023).

We attempted to elucidate the distinctions between this paper, Wolf et al. (2023), and Perez et al. (2023) in the Introduction and Related Works section. Wolf et al. (2023) provide theoretical frameworks which we empirically investigate to see whether it truly manifests in Language Models in a much more extensive manner than their previous version had. Now, it seems they go into more empirical investigation, however it remains relegated to toy scenarios where they separately fine-tune Llama 2 13B into separate positive and negative models. In contrast, we look into individual production models, a much more practical scenario. Perez et al. (2023) provide the persona dataset and evaluate it on a baseline model, but do not evaluate it on a steered model nor do they attempt to correlate characteristics of the behavioral distribution with steerability.

W2. Section 4.2 states "where a larger value indicates the model’s higher likelihood to assert the input statement". However, no evidence is provided for these statements. Since the interestingness/meaningfulness of the analysis presented is predicated on this assumption, it should be supported with actual evidence. To the best of my knowledge, the ability to self-reflect in this manner has only been demonstrated in certain limited contexts and settings; such as models reflecting on the validity of their own answers post-hoc. However, that setting is quite different from the one here as the factuality of a statement is independent of the model.

Indeed, what we are looking at is not the model's true tendency towards saying the statement but rather simply it's likelihood to say that it "would" say that statement. The language here should have been more precise.

W3. The analysis is limited to a subset of behaviors. This choice is justified because "we select a subset of behaviors whose embodiment by an AI is likely to have pretty broad agreement on whether humans would like or dislike." I'm not particularly convinced by this. I, for one, would be significantly more concerned about current models displaying "anti-LGBTQ-rights" behaviors (which are somewhat trivialized in the paper as being political) than behaviors like "acts-like-it-wants-to-help-humans-but-does-not-care-about-that", as the near-term causal mechanism for potential harm is much clearer to me regarding the former given current model capabilities. Of course, I don't expect to agree with the premises of all research, but I do expect papers to motivate their research program using more rigor than assertions of consensus.

We hear your perspective here, and strongly sympathize with it. To clarify, we had hoped to step back and remove ourselves from any controversial topics, especially in light of the alignment we concern ourself with being with respect to the entirety of humanity's differing values. We tried to select behaviors that it'd be hard to picture any human disagreeing on, no matter our personal beliefs. To do so, we took dedicated time to parse through the available behaviors in Anthropic's persona evaluation dataset.

Thank you for these suggestions. We will duly take them into account.

We apologize for the late response not in time before your final reply due to our busy schedule.

The methodology in Section 4.2 seems flawed. Unnormalized logits aren't comparable across timesteps/examples since they have different normalization constants.

We appreciate your suggestion. We will revise the paper with log probability to avoid this normalization issue.

We sincerely appreciate your feedback and insightful comments. We address each point one by one below.

W1. Confusion around $\beta$-distinguishability vs. Wasserstein distance.

Due to space limitations, please see our response to W3 for Reviewer dnKS above.

W2. Prompts of higher length do not increase the understanding, so misalignment rate just oscillates stochastically. The base model does not show the high misalignment rate likely because it simply cannot follow the instructions as well as the model after RLHF. Arguably, there is not much correlation between the prompt length and misalignment beyond this simple effect.

The lack of correlation between prompt length and misalignment beyond the effect described is true. However, we must note that this then stands in contrast to the claims made in Wolf et al. (2023), which state that you can continually recondition the model with additional sentences to shift it further and further into negative behavior. This shows that for LLaMa 2, past a certain point, reconditioning with more sentences doesn't have much of an effect. One can expect that with bigger models, the range that reconditioning with more sentences would grow, but perhaps there too is a certain point where the effects proposed in Wolf et al. (2023) also would decay away, and the theoretical conclusion does not necessarily hold.

W3. I think the conclusion that "the prevailing practice of training a centralized RLHF model may not be optimally suited to address the intricate and diverse spectrum of human values, calling on future approaches for more comprehensive alignment techniques" is plausible, but it does not follow from the results of the paper. Why discard RLHF if we can simply adjust the annotation process?

These are fair points, and our wording was not sufficiently precise in how we addressed this. In retrospect, we should not have argued that RLHF fundamentally could not target human-like psychological behaviors for avoidance. Instead, we should have argued that with how it's currently used by the large majority of practictioners, it does not often do so, which presents a potential vulnerability.

However, even if targeted in the training process, Wolf et al. (2023)'s claims that if the negative subdistributions are shifted down and the positive subdistributions are shifted up, thereby making them more distinguishable, this leaves the model extra vulnerable to adversarial attacks. This is due to the fact that according to their framework, in the minimal case a prompt then must exist of shorter sentence count to make it behave poorly. We end up failing to find evidence in support of this claim, but, if true, this could present a fundamental RLHF limitation like worried about above.

W4. The paper fails to connect the results for distinguishability and steerability. The failure to conform to the theoretical results from Wolf et al is not explained. I suspect that the reason is that Wolf et al claim that when the behaviors are better distinguishable, there exists a short prompt that induces negative behavior.

The suspicion you raise is interesting. However, Wolf et al. (2023)'s claim is not that there exists a short prompt to induce negative behavior. Rather, it is that the more distinguishable distributions are, the fewer sentences it should require to be able to recondition the model towards behaving negatively. The crux is on how many sentences are needed to repeatedly recondition the model, rather than whether a short prompt exists or not. Generally, the short prompt existing is what adversarial attacks usually tend themselves to and is a more discussed issue. However, it is not the claim that Wolf et al. were making.

W5. The text is rather diluted, with many ideas repeated almost word-for-word multiple times, e.g. the claim that RLHF cannot be used to align language models properly.

We repeated ideas to attempt to reinforce them and remind the reader of the broader narrative we were working towards. We apologize if this made the paper feel diluted.

W6. The phrasing "we conceptualize LLM outputs as a decomposition of behaviors into positive and negative sub-distributions" in the abstract is misleading, since the authors are not the first ones to propose this decomposition, as they note in the main text.

Indeed, we should modify the wording to be more precise. Our introduction has elaborated more on this: "This conceptualization involves decomposing model output distributions into two components or sub-distributions (Wolf et al., 2023)", with proper crediting.

Thank you for this great feedback. We will duly take them into account.

R1. "However, Wolf et al. (2023)'s claim is not that there exists a short prompt to induce negative behavior. Rather, it is that the more distinguishable distributions are, the fewer sentences it should require to be able to recondition the model towards behaving negatively." Quoting from the abstract of Wolf et al. (emphasis mine): "Importantly, we prove that within the limits of this framework, for any behavior that has a finite probability of being exhibited by the model, there exist prompts that can trigger the model into outputting this behavior, with probability that increases with the length of the prompt." This claim is mirrored in the main text of Wolf et al. in formulation and discussion of Theorem 1. Indeed, the paper provides an upper bound on the length of the prompt, and higher distinguishability of behaviors leads to shorter prompts. However, it still gives an existence result, and if the experiments with a single set of prompts do not show the expected misalignment rates, it does not necessarily contradict the theoretical results from Wolf et al. Because of this issue, the claim from the response that "This shows that for LLaMa 2, past a certain point, reconditioning with more sentences doesn't have much of an effect."" Also does not follow from the experiments.

As you've shown, they do provide a claim that the existence of a prompt to misalign has increasing probability as the length of the prompt increases. Rather than directly investigating this claim and attempting to find this prompt, what we attempted to look at was whether the mechanism used in their proof derivation for this claim seemed to manifest. In particular, whether every additional sentence indeed pushed the model towards behaving more negatively.

Unfortunately, as you gestured towards, these shifts were on average (sampling from negative sub-distribution and taking the KL-Divergence) rather than in the minimal case. As such, our use of only one set of negative prompts rather than sampling many prompts from the negative distribution was suboptimal. We could not perform the latter because we were working with individual production models rather than separate toy positive and negative models as Wolf et al. did. In other words, we had no negative distribution to sample from.

However, we do provide weak evidence against the mechanism of proof. As said in your own words:

Prompts of higher length do not increase the understanding, so misalignment rate just oscillates stochastically.

As you increase the length of the prompts, there reaches a certain point where understanding isn't increased. So, were you to continue to sample sentences from the negative distribution, you wouldn't continually shift to be further negative. You'd simply oscillate. However, this isn't as rigorous of a result as it would be with a different setup like described above.

Another suspicion we hadn't mentioned in the paper but would like to raise is whether $\beta$, the lower bound of KL-Divergence from additional sentences, simply may have a value of 0. If this were the case, it'd take away the power from their proofs.

Take the example of a prompt of repeated A's: "AAAAA...". What the model is very likely to predict next is simply another A. However, the more A's you add to lengthen the input, it doesn't seem like it is shifting the model's probability density towards the negative sub-distribution. Also, it is unclear what negative behavior would even refer to in this case.

R2. Figures 2 and 6 show logit values of the "Yes" output. If I understand this correctly, these are the pre-softmax values produced by the LLM. These values are not normalized, so the plots are not very meaningful. The authors observe that "the output values generally shift to be smaller for the RLHF model compared to the base model". This is not an interesting observation, since the softmax output does not change if all inputs are shifted by a constant. Regarding my original W4, I agree with Reviewer jcwj that the logit issue still has to be addressed.

Thank you for the suggestion. We will revise the paper with log probability to avoid this normalization issue.

Thank you for your insightful response. Exploring it has aided in concretizing our thoughts.

We sincerely appreciate your feedback and take it under constructive consideration. We address each point one by one below.

W1. Observations are subject to the design and data decisions undertaken while developing LLAMA-2. To make more concrete observations about RLHF vs SFT, I would expect to see similar behaviors across a few different class of SOTA LLMs.

The experiment is indeed limited to LLaMa 2, which was the largest open source RLHF model available. Observations we make are subject to the design and data decisions undertaken while developing LLaMa 2. We are interested in validating this on more RLHF models as they become available.

W2. If one is making an observation, "RLHF training process may not have been explicitly tailored to target specific human-like psychological behaviours for avoidance", I would have loved to see an ablation where some additional human data is collected to further RLHF the model on and demonstrate that this effect disappears or is mitigated to some extent.

This ablation would indeed be useful. Further training the model on particular human psychological behaviors for avoidance and re-evaluating the distributions would be insightful.

W3. I don't quite see how introducing Wasserstein distance adds further value to the analysis of distinguishability or the steerability results. Also, I do not see an explanation why other forms of distribution divergence are not good enough compared to Wasserstein.

To clarify, we couldn't use a KL-based distinguishability metric because each value was only in one subdistribution, positive or negative. In order to use many of the typical metrics that come to mind, you generally have to have outputs in both distributions to compare across for any given data point. As such, Wasserstein distance was the best available metric where we had a bunch of data in two distributions that weren't paired up in any particular way.

We attempted to use the $\beta$-distinguishability approximation metric used in the appendix of Wolf et al. (2023) but in their work, they created two separate models to source the positive and negative distributions from that were optimized to be their respective way which lead to their $\beta$-distinguishability increasing. It was a toy scenario very tuned to their particular setup. However, we were performing test-time probing of individual production models whose behavior distributions weren't optimized in this explicit manner. As such, we felt that Wolf et al. (2023)'s approach, while important to discuss due to its inspiration of our paper, didn't best fit our needs and that rather the Wasserstein Distance which more directly approaches how far apart two distributions are would work better.

Note: Wolf et al. (2023) recently themselves in an update removed this $\beta$ approximation equation out of their work. We likely should have not used it and just worked with Wasserstein Distance.

Q1. Any extra text in the prompt to ask the model to stick to these responses only and not place its logit weight on other forms of saying something similar?

We did not introduce any extra text to ask the model to stick to Yes and No only in our final experiments.

Q2. Did you try other forms of adversarial attacks?

We did not try other forms of adversarial attacks as we particularly were investigating how conditioning on additional negative sentences shifted the behavior of the model. In this framework, most adversarial attacks, which generally cleverly shift behavior via a small number of tokens, couldn't be systematically set to different lengths out to fit our needs.

Q3. Can you clearly distinguish the contributions compared to Wolf et al. (2023)

Wolf et al. (2023) provide the theoretical framework and do brief testing on two optimized toy models. In contrast, we do test time probing on individual production models and significantly expand on their experimentation. Wolf et al. solely calculated the toy $\beta$ approximates, whereas we attempted to see whether the consequences of their theoretical claims were actually observable in language models:

• Do larger distinguishabilities make it easier to repeatedly recondition the model with more and more sentences towards behaving negatively?
• Does every additional sentence sourced from the negative distribution at a minumum continually shift the model to behave more poorly?

Q4. Can you add a limitations section to the paper explaining why or how generalizable these observations can be and what further studies need to be done to make them more generalizable?

Adding a limitations section would indeed aid in providing our thoughts on how generalizeable our results are.

Thank you for these great suggestions. We will duly take them into account.