Human Feedback is not Gold Standard

One big question not addressed in the paper is how the biases in human judgement scale with human capabilities (e.g. question difficulty, annotator competence) and model capabilities (e.g. model size). Are humans more biased when evaluating responses they’re less competent to evaluate (e.g. medical advice, science explanations)? Are larger models more susceptible to exploit human biases? Overall, I think a good perspective for grounding this paper is scalable oversight [1]: the problem of evaluating agents more capable than the evaluators. LM capabilities will soon surpass human capabilities on many tasks and it’s critical to know whether human feedback will be increasingly misleading. That’s of course a big open problem and I don’t take not addressing it against the paper, it but it would be relatively easy to do experiments with models of the same family but different size or group questions by (perceived) difficulty.

This is a really interesting question! Our contribution in this paper is to carefully confirm that the effect exists at all. It would be great to see future work investigate the dynamics of this effect, and how it changes with task, model, problem domain etc.

I’m not sure I’m convinced that “Human evaluation is necessary” (as stated in the conclusion). Techniques such as RL from AI feedback have been shown to be as effective as RLHF for increasing the harmlessness of LMs [2]. They ultimately rely on humans, but these are human experts involved in prompt engineering and designing rules and constitutions used for generating training data for SFT and training preference models. A big open questions is whether LMs-as-evaluators inherit human biases, even when carefully prompted. A simple experiment addressing this question would involve having an LM play the role of an evaluator and conducting a similar analysis for its judgements.

It's true that "human evaluation with crowdworkers" is not always necesary, but language is an inherently human construct and so we would argue that humans are likely to be required to some degree for the foreseeable future. Using models for evaluation is more scalable but these models were still trained on data (ie language) that was originally generated by humans. Further, more targeted experiments would be valuable to confirm whether using LLMs as evaluators mitigates or exacerbates the biases we identify in this paper.

The crowdworkes were asked to rate LM responses according to each criterion in series and then to give an overall score. I wonder such ordering of the task is not itself a confounder. Would overall scores be different if the crowsworkers were not asked to give scores for criteria before? Would a score for Formatting be different if the crowdworkers were not ask for Factuality before? Do authors control for this anyhow?

Yes, we accounted for this - see the first paragraph in Sec 2.1. We used one groups of annotators to collect overall preference scores, and a separate group for the error annotations (who were also asked for overall ratings). You are correct that there is a 'priming' effect: Fig. 7 in the appendix shows a higher correlation between error scores and overall scores for the group that performed the error annotation, indicating that this did indeed affect their decision. The main results in the paper are based on the 'unprimed' annotations from the separate group, but we consider this an auxiliary finding of interest.

It’s subtly incorrect to say that “Perez et al. (2023) identified similar ‘inverse-scaling’ behaviour, where training on a RLHF objective worsens sycophancy”.

We have discussed this and agree with your observation: we have updated our paper.

QA: This is not really a flaw, but it would be much better if the experiments were done on responses from frontier models (such as GPT-4, Claude 2 and possibly LLaMA-2-70B). Was budget the limiting factor?

Our experiments were run using the most performant models available with permissive licenses at the time the experiments were run. GPT-4 and Claude were not available to us for legal reasons.

QB: A minor remark: The idea of predicting human preference judgements using a logistic regression model on top of human-interpretable features was recently explored in two concurrent papers that might be worth taking a look at: https://arxiv.org/abs/2310.13548 and https://arxiv.org/abs/2310.13011

Thank you! These were both released after the ICLR submission deadline, but we will add them to our related work. We feel that the existence of related contemporary work serves to highlight the importance of this line of research and its growing interest in the community.

The paper feels a bit ad-hoc; the properties tested seemed kind of arbitrarily chosen, but the specific set tested should have a significant impact on the learned lasso weights (e.g., if one feature is more predictive than the rest, the lasso weight would put all of it on that)

We disagree that Lasso will choose a single feature, since the features are not all perfectly correlated. Lasso will include any features that lead to a large enough predictive improvement to counteract the penalty term. However, we have updated the paper with more details of how the error types were chosen.

Some aspects of the paper do not engage with prior work. For example, it claims that “human feedback the de facto standard for evaluation” with no citation, and does not engage with the extensive work benchmarking LLMs

Thanks for pointing this out - you are correct that we should have included references to LLM benchmarking. We have updated the paper.

The RLHF increases assertiveness claim is tested indirectly; it compares Llama 2 13B and Command 52B, which have many confounders beyond RLHF.

Our results do show that Llama2 exhibits higher assertiveness at equivalent quality, but it's true that we cannot be certain why this is the case. Since the major difference between the two models is the RLHF training, we suggest that this may be the cause of the difference.

One of our primary contributions is to clearly demonstrate the biasing effect of assertiveness on perceived quality. To the best of our knowledge, this is the first work that demonstrates this for LLMs and we hope this research will provide a basis for future efforts to build on.

QA: How do the lasso weights change when you optimize on different subsets of the properties?

We have added a sensitivity analysis of the Lasso regression to the paper, see Fig 9 in the appendix. The changes in regression weighting when we leave out each feature in turn are fairly small, and the features that get selected do not change, indicating that the regression is stable. We hope that this addresses your concern.

QB: How much does the choice of ten properties you label for matter? What would happen if you included different properties?

I'm not sure I fully understand the question - clearly, a different set of properties would lead to different results. However, our intention was to establish a set of error types with as much coverage as possible. We welcome any suggestions for errors that have been ommitted.

I wish there were discussion on not just the problem of individual judgments being ambiguous, but that standard RLHF is optimizing towards a single user preference per example rather than considering factors resulting in a distribution over judgments for a number of different annotators (a point that is made e.g., as a motivation for jury learning, Gordon et al. 2022).

Thanks for the pointer to the jury learning paper, we have added this to our discussion. We briefly mention some related work on modelling annotator distributions by Wang et al., 2023. Collecting multiple annotations will partially mitigate the problem, but care still needs to be taken when aggregating to account for the biases that we present here. There are many issues with RLHF but our intention was to perform a targeted analysis of whether preference scores are a good training/evaluation metric.

I would also like more discussion on what to do in light of these findings. There were hints at one possible perspective of a solution -- optimizing nebulous "preference" is far from what people do when using language, which is more based on utility and communicative success (which preference is not tied to at all).

Thank you, this is a very valid point! Testing possible mitigation strategies is outside the scope of this paper, but we have added more detailed suggestions for possible future directions.

I think the analysis in Section 4 is conflating a few different factors in comparing models -- the fact that Command is trained off-policy and Llama 2 is on-policy isn't discussed in enough depth for me to be convinced this is relevant to mention. I don't have a good intuition on why that might influence the model behavior, especially if the two models are trained on different data as well. Following the above, I would have liked to see an experiment that directly shows how optimizing for human preference influences outputs across different dimensions explored in this paper (e.g, assertivness). [...]

We are the first to demonstrate the existence of a biasing effect due to assertiveness, which is not a trivial finding. We believe it is unfair to critique the paper for not also including an additional large-scale experiment investigating the effects of RLHF.

We disagree that the only conclusion is that "all the models conflate quality and style": the results definitely show that Llama 2 exhibits higher assertiveness at equivalent quality. However, it is a valid criticism that we cannot be sure why this is. We tried to qualify the findings in Section 4 as "preliminary evidence" since we did not directly test the same models with and without RLHF. A targeted set of experiments, using the same base model and varying the number of RLHF steps (and other factors), is planned for the future but exceeds the scope of this paper.

I would also like to see an experiment controlling for factuality, like in Section 3.2. E.g., if it's possible to even post-hoc separate the results in Figs 3 and 4 by whether an expert judged the output as factual or not. Basically, do you find that assertive false responses are rated as factual by annotators more often than inassertive true responses?

Here are the annotator factuality error rates, grouped by preamble type and expert factuality judgement:

So, it's not the case that "assertive false responses are rated as factual by annotators more often than inassertive true responses", but the factual errors are much less likely to be spotted in assertive outputs. Note that there are only 7 examples that are inassertive and false, so the statistical power of this analysis is weak, and we therefore will not include it in the paper. Expert annotations are unfortunately difficult to scale!

I think the main table showing expert judgments for Sec 3.2 is in the appendix, but in my opinion it should be in the main paper.

We would like to include the full results in the main flow of the paper, but unfortunately space is limited. We have the choice to either show annotator and expert judgements alongside each other and require the reader to take the difference, or to show just the difference and include the breakdown of results in the appendix. Presenting the results as differences demonstrates most clearly the findings we are trying to present, without excess complication, but we include the full detailed results in the appendix for completeness.

it is not clear if this error type categorization is comprehensive enough; authors haven't provided any empirical/experimental support for these error categories.

Thank you for the feedback! The error categories were based partly on the well-established Gricean Maxims (e.g. the Maxim of Quantity prohibits repetition), partly on experimental work by Xu et al. who asked human evaluators to give justifications for their scores, and influenced by the factors that users care about when using LLMs in production environments. We did our best to account for as wide a range of errors as we could, but it's possible that we have missed something. We have updated the paper with more details.

the datasets studied may not be comprehensive enough to have captured some of the important dimensions of AI safety (toxicity, attribution, etc)

We selected datasets that are well-scoped (such that evaluation is possible) but are still challenging and require long-form generation. Further work is definitely needed - but there is a lack of datasets with permissive licenses for realistic use cases.

Besides the Lasso regression model, other interpretable/explainable ML methods might have shed more light on error categories and their correlation with both human preference and model output.

Thanks for the feedback - we have released the full dataset and welcome future work investigating more detailed interactions between error types and preferences.

authors note that human preferences are mostly absolute or relative - there isn't any comparison between the two in this research. Since the paper provides insights into understanding these human preferences, more focus on various types of human preferences would have strengthened the finding of the paper

Time and budget constraints meant that we used a fixed protocol, based on RankME (Novikova et al., 2018), and focussed on identifying whether the biasing effects exist at all. You're right that it would be really interesting to run our experiments with different protocols: varying the number of outputs evaluated at a time, using absolute/relative scores, different instructions etc. Future work could investigate whether the strength of the effect varies with annotation protocol.

There are other important biases from AI safety perspective that isn't explicitly studied in the paper (could possibly make the paper more relevant) - hallucinations, more fine-grained categorization of inconsistencies, toxicity, etc Authors very briefly touch upon the AI safety implications but do not give justice to this very important piece of information in the paper - what are the AI safety implications as a result of using human preferences for training and evaluation.

Safety is an important area of research, and we agree that there would be a lot of value in extending our study to consider the interaction between assertiveness/complexity and various types of unsafe LLM behaviour. However, we believe that doing this justice would be another whole paper in itself.

In this paper we confirm that preference scores are a) an imperfect measure of important error types, and b) biased by assertiveness. These observations hold true even for a (hypothetical) perfectly safe model. We hope it will serve as a starting point for further work, including experiments specifically targeted at the wide range of important safety concerns.