Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models

We are motivated to find that the reviewer finds our work (a) important for understanding the effect of feedback protocols, (b) widely applicable to both human and AI, and (c) useful for informing people how people design feedback protocols.

Q: Why were Dolly, User-orient, and SuperNI selected as the tasks of interest?

A: We thank the reviewer for their insightful question on our dataset setup. As mentioned in Section 3, the instructions are designed to present a collection of queries that could potentially be posed to a text-based AI assistant in real-world scenarios. We clarify that Dolly, User-Orient, and SuperNI are all human-written instruction datasets. Specifically, Dolly and User-orient are open-ended instructions and responses while we select a subset of SuperNI datasets that cover a wide range of public NLP datasets with long responses (Table 11). We also point that there is no consensus within the community about the datasets to be used for feedback acquisition and subsequent reward modeling. Hence, we focused on high quality datasets that are used for instruction tuning.

Q: Additionally, the generalizability of the results may be limited by the choice of model in the various experiments.

A: We thank the reviewer for their query regarding our experimental setup.

• While it is indeed possible to change various components of the setup, the critique holds for any LLM paper as LLMs are rapidly evolving and increasing in number. Our choices reflect (a) strong models and practices which have had numerous pick ups by the community at large (e.g., GPT-3.5-Turbo, Alpaca-7B), (b) finite compute and labeling budget available for conducting the experiments in an academic setting.
• In our experiments, we generated $30$K candidate responses and got them annotated by humans and AI which costs ~$2000 USD in total. Although it would be interesting to generate candidate responses from other models, the subsequent annotation is time-consuming and out of our academic budget.
• We also clarify that the win-rate against davinci-003 has been a de facto evaluation metric for many influential works on LLM alignment [1,2,3,4] We will mention this argument in the revised paper.

Following up on the reviewer’s suggestions, we do run additional experiments for (a) using GPT-3.5-Turbo for consistency analysis and (b) using LORA Alpaca-7B as a reward model.

(a) GPT-3.5-Turbo for Consistency Analysis

By default, we use GPT-3.5-Turbo with temperature = 0. Here, we perform an inconsistency analysis on 500 comparisons between the ratings and rankings feedback for (i) GPT-3.5-Turbo at temperature = 0.5, (ii) GPT-3.5-Turbo-0613, (iii) GPT-4. Here are the results:

The standard error for the above numbers is 4% with 95% confidence. We find that all the model choices suffer from the inconsistency problem. This indicates that our results are not restricted by the choice of the AI annotator, and are indeed observable in different models too.

(b) Choice of the reward model

• We clarify that there is no clear consensus on the “right” choice of the reward model. We believe that any good language understanding model could act as a reward model. Prior works [1,6] use decoder-only architecture for the reward model. These models are normally at a scale of 10-100 billion parameters and harder to train in the compute efficient manner without engineering tricks. Hence, we chose Alpaca-7B in our setup, and LoRA is just a method to finetune it in a parameter efficient manner.
• On the other hand, [5,7] have used BERT transformer encoder models, which usually have less parameters, say 500M-1B parameters. Finetuning BERT models achieve very good performance on many NLP tasks. Hence, we repeat train reward models on the Roberta-large architecture.
• Specifically, we finetune a robert-large model on the ratings data using the regression loss. In addition, we train another roberta-large model on the rankings data using the negative log likelihood loss. We use these two finetuned models to perform Best-of-64 policy. Subsequently, we use GPT-3.5-Turbo to provide rating and ranking feedback for the rating/ranking Best-of-64 policy.

We present the results below:

Thanks again for your insightful feedback on our work! We've carefully worked to address your comments/questions and would like to note that the end of the discussion phase is coming soon. Are there any further questions or concerns we should discuss?

Hi,

We wanted to highlight that we have uploaded the revised paper with the new results and setups. Addressing your suggestions, we have added Section E.2 (variation in AI annotator), Section N (RoBERTA-Large), Section L (Task-specific results), Section I (AI prompts), Section J (human UI screenshot), Section P (qualitative examples).

Please let us know if there are any further questions.

Q: The evaluation inconsistency seems straightforward: the performance of reward models will be affected by the format of input data. It is better to convert the rating feedback to the ranking format first and then use it to train an NLL reward model (just like the ranking feedback) and then compare the performance.

A:

• In our work, we established that the feedback data obtained from humans and AI suffers from inconsistency problems. We believe that our work is unique as it shows that this has a direct impact on the alignment and evaluation of LLMs through empirical evidence.
• The choice of reward models was naturally made to reflect the nature of the feedback protocol. Specifically, the rating protocol provides us with a scalar score hence the regression model makes sense, and the ranking protocol does not provide us with scalar scores hence we use a negative log likelihood (NLL) objective function.
• We observe that 57.4% of the pairwise comparisons are considered “equal” when we convert the rating data into the ranking format (Row 1 of Table 2(a)). In practice, this means that we need to filter almost 60% of the rating data if we want to train a ranking format reward model. This is not the most optimal way to use the rating data when we know that a better regression method exists that can utilize the complete data. Hence, we train a regression reward model in our original setup.

However, we still perform the experiment as requested by the reviewer.

1. We train two reward models, of roberta-large architecture, on (a) ranking data with NLL loss and (b) rating data converted to ranking with NLL loss.
2. We perform Best-of-n sampling with n = 64
3. We evaluate the model’s win-rate against davinci-003 with a rating and ranking evaluation scheme.

We present the results below:

We highlight that the evaluation inconsistency is still evident when the data format used for training the reward models is kept identical. This indicates that our finding is not confounded by the reward model’s data format but the nature of the feedback protocol data itself i.e., rating and ranking. We will update this result in the paper.

Q: Will the collected feedback be released?

A: Yes, we will release the data upon acceptance. To support the claim, we have uploaded the AI feedback data in the supplementary material.

Q: Is there any calibration on the rating scores?

A:

• We clarify that we collect the rating data on a scale of 1-7 from AI and humans where {1=Very poor, 2=Poor, 3=Below Average,4=Average,5=Above Average,6=Good,7=Excellent}.
• We provide an annotation guideline in the prompt to the AI explaining the various quality dimensions along which the rating score decision must be based along with two in-context examples – one for low score scenario and second for the high score scenario. In addition, we perform rating data collection for each instance in a new session so that the LLM maintains its same calibration/understanding for the rating scores throughout the annotation process. We will add the prompt to the revised paper.
• We provide the same annotation guidelines as AI to humans. Differently from AI, the humans can (re)calibrate themselves as the annotation progresses. As we ask four workers to annotate the same instance, the small differences caused by the worker’s calibration will be averaged out. Specifically, the feedback data will not be biased towards a specific annotator's calibration, instead reflect the average belief about the response quality under the rating/ranking feedback protocol. We will add the UI screenshot of the human annotator interface in the revised paper.

We thank the reviewer for their helpful feedback.

Q: This paper does not explore the influence of feedback protocols on common alignment methods (such as RLHF[1], RRHF[2], RLAIF[3], etc.). The alignment in this paper just applies the reward models to select the best out of n generation, which is only affected by the performance of reward models.

A: We thank the reviewer for their thoughtful question. Firstly, we highlight that the best-of-n (rejection sampling) is also a common method in RLHF literature for aligning LLMs. As mentioned in Section 2.3, we choose rejection sampling as due to simplicity, simplicity, and robust performance. For instance, [1] finds that the win-rate of the Best-of-n policy outperforms the base model by 10% absolute points on AlpacaEval. We do not perform PPO since it requires atleast 8 80GB GPUs according to the AlpacaFarm implementation, which is outside our academic budget. We attempted to train PPO at our compute scale by reducing the effective batch sizes and using LoRA models but it did not run stably. Following the reviewer’s feedback, we performed an additional experiment with rejection sampling finetuning (RSFT) used in three highly cited LLM alignment papers, including LLaMA 2 [2,3,4]. Specifically, our setup is as follows:

1. We prompt Alpaca-7B with 5K instructions from Alpaca-52K data.
2. We generate 64 responses for every instruction.
3. We use our rating and ranking reward model to select the best response from the 64 responses.
4. We finetune two Alpaca-7B models with the 5K instruction-responses data. (a) One where the responses are chosen from the rating reward model. (b) Second where the responses are chosen from the ranking reward model.
5. Post finetuning, we sample a single response from the finetuned Alpaca-7B with 553 evaluation instructions
6. We calculate the win-rate against DaVinci003 using the rating and ranking protocol using ChatGPT.

Here are the results for this experiment where baseline is the base Alpaca-7B model:

We find that the evaluation inconsistency persists under this alignment algorithm too. It indicates that the choice of feedback protocol for the evaluation favors the same feedback protocol used for training the reward models and subsequently finetuning the base LLM. We will add this result in the updated paper. We hope that this experiment answers the reviewer’s comment on the generalizability of the evaluation protocol to other algorithms.

[1] AlpacaFarm: https://arxiv.org/abs/2305.14387
[2] RRHF: https://arxiv.org/abs/2304.05302
[3] LLaMA2: https://arxiv.org/pdf/2307.09288.pdf
[4] Constitutional AI: https://arxiv.org/pdf/2212.08073.pdf
[5] AlpacaFarm Code: https://github.com/tatsu-lab/alpaca_farm#running-reference-methods

Thanks again for your insightful feedback on our work! We've carefully worked to address your comments/questions and would like to note that the end of the discussion phase is coming soon. Are there any further questions or concerns we should discuss?

We are excited to find that the reviewer finds our work (a) interesting in an important area, (b) sheds light on unexplored issues, (c) and with positive and good scientific contributions to the ICLR community. Here, we address your comments in detail.

Q: Section 3.1 length distributions

A:

• We thank the reviewer for their pertinent question. In Section 3.1, we mention that the max length of the Alpaca generated response is 128. This avoids humans to rank very large responses (>128 tokens). In Table 6 and 7, we show that the human and AI is indeed not biased towards longer responses or towards the number of unique words in our setup. However, the results might differ with the changes in the feedback collection setup. We further identify the distinctions between our setup and the previous ones to further understand such differences.
• [1] mentioned verbosity bias examined verbosity bias when using LLMs to collect feedback data by constructing a repetitive list attack in which responses with a list were prepended with a paraphrased version of the list (generated by GPT-4) which contains no new information. For example, model A has two pointer responses and model B has the same two pointers + their paraphrased versions (which does not add any new information), the annotators prefer model B response. We believe that this setting largely differs from our setting since we consider two different responses with no further intervention such as a repetition attack. Furthermore, [1] conducts their experiment on just 23 model responses [3] whereas we perform 2K responses. We will add this discussion in our updated paper.
• Similarly, [2] talks about verbosity bias in preference labeling from humans and AI for creative writing tasks. In our work, we consider a broad range of instructions sourced from 3 datasets as discussed in Table 1. In addition, they conduct their experiments on just 300 responses whereas we perform our analysis with 2K responses. We will also include this discussion in our paper.

[1] LLM-as-judge: https://arxiv.org/pdf/2306.05685.pdf
[2] Verbosity Bias: https://arxiv.org/pdf/2310.10076.pdf
[3] Data: https://github.com/lm-sys/FastChat/blob/main/fastchat/llm_judge/data/mt_bench/reference_answer/gpt-4.jsonl

Q: Section 4.2: Qualitative Analysis

A:

We thank the reviewer for their insightful question. We agree that it is important to understand the cause of inconsistencies quantitatively and qualitatively. Here, we describe our qualitative analysis for more clarity:

1. In Section 4.2, we show that the feedback data collected from humans and AI is highly inconsistent and analyze their behaviours.
2. Further, we quantify the closeness with the model responses from the lens of rating and ranking feedback protocol. Overall, this analysis suggested that the perceived closeness between the quality of the responses leads to more inconsistent feedback.
3. To investigate this further, we conduct another small scale experiment. Specifically, we sample a few consistent and inconsistent responses, and then ask the annotators to provide their rating/ranking feedback along with their natural language explanation. We agree with the reviewer that such experiments should be better documented. As a result, we will include the human annotation interface in the updated paper. We believe that our small scale experimental setup is a good design under a finite budget for a qualitative study.
4. Post-feedback collection, we study the human responses and natural language explanations for both inconsistent (Table 9 and 10) and consistent responses (Table 11 and 12).
5. We notice that the natural language explanations from humans do explain their choice well. However, the quality aspects that they focus on changes with the feedback protocol.
6. Specifically we look at the inconsistent feedback example in Table 9. In the ranking setup, the annotators perceived Response 2 as more clever (“making a riddle..”) or correct (“makes more sense..”) over Response 1. Whereas, in the rating setup, the Response 1 gets higher average score as the annotators focussed on the answer’s faithfulness to the task instruction (“Response 2 does not meet the task structure requirements”..)
7. To provide more evidence, we will update the revised paper with more qualitative examples in the Appendix.

Dear authors,

Thank you so much for your response. I appreciate your detailed response, but I think the analyses could have been improved, or the descriptions are still not clear to me.

Re: length bias

Regarding the length issue, now I think the different finding might be coming from a much shorter maximum length---the max length of 128 seems to be really short for instruction-following queries. While I am not super familiar with [1], [2], [3] and their evaluation setups might be different, several papers analyzing the relationship between human/model preferences and length on the setup similar to this work e.g., Singhal et al. (2023) or Wang et al. (2023) set maximum length much higher. In my opinion, if we want to draw some conclusions on the length bias in human/model evaluations, we should set the maximum length longer.

Also I am confused by the sentece.

This avoids humans to rank very large responses (>128 tokens).

Do you mean that you set the maximum tokens to be 128 as it's going to be expensive if we ask human to rank long sequences?

Re: qualitative analysis

Could you clarify the exact number you sampled, the overall statistics of qualitatively analyzed data, and representative trends? For instance, as in Xu et al. (2023), you can meta-analyze the human explanations to understand what common cases are.

we sample a few consistent and inconsistent responses, and then ask the annotators to provide their rating/ranking feedback along with their natural language explanation. We agree with the reviewer that such experiments should be better documented. As a result, we will include the human annotation interface in the updated paper. We believe that our small scale experimental setup is a good design under a finite budget for a qualitative study.

I agree that it is difficult to scale up qualitative analysis, but due to the subjective nature of qualitative analysis, qualitative analysis should be carefully designed and a paper should provide clear details of the experimental designs to make it reproducible. For me, it's hard to draw conclusions without knowing the exact details and stats of this analysis.

Overall, I think the initial motivation is interesting and I support the paper, while I felt the analysis could have been improved, and the findings may not be fully applicable to more general settings (e.g., larger LMs, longer sequences, different way of RLHF, such as PPO). Therefore, I keep my score (6).

Thanks again for your insightful feedback on our work! We've carefully worked to address your comments/questions and would like to note that the end of the discussion phase is coming soon. Are there any further questions or concerns we should discuss?

We are excited to find that the reviewer really liked our work and found it (a) essential and timely, (b) well structured and written, and (c) extensive and carefully designed.

Q: Annotate the same examples using another batch of human or sample from the same model (GPT-3.5-Turbo) using different temperature, what would be the disagreement?

A: We thank the reviewer for their insightful question. Firstly, we provide further clarifications on collecting the feedback data from the humans.

1. During the feedback acquisition, we create two exclusive groups of humans – one that provides the rating feedback and ranking feedback.
2. For the rating protocol, four humans rate an individual response for a given instruction.
3. Similarly, four humans rank a pair of responses for a given instruction under the ranking protocol.
4. As described in the Agreement Analysis (3.1), we consider three annotations as the predicted human preferences and the fourth annotation as the human gold label.
5. Table 2(b) indicates consistency analysis of the predicted human preferences i.e., randomly selecting 3 annotators out of 4 annotators.
6. To provide further analysis as requested by the reviewer, we calculate the consistency evaluation for a different batch of randomly selected 3 annotators four times.
7. We find that the standard deviation of the inconsistency analysis for four splits of the humans evaluators is 1.6%. This indicates that the inconsistency analysis is robust to the batch of humans used for annotation.

To provide more evidence for the same model using a different temperatures – 0, 0.2 and 0.5 for 1000 comparisons. We find that the standard deviation in the disagreements between these temperatures is 2.7%. This indicates that the inconsistency analysis is also robust to the different model temperatures used for generating feedback data. We will add these statistics in our updated paper.

Q: Root cause of the difference

A:

We agree that the root cause of the difference between the feedback protocols is quite important. Towards this, we highlight our quantitative and qualitative findings in the Section 4.2:

1. We show that the feedback data collected from humans and AI is highly inconsistent and analyze their behaviours.
2. Further, we quantify the closeness with the model responses from the lens of rating and ranking feedback protocol. Overall, this analysis suggested that the perceived closeness between the quality of the responses leads to more inconsistent feedback.
3. To investigate this further, we conduct another small scale experiment. Specifically, we sample a few consistent and inconsistent responses, and then ask the annotators to provide their rating/ranking feedback along with their natural language explanation. We agree with the reviewer that such experiments should be better documented. As a result, we will include the human annotation interface in the updated paper. We believe that our small scale experimental setup is a good design under a finite budget for a qualitative study.
4. Post-feedback collection, we study the human responses and natural language explanations for both inconsistent (Table 9 and 10) and consistent responses (Table 11 and 12).
5. We notice that the natural language explanations from humans do explain their choice well. However, the quality aspects that they focus on changes with the feedback protocol.
6. Specifically we look at the inconsistent feedback example in Table 9. In the ranking setup, the annotators perceived Response 2 as more clever (“making a riddle..”) or correct (“makes more sense..”) over Response 1. Whereas, in the rating setup, the Response 1 gets higher average score as the annotators focussed on the answer’s faithfulness to the task instruction (“Response 2 does not meet the task structure requirements”..)
7. To provide more evidence, we will update the revised paper with more qualitative examples in the Appendix.

Thanks again for your insightful feedback on our work! We've carefully worked to address your comments/questions and would like to note that the end of the discussion phase is coming soon. Are there any further questions or concerns we should discuss?

Hi,

We wanted to highlight that we have uploaded the revised paper with the new results and setups. Addressing your suggestions, we have added Section E (variation with annotators), Section I (AI prompt), Section J (human UI screenshot), and Section F (pairwise rating), Section P (more qualitative examples).

Please do let us know if there are any further questions. If all your questions have been resolved, we would be grateful if you would consider raising your score.