Chain of Hindsight aligns Language Models with Feedback

Dear Reviewer us9M,

Thank you so much for your positive review and insightful comments. We appreciate your feedback and have incorporated your suggestions into the revised version to improve it.

#1: Regarding your suggestion on larger and more recent LLMs

We have conducted experiments using the latest 7B LLaMA model. In our study, we compared SFT, RLHF, and CoH against Koala, a state-of-the-art LLaMA-based chatbot, serving as a robust calibration baseline. Our comparison (see Figure 9) of these methods applied to LLaMA on HH and summarization datasets with Koala used pairwise human evaluation. Notably, Koala is developed from the ShareGPT dataset, known for its superior quality and greater diversity compared to open-source datasets like HH and summarization. The results indicate that our CoH approach performs on par with Koala, and when combined (CoH + Koala), it slightly outperforms Koala based on human ratings. However, both C-SFT (conditional SFT) and SFT significantly lag behind Koala, highlighting the effectiveness of CoH in leveraging human preferences for learning.

#2: Clarification on "past tokens" and masking ratio

By “past tokens,” we refer to the causal tokens visible to a given token. Our approach randomly masks past tokens to prevent overfitting during training. The masking ratio was determined through preliminary experiments; we found that a larger ratio hindered training efficiency, while a smaller one made negligible difference.

#3: On regularization and pretraining data

For the regularization, we used approximately the same number of tokens as the human feedback data. This approach resulted in about a 40% increase in wall clock time for training. We will clarify this further in our revision.

#4: Suggestion for GPT-4 based evaluation

While we primarily focused on human evaluation for its reliability, we acknowledge its limitations in terms of scalability and expense. We have conducted a GPT-4 evaluation for LLaMA-trained SFT, PPO-RLHF, and CoH, following the MT-Bench evaluation framework. Our results show that CoH achieves the highest score, significantly outperforming SFT and PPO. We will include results of GPT-4 based evaluation for more tasks in the revision.

We appreciate your efforts in using our open-source code. The issue you encountered with model generation quality could be due to the absence of regularization, leading to overfitting, or incorrect hyperparameter settings. We plan to enhance our codebase with more detailed instructions and example scripts to aid future research and ease of use.

#5: Does inference process need post-processing No extra post-processing is required during inference. The model is designed to generate quality responses directly. However, we are exploring potential post-processing methods that might include using search algorithms to select the best outputs or combining various token outputs for exploration. These are promising directions for future research.

#6: previous submission review feedback and revisions In our ICML 2023 submission, reviewers commended the novelty of our work in utilizing pretrained language models aligned with human feedback, as well as the strong empirical results in summarization and dialogue tasks. A primary concern was the absence of an RLHF baseline, which we have now addressed in this submission. Our updated comparison includes both automatic metrics and human evaluation, showing our model's substantial superiority over RLHF.

#7: Qualitative model inference examples We have included examples of model inference outputs in the appendix and plan to add more in the camera-ready version.

Once again, thank you for your valuable feedback.

Dear reviewer KdNj,

Thank you so much for your positive review and insightful comments. We appreciate your feedback and suggestions, which we have incorporated into the revised version to improve it.

#1: More discussion on RLHF

PPO-based RLHF suffers from reward over-optimization, as indicated in [1]. This paper also suggests training a very large model to address this issue. Despite experimenting with large models (up to 13B parameters due to compute constraints) for our PPO RLHF, we observed no significant improvements. Our hypothesis is that an even larger-scale reward model might be necessary. Additionally, we have tuned the PPO pipeline using the HuggingFace TRL library and experimented with varying the pretraining data regularization term for PPO RLHF training. Due to the complexity of reward learning and PPO RL training with multiple models, achieving optimal performance is challenging. CoH's advantage lies in its likelihood-based training, which aligns with the pretraining objective, thereby facilitating tuning and ensuring effective performance. We anticipate that future research will further explore the strengths and weaknesses of PPO RLHF and CoH, potentially leading to improved training algorithms.

[1] Scaling Laws for Reward Model Overoptimization by Leo Gao, John Schulman, Jacob Hilton

#2: On using 'Good:' prompts at inference time

We agree that prompting CoH-trained LM with other instructions at inference time could improve performance. To this end, we have conducted experiments to compare prompting model with Good:, and prompting model with Good: followed by another round of Better:. The results shown in Figure 8 shows that CoH-trained LM can learn to consider the initially generated response and improve upon it.

#3: More discussions about the performance relative to C-SFT.

CoH differs from C-SFT in that it allows the model to reference a bad response when predicting a good one, and vice versa. This approach is crucial as it enables the model to learn directly from feedback and examples. As shown in experiments, CoH outperforms C-SFT substantially.

#4: Clarification on 'CoH w/o lang'

By this, we mean consistently training with the "Good: {good} Bad: {bad}" template. The term “fine-grained language feedback” refers to the use of a variety of task-dependent language templates, such as "A good summary: {good}, a worse summary: {bad}" and "You are a helpful assistant: {good}, you are an unhelpful assistant: {bad}". These templates are detailed in the box at the bottom of page 3 and in Table 6. We will clarify this in the revised manuscript.

#5: On scaling to N-way rankings

CoH can be readily adapted for N-way rankings by structuring N-way ranked outputs into a sequence. However, a potential challenge arises if N is significantly large, as this would proportionally increase the context size. Regarding the WebGPT dataset, each prompt includes two answers, which we rank according to the accompanying scores. We will provide more details about this in our revision.

#6: Importance of loss masking during CoH training

Loss masking is beneficial for stabilizing training, as predicting control tokens can introduce noise into the prediction of desired outputs. It also aids in regularizing the model and preventing overfitting to specific training examples. While CoH is effective without loss masking, we included it as it enhances performance without additional computational costs.

Once again, thank you for your valuable feedback.

Dear reviewer aon5,

Thank you so much for your positive review and insightful comments. We appreciate your feedback and suggestions, which we will incorporate into the revised version to improve it.

#1: Consistency in terminology

We acknowledge the inconsistency you pointed out in our use of terms 'helpful/unhelpful' and 'good/bad' across different sections. We agree that maintaining a consistent terminology is crucial for clarity, and we will revise our manuscript to ensure uniformity in the terms used throughout the paper. However, note that we train with a variety of natural language feedback (see Table 6 in the paper).

#2: Include a discussion on related studies

We appreciate your suggestion to include the studies “RAFT” and “Direct Preference Optimization”. These are indeed significant contributions to the field. We will incorporate these references into our paper and discuss how our work aligns with, and differs from, these studies.

In addition to these specific changes, we will thoroughly review our manuscript once more to ensure clarity and coherence in our presentation. We believe that addressing these points will strengthen our paper and better communicate our contributions to the field.

Once again, thank you for your valuable feedback.

Dear reviewer WQPV,

Thank you so much for your positive review and insightful comments. We appreciate your feedback and suggestions, which we have incorporated into the revised version to improve it.

#1: Relation to rejection sampling

We appreciate the question, CoH leverages in-context power of large language models to learn from a sequence of examples and feedback, while rejection sampling is an orthogonal direction that uses a reward model to select top N model outputs for finetunning. CoH and rejection sampling can be directly combined to potentially further improve the performance of CoH, which is a promising future direction.

#2: Broader scope of testing

We acknowledge that broader scope of testing as an area for future improvement. Our initial focus was on demonstrating the effectiveness of CoH in tasks where feedback dynamics are crucial, such as dialogue and summarization, and evaluated on the primary dataset in each category. However, we agree that broader testing across diverse datasets is essential for validating the method's applicability in a wider context. In future work, we aim to include a more varied set of benchmarks, including code generation from unit tests feedback, to better evaluate the method's generalizability and performance in different scenarios.

Once again, thank you for your valuable feedback.