Enabling Lanuguage Models to Implicitly Learn Self-Improvement

Thank you for the thoughtful feedback! We are happy that you find our approach interesting, novel, and effective! We understand you may have some confusion and concerns, which we address as follows.

W1: The paper motivates why using RLHF in output improvement is better than just prompting the LLM to give feedback, but it is unclear why we should apply RLHF for the improvement, instead of directly applying RLHF to the original LLM. It would be better to see the comparison in the evaluation.

The original LLM (policy model in Section 3 and “LLMs” in Figure 1) that generates the initial response is already RLHF-finetuned. PIT takes initial responses generated by the original LLM as input, and generates improved responses. PIT and the original LLM were finetuned with the same starting pre-trained checkpoint and data, but PIT is not a continual training of the original LLM (Continual training may not be suitable since their input formats are different. Appendix A illustrates more details). PIT only utilizes the output of the original LLM during training and inference time and does not use its weights. We compare PIT with (1) original responses given by the original LLM and (2) Improved responses given by Self-Refine on the original LLM to show PIT is better than the original LLM with Self-Refine.

We apologize for not making ourselves clearer and causing you some potential confusion. We demonstrate our workflow in Appendix A in the revised paper to make our approach clearer and more organized. We hope these extra demonstrations are helpful for a better understanding of our approach and can clarify this concern.

W2: Comparing PIT with Self-Refine seems unfair as PIT requires more human annotation as it requires additional human preference labels.

The training data of PIT is exactly the same as the data we use to train the RLHF-finetuned model. Thus, PIT needs no extra data. A more detailed workflow demonstration can be found in Appendix A in the revised paper. Again, we apologize for the confusion and for not making this point clearer.

W3: The description of PIT is a bit unclear as the model used in each component of the method is not clearly stated.

We make a more detailed workflow demonstration containing the input format training algorithm blocks in Appendix A in the revised paper. We believe general response GR2 will also be helpful for understanding. We hope now the framework is clearer.

We hope our reply addresses your questions, and please let us know if you have further questions. We would be happy to continue the discussion if needed.

We thank the reviewer for the detailed and insightful feedback! We are excited that you think our method is convincing, interesting, reasonable, intuitive, and intriguing! Here are our responses to clarify your questions:

W1.1: The computational cost and the execution difficulty should be more clearly stated: while the method seems to lead to a stronger gain than self-refine, such an improvement is not always significant and sometimes self-refine seems to be stronger by some metric (e.g., in terms of GPT-4 evaluation), although the authors have given possible explanations as to why they occur.

We understand the reviewer’s concern about GPT-4 metrics, so we introduce human evaluation in this case. GPT-4 is well-known for many biases, such as position bias [1,2], format bias [3], and knowledge bias [3]. Therefore, we introduce human evaluation when GPT-4 disagrees with the reward model. Our human evaluation detects several false negative cases in GPT-4, like long-content preference, as stated in Section 4.4. Therefore, we trust human evaluation in this case, which shows PIT is still apparently better than Self-Refine (Table 1).

[1] Wang, Peiyi, et al. "Large language models are not fair evaluators." arXiv preprint arXiv:2305.17926 (2023).

[2] Zheng, Lianmin, et al. "Judging LLM-as-a-judge with MT-Bench and Chatbot Arena." arXiv preprint arXiv:2306.05685 (2023).

[3]Zhu, Lianghui, Xinggang Wang, and Xinlong Wang. "JudgeLM: Fine-tuned Large Language Models are Scalable Judges." arXiv preprint arXiv:2310.17631 (2023).

W1.2 On the other hand, self-refine as a prompting-based method is much easier to execute and cheaper I think the paper would benefit if the authors would give more detailed account on the computational cost, including a comparison with the baseline methods

Thanks for pointing out this important discussion. A short answer to the computational cost concern is that our method does not bring extra computational cost compared to Self-Refine during inference and, in principle, will even have a bit lower computational cost due to fewer token inputs (without rubrics). A more detailed explanation can be found in the general response GR2.

For the execution difficulty, we would like to address that Self-Refine is more challenging to execute than it looks since it needs careful prompt/rubric design, which is one of our motivations. From the end-user perspective, our method is more friendly since the end-user only needs to provide the query and a candidate response without putting effort into designing prompts.

Overall, we appreciate the reviewer pointing out this important discussion, and we already added this to Appendix B of the revised paper.

We thank the reviewer for the valuable feedback. We sincerely appreciate your time in reviewing our paper.

A potential method to tackle the training computational cost issue is to incorporate PIT into the original model by jointly training the original model on both the PIT training objective and RLHF objective, so that we can integrate PIT functionality into the original model without training a separate model. We thank the reviewer for bringing this discussion; this could be an interesting future work :)

We appreciate the reviewer for the helpful feedback. We are happy to see that you find our method novel and practical! Here are our responses to your concerns:

W1: In the experiment, the authors compare PIT with prompt-based methods (self-refine), while there is a lack of comparison with other reinforcement learning related methods like [1]. Can other RL methods help to self-improve the response? [1] Song F, Yu B, Li M, et al. Preference ranking optimization for human alignment[J]. arXiv preprint arXiv:2306.17492, 2023.

We thank the reviewer for pointing out related resources. We added a discussion comparing our algorithm with the mentioned PRO paper in Section 3 (highlighted with green color). In short: PRO and similar papers such as DPO [1], focus on LLM alignment which is not what our work aims for. We focus on triggering the self-improvement capability implicitly from data. Therefore, PIT is not directly comparable to these methods.

More specifically, RLHF, DPO, and PRO focus on “How to make LLM generate responses preferred by humans given an input $x$”, whereas PIT focuses on “How to get a better response preferred by humans given an input $x$ and a reference response $y$, where the reference response is generated by policy models (LLMs) trained with RLHF, DPO or PRO.” Appendix A shows more details of PIT, which may help understand that PIT has different optimization goals compared to PRO, etc.

[1] Rafailov, Rafael, et al. "Direct preference optimization: Your language model is secretly a reward model." arXiv preprint arXiv:2305.18290 (2023).

We hope our reply addresses your concerns, and please let us know if you need further clarification. We are happy to continue discussions if needed!

We thank the reviewer for the constructive feedback! We are glad you find our method novel and the potential to guide future directions. We address your concerns in the following paragraphs.

W1: The framework may not incorporate new information or data during the self-improvement cycle, potentially limiting the scope of learning to alignment with initial human preferences.

We understand the reviewer's concern about the scenario of new incoming preferences. This is a challenging question for the conventional RLHF, too, not only our approach. In the conventional RLHF setting, one needs to re-train the reward model or ensemble different reward models. PIT can adopt the same technique to handle new incoming preferences as conventional RLHF.

W2: The iterative nature of the proposed self-improvement process could result in increased computational time and cost in inference.

In fact, the inference cost is the same as Self-Refine and we apologize for not directly pointing this out in the paper. Detailed discussions can be found in the general response GR2.

Both Self-Refine and PIT iteratively replace candidate responses with improved responses (See Algorithm 2 in Appendix A in the revised version for more details). We evaluate the self-improvement of both methods under one iteration (Table 1, 2, and Figure 3) and multiple iterations (Table 3). Both methods have identical computational costs given the same number of iterations.

Q1: Can you clarify if the three models depicted in Figure 1, i.e. LLMs, PIT, and PIT Reward Model? Are they distinct or the same model?

We apologize for the confusion. In our experiments, all models start with the same pre-trained language model, i.e., PaLM 2 (Bison). “LLM” (or the policy model $\text{M}\_\text{P}^\text{RL}$ in Section 3) is the model fine-tuned on the dataset (e.g., HH-RLHF) following the normal RLHF pipeline. “PIT” (i.e., $\text{M}\_\text{PIT}^\text{RL}$ in Section 3) is the model fine-tuned with our reformulated RLHF pipeline (Equation 4 and 5) on the same dataset (e.g. HH-RLHF). “PIT Reward Model” (i.e., $\text{R}\_\text{PIT}$ in Section 3) is the model fine-tuned with our reformulated reward model training objective (equation 2) on the same dataset (e.g., HH-RLHF). “PIT reward model” offers the reward signal in equations 4 and 5 when training “PIT”.

During inference, we first get a candidate response from “LLM”, then self-refine will design prompts and let “LLM” self-improve the candidate response. Differently, our approach uses “PIT” to improve the candidate response.

A more formal and detailed demonstration including input formats, training, and inference details can be found in Appendix A in the revised paper. We also added some clarifications in Section 3 to help readers understand their differences (modifications are highlighted with green color).

Q2 and Q3: How to train with objectives (4) and (5)? An algorithm block is easy to illustrate. The paper lacks a detailed algorithmic flow for training with objectives (4) and (5). Could you provide an algorithmic block to illustrate the training process?

We have provided these details in our revised paper Appendix A. In short, to train with objective (4), we sample $x$ and $y\_\text{ref}$ from data $\mathcal{D}\_\text{RL}$ and use RL for optimization. To train with objective (5), we sample $x$ from data $\mathcal{D}\_\text{RL}$ and $y\_\text{ref}$ from $\text{M}\_\text{P}^\text{RL}(\cdot | x)$, and then use RL for optimization.

Q4: How to determine the number of iterations for the self-improvement process for inference?

This is a great question! It is a common problem for all self-improvement methods during inference. It is challenging to determine the stopping criteria for both Self-Refine and PIT. We show preliminary analysis in the paper (Section 4.7) -- there is no guarantee that every iteration can lead to improvement. Table 3 shows that more iterations do not denote better response qualities for both Self-Refine and our method. Nevertheless, our method is better than Self-Refine under all iterations, showing its effectiveness.Further research is needed to investigate the stopping criteria of self-improvement methods. One potential method is gradually reducing the temperature for the next iteration until the response is repeated.

We hope we clarify your concerns and questions, please let us know if you have further questions and we will be happy to discuss further.