Language Model Self-improvement by Reinforcement Learning Contemplation

Q6: Regarding questions about the experiments.

Comment 1: Do the RLAIF and other baselines also get to use chain of thought for the reasoning tasks, like your method?

Response 1: Yes. All baselines in our experiments also use CoT prompt: “Let’s think step by step”.

Comment 2: What are the prompts used for RLAIF / other baselines?

Response 2: We use the same prompts for all baselines and RLC. The prompts we used in the experiments are listed in Table 6 in Appendix for the details of prompts.

Q8: I don't really see why you distinguish between RLCAI and RLAIF.**

We simply use two terms to indicate they are from different works. We mainly use the term ‘RLAIF’ in the whole paper.

Reference：

[1] Sun Z, et al. Salmon: Self-alignment with principle-following reward models, 2023.

[2] Yang C, et al. Large language models as optimizers[J]. arXiv preprint arXiv:2309.03409, 2023.

[3] Bai Y, et al. Constitutional ai: Harmlessness from ai feedback, 2022.

[4] Lee H, et al. RLAIF: Scaling reinforcement learning from human feedback with ai feedback, 2023.

We hope that these responses can address your concerns and questions. If you had any further concerns, please let us know.

We are grateful for the time and effort you put into reviewing our work. Below we address each of your concerns and questions.

Q1: I'm not sure that the idea of evaluation being easier than generation is particularly new.

It is an intuitive idea that evaluation is easier than generation. Thus some previous works may use this idea more or less. They commonly use the evaluation ability of LLM heuristically (e.g., RLAIF, Self-Refine, etc.). In contrast, our study offers a thorough experimental validation of the performance disparity between evaluation and generation capabilities in language models.

Q2: Regarding the BigBench-Hard datasets used in the experiment.

Comment 1: On a related note, these tasks seem a bit nonstandard for RLAIF evaluation.

Response 1: The key insight of RLC is “LLM self-improve by contemplation”, which can help LLM to improve continuously on the given tasks. As for RLAIF, There is also a lot of work to use text generation tasks([1], [2], [3], [4]) such as summarization and reasoning tasks such as BBH ([1], [2]).

Comment 2: I'm curious whether the results hold up when you do the classic tasks like harmlessness and helpfulness.

Response 2: Good point. We have conducted additional experiments employing RLC to train FLAN-T5-Large in generating negative sentences. Specifically, we provide language models with a continuation prompt: "Continue with a story starting with 'I just went to the cinema'". Subsequently, the model's output is evaluated as positive, neutral, or negative, with corresponding rewards of 0, 1, and 2, respectively. We collected 200 examples of model output and subjected them to human evaluation. The results indicate that 92% of the generated sentences were negative, demonstrating the effectiveness of RLC in producing the desired output. This suggests that the approach can be extended to other tasks, such as harmlessness and helpfulness, to further validate its efficacy.

Q3: Is it fair to use Flan-T5 to evaluate outputs for your model, while using finetuned GPT2-Large (presumably weaker) to compare outputs for your baseline?

We use GPT2-Large due to it has similar size (774M) as FLAN-T5-Large (780M), and GPT series models have been applied as reward by ChatGPT. However, we recognize that FLAN-T5-Large could be more proper thus we conduct additional experiments that replace GPT2-Large with FLAN-T5-Large. The experimental results are shown in the following table. The experiment results show that there is no significant change in the performance of RLAIF (FLAN-T5-Large). Of course, it's better than RLAIF (GPT-2 Large).

Q4: Using language models to directly output 1-10 scores seems like it might suffer from poor calibration. Why not e.g., try to finetune with a regression head on top rather than just using the LM out of the box? Would that improve performance?

Implementing fine-tuning with a regression head appears promising for evaluating text quality. However, this approach necessitates supervised data for effective fine-tuning, while this paper mainly focuses on improving language models without external supervision.

To address potential calibration issues, a possible solution is to reduce the evaluation score range. This adjustment aims to enable the language model to more accurately assess text quality.

Q5: The numbers in 6.4 don't seem very convincing for what you're claiming, if anything it seems like the performance went up more on the unrelated tasks (object counting, penguins)?

Apologize for any confusion caused by the presentation in the original version. The increase in performance is not necessarily correlated with the similarity between the training tasks and the evaluation tasks. For instance, Logical Deduction (3) (+1.5) and Tracking Shuffled Objects (5) (-0.1) both have tasks akin to the training tasks. It is important to note that the tasks in question were not seen during the training process. Despite this, the RLC method demonstrates an overall improvement in performance on average. We will update the paper to rectify any misleading presentation and provide clearer explanations.

Hi reviewer jzQi. As the discussion period comes to an end, we want to follow up to see if the response addresses your concerns. If you have any further questions, please let us know. Thank you again!

Thanks for your timely response. We are glad to find our response has addressed your proposed issues.

We understand your concerns about the sample generation number of LMSI methods. While we recognize that employing multiple sample generations could potentially enhance the performance of LMSI methods, it's essential to highlight that our experimental focus lies in assessing the efficacy of various methods in enhancing language models. Our evaluation primarily measures the performance disparity between DG (which involves a single sampling). Consequently, in our experiments, LMSI methods are restricted to a single sampling instance.

However, we agree that conducting experiments combining LMSI methods with multiple sampling instances would be valuable. This approach could effectively showcase performance enhancements over these varied sampling methodologies, such as 'best-of-N', thereby enriching the experimental outcomes. We assure you that the experiments will be conducted and integrated into the future version of the paper.

———

If you had any further questions, we are glad to discuss.

Q4: Would appreciate a discussion (backed by experiments) on the RLC/RLAIF updates compared to doing more sampling and non-invasive self-evaluation.

Thank you for your suggestion. Both RLC and RLAIF are invasive LMSI methods. In contrast, non-invasive methods, such as Self-Consistency, do not require additional training for language models. This distinction results in different trade-offs between these approaches.

Language models trained using invasive methods like RLC and RLAIF can generate responses more rapidly, making them suitable for applications with stringent response time requirements, such as chatbots. However, these methods necessitate additional training, which increases computational costs and complexity.

On the other hand, non-invasive methods do not require further training of the language models, reducing computational costs and simplifying their implementation. While these methods may not be as fast in generating responses, they offer a more resource-efficient alternative.

We suggest that both invasive and non-invasive LMSI methods are closely related to the concept of autonomous agents, which must solve tasks and continuously improve. These two categories of methods can naturally cater to these essential capabilities. We will incorporate this discussion, along with relevant experimental results, at the end of Section 6 (Experiments) to provide a comprehensive comparison of these approaches.

Q5: It is not clear to me how accuracy is computed in experiments of section 4.1.

Apologize for any ambiguity in our initial presentation.

To assess the accuracy of text generation, we use human evaluation. A generated text is considered "correct" if it inclusively incorporates all the concepts presented in the question.

To assess the accuracy of text evaluation, we first prompt the same LLM to self-evaluate if the generated text contains all the specified concepts, utilizing the prompt given in Table 7. Then we mark the evaluation result as “correct” if the LLM's evaluation result is same to that of human.

Q6: What is the difference between best-of-N and w/ SE in section 4.3?

In Section 4.3, we present two distinct implementations of the W/ SE method for the BigBench-Hard and summarization tasks. For the BigBench-Hard task, the W/ SE approach involves regenerating an answer if the language model evaluates the previous response as incorrect, which is different from best-of-N (i.e., choose best one from N samples). In contrast, for summarization tasks, W/ SE generates three potential answers and selects the optimal one based on evaluation results. This latter approach is similar to the best-of-N method.

Q7: Discussions about the experiment results.

Comment 1: Do you have an intuition why preference-based RLAIF is weaker than RLC?

Response 1: A key distinction between RLAIF and RLC lies in the method of obtaining rewards. In RLC, rewards are directly derived from self-evaluation outcomes, whereas RLAIF relies on rewards trained from preference data labeled by the language model. This difference in reward acquisition may contribute to the weaker performance of RLAIF compared to RLC.

Comment 2: any idea why RLC is stronger than RLFT on “Reasoning about Colored Objects”?

Response 2: The RL tasks in our experiments are characterized by sparse reward problems, where agents receive a binary 0-1 reward signal upon generating complete answers. This may make it challenging for language models to learn from sparse reward signals in certain tasks. As observed in Figure 7 of the Appendix, RLC and RLFT exhibit similar training curves. A potential solution to address this issue is to replace the binary reward with a softer one, such as using the language model's probability of evaluating an answer as correct or incorrect as the reward.

Q8: It is often not really clear what model is used (4.2, 4.3, 6) and why results are not reported for different model sizes.

In Sections 4.3 and 6, we employ the Flan-T5-Large model, while for Section 4.2, we utilize the Flan-T5-XL model. Our primary objective in Sections 4.2 and 4.3 is to validate the hypothesis that "language model can self-improve using it evaluation ability". These experiments in Section 4 only serve as a validation step, rather than the main results of the paper. Therefore, we have not conducted experiments with varying model sizes. For experiments in Section 6, we have conducted experiments with variety of FLAN-T5 model sizes, as the results shown in Figure 6.

We hope that our response has addressed your concern and questions satisfactorily. If you had any further concerns, we are glad for discussion.

Thank you for your careful review and comprehensive comments. Please find our response below.

Q1: Section 4.3 looks a bit weak to me & Why doing only 3 samples？

We would like to clarify that the experiment in Section 4.3 aims to verify the potential of text evaluation to improve text generation. We focus more on investigating whether the LM can improve when equipped with self-evaluation mechanism.

Consequently, we employ a straightforward approach in which the language model generates output while simultaneously evaluating its performance. The experimental results, based on three samples, indicate that language models exhibit improvement when utilizing self-evaluation, thus supporting the central motivation of this paper.

Q2: The novelty of the method.

We acknowledge the prior work, RLAIF, shares the same setting with RLC. However, we would like to clarify their differences in motivation and framework. the main point of our work is to explain why a kind of RLAIF functions effectively. We provide evidence that evaluation tasks are simpler than generation tasks for LLMs. This difference can be leveraged to boost the performance of LLMs of various sizes.

In contrast, prior RLAIF studies relied mostly on comprehensive experiment results, some of which necessitated the LLMs to be robust enough. The experimental findings (referenced in Table 4) illustrate this particular limitation.

Furthermore, RLC offers a more straightforward implementation for LMSI, eliminating the need for gathering preference data and training a reward model.

Q3: Regarding suggestions about additional experiments.

Following your suggestion, we have conducted additional experiments.

Comment 1: would be interesting to have a more complete view: e.g. using smaller/bigger evaluation models. Similarly to the RLAIF baseline that uses GPT2 through the RLAIF baseline in 6.2 but not using RLC.

Response 1: Thank you for your suggestion. It’s a great direction for RLC to explore. We have conducted supplementary experiments employing FLAN-T5 models of different sizes as evaluation models, specifically FLAN-T5-Small (80M), FLAN-T5-Base (250M), and FLAN-T5-Large (780M). The results demonstrate that, generally, larger evaluation models yield improved performance. Notably, even the smaller evaluation model, FLAN-T5-Small, delivers satisfactory results, supporting the claim that evaluation is a less complex task.

Comment 2: Could you plot the evaluation of the LM when fine-tuning from RLC (for instance on the Figure 7 from appendix), as well as the heuristic metrics (BLEU, BERTScore, ROUGE) when applicable?

Response 2: Good point. This result would be helpful to understand what happens during the RLC training process. However, these heuristic metrics (BLEU, BERTScore, and ROUGE) are not applicable to BigBench-Hard tasks as these metrics cannot be directly calculated. As supplementary, we plot the episodic return curves during the training process, as shown in Figure 8 in Appendix in the revised version of manuscript. The experiment results show that the answer accuracy has a consistent trend as the episodic return, demonstrating the effectiveness of the self-evaluation.

Q4: What is the statistical significance of the gains in Tables 2 and 4?

We understand your concern that the improvement in Table 2/4 is not significant. We would like to clarify the experiment results as follow:

1. Concerns about the significance of the gains.

   It is important to note that Reasoning dataset is inherently challenging for language model enhancement. We specifically utilized the "Hard" subset within the Reasoning task. For BigBench-Hard [5] dataset, an advanced model from OpenAI, text-avinci-001, fail to answer with a high accuracy [5] on Tracking Shuffled Objects. Moreover, some previous methods exhibit poor improvements in reasoning tasks. For example, self-refine [1] exhibits 0% and 0.2% enhancements with GPT3.5 and GPT4 model, respectively. Furthermore, ReAct [2] demonstrates performance degradation on HotpotQA (a popular reasoning task).

2. Distinction between Tables 2 and 4:

   Table 2 demonstrates the potential of self-evaluation in improving text generation without incorporating the proposed method. It reveals that a language model can improve its performance using a self-evaluation mechanism. In contrast, Table 4 presents the results of the proposed RLC method, which surpasses the DG method on 11 out of 12 tasks. These findings substantiate the effectiveness of the RLC method.

3. Regarding the standard deviation: To ensure the robustness of our experimental setup, we conducted all experiments using three distinct seeds, thereby reducing the impact of deviations.

References:

[1] Madaan A, et al. Self-Refine: Iterative refinement with self-feedback, 2023.

[2] Yao S, et al. ReAct: Synergizing reasoning and acting in language models, 2023.

[3] Shinn N, et al. Reflexion: an autonomous agent with dynamic memory and self-reflection, 2023.

[4] Wei J, et al. Chain-of-thought prompting elicits reasoning in large language models, 2022.

[5] Suzgun M, et al. Challenging big-bench tasks and whether chain-of-thought can solve them, 2022.

We hope the response has addressed your concerns. But if you have any further questions, please let us know.

Thank you for your time and valuable comments. We have taken every comment into consideration. Please find the response below.

Q1: For there is a family of work in evaluating self-generation (https://arxiv.org/abs/2210.03629), how is this work different?

The work [2] you mentioned and its related works ([1, 3]) lie in the field of language model self-correction. The primary differences between our work and these previous works can be summarized as follows:

1. Motivation: We would like to highlight that the main point of our work is to explain why a kind of RLAIF functions effectively. We provide evidence that evaluation tasks are simpler than generation tasks for LLMs. This difference can be leveraged to boost the performance of LLMs of various sizes. Meanwhile, previous studies relied mostly on comprehensive experiment results, some of which necessitated the LLMs to be solid enough.
2. Implementation: These previous works refining their output during the generation process by repeated self-evaluation and revision. In contrast, invasive methods update LM’s parameters to improve its capability.
3. Inference speed: RLC offers a direct generation process after training, contrasting with these previous works, which entails slow generation due to multiple rounds of evaluation and revision.
4. Method justification: We justify the performance gap between text evaluation and generation through comprehensive experiments in Section 4, while previous works typically utilize the evaluation ability of LLM heuristically.

We would like to discuss and highlight these differences in Section 2 (Related Work).

Q2: This work uses CoT, do the baselines use CoT as well? How much of the gain is coming from CoT? This work is missing ablations that quantify the gains from the primary contribution.

In our experiments, we utilize the CoT prompt [4], "Let's think step by step," for all baseline models. Consequently, the performance gain of RLC can be assessed by comparing its answer accuracy to that of the DG method. We apologize for any ambiguity in the original manuscript. To address this, we will revise Section 6.1 (Experimental Setup) to provide a clearer explanation of the experimental design.

Q3: Regarding the paper organization.

We believe there could be some misunderstanding about the paper's organization. Please find our response below.

Comment 1: The main contribution of this work starts on page 5.

Response 1: Sorry for the potential misleading about the paper organization. We would like to clarify that Section 4 (starting from page 3) presents an important contribution of this paper: the verification of the performance gap between text generation and evaluation. This performance gap serves as the foundation of the design of RLC. Meanwhile, we have some discussions about the potential of language model self-improvement leveraging such a gap.

Comment 2: I suggest that the authors drastically truncate the background content and use the remaining room to analyze the primary contribution.

Response 2: Thanks for your suggestions. We would like to simplify the related work section and add more discussions about the differences between this work and previous works. Besides, we will update Section 5 (Method) to discuss how RLC utilizes self-evaluation, an its differences in comparison to prior works.

Hi reviewer BB78. We wanted to follow up to see if the response addresses your concerns. If you have any further questions, please let us know. Thank you again!

Thanks for your positive comments and constructive feedback on the paper. Below we address each of your concerns and questions.

Q1: The datasets considered are a bit toy. It would be great to see other experiments from other domains (maybe something like math word problems?)

Thanks for your suggestion. In our experiments, we primarily employed the BigBench-Hard and CNN-Daily-Mail datasets, which are commonly utilized to assess various capabilities of LLM. To address your concern, we have incorporated an additional dataset, MathQA [1], renowned for its diverse and challenging problem set. The experimental results, as presented in the table below, demonstrate that RLC improves answer accuracy from 14.9% to 19% on the MathQA dataset. This finding suggests that RLC can be effectively applied to other datasets, thereby showcasing its versatility and applicability.

Q2: The approach is limited to having a training dataset …

Comment 1: limited to have a training dataset

Response 1: In this work, we consider the language model self-improvement with a training dataset, which is a common setting in LMSI studies. We are encouraged that you are also interested in training the language models without a dataset, which is also the ultimate goal of RLC. To address this, we have conducted supplementary experiments aimed at improving language models without relying on a dataset. Specifically, we provide language models with a continuation prompt: "Continue with a story starting with 'I just went to the cinema'". Subsequently, the model's output is evaluated as positive, neutral, or negative, with corresponding rewards of 0, 1, and 2, respectively. We collected 200 examples of model output and subjected them to human evaluation. The results indicate that 92% of the generated sentences were negative, demonstrating the effectiveness of RLC in producing the desired output. This suggests that the approach can be extended to other tasks, such as harmlessness and helpfulness, to further validate its efficacy.

Comment 2: I am a bit concerned that there might be stronger baselines that make use of that unlabeled training dataset

Response 2: To the best of our knowledge, the baselines in our experiments are representative methods in the domain , encompassing invasive methods (such as self-training, RLAIF, Best-of-N) and non-invasive methods (SC, Self-Refine), etc. We remain open to conducting further experiments with any additional baselines that demonstrate proficiency in LMSI.

Q3: It's not clear to this reviewer what the model is learning here & It would be great to have a version of Figure 6 (performance of RLC across sizes of LMs) comparing with "Self-train" instead of with no finetuning.

We agree that utilizing the Self-train method as a baseline can provide valuable insights into the learning process of the language model. The Self-train method aims to improve the language model's ability to generate accurate answers.

As per your recommendation, we have updated Figure 6 (please refer to the revised version), adding Self-train method. We observe that RLC consistently outperforms the Self-train method across various language model sizes in terms of answer accuracy.

This results indicate the language model could learn new knowledge during the reinforcement learning contemplation process, in addition to the accurate answer.

Q4: For the experiments, how do the non-PPO trained models generate answers…

In the experiments, all models generate answers using an auto-regressive approach to generate token sequences. These models are not restricted to outputting valid options, which is consistent with the experimental settings employed in prior research [2]. Consequently, language models may produce answers that fall outside the set of candidate options, which would be evaluated as ‘wrong’ during the evaluation process.

It is important to note that, PPO-trained models (RLC, RLAIF, RLFT) could generate the answer out of the candidate options, as these models also output the token sequence, rather than the discrete actions like in traditional RL.

References above:

[1] Amini A, Gabriel S, et al. MathQA: Towards interpretable math word problem solving with operation-based formalisms, 2019.

[2] https://github.com/FranxYao/chain-of-thought-hub/tree/main/BBH

We hope that these responses can address your concerns and questions. If you had any further concerns, please let us know.

Thank you for carefully reviewing our paper and providing constructive comments. We hope that our response has addressed your concerns, but if we missed anything please let us know.

Q1: The novelty of the work is limited.

We would like to highlight that the main point of our work is to explain why a kind of RLAIF functions effectively. We provide evidence that evaluation tasks are simpler than generation tasks for LLMs. This difference can be leveraged to boost the performance of LLMs of various sizes. Meanwhile, prior RLAIF studies relied mostly on comprehensive experiment results, some of which necessitated the LLMs to be robust.

Q2: The effectiveness of the proposed approach is demonstrated using the 780M model of Flan-T5 &  Is the proposed approach still effective for larger models?

We appreciate your suggestions and have conducted experiments on FLAN-T5-XL (3B), which is about 4 times larger than FLAN-T5-Large (780M). The experiment results are presented in the following table. We observe that RLC can be applied to larger language model. As future work, we will try to conduct experiments with larger model (e.g., FLAN-T5-XXL (11B), LLaMA2, etc).

Q3: I am wondering if the big accuracy gap between generation and accuracy still exists in decoder-only models.

This is an interesting question. In the development of the method, RLC does not impose specific requirements on the model architecture. This versatility allows for the exploration of the accuracy gap in various models, including decoder-only configurations.

Q4: Figure 2 suggests that the accuracy gap between generation and evaluation becomes smaller as the model size increases.

As discussed in Section 4.1, evaluation accuracy is influenced by both the evaluation capability and the quality of the generated text. As the model size increases, the language model can produce higher quality text that becomes more challenging to differentiate. Consequently, the gap may narrow when the language model exhibits exceptional text generation abilities. We posit that the experimental results in Section 4.1 provide an intuitive representation of the performance disparity between text evaluation and text generation.

Q5: Is it true that GPT-2 Large was used in Lee et al. (2023)? It seems to me that they used PaLM 2.

Sorry for the unclear presentation regarding the GPT-2 Large model of RLAIF. It is true that Lee et al. employed the PaLM 2 model as the LM labeler. However, the size of the model is positively correlated with its capabilities, and if the model is too large (such as PaLM 2), it is difficult to compare whether it is the improvement brought about by capabilities or methods. Therefore, in order to ensure fairness in comparison, in our study, we utilize the GPT-2 Large (774M) model as both the labeler and reward model, which is comparable in size to the FLAN-T5-Large (780M) model.

Furthermore, to further verify the impact of the reward model, we conducted supplementary experiments in which we replaced the GPT-2 Large model with the FLAN-T5-Large model, aligning our implementation with the RLC that employs FLAN-T5-Large as the evaluator. The results of these experiments are presented in the table below. We can observe that RLAIF shows close performance on different model. In the future, we will also try to conduct experiments with different types of models for RLAIF and RLC to compare their performance.

Thanks again for the careful and timely response. We are glad to any further discussions.

Dear reviewer scBc,

Regarding your concerns about (Q3) if the big accuracy gap between generation and accuracy still exists in decoder-only models, we have conducted supplementary experiments using the widely-recognized decoder-only model, ChatGPT (GPT-3.5-turbo), on the CommonGen task. These experiments encompass varying levels of difficulty, as determined by the number of concepts involved. The updated experimental results are presented below:

The experiment results indicate that evaluation accuracy generally surpasses generation accuracy. When the generation task is relatively simple (concept number ≤ 8), the language model can achieve high generation accuracy, resulting in a slightly higher generation accuracy compared to evaluation accuracy. However, as the generation task becomes more complex, a performance gap between generation and evaluation emerges. This experiment demonstrates that the accuracy gap persists in decoder-only models. We would like to conduct further investigation with additional decoder-only models to provide a more comprehensive results.