Teaching LLMs to Teach Themselves Better Instructions via Reinforcement Learning

Q1: Performance Improvement in Figures 4 and 5.

• A1: Thanks for your valuable comments. In our comparative experiments involving llama-2 models, as shown in Figure 4, our method exhibits significant enhancements in the performance metrics of TruthfulQA and MMLU. Notably, TruthfulQA, a benchmark encompassing a diverse array of 817 questions spanning 38 distinct categories, covering a broad spectrum of domains, including but not limited to health, law, finance, and politics, demonstrates marked improvement. Similarly, our method reflects notable advancements in MMLU, a comprehensive evaluation criterion assessing a text model’s multitask accuracy across domains such as elementary mathematics, US history, computer science, law, etc. Conversely, in the case of the HellaSwag and AI2 Reasoning Challenge benchmarks, llama-2-7b slightly outperforms our method. This discrepancy in performance could be attributed to several factors, including the adequacy of the number of instructions provided and the nature of the initial dataset, which may have placed a greater emphasis on multi-task learning capabilities than specific task performance.
• In Figure 5, Our model, denoted as lunyu-7b-v1.1, was trained as an extension of llama-1-7b. Our approach has yielded substantial improvements in performance across the TruthfulQA and ARC benchmarks, while achieving a comparable level of performance in the HellaSwag benchmark. However, in the MMLU benchmark, our model exhibits slightly diminished performance relative to llama-1-7b. These results can be attributed, in part, to the potential insufficiency of instructional data. It is noteworthy that we utilized approximately 5% of the training data of WizardLM's, which may not have provided an ample instructional basis for our model. Nevertheless, even within the constraints of this relatively small dataset, our approach has demonstrated the capability to markedly enhance reasoning and multitasking proficiencies when compared to llama-1-7b. Furthermore, it has achieved performance levels comparable to those of WizardLM-7b and, notably, has exhibited remarkable enhancements in overall performance compared with llama-1-7b.

Q2: Also, is the comparison to WizardLM7b in Fig 6 fair, given that you used WizardLM13b extensively in your pipeline?

• A2: WizardLM-13 presents the unrestricted accessibility without imposing any querying costs. However, it is imperative to emphasize that employing ChatGPT for queries may engender substantial expenses. Consequently, from the perspective of saving querying costs while generating high-quality dataset, it is fair. Particularly, the query count of WizardLM-13B is only 896, even with the query count of WizardLM-13B, our method can still reduce the query count by 94.13% to reach comparable outcomes, emphasizing a more economical and sustainable strategy for LLM training.

Q3: why train policy with a 13B model?

• A3: Thanks for your constructive comments. Generally, if the model is larger, the model performance is more powerful, since the WizardLM-13B performance is better than WizardLM-7B model, and is open-source, which could save query costs, thus it is a better choice to use it for training our policy.

Q4: final downstream performance.

• A4: I appreciate your acknowledgment. We evaluate model performance on a well-established benchmark that can provide a more robust and objective assessment, particularly when assessing the quality of generated instructions, which can be inherently challenging to evaluate in isolation.

Q5: Typos.

• A5: Thanks for your carful review, we have fixed the typos.

Q6: ChatGPT/GPT-4 and RL model uses.

• A6: ChatGPT/GPT-4 serve as the generative model for producing intricate instructions and corresponding responses. It is noteworthy that the trained RL model plays a pivotal role in teaching ChatGPT to produce high-quality instructions and responses.

Dear Reviewer tJXC,

Thank you for your constructive and valuable reviews, which have significantly enhanced the quality of our paper.

• First, we have outlined our key contributions, which include low costs, the use of high-quality data, enhanced model privacy performance, and potential generalizability to other areas. We have also elucidated the performance improvements as illustrated in Figures 4 and 5. Furthermore, our training process requires only a limited number of samples from WizardLM-13B (371 samples for the revised version, and 896 samples for the initial version). This demonstrates that our method can significantly reduce the query count of WizardLM-13B by over 94% while achieving comparable results, indicating a more cost-effective and sustainable approach to LLM training and data quality enhancement.

• Second, we have clarified performance evaluation on a well-established benchmark that can provide a more robust and objective assessment.

• Third, we have described the use of ChatGPT (GPT-3.5) and GPT-4 in our work, where we developed a method for training a policy that teaches ChatGPT or GPT-4 to generate high-quality data, specifically focusing on steps 3 and 4 in the latest version of our manuscript.

As the deadline for discussion is approaching, we highly value your reviews and eagerly await your feedback. We kindly ask you to review our responses once more and let us know if they have fully or partially addressed your concerns. We are immensely grateful for your time and effort in reviewing our paper.

With gratitude,

Authors of Paper 9302

Q1: Directly evolving instructions through WizardLM or Tree-Instruct prompts also avoids the need for training a large language model. The benefit brought by their method is constrained.

• A1: The direct utilization of WizardLM or tree-instruct methodologies may incur substantial costs in terms of both querying GPT costs and training time. In the case of WizardLM, the utilization of random samples for instruction poses a challenge, as it is not feasible to discern which instructions are optimal for instructing LLMs effectively. Concerning tree-instruct, the requirement to formulate well-structured tree instructions to facilitate enhanced LLM learning is an explicit challenge. In contrast, our proposed methodology introduces the incorporation of RL techniques to design tree instructions, thereby contributing to improving LLMs’ performance with small data. Notably, the dataset employed by WizardLM outpaces our dataset in terms of scale, exceeding it by a substantial margin of over 94%. Nevertheless, our proposed approach, despite its reliance on a comparatively smaller dataset, showcases the capacity to achieve performance levels that are on par with those achievable by models trained on considerably larger datasets, such as WizardLM.
• Our method can efficiently fine-tune LLMs with the high-quality small dataset, and we also leverage the discrete action representation to continuously encode context-aware selection of actions in the policy. For details, please see the common response' contribution part.

Q2: Whether RLHF will further facilitate human alignment is not verified in this paper.

• A2: Thanks for your constructive comments. The alignment of RLHF with human values is an important topic. However, our method differs from RLHF, and the training data is derived from teaching advanced models, such as ChatGPT (GPT-3.5) and GPT-4. We believe the data from these advanced models aligns well with human values.

Q3: Involving RL in the stage of SFT is computationally expensive.

• A3: Our policy is trained utilizing a modestly-sized model, such as the WizardLM-13b model, and the training process requires less than one hour (the query count of WizardLM-13B is only 896), conducted on a hardware setup consisting of an A100 GPU and 16-core CPUs. Once our policy is adequately trained, it becomes a valuable asset to teach any advanced LLMs (such as ChatGPT) for the generation of complex instructions and corresponding responses, obviating the need for further policy training. Subsequently, upon acquisition of the requisite dataset comprising instructions and responses, we proceed to fine-tune a base model using a supervised learning approach, devoid of any RL training at this juncture.

Dear Reviewer 6muu,

Thank you for your constructive and valuable reviews, which have significantly enhanced the quality of our paper.

• First, we have outlined our key contributions (low costs, high-quality data, model privacy performance, and general for other areas potentially) and clarified the disadvantages of directly evolving instructions through WizardLM or Tree-Instruct prompts.
• Second, our method differs from RLHF, and the training data is derived from teaching ChatGPT (GPT-3.5) and GPT-4. We believe the data from these advanced models aligns well with human values [OpenAI-learning-from-human-preferences, 2017; Christiano, P. F., et al., 2017; Ouyang, L., et al., 2022] .
• Third, In the STF stage, our method does not require RL training, and our approach is designed to train models more efficiently using small, high-quality datasets rather than relying on large volumes of data. We kindly request that you review our responses once again and inform us whether they have fully or partially addressed your concerns.

As the deadline for discussion is approaching, we highly value your reviews and eagerly await your feedback. We are immensely grateful for your time and effort in reviewing our paper.

• [OpenAI-learning-from-human-preferences, 2017] https://openai.com/research/learning-from-human-preferences, June 13, 2017.
• [Christiano, P. F., et al., 2017] Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., & Amodei, D. (2017). Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30.
• [Ouyang, L., et al., 2022] Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., ... & Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730-27744.

With gratitude,

Authors of Paper 9302

Q1: Contributions.

• A1: See the common response’ contribution part.

Q2: external models usage, and Steps 4 & 5 for generating instructions and responses.

• A2: The external models include WizardLM-13B, ChatGPT(GPT-3.5) and GPT-4. As illustrated in our manuscript, during training RL, WizardLM-13B is used as an environment to help search for a policy (Step 3). The policy is leveraged to teach LLMs (such as ChatGPT) to generate high-quality data comprising complex instructions (Step 4) and their corresponding responses (Step 5). Particularly, our methodology is applicable beyond the specific external models used, indicating that the contribution is not limited to the capabilities of ChatGPT and GPT-4 but is instead a generalizable strategy. This approach is anticipated to substantially reduce computational demands as well as the number of queries to models such as ChatGPT, thus yielding a notable decrease in associated costs and generating high-quality small data (In the latest version of the manuscript, we have merged steps 4 and 5 into a single step, which is now referred to as step 4).

Q3: Performance analysis (Fig 4 & 5).

• A3: Thanks for your helpful comments. In our comparative experiments involving llama-2 models, as shown in Figure 4, our method exhibits significant enhancements in the performance metrics of TruthfulQA and MMLU. Notably, TruthfulQA, a benchmark encompassing a diverse array of 817 questions spanning 38 distinct categories, covering a broad spectrum of domains, including but not limited to health, law, finance, and politics, demonstrates marked improvement. Similarly, our method reflects notable advancements in MMLU, a comprehensive evaluation criterion assessing a text model’s multitask accuracy across domains such as elementary mathematics, US history, computer science, law, etc. Conversely, in the case of the HellaSwag and AI2 Reasoning Challenge benchmarks, llama-2-7b slightly outperforms our method. This discrepancy in performance could be attributed to several factors, including the adequacy of the number of instructions provided and the nature of the initial dataset, which may have placed a greater emphasis on multi-task learning capabilities than specific task performance.

• In Figure 5, Our model, denoted as lunyu-7b-v1.1, was trained as an extension of llama-1-7b. Our approach has yielded substantial improvements in performance across the TruthfulQA and ARC benchmarks, while achieving a comparable level of performance in the HellaSwag benchmark. However, in the MMLU benchmark, our model exhibits slightly diminished performance relative to llama-1-7b. These results can be attributed, in part, to the potential insufficiency of instructional data. It is noteworthy that we utilized approximately 5% of the training data of WizardLM's, which may not have provided an ample instructional basis for our model. Nevertheless, even within the constraints of this relatively small dataset, our approach has demonstrated the capability to markedly enhance reasoning and multitasking proficiencies when compared to llama-1-7b. Furthermore, it has achieved performance levels comparable to those of WizardLM-7b and, notably, has exhibited remarkable enhancements in overall performance compared with llama-1-7b.

Q4: TruthfulQA performance of llama-2-chat-7b is under-reported as 38.98, Table 14 from the Llama2 paper reports the performance of the 7B chat model to be at 57.04.

• A4: As illustrated within our manuscript, in an endeavor to save computational resources, we have uniformly configured all models to operate in a float16 format and evaluated all models' performance on a well-known benchmark, lm-evaluation-harness. It is worth noting that the performance metrics reported in the llama2 paper may have been derived from models utilizing a float32 format. In our own investigations, we have ascertained that models employing a higher-precision float format tend to yield superior performance scores. Consequently, it is important to acknowledge that the performance scores we have examined may exhibit disparities in comparison to those presented in the llama2 paper, due to the aforementioned difference in precision format.

Q5: Discrete value-based action space $S$.

• A5: The instruction actions are similar to WizardLM, but it is a little different from the instruction writing. Particularly, in our study, the discrete value-based action space means the instruction items. However, in RL policy searching, after mapping the instruction actions into a discrete value-based action space, we further map the discrete action space into a continous space for searching a policy. This facilitates the effective search for a high-quality policy. Such a policy is capable of teaching LLMs to generate complex instructions and responses with enhanced effectiveness. Conversely, in WizardLM, they randomly sample the instruction items, which may not be effective in generating high-quality data.

Q6: How is TRPO used here?

• A6: In the process of delineating the action and state spaces, TRPO is employed to ascertain an optimal policy. TRPO operates on a framework where the state and corresponding rewards, as yielded by the environment, constitute the inputs. Upon processing these inputs, TRPO formulates a sequence of actions that are then executed within the environment with the objective of enhancing the cumulative reward. It is imperative to note that the optimization is conducted over entire trajectories of rewards as opposed to singular, instantaneous rewards, thereby emphasizing the sequential decision-making aspect inherent in RL. However, TRPO is not the only choice, The choice of TRPO is motivated by its ability to rigorously handle advantage function values in policy searching, backed by a theoretical underpinning.

Q7: The rejection sampling approach.

• A7: Our methodology shares a conceptual alignment with the principles underpinning the rejection sampling approach, but differs in some important technical aspects to address some challenges, including but not limited to scalability and contextual comprehension. Notably, the process of learning a trajectory policy within the rejection sampling framework can be time-consuming and susceptible to the curse of dimensionality, primarily stemming from the exponential growth in the number of required samples. Additionally, it does not inherently guarantee the realization of monotonically improving rewards. In contrast, our method relies on interactive engagement with a contextually aware environment, notably represented by WizardLM-13B. In this interactive paradigm, to learn a trajectory policy, our approach elicits feedback in the form of rewards contingent upon the actions undertaken. Consequently, it undergoes an iterative learning process aimed at enhancing decision-making efficacy within the context, thus fostering continuous improvement through the feedback loop.

Q8: Why are LLMs called Advanced LLM in Fig 1 & 2?

• A8: Our training models fall within the category of base models, while advanced models, exemplified by ChatGPT, represent a higher level of sophistication and capability within this category.

Dear Reviewer 1H8F,

Thank you for your constructive and valuable reviews, which have significantly enhanced the quality of our paper.

We have outlined our key contributions (low costs, high-quality data, model privacy performance, and general for other areas potentially), explained models' performance on a well-known benchmark, lm-evaluation-harness, and elucidated how our method works, focusing on the main steps outlined in steps 3 and 4 of the latest version of our manuscript.

As the deadline for discussion is approaching, we highly value your reviews and eagerly await your feedback. We are immensely grateful for your time and effort in reviewing our paper. We kindly ask you to review our responses once more and let us know if they have fully or partially addressed your concerns.

With gratitude,

Authors of Paper 9302

Dear reviewers,

We would again like to thank all the reviewers for their constructive feedback, which significantly improved our paper's quality.

We have completed new experiments involving training a policy with WizardLM-13B. Remarkably, this training only necessitates 371 samples from WizardLM-13B to achieve a policy performance comparable to that of our initial submission. This indicates that, despite the query count of WizardLM-13B, our method can still reduce the query count by 94.21% to attain similar results. It underscores a more economical and sustainable approach in LLM training and enhancing data quality.

We hope our responses and revisions to the paper address the reviewers' concerns. If you have any additional comments or concerns, please do not hesitate to let us know; we are happy to address them.

With gratitude,

Authors of Paper 9302