Q5: Are LLMs good at self-feedback

To support the claim that LLMs are capable of self-feedback, we can refer to studies like [1,2,3]. These references suggest that LLMs can indeed engage in meaningful self-improvement.

we found that meta-skill learning is critical to the self-feedback ability. After meta-skill learning, the accuracy of self-feedback to identify the correctness of an answer is 70% (Without meta-skill learning, the accuracy is limited to 33%, as the model tends to self-feedback all responses as correct.).

We presume that self-feedback ability might depend on several factors, such as the model's intrinsic ability, prompt design, problem domain, etc.

[1] William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802, 2022.

[2] Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023.

[3] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651, 2023.

Q6: When to stop training and self-refinement

we use common regularization techniques to prevent overfitting, including monitoring validation set performance and early stopping.

Regarding self-refining during inference, we consistently apply self-refinement once across all models for a fair comparison. Besides, our additional tests indicate optimal performance with 3-4 self-refinement iterations. Beyond 4 iterations, performance may decline.

Q7: Test cases in EvolInstruct test set

The EvolInstruct test set comprises 218 test examples, We will include this information in section 4.1.1 of the revised paper.

Q8: Figure 3 explaination

Figure 3 shows the comparison test result in the general domain. As noted in sec. 4.2.2, "We follow the evaluation procedures outlined in [1], which address the order bias issues identified in the evaluation methods proposed by [2]". Specifically, we test different models (Vicuna, Vicuna fine-tuned on $D_{QA}$, apply SELF to Vicuna fine-tuned on $D_{QA}$, referred to as Vicuna, Vicuna + $D_{QA}$, and Vicuna + $D_{QA}$ + SELF) on the Vicuna test set and EvolInstruct test set. We then employed GPT-4 to assign an assessment score for each test response of different models.

In Figure 3, the color coding is as follows:

• Blue indicates test cases where the model under evaluation performs better than the baseline model (Vicuna), as judged by GPT-4.

• Yellow indicates test cases of equal performance.

• Pink indicates test cases where where model under evaluation performs worse than the baseline model.

From this figure, several key findings emerge:

• First Block: After being fine-tuned on $D_{QA}$, the model demonstrates improved performance compared to the original Vicuna model, as evaluated by GPT-4.

• Second Block: The application of the SELF framework to Vicuna + $D_{QA}$ enhances performance on both test sets.

• Third Block: Further improvements are observed when Vicuna + $D_{QA}$ + SELF applied self-refinement during inference.

These results demonstrate that SELF is effective in the general domain. We recognize the need for clarity and will add more detailed explanations in our paper to facilitate better understanding.

[1] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023.

[2] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/

Q9: Confusion of 'self-curated ' and 'self-filtered'

Apologies for the confusion. Here's a clarification:

self-curated data: This is the data after applying self-refinement to self-generated responses.

self-filtered data: This data undergoes one extra self-feedback step to the self-refinement response. We filter out the data that is judged by self-feedback as incorrect.

The purpose of Section 4.4: We are investigating whether self-feedback can further improve the self-evolution training data quality and thus result in better test performance.

We will update the paper to better explain these concepts and our experiment settings.

W4: Poorly written & Local issues and typos:

We appreciate your guidance regarding the local issues and typos and the suggestions for incorporating more empirical study findings into the Introduction of our paper. We will thoroughly review each suggestion and make revisions. We aim to release the revised paper as soon as possible.

Moreover, we would like to clarify your concerns regarding the last point in Local issues and typos:

'Restart Training' involves combining the meta-skill learning corpus with all rounds of self-evolution training data to train the model.

"Continual Training (Mixed Data)" involves training the model with self-evolution data from all rounds.

'Continual Training ($D^t_{self}$ Only).' refers to training the model with the $t$th round self-evolution data in a sequential manner, round by round.

We will update the manuscript to reflect this distinction more clearly

Q2: Comparison with reinforcement learning based method

SELF differs from reinforcement learning (RL) methods in its use of informative natural language feedback instead of information-sparse scalar rewards.

We also conducted an additional RL-based experiment based on trlx. For a fair comparison, we use the same SFT model (Vicuna + $D_{QA}$) for SELF and RLHF. The reward model was trained on pair-wise comparison data (refined response is assumed better than original response) derived from the meta-skill learning corpus in SELF. Although we follow the RLHF framework to conduct experiments, the comparison data was provided by GPT4 instead of humans, which can be seen as a form of reinforcement learning without human feedback. The results are as follows:

RL vs. SELF: RL achieved a 25.55% accuracy on GSM8K, lower than SELF's 27.67%. We found that the reward model often fails to identify the correct response, leading to limited performance improvements. For instance, 76% of incorrect answers were assigned higher scalar rewards than correct answers on the GSM8K test set. Unlike RL-based methods, SELF leverages natural language feedback, which provides a more accurate evaluation (only 28% incorrect answers were judged as correct).

Q3: Apply SELF to stronger LLMs and the variant of SELF

Previously, it was widely believed that self-improvement capabilities were exclusive to large language models (LLMs) such as GPT-4 and GPT-3.5. However, our research in the SELF framework reveals that even smaller models can acquire such abilities. This is evidenced in Table 1, which shows that initial Vicuna models lacked self-improvement capabilities, yet our meta-skill training effectively endowed it with self-refinement abilities.

We did not mean SELF can only be used in small LLMS. Instead, we have observed improvement of SELF in a more robust baseline model, i.e., VicunaV1.5, which was fine-tuned from Llama2-7b.

To address your question, we experiment with SELF with OpenLlama-3b(smaller LLM), VicunaV1.5(stronger LLm), and Vicuna(in paper), demonstrating its effectiveness across different models.

The method of "a refiner network improves another, smaller downstream network" can seen as a variant of SELF. It is interesting to explore in future work.

Q4: Confusion about RLHF's limitation and one more limitation of RLHF(limited human availability)

"RLHF involves complicated iteration between the models and reward functions, requiring many hyper-parameters tuning" means: challenges in reinforcement learning, particularly the iterative inference between policy and reward models and the complexity of numerous hyperparameters (e.g., minibatches, discount factor, etc.).

We will update the 'Related Work' section of our paper to provide references and a clearer description of these aspects and also discuss the expensive human availability of RLHF.

W3: More comparison experiments

We have conducted 6 additional experimental comparisons to form a more solid work. We will include details of these experiments in our revised paper. The key experiments include:

• RL-based Baseline Comparison: We compare the SELF framework with RL-based baselines (Response to Q2).

• Self-Consistency Filtering Analysis: This explores the contribution of meta-skills and self-consistency to the generation of the self-evolution training corpus.

  Model	Acc. of Training Data(%)	Acc. on Test set(%)
  self-consistency filtered (5x majority)	28.27	26.77
  self-refinement revised	29.89	26.90
  meta-skills filtered	44.10	27.67

  This study demonstrates self-refinement as a necessary component of SELF. Meta-skills comprising self-refinement and self-feedback form a robust foundation for the self-evolution training process.

• SELF Adaptability with OpenLlama3b/VicunaV1.5(Llama2-7b): This examines the adaptability and extensibility of SELF across models of varying sizes and capacities (Response to Q.3).

• Impact of Meta-Skill Learning Quality: We investigate how the quality of meta-skill learning influences the self-evolution process.

  Training Stage	Direct Generation(%, GPT-3.5-turbo/GPT4)	Self-Refinement(%, GPT-3.5-turbo/GPT4)
  Vicuna + meta-skill learning	24.84/25.39 (0.55$\uparrow$)	25.22/28.28 (3.06$\uparrow$)
  Vicuna + meta-skill learning + SELF	25.11/27.67 (2.56$\uparrow$)	25.47/29.34 (3.87$\uparrow$)

  The experiments showcase better performance metrics across our SELF framework by using GPT-4 to generate the meta-skill corpus, compared with GPT-3.5-turbo. This study affirms the critical role of meta-skill training data quality in instilling self-feedback and self-refinement in the Vicuna model.

• Single vs. Multiple Rounds of Self-Evolution: Given the same number of prompts, compare the effect of training with a single round versus training iteratively, to assess the difference between a static and an improved model as a self-evolution training data generator.

  Training Method	Direct Generation (%)	Self-Refinement (%)
  SELF (Single Round)	28.40	30.55
  SELF (Iterative)	29.64	31.31

  The comparison reveals a higher performance of iterative training over that of single round training. It highlights the advantages of iterative training in leveraging improved LLMs across rounds for enhanced training data quality and subsequent test performance.

• SELF with 7.5k Meta-Skill Corpus vs. Supervised Fine-Tuning on 7.5k GSM8k training data.

  Training Method	Direct Generation(%)	Self-Refinement(%)
  Vicuna + $D_{QA}$(7.5k)	28.05	-
  Vicuna + meta-skill learning (7.5k)	31.23	32.98
  Vicuna + meta-skill learning (7.5k) + first round Self-Evolution	35.43	36.22
  Vicuna + meta-skill learning (7.5k) + first and second round Self-Evolution	37.87	38.12
  Vicuna + GSM8K training data (human-annotated, 7.5k)	35.70	-

  In comparison to supervised fine-tuning on the GSM8K 7.5k training set, the SELF approach achieves a higher accuracy of 37.87%, surpassing the fine-tuned model's accuracy of 35.70%. It's essential to highlight that the reported 29.64% in Table 2 of our paper stemmed from a 3.5k meta-skill learning corpus and to ensure a fair comparison, new experiments using an expanded 7.5k meta-skill data demonstrate the effectiveness of SELF, with performance reaching 38.12%, outperforming the supervised fine-tuning result.

These experimental results will be detailed in the appendix of the revised paper due to the scope limitation.

W1 & Q1: Discussion about self-consistency

Self-consistency [1] works by sampling a diverse set of reasoning paths and then selecting the most consistent answer (We sample 5 times in our experiments), a straightforward and effective method. Self-refinement enables models to evaluate and improve their outputs.

Self-consistency operates without additional finetuning while self-refinement applies to broader domains, not just those with unique correct answers, as demonstrated in Section 4.2.2 (General Test).

Section 4.2.1. shows that self-refinement can complement self-consistency. Integrating these two strategies leads to a better performance.

We will incorporate a thorough discussion and citation of [1] in the revised paper.

[1] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- er, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022a.

W2: Missing reference of RL-without human feedback

We will revise our paper to include a discussion on RL-related methods in the 'Related Works' section. We have conducted an additional RL experiment, the results of which will be detailed and compared in the response to Q2.

W5: SELF versus Supervised Fine-tuning baseline

In addressing the comparison between the SELF framework and supervised fine-tuning using the GSM8K training set, we present the following results:

Supervised Fine-Tuning vs SELF: The Vicuna model, when fine-tuned on the GSM8K 7.5k training set achieves an accuracy of 35.70%, which is lower than of SELF (37.87%).

It is important to note that the 29.64% in Table 2 of our paper originated from a 3.5k meta-skill learning corpus. To ensure a fair comparison with the Supervised Fine-tuned model, new experiments were conducted using an expanded 7.5k meta-skill data as shown above.

Specifically, using 7.5k unlabeled training prompts to construct the meta-skill learning corpus, The baseline model Vicuna + $D_{QA}$ achieves 28.05%. After meta-skill learning, the initial result is 31.23%, which improves to 32.98% after self-refinement. The performance further increases in subsequent self-evolution rounds, with the second round reaching 37.87% to 38.12%, surpassing the supervised fine-tuning result(35.7%).

Advantages of SELF: SELF's main advantage is its ability for continuous improvement and adaptation. Unlike supervised fine-tuning, SELF does not rely on human or external LLM (GPT3.5/GPT4) to annotate training data in the self-evolution training. As noted in the response to weakness 4, providing language feedback to the model's incorrect responses can also improve its question-answering (QA) capabilities. This is another advantage of SELF.

Q1: Self-refinement is a necessary component for SELF

We note that performance degradation occurs only in models lacking meta-skill learning in Table 1 of our paper (Vicuna, and Vicuna with $D_{QA}$). In contrast, models equipped with meta-skill learning consistently exhibit performance improvement via self-refinement. Table 1 demonstrates that SELF, when applied with self-refinement, achieves an accuracy of 31.31%, surpassing the 29.87% obtained using self-consistency. Furthermore, while self-consistency is limited to problems with unique answers, self-refinement can be applied to a more general domain as evidenced in Section 4.2.2(General Test).

We also conducted an ablation study to compare the effect of self-refinement and self-consistency on self-evolution training data quality and their subsequent impact on test performance:

In the above table, "Acc. of Training Data" refers to the accuracy of self-generated training data post-filtering/refinement, while "Acc. on Test Set" indicates the model's test performance after fine-tuning such training data.

As shown in the table, self-refinement produces higher quality training data compared with self-consistency, and results in better fine-tuned model performance. The final row demonstrates that further filtering the self-refinement data with self-feedback can improve both training data accuracy and test performance.

The study clearly shows that self-refinement is a necessary component of SELF. Coupled with self-feedback, self-refinement establishes a robust foundation for the self-evolution training process. Moreover, self-refinement can improve the model's performance during inference.

Q2: Some related work for debugging/checking is missing

We appreciate your suggestion. We will update our paper to add citations of these works into related works and discuss how SELF is related.

W1(a): Training corpus collection pipeline

In Section 3.1.1 of our paper, we have outlined the pipeline for collecting the training corpus for meta-skill learning, including self-feedback and self-refinement. In the revised version of our paper, we will provide more comprehensive details.

W1(b): Model for generating feedback and refinement

For generating the meta-skill learning corpus, we employed GPT-4 owing to its superior refinement capabilities

W1(c): Hyperparameters

The hyperparameters used during the training were described in Appendix A.1.1 of our paper. These parameters were consistently applied across all training methods in our experiments.

W2: Combining all data into a single round versus multiple rounds (iterative)

To address your concerns, we conducted the following experiments:

Single Round vs. Iterative Training: In a single round, the performance is 28.40% for direct generation and 30.55% for self-refinement. The iterative approach shows higher scores: 29.64% for direct generation, 31.31% for self-refinement.

Advantages of Iterative Training: The iterative method benefits from improved LLMs in later rounds, producing higher-quality training data and, consequently, enhanced test performance.

W3: About training over QA data during the meta-skill learning stage

(1) The QA data in Table 2, denoted as $D_{QA}$, consists of pairs of prompts and pseudo answers provided by GPT4. It does not include ground-truth labels, which are noted in section 3.1. Moreover, pseudo answers in $D_{QA}$ are extracted from meta-skill learning data $D_{meta}$.

(2) We combine both $D_{QA}$ and $D_{meta}$ as we explain in Section 3.1: "Given that the data structure in $D_{meta}$ diverges from conventional direct question answering formats, we also employ a dataset composed of pairs of questions and refined answers, denoted as $D_{QA}$, during meta-skill training."

To eliminate any confusion, we will revise our paper, specifically Section 3.1, to clearly explain how $D_{QA}$ and $D_{meta}$ contribute to the meta-skill learning.

W4: $M_{\text{meta}}$ can also boost reasoning performance

In the caption of Table 2, "The right arrow indicates the performance improvement by Self-Refinement" indicates that the table displays results both without and with the application of self-refinement. The second row shows the result '25.39 → 28.28'. Here, the 25.39% reflects the direct generation performance, which is an improvement without self-refinement compared with the 24.49% (first row). The subsequent increase (+2.89%) to 28.28% represents the additional gain brought by self-refinement. It is evident that the inclusion of the meta-skill learning corpus ($D_{\text{meta}}$) leads to enhancements in both direct generation and self-refinement outcomes.

As we have discussed in Section 4.3(1) ("Integration of Meta-skill Training Data $D_{\text{meta}}$ Elevates Direct QA"), providing language feedback to the mistake of the model can improve the performance. The positive impact of $D_{\text{meta}}$ is an interesting finding and is worthy of future research.

Thank you for your efforts! However, I must admit that after spending a considerable amount of time reading the paper, I am still quite confused. I couldn't grasp the core contribution of the paper initially. I suggest that the authors rewrite sections 3 & 4 and resubmit to the next conference. The current version cannot be accepted by ICLR.

Q2: An upper limit or plateau in the self-evolution process and when to stop

Table 1 in our paper illustrates that iteratively enhancing model capabilities with increased self-evolution training corpus progressively improves performance. However, we presume that this improvement may exhibit diminishing returns over time.

To determine the optimal point for halting the self-evolution, we employ a validation set to monitor performance variance. When the model performance converges, indicating minimal gains from additional training, it is appropriate to stop the self-evolution process.

Additionally, the upper bound of self-evolution effectiveness is influenced by several factors. We have done additional experiments that indicate that stronger baseline models lead to more significant improvements in self-evolution performance, thereby raising their learning upper limits. The result is as follows:

This indicates that starting with a more robust model can enhance the efficacy of the self-evolution process.

Q3: Prevent the inclusion of unsafe or unethical knowledge in the self-evolving training

To prevent unsafe or unethical knowledge from entering self-evolving training, we can train the model with meta-skills to recognize and revise unethical or unsafe content. We then apply these skills during the self-evolution phase. By integrating this mechanism into the self-evolution process, we aim to ensure that the model remains aligned with ethical and safe guidelines.

Q4: Dependency of SELF on the starting model quality

To explore how SELF performs with different starting model qualities, we conducted experiments using the OpenLlama-3b model [1], a smaller LLM, along with a stronger LLM, VicunaV1.5(finetuned from Llama2-7b), on the GSM8K dataset. This allowed us to assess SELF’s adaptability to model quality. The results were as follows:

We conclude that SELF demonstrates robustness by consistently enhancing the performance of models from smaller to stronger. Its efficacy is affected by the capability of the base model, as the stronger model acquires more benefits from SELF.

[1] Xinyang Geng and Hao Liu. Openllama: An open reproduction of llama, May 2023. URL https://github.com/openlm-research/open_llama

Q5: Computational overhead and more expensive cost of SELF compared to regular supervised training

The SELF framework consists of two primary steps:

1. Creating self-evolution training data.

   In contrast to regular supervised fine-tuning, which often requires extensive manual data annotation—a laborious and time-consuming process—SELF automates the generation of its training data. This automation significantly reduces the effort and time typically associated with data annotation.

2. Fine-tuning the model with this newly generated data.

   It's important to note that the actual training phase in SELF is similar to the Supervised Fine-Tuning (SFT) process. Therefore, SELF does not introduce additional computational costs in the training phase.

When considering the overall process, SELF effectively reduces both the time and monetary costs associated with data collection, making it a more efficient alternative to traditional supervised training methods.

W1: The analysis on sensitivity of SELF to the quality of meta-skills.

Acknowledging the importance of the quality of meta-skill training data, we conducted experiments to examine the effect of using different models (GPT-3.5-turbo vs GPT-4) for corpus construction.

The experiments indicate that with a higher quality meta-skill learning corpus from GPT4 compared with GPT-3.5-turbo, all performance metrics in our SELF framework show improvement. This is particularly significant when applying self-refinement, where the increase in higher quality meta-skill data surpasses 3%.

Our results confirm that the quality of the meta-skill training data is crucial for instilling self-feedback and self-refinement into the Vicuna model. These findings underscore the importance of using high-quality models for corpus construction in meta-skill training.

W2: Insight on how self-evolving training affects model internals and learned representations.

We have conducted thorough experiments to investigate how different factors influence self-evolving training. Exploring how self-evolving training affects model internals and learned representations is a worthwhile research direction. Due to limited resources and scope, we leave this for future work.

W3: Comparisons to related human preference alignment methods.

We have conducted additional experiments to compare the SELF framework with Reinforcement Learning from Human Feedback (RLHF). For a fair comparison, we use the same SFT model (Vicuna + $D_{QA}$) for SELF and RLHF. The reward model was trained on pair-wise comparison data (refined response is assumed better than original response) derived from $D_{meta}$, the meta-skill learning corpus in SELF. The results are as follows:

The experimental analysis is as follows:

RLHF achieved a 25.55% accuracy on GSM8K, lower than SELF's 27.67%. The reason is that the reward model often fails to identify the correct response via a single scalar value, leading to limited performance improvements. Specifically, 76% of incorrect answers were assigned higher scalar rewards than correct answers on the GSM8K test set. Unlike RLHF, SELF leverages natural language feedback, which provides a more accurate evaluation (only 28% incorrect answers were judged as correct).

Q1: The robustness of SELF to noise in self-generated data and the effectiveness of meta-skills in filtering.

We conduct the following experiments to illustrate the robustness of SELF to noise in self-generated data and the effectiveness of meta-skills in filtering:

The second column means the accuracy of self-generated training data. The third column means the test performance of the model finetuned on such data.

Experimental analysis:

(1) Unfiltered: This strategy utilizes self-refinement to refine the self-generated data without filtering.

(2) Meta-Skills Filtered: After applying self-refinement, we utilize self-feedback to filter the data, and there is a significant improvement in the accuracy of training data (44.10%). The performance of the finetuned model also improves (27.67%). We observe that the performance improvement is not as significant as the accuracy improvement of self-generated data. We hypothesize the reason is that the self-generated data size shrinks after applying filtering (from 4k to 1.8k).

By solely using self-refinement, the performance of the finetuned models still improves. This shows the robustness of SELF to noise in self-generated data. The accuracy of the self-refinement data significantly improves with the help of self-feedback meta-skill and results in better fine-tuned model performance. This shows that self-feedback has a strong ability to filter bad data.