Self Guided Exploration for Automatic and Diverse AI Supervision

Dear reviewer 1gMq,

Thank you so much for your positive review and insightful comments. We appreciate your feedback and suggestions, which we have incorporated into the revised version to improve it.

#1: Can exploration principles be used for simple data augmentation to improve performance?

It's an insightful question whether data augmentation based on simple rules improves performance. Augmenting text with simple rules is an interesting baseline to consider; we have conducted experiments to study it. Unlike image augmentation, text augmentation is nontrivial; therefore, we opted to use Vicuna, which is the same model used in EAI, to augment the GSM8K and MATH training split. We augment each data point multiple times to obtain the same number of data points as generated by EAI, and use EAI's exploration principles as rules for data augmentation. Concretely, for each question, we ask Vicuna to follow a principle such as 'rephrasing' to augment the question, and we sample multiple responses. The results shown in the table below reveal that data augmentation improves performance, especially when all principles are combined, which outperforms human-curated data. It also shows that EAI performs substantially better than all the data augmentation variants, highlighting the importance of in-context exploration and the effectiveness of the EAI framework.

#2: Some comparisons with human designed rules would be interesting. Moreover, the ablation is not complete, how about only do one principle at a time? Will the performance drop a lot?

Thank you for the suggestion to study one principle at a time and to classify the generated data into categories. To this end, we have conducted additional ablation experiments to study how each principle works. The results are included in Table 3, which is attached below. This analysis reveals that removing any one of the principles reduces performance; using only one of them is suboptimal; among the principles, rephrasing and restructuring are the most effective; and combining all principles achieves the best results.

#3: It is also helpful to release the questions/answers generated on GSM8k and MATH for future comparisons and analysis.

We recognize the value of this for future comparisons and analysis, and will release both the generated dataset and the codebase used to create it.

Dear reviewer uvsC,

Thank you so much for your review and for highlighting the idea of principle based exploration is well presented, and the research problem is interesting. We appreciate your feedback and suggestions, which we have incorporated into the revised version to improve it.

Your main concern is comparing with stronger baselines. We will address your concern and answer your clarification questions below. Please let us know if our response satisfies you.

#1: Primary complaint: compare with much stronger baselines like WizardMath and MAmmoTH

We have conducted experiments comparing EAI with WizardMath and MAmmoTH. Previously, we did not directly compare them because their methods involve generating data from much more powerful models such as GPT-4 to improve LLaMA, while our experiments focus on generating data from LLaMA itself. Since we do not have access to GPT-4's weights, distilling GPT-4 is not transparent. Additionally, GPT-4 may change or discontinue service in the future, which could hinder reproducibility. Furthermore, the terms of service of OpenAI impose legal requirements on distilling their proprietary data. Therefore, we focused on enhancing LLaMA and Vicuna using the outputs generated by these models.

That being said, we agree with the reviewer that a comparison with WizardMath and MAmmoTH can make the paper more convincing. We included experiments where we applied EAI to GPT-4 and then finetuned LLaMA with the resulting data, following the approaches of WizardMath and MAmmoTH. The results in Table 1 show that EAI performs substantially better than both approaches on both GSM8K and MATH, achieving new state-of-the-art performance and demonstrating EAI's effectiveness in generating diverse outputs.

#2: The paper only evaluates the proposed algorithm on a very narrow set of problems, namely only two benchmarks, and both relating to math problems. A more thorough evaluation on a variety of benchmarks covering different types of problems would be much more convincing re: the scalability of the method.

We focused on evaluating EAI on GSM8K and MATH benchmarks because these two are the main benchmarks used in our baselines publications. We agree with the reviewer that more thorough evaluation on a variety of benchmarks can make the paper more convincing. To this end, we conducted experiments on the code generation task. Results are shown in Table 5 in the paper and attached below as well. Evaluated with 3-shot MBPP test split and 0-shot HumanEval, EAI substantially outperforms all baselines and improve LLaMA2 and CodeLLaMA in their code generation performance, achieving 2.8% absolute increase in 3-shot MBPP and 1.9% increase in 0-shot HumanEval, confirming its effectiveness in exploring diverse supervision.

*#3: Justification for picking the number 48K for number of generated data points

Regarding picking the number 48K as the number of generated data points, it is the maximum number of generated data points reported by RFT and we choose it to achieve a fair comparison and to reduce inference cost. We have made this more clear in revision. We have conducted more experiments to scale up 48K to 96K to study the scaling trend more rigorously.

Regarding the trend of scaling, Figure 4 shows the performance of different methods along with the number of generated data points, EAI keeps outperforming RFT and scales well from 8K to 48K to 96K, while baselines saturate after 40-50K. This result confirms the effectiveness of EAI and we believe that further improvements can be done by doing large-scale exploration, which we leave for future work.

#4: How does EAI work without any “seed” data

For zero-shot exploration without human-provided “seed” data, we start by instructing Vicuna with instructions like generating a high school level math question and randomly sample 32 questions as the “seed” dataset. We will make this more clear.

Dear reviewer td87,

Thank you so much for your review and insightful comments. We appreciate your feedback and suggestions, which we have incorporated into the revised version to improve it.

#1: The method is still relying on humans to provide exploration principles.

It is true that humans provide exploration principles to LLM to guide autonomous exploration. It is much easier to provide these principles than to manually curate data. We used a set of four exploration principles including paraphrasing questions, coming up with a new topic, coming up with a new scenario, and restructuring questions, which are simple to come up with and does not require domain expertise. In contrast, humans to curate the dataset is both time consuming and require domain expertise. Compared with human supervision, the EAI framework generates diverse AI supervision with minimal human input, is both scalable and effective. To make EAI even more scalable, we plan to investigate generating principles by LLM itself in future work.

#2: would love to see more experiments on different kinds of benchmarks

We focused on evaluating EAI on GSM8K and MATH benchmarks because these two are the main benchmarks used in our baselines publications. We agree with the reviewer that more thorough evaluation on a variety of benchmarks can make the paper more convincing. To this end, we conducted experiments on the code generation task. Results are shown in Table 5 in the paper and attached below as well. Evaluated with 3-shot MBPP test split and 0-shot HumanEval, EAI substantially outperforms all baselines and improve LLaMA2 and CodeLLaMA in their code generation performance, achieving 2.8% absolute increase in 3-shot MBPP and 1.9% increase in 0-shot HumanEval, confirming its effectiveness in exploring diverse supervision.

#3: Include an ablation study on critic

We recognize the importance of an ablation study to isolate the effect of the critic component in our framework. In the revised version, we include a detailed ablation study with its results shown in Table 4, comparing EAI with and without the critic component on GSM8K, MATH, MBPP, and HumanEval benchmarks.

The results are also attached below for convenience. They show that EAI without the critic significantly outperforms the established baselines, demonstrating the effectiveness of principles guiding context exploration. Reintroducing the critic into the EAI framework substantially improves these results, achieving significantly better performance than Self-instruct, RFT, and EAI without the critic.

A qualitative analysis provided in Appendix A reveals how the critic aids in guiding exploration. Both the quantitative and qualitative results demonstrate that the critic is crucial for achieving optimal exploration outcomes; removing the critic leads to substantially lower performance.

#4: Include T-SNE results in addition to diversity gain.

We appreciate your suggestion to include more visualization for the diversity experiment. In response, we integrate T-SNE plots of the embeddings in Figure 3 of our revised manuscript. Compared with human curated GSM8K and RFT generated data points, EAI produces substantially more diverse data points, providing more compelling evidence of its effectiveness.

#5: More discussions on connection to unsupervised RL pretraining and exploration

We will elaborate on the connection between our framework and unsupervised reinforcement learning (RL) in the revised paper. Although our method primarily uses prompting, it aligns with the principles of unsupervised RL by learning from the environment (in this case, the responses from the LLM) and adapting accordingly. Unsupervised RL like APT tries to visit states that are different from states sampled from replay buffer through reinforcement learning, EAI instead tries to generate data that are different from data sampled from replay buffer through principles guided in context exploration. We will also clarify how EAI contributes to the learning of unsupervised exploration over time through diverse in context examples, bridging the perceived gap.

In conclusion, we believe that addressing these points substantially strengthens our submission. We hope that our response addresses your concerns. If our response addresses your concerns, please consider revising the score accordingly, otherwise, please let us know if anything further is required. We look forward to hearing from you.

Dear reviewer 7sTU,

Thank you so much for your positive review and insightful comments. We appreciate your feedback and suggestions, which we have incorporated into the revised version to improve it.

#1: From Table 2, it seems that a larger replay buffer can achieve more improvements. What if we use an even larger one (e.g., 12 or 16)? Will the performance keep increasing or converge?

We conducted experiments to investigate this. We scale up the number of samples from replay buffer from 8 to 12 and 16, and evaluate the performance. The following table presents the results of an experiment examining the impact of varying the number of samples on GSM8K and MATH. As the number keeps increasing, the performance on both datasets keep increasing, highlighting the significance of having larger input size for exploration. LLM with larger context window size can fit in more samples, and potentially lead to more effective exploration.

#2: What is the training efficiency of EAI?

EAI is compute intensive due to it involves sampling from large language models. Using 8x A100, generating 48K samples with EAI takes about 12 hours. As inference approaches keep advancing, the training efficiency of EAI will keep increasing.

#3: Include a detailed analysis of the quality/diversity of the generated content.

We appreciate your suggestion to include more visualization for the diversity experiment. In response, we integrate T-SNE plots of the embeddings in Figure 3 of our revised manuscript. Compared with human curated GSM8K and RFT generated data points, EAI produces substantially more diverse data points, providing clearer and more compelling evidence of its effectiveness.

#4: Include qualitative results of the generated content.

We appreciate your suggestion to include qualitative analysis. A qualitative analysis provided in Appendix A, which reveals how the EAI accomplishes exploration. Actor follows principles to generate diverse samples and the critic provides feedback to aid the exploration.

#5: Potential issues of critic and actor being the same LLM.

Across all four benchmarks we evaluated, we did not observe any degeneration due to exploration. Since the actor is guided by a set of exploration principles, such as 'restructuring questions', its outputs are steered towards high-quality data. Moreover, the critic's method of evaluating the actor’s outputs, akin to widely used self-reflection techniques, further enhances the quality of exploration. Furthermore, the benefits of diverse samples may outweigh the drawbacks of a potentially incorrect subset of samples.

#6: Other suggestions.

Thanks for suggesting and we have incorporated them to improve the paper.

In conclusion, we believe that addressing these points substantially strengthens our submission. We hope that our response addresses your concerns. If our response addresses your concerns, please consider revising the score accordingly, otherwise, please let us know if anything further is required. We look forward to hearing from you.