Let Large Language Models Find the Data to Train Themselves

Thank you for your thoughtful review and valuable feedback. We appreciate your recognition of our work’s well-motivated. Below, we address your concerns in detail.

Q1: Regarding the implementation of ICL for policy model updating.

A1: We appreciate the reviewer's insightful comment regarding the implementation of ICL for policy model updating. We would like to provide further clarification, the policy model requires frequent updates with a complexity of O(trajectory sampling number * target task number), resulting in approximately 40,000 updates per experiment. Given this computational intensity, implementing traditional fine-tuning approaches would be prohibitively expensive. Consequently, we adopted in-context learning as our primary methodology across all experimental conditions to maintain computational efficiency while preserving performance.

Furthermore, extensive research has demonstrated that ICL can achieve comparable or even better effectiveness than traditional parameter fine-tuning [1][2][3][4][5]. We will include these analyses in the updated version.

References:

[1] Exploring the relationship between in-context learning and instruction tuning. arXiv preprint, 2023.

[2] Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation, ACL 2023 Findings, 2023.

[3] Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers. ACL 2023 Findings, 2023.

[4] In-Context Learning with Long-Context Models: An In-Depth Exploration. arXiv preprint, 2024.

[5] Many-Shot In-Context Learning. NeurIPS, 2024.

Q2: Regarding the comparison between ADS and baseline methods.

A2: Since more than one reviewer raised this question, we have responded to it in the global rebuttal section (Q1) to save space.

Q3: Regarding the efficiency of our method.

A3: We sincerely appreciate your insightful feedback regarding the efficiency of our method. Our approach is divided into two parts: optimizer refinement ("learn to learn") and policy optimization ("learn").

Optimizer refinement requires frequent updates and evaluations of the policy model during the training process, resulting in a time complexity of O(trajectory sampling number * target task number). We acknowledge that the optimizer refinement process requires substantial initial overhead.

However, once implemented, the subsequent costs for policy optimization become minimal. This automated approach serves to replace the traditional, labor-intensive process of manual training data collection and curation, which typically involves extensive trial and error, thereby significantly enhancing model adaptation efficiency for learning any tasks.

Furthermore, as discussed in the limitations section, we acknowledge that the proposed ADS framework represents a conceptual prototype implementation of the automated data collection process. There remains substantial further investigation before practical implementation can be achieved. We will incorporate these analyses into the updated version of our work.

Q4: Regarding the comparison between ADS and RLAIF and AutoDetect.

A4: We thank the reviewer for their insightful question regarding the comparison between ADS and AutoDetect. To clarify, the goal of our proposed ADS approach is to automatically collect valuable training data to enhance the performance of the target task, which distinguishes it from the aforementioned methods.

The RLAIF approach primarily focuses on leveraging rewards from AI to replace human feedback in reinforcement learning. This method relies on existing training data and does not involve the collection of new data.

In contrast, the AutoDetect method requires a powerful LLM, such as GPT-4, to identify weaknesses in a weaker policy model through iterative question generation and answer evaluation. This approach depends on GPT-4 for weakness identification and distills data from GPT-4.

We acknowledge that the word "training" in our paper can be controversial. In-context learning is a new learning paradigm that significantly diverges from traditional machine learning approaches and primarily works with large language models. On one hand, a number of studies have studied its effectiveness, efficiency, and underlying mechanisms, particularly in comparison to fine-tuning, suggesting that it is a strong alternative to fine-tuning [1-5]. On the other hand, yes, it is usually categorized as an inference-time learning method.

To avoid future confusion, we can replace all related wordings of "training" with "teaching" in our revised manuscript. Please note that the core contribution of this paper is the introduction of an automatic data collection and curation framework for targeted performance enhancement. Generally, the specific methods used to learn from the resulting data do not affect the overall framework. We can opt for fine-tuning when computational costs are more affordable.

Thank you once again for your valuable suggestions. We greatly appreciate your feedback and would like to know if this helps clarify the contribution of our paper or if you have any further comments on this matter.

Thank you for your thoughtful review and valuable feedback. We appreciate your recognition of our work’s novelty and significance. Below, we address your concerns in detail.

Q1: Regarding the intrinsic advantages of the API trajectory optimized by the optimizer model in ADS.

A1: We appreciate the reviewer’s insightful comment regarding the inherent advantages of API trajectories generated through our optimizer model. Within the ADS framework, the optimizer model is specifically designed to identify and curate specialized training datasets that enhance policy performance by generating appropriate textual API calls to interact with various resources and environments. Through iterative reinforcement learning, the optimizer model is incentivized to detect and address performance weaknesses, thereby facilitating the progressive development of self-knowledge. This process systematically increases the probability of generating optimal API trajectories, ultimately leading to continuous self-improvement of the system.

Q2: Regarding the comparison between ADS and strong baseline strategies.

A2: Since more than one reviewer raised this question, we have responded to it in the global rebuttal section (Q1) to save space.

Q3: Regarding the generalization of ADS.

A3: Thank you for the insightful question. We would like to clarify that the primary objective of our ADS framework is to develop a generalized optimizer model capable of addressing a wide spectrum of target tasks. To achieve this, we have meticulously compiled a diverse dataset comprising approximately 10,000 target tasks for optimizer training and validation.

Additionally, in our experiments, despite the evaluation of 1,000 in-house target tasks, we have also validated our optimizer model's generalizability on three established public benchmarks: AlpacaEval 2.0, Arena-Hard, and MT-Bench. These benchmarks span multiple domains and task types, and importantly, are entirely independent of our training dataset, thereby providing robust evidence of our model's generalization applicability.

Q4: Regarding the function and importance of the three APIs in ADS.

A4: Thank you for raising this important point. As shown in Section 4.1, these three APIs are designed to facilitate the acquisition, utilization, and enhancement of knowledge, respectively. The Information Retrieval API employs both sparse and dense retrieval to retrieve relevant documents from external knowledge databases such as Wikipedia, thereby supporting knowledge acquisition, analogous to the pre-training stage of LLMs. The Demonstration Generation API utilizes the policy model to generate appropriate exemplar instruction-response pairs, tailored to various knowledge application scenarios, reminiscent of the alignment stages of LLMs. The Question Answering API resorts to the wisdom of human experts, mimicking how humans learn from each other.

We also demonstrate the ablation experimental results in the table below. We can observe that the proposed ADS framework with all of these three APIs demonstrates the best performance.

Q5: Regarding the splitting of observed and held-out instructions.

A5: Thank you for raising this thoughtful question. As discussed in Section 4.2 (line 226-233), we divide the instructions for each target task into two distinct sets: one set of observed instructions for trajectory generation, and another set of held-out instructions for performance evaluation. This splitting serves a crucial purpose: it enables us to verify that the optimizer is effectively learning to enhance task-level performance (i.e., acquiring valuable training data that improves overall task capability) rather than merely optimizing for instance-level solutions (i.e., developing specific solutions for given instructions). This distinction is important because superior performance on observed instructions does not necessarily generalize to improved performance on held-out instructions, potentially indicating overfitting rather than effective learning.

Q6: Regarding the overlap between generated instructions and original instructions.

A6: Thank you for raising this important point. In our methodology for generating new task-specific instructions (which serve as observed instructions), we follow the implementation in Alpaca [1] to ensure instruction diversity. Specifically, we prompted the LLM to generate instructions that were different from the original ones. To maintain distinctiveness, we employed a filtering mechanism whereby any generated instructions with a Rouge-L similarity score exceeding 0.7 when compared to the original instructions were eliminated. We will incorporate these details in the revised version.

References:

[1] Stanford Alpaca: An Instruction-following LLaMA model. GitHub repository, 2023.

Q7: Regarding the reasons for performance improvements achieved by cost control.

A7: This is an insightful observation. The cost-control mechanism, by selecting responses from among those with top-tier rewards while prioritizing lower costs, inherently promotes greater response diversity compared to pure reward maximization approaches. This enhanced diversity potentially yields two significant benefits: first, it contributes to more generalized performance improvements across various tasks, and second, it helps mitigate reward over-fitting issues. These additional discussions will be incorporated into our revised manuscript.

We sincerely appreciate the reviewer's insightful comment regarding the utilization of the QA API within the ADS framework.

We would like to clarify that our proposed ADS framework does not overly rely on the QA API; for instance, on 1,000 in-house tasks using Gemini-2-9B-Instruct, the number of QA API calls was reduced by 59.6% compared to the ablation baseline that solely depends on the QA API. The proportion of QA usage is only about 44.2% among all three APIs. It is important to note that the QA API necessitates a strong external LLM to serve as a proxy for human annotation, making it the most cost-intensive among the other APIs. The ablation study actually shows that our ADS framework achieves larger performance gains at reduced operational costs, particularly with much fewer QA API calls.

Thank you once again for your valuable suggestions. We greatly appreciate your feedback and hope that above information helps address your concern regarding the effect of the QA API.

Q5: Regarding the implementation of ADS to different policy models.

A5: Thank you for this valuable suggestion. In our experiments, we have explored two different policy models, including Qwen-2-7B-Instruct and Gemma-2-9B-Instruct, both of which are famous open-source LLMs in the research community. As illustrated in Figure 2 and Table 2, our proposed ADS framework, when integrated with these diverse policy models, demonstrates significant performance enhancements across a wide range of in-house and public test tasks.

Thank you for your thoughtful review and valuable feedback. We appreciate your recognition of our work’s innovation and robustness. Below, we address your concerns in detail.

Q1: Regarding the comparison between ADS and baseline methods.

A1: Since more than one reviewer raised this question, we have responded to it in the global rebuttal section (Q1) to save space.

Q2: Regarding the contribution of individual API and iterative refinement.

A2: Thank you for raising this important point. As shown in Section 4.1 (line 226-233), these three APIs are designed to facilitate the acquisition, utilization, and enhancement of knowledge, respectively. The Information Retrieval API employs both sparse and dense retrieval to retrieve relevant documents from external knowledge databases such as Wikipedia, thereby supporting knowledge acquisition, analogous to the pre-training stage of LLMs. The Demonstration Generation API utilizes the policy model to generate appropriate exemplar instruction-response pairs, tailored to various knowledge application scenarios, reminiscent of the alignment stages of LLMs. The Question Answering API resorts to the wisdom of human experts, mimicking how humans learn from each other. We also demonstrate the ablation experimental results in the table below. We can observe that the proposed ADS framework with all of these three APIs demonstrates the best performance.

To illustrate the effectiveness of the iterative refinement process in ADS, our original submission conducted a series of experiments and demonstrated the iterative training results (iteration 0-3) in Figure 2 and Table 2. Our findings indicate that iterative ADS boosts consistent performance improvements for both in-house test tasks and generalized public benchmarks.

Q3: Regarding the analysis of tasks that ADS does not perform as expected.

A3: Thank you for raising this important point. In Figure 3, we have shown that for categories like editing, creative writing, and debugging, our ADS only has slight improvements or maintains comparable to the baseline. This limited enhancement can be potentially attributed to the inherent nature of these tasks, which primarily involve format and style rewriting, as well as fragment modifications, presenting inherent challenges for optimization through in-context learning from acquired training data. These additional analyses will be incorporated into the updated version of the paper.

Q4: Regarding the details of reinforcement learning.

A4: We sincerely appreciate your insightful feedback regarding the details of reinforcement learning. To facilitate the optimizer’s decision-making process in obtaining optimal training data, we propose a reinforcement learning strategy for optimizer training. This strategy leverages feedback reward signals from the policy to concurrently maximize task performance and minimize computational costs. As described in Section 4.4 (line 307-311), the reinforcement learning process comprises two primary phases. Initially, we implement a warm-up rejection sampling procedure, which fine-tunes the optimizer model on trajectories that demonstrate the highest reward among multiple sampled responses. Subsequently, we engage in an iterative process to update the optimizer model through direct preference optimization. This process involves selecting a "chosen" trajectory (exhibiting the highest reward) against a "rejected" trajectory (displaying the lowest reward). The objective of this approach is to enable the optimizer model to amplify the gap between these high-quality and low-quality trajectories, therefore increasing the probabilities of generating valuable API trajectories.

Thank you for your replies. Most of my concerns have been addressed.

Thank you for your thoughtful review and valuable feedback. We appreciate your recognition of our work’s promisingness and effectiveness. Below, we address your concerns in detail.

Q1: Regarding the comparison between ADS and baseline methods.

A1: Since more than one reviewer raised this question, we have responded to it in the global rebuttal section (Q1) to save space.

Q2: Regarding the difference between ADS and RAG.

A2: Thank you for your insightful question. The distinctions between ADS and RAG can be summarized across three key dimensions:

1. Motivation: ADS primarily aims to identify valuable training data for a specific task through environmental interactions via various APIs (including but not limited to Information Retrieval), ultimately enhancing overall task performance. In contrast, RAG focuses on retrieving relevant information for individual queries, with the primary objective of improving response quality for specific queries rather than optimizing global task performance.
2. Implementation: The fundamental component of ADS is the optimizer model, which dynamically generates appropriate API calls and API parameters at the task level. This differs significantly from traditional RAG, where its retrieval process solely and directly matches the given instructions to texts in a database.
3. Framework: ADS incorporates more comprehensive APIs beyond Information Retrieval, including Demonstration Generation and Question Answering. This integration of diverse APIs results in a more complicated and effective framework compared to RAG.

Q3: Regarding the use of the Llama model in our experiments.

A3: We sincerely appreciate your insightful feedback regarding the use of the Llama model in our experiments. To address your questions, we would like to clarify that, since our training data is generated by Llama-3-8B-Instruct, we exclude it from our choices of policy models to avoid any potential biases and ensure a fair comparison. This explanation is also mentioned in Section 4.4 (line 323) of the current manuscript. Regarding the Question Answering API, which is designed to generate responses to given questions and is independent of the training data, we opt for the strong Llama-3.1-70B-Instruct model as a proxy for human annotation in practice for efficiency and reproducibility. It is worth noting that alternative LLM such as GPT-4 or Claude-3.5 could also serve as the strong model.

Q6: Regarding the generalization of ADS.

A6: Thank you for the insightful question. We would like to clarify that the primary objective of our ADS framework is to develop a generalized optimizer model capable of addressing a wide spectrum of target tasks. To achieve this, we have meticulously compiled a diverse dataset comprising approximately 10,000 target tasks for optimizer training and validation. Additionally, in our experiments, despite the evaluation of 1,000 in-house target tasks, we have also validated our optimizer model's generalizability on three established public benchmarks: AlpacaEval 2.0, Arena-Hard, and MT-Bench. These benchmarks span multiple domains and task types, and importantly, are entirely independent of our training dataset, thereby providing robust evidence of our model's generalization applicability.

Q7: Regarding the requirements of clustered tasks.

A7: We appreciate the reviewer’s insightful comment regarding the requirements of clustered tasks. Our proposed approach primarily aims to enable LLMs to identify valuable training data that can enhance their performance on specific target tasks. In our experiments, we prompt LLMs to cluster a large instruction dataset into splits of specific tasks, ensuring the reproducibility of our research. We believe that in practical applications, users can simply formulate a small number of exemplar instructions according to their needs.

Q8: Regarding the case studies of ADS.

A8: Thank you for raising this important question. In our original submission, we present case studies in Table 4 (Appendix C). We can observe that our optimizer model, when provided with a few task-specific instructions, is capable of executing a three-step process. First, it generates a comprehensive analysis of the fundamental requirements to solve the target task. Subsequently, it produces self-reflective evaluations to identify potential limitations. Finally, it develops corresponding API trajectories to construct suitable training data, thereby enhancing task-specific capabilities.

Q9: Regarding the construction of API trajectories in rejection sampling and direct preference optimization.

A9: Thank you for raising this important question. As discussed in Section 4.4 (line 307-311), for reward maximization optimization, the construction of API trajectories is as follows:

1. In rejection sampling, the chosen trajectory is the one with the maximum reward from multiple sampled trajectories.
2. In direct preference optimization, we select the paired chosen and rejected trajectories with the maximum and minimum rewards, respectively.

For the cost-control mechanism, we introduce a cost tier parameter τ ∈ [0, 1] to control the trade-off between rewards and costs. Trajectories within the top-tier rewards ranging [(1−τ )Rmax+τRmin, Rmax] are considered to have similar performance. From this subset, we select the trajectory with the lowest cost as the chosen trajectory. Conversely, for the reject trajectory, we select the one with the highest cost within [Rmin,(1 − τ )Rmin + τRmax]. We will make the above clearer in the next version.

Q10: Regarding the cost of API calls considered only when selecting trajectories.

A10: Yes, you are right. The API call costs are exclusively considered during the selection of trajectories from a given reward-tier trajectory group. Specifically, for rejection sampling, we select the chosen trajectory with the minimal cost among those in the highest reward tier. Regarding direct preference optimization, we select the lowest-cost trajectories from the top reward tier for chosen trajectories, and highest-cost trajectories from the bottom reward tier for rejected trajectories.

Q11: Regarding the details of the prompts used in ADS.

A11: We would like to note that all these prompts are detailed in Appendix A of our original submission. For individual API, we utilize zero-shot prompting. For the optimizer model's API trajectory generation, we implement a two-shot prompting, incorporating examples both with and without API calls. Regarding the evaluation prompts, we maintain consistency with the official implementations by using zero-shot prompts throughout the assessment process.

Q12: Regarding the details of the internal test set.

A12: Thank you for raising this important point. Our internal test set comprises 1,000 target tasks, which were constructed from the Magpie dataset. To ensure robust evaluation, for the test set, we expanded each task's initial five instructions to 100 through the Self-Instruct method, where three instructions serve as observed samples while the remaining 97 are maintained as held-out examples. We have provided a detailed description of the test set construction method in Section 4.2 (lines 254-262), with comprehensive statistical analyses in Table 3 (Appendix B).

Thank you for your thoughtful review and valuable feedback. We appreciate your recognition of our work’s innovation and significance. Below, we address your concerns in detail.

Q1: Regarding the comparison between ADS and baseline methods.

A1: Since more than one reviewer raised this question, we have responded to it in the global rebuttal section (Q1) to save space.

Q2: Regarding the influence of the Magpie dataset.

A2: Thank you for your valuable suggestion. Firstly, we opted for the Magpie dataset due to its comprehensive coverage of diverse alignment tasks, encompassing various domains, difficulty levels, and intents. This aligns with the objective of our ADS framework, which aims to identify suitable data for training policy models for each specific task. Furthermore, to ensure a fair comparison, we fine-tuned our base LLMs using the Magpie dataset and conducted evaluations on public benchmarks.

Our empirical results indicate that fine-tuning the LLM with Magpie data significantly decreased performance. The reason is that the base LLMs (Qwen-2-7B-instruct and Gemma-2-9-Instruct) have already undergone extensive fine-tuning with high-quality training data. Compared to that, the data quality of Magpie may be relatively low. These findings confirm that the improvements observed in our experiments are not attributed to the Magpie dataset, indicating the effectiveness of the proposed ADS method. We will incorporate these findings into the revised manuscript.

Q3: Regarding the influence of generated data length on ADS.

A3: We appreciate your insightful feedback and would like to offer some clarification. It is important to note that when employing in-context learning for policy updating, the scale of generated data is inherently constrained by the model's context window capacity. However, state-of-the-art LLMs typically operate with context windows of approximately 32,000 tokens. Our empirical observations indicate that this scale of training data is sufficient for the effective optimization of a given target task.

Q4: Regarding the analysis of input task-specific dataset size to the optimizer model.

A4: We appreciate your insightful question. In our implementation, we utilize the task-specific examples as a representative for the target task. Since successful task completion requires various fundamental capabilities and skills, we speculate that more observed examples could represent a broader spectrum of task scenarios, therefore achieving better performance. Currently, we are conducting experiments to scale the example number, and we will update the results once the experiment is concluded.

Q5: Regarding the extension from ICL to fine-tuning.

A5: We appreciate the reviewer's insightful comment regarding the implementation of ICL for policy model updating. We would like to provide further clarification, the policy model requires frequent updates with a complexity of O(trajectory sampling number * target task number), resulting in approximately 40,000 updates per experiment. Given this computational intensity, implementing traditional fine-tuning approaches would be prohibitively expensive. Consequently, we adopted in-context learning as our primary methodology across all experimental conditions to maintain computational efficiency while preserving performance. Furthermore, extensive research has demonstrated that ICL can achieve comparable or even better effectiveness than traditional parameter fine-tuning [1][2][3][4][5]. We will include these analyses in the updated version.

References:

[1] Exploring the relationship between in-context learning and instruction tuning. arXiv preprint, 2023.

[2] Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation, ACL 2023 Findings, 2023.

[3] Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers. ACL 2023 Findings, 2023.

[4] In-Context Learning with Long-Context Models: An In-Depth Exploration. arXiv preprint, 2024.

[5] Many-Shot In-Context Learning. NeurIPS, 2024.