Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning

Thanks for the constructive comments.

Q1: Different numbers of agents or agent pairs on the results.

A1: We have conducted experiments to explore the impact of varying the number of agent pairs on the results. As shown in Table 1, we observed that as the number of agent pairs decreases, the model's performance exhibits a corresponding decline. This finding highlights the importance of the number of agents in achieving optimal results. We will include these experimental results and a discussion of their implications in the revised manuscript.

Table 1. Performance using different agent pair

Q2: The authors should also specify the exact number of agents used in the experiments in the "Implementation Details" section.

A2: Thank you for your insightful suggestion. In our experiments, we utilized a total of 10 agent-pairs. We will include this specific information in the "Implementation Details" section of the revised manuscript to ensure clarity and completeness.

Q3: Could you provide some specific cases? For example, demonstrate how the outputs from different agent pairs differ.

A3: we have included a specific case in the table below to demonstrate how the outputs from different agent pairs differ briefly. Due to the character limitations of this rebuttal, more details are provided in the supplementary PDF attached to the global rebuttal. We kindly ask you to review the detailed examples there.

• Instruction: Can you suggest a highly specialized conference dedicated to exploring advanced machine learning algorithms and their practical applications in the vast field of artificial intelligence? We are looking for keynote speakers who are prominent figures in the field of AI research, and the event should provide attendees with a chance to participate in immersive, hands-on workshops, as well as poster sessions highlighting groundbreaking research in this field. Additionally, the conference should offer insights into the latest trends and innovations in the world of machine learning.
• Agent-Pair A: I recommend checking out the Conference on Neural Information Processing Systems (NeurIPS). It is one of the most prestigious conferences in the field of machine learning and artificial intelligence. NeurIPS features keynote speakers from top universities and research institutions, as well as hands-on workshops and poster sessions showcasing cutting-edge research in the field. The conference also offers a chance to learn about the latest trends and innovations in machine learning.
• Agent-Pair B: The conference you are looking for is the International Conference on Machine Learning (ICML). It is a highly specialized conference that focuses on exploring advanced machine learning algorithms and their practical applications in the field of artificial intelligence. The conference is organized by the International Machine Learning Society (IMLS) .......
• Agent-Pair C: I would highly recommend the NeurIPS (Conference on Neural Information Processing Systems) conference for your needs. NeurIPS is one of the largest and most prestigious annual conferences in machine learning and artificial intelligence. It attracts leading researchers, engineers, and students in the field, providing an ideal setting for sharing new research, ideas, and innovations..........
• Agent-Pair D: Certainly! I recommend the "Machine Learning and Artificial Intelligence Conference" which will be held in San Francisco, California. The conference will feature keynote speakers such as Andrew Ng, Yoshua Bengio, and Geoffrey Hinton, who are all renowned experts in the field of AI research. Attendees will have the opportunity to participate in immersive, hands-on workshops, as well as poster sessions highlighting groundbreaking research in machine learning. The conference will also offer insights into the latest trends and innovations in the world of machine learning.
• Agent-Pair E: One highly specialized conference that meets your criteria is the "International Conference on Machine Learning" (ICML). ICML is one of the premier events in the field of machine learning and attracts top researchers, practitioners, and industry leaders. The conference features keynote speeches by renowned experts, in-depth workshops, and numerous poster sessions showcasing cutting-edge research.\n\nHere are some key details about ICML:........Another conference that you might consider is the "NeurIPS" (Conference on Neural Information Processing Systems), which is another major event in the AI and machine learning community. NeurIPS has a similar format and focus on advanced research and practical applications.
• Agent-Pair F: .........

Q4: What impact would removing some or adding additional agent pairs have on the results?

A4: Thank you very much for your insightful suggestion. We have conducted additional experiments to investigate the impact of varying the number of agent pairs on our model's performance. As illustrated in Table 1, we observed that reducing the number of agent pairs tends to result in a gradual decline in the model's effectiveness. This trend indicates that the performance of our model is positively correlated with the number of agent pairs. We hope these findings address your concerns satisfactorily.

Dear Reviewer 4rks,

We sincerely appreciate the time you’ve taken to provide valuable feedback on our paper. In our rebuttal, we have thoroughly addressed all of your initial concerns and included the requested experimental results. If you have any further questions or concerns, we would be happy to discuss them with you. Additionally, we welcome any new suggestions or comments you may have!

Thanks for the constructive comments.

Q1：What is the overhead of this proposed method, like wall-clock time?

A1：Thank you for your insightful question regarding the overhead of the proposed method.

The computational overhead of our proposed method primarily depends on the inference computational load of the various Large Language Models (LLMs) used. To provide a clearer understanding, let us consider specific LLMs use in this paper:

• LLM Qwen-14B: During inference with a sequence length of 256 tokens, the computational load is approximately 4×10^12 floating point operations (FLOPs).
• LLM Phi-2-2.7B: For the same sequence length, the inference computational load is around 7×10^11 FLOPs.
• LLM ChatGPT: Given that ChatGPT is a proprietary model, we lack precise data on its computational requirements.

Nonetheless, for estimation purposes, we can approximate the overall computational cost. Assuming an iterative process involving multiple LLMs (e.g., 10 LLMs) and a large dataset (e.g., 70,000 samples), the total computation without our methods can be roughly estimated as:

4×10^12 FLOPs (Qwen-14B) × 10 LLMs × 70,000 samples = 2.8×10^18 FLOPs

While, when the Agent-Pairs Sampling and Instruction Memory Bank were employed, 5 of 10 LLMs are employed to generate data , therefore, total computation can be significant reduced and roughly estimated as:

4×10^12 FLOPs (Qwen-14B) × 5 LLMs × 70,000 samples = 1.4×10^18 FLOPs

This estimation highlights the significant computational requirements, largely dominated by the inference processes of the LLMs. Other components of our method contribute negligible computational overhead compared to the inference load of the LLMs.

Q2：stability: can this method be scalable to large scale dataset optimization, like web-crawl data?

A2：We appreciate your inquiry about the scalability of our method. Our approach focuses on enhancing the quality and diversity of instruction datasets, and it is inherently scalable to larger datasets. The methodology we proposed is not constrained by dataset size, thus it can be effectively applied to extensive datasets.

Specifically, there are existing studies that leverage large models to synthesize and process pre-training data [r1]. Our method can be integrated with these existing approaches to optimize and handle large-scale pre-training datasets such as web-crawl data. This integration is a promising area for our future research, and we are committed to exploring this direction further.

References:

[r1] : Nemotron-4 340B Technical Report[J]. arXiv preprint arXiv:2406.11704, 2024.

Dear Reviewer b1DV,

We sincerely appreciate the time you took to provide valuable comments on our paper. In our rebuttal, we have addressed all of your initial concerns and included the requested experimental results. If you still have any question or concern, we would be glad to discuss them with you. Additionally, we welcome any new suggestions or comments you may have!

Regards

Thanks for your constructive comments.

Q1：What is the size and composition of the datasets used in your experiments?

A1：The datasets used in our experiments consist of 70,000 samples based alpaca evol instruct datasets [r1]. Each sample is paired with an instruction and a corresponding response. Here is an example:

• Instruction: Can you suggest a highly specialized conference dedicated to exploring advanced machine learning algorithms and their practical applications in the vast field of artificial intelligence? We are looking for keynote speakers who are prominent figures in the field of AI research, and the event should provide attendees with a chance to participate in immersive, hands-on workshops, as well as poster sessions highlighting groundbreaking research in this field. Additionally, the conference should offer insights into the latest trends and innovations in the world of machine learning.

• Response: I recommend checking out the Conference on Neural Information Processing Systems (NeurIPS). It is one of the most prestigious conferences in the field of machine learning and artificial intelligence. NeurIPS features keynote speakers from top universities and research institutions, as well as hands-on workshops and poster sessions showcasing cutting-edge research in the field. The conference also offers a chance to learn about the latest trends and innovations in machine learning.

Q2: Have you compared your framework's performance using standard benchmarks like MMLU or those from Hugging Face?

A2: We evaluated our framework on the standard benchmarks from the Open LLM Leaderboard (including MMLU). As shown in Table 1, although the training datasets we used in our paper focus more on conversational performance, our framework still achieved improvements across multiple standard test sets.

Table 1. Performance on Open LLM Leaderboard

Q3: What would be the impact of using only Mistral-ChatGPT for 70,000 iterations on the diversity and quality of the generated data?

A3: Thank you for your insightful question regarding the impact of using Mistral-ChatGPT for 70,000 iterations on the diversity and quality of the generated data. We have conducted a detailed analysis to address your concerns.

Firstly, we generated 70,000 data samples exclusively using the agent-pair Mistral-ChatGPT. In terms of diversity, we observed a decline in the variation of responses for the same instructions. Regarding quality, some of the data generated was of lower quality compared to using the all of agent-pairs, but it was still superior to the baseline data.

To further assess the implications, we trained the model using this generated dataset. The performance metrics are presented in the Table 2. The results demonstrate that relying on Mistral-ChatGPT for data generation negatively impacts the model's overall capability compared to using the full set of agent-pairs, but it is still superior to the baseline. This confirms the effectiveness of our approach.

Table 2. Performance of Mistral-ChatGPT agent pair

References:

[r1] WizardLM: Empowering large pre-trained language models to follow complex instructions[C]//The Twelfth International Conference on Learning Representations. 2024.

Dear Reviewer kHqZ,

Thank you for your thoughtful feedback and valuable comments. We appreciate the opportunity to further clarify our methodology.

Regarding the metric used to calculate diversity, we employed cosine similarity as the primary metric. Specifically, we calculated the cosine similarity between the sentence embeddings of the datasets. The diversity metric is inversely related to the cosine similarity; hence, lower cosine similarity indicates higher diversity in the generated data. In our analysis, we found that the Mistral-ChatGPT dataset exhibited a cosine similarity of approximately 20.8%, which indeed suggests a higher degree of similarity (and thus lower diversity) when compared to the dataset generated using all agent pairs, which had a cosine similarity of 6.9%.

We hope this clarification adequately addresses your concerns, and we are grateful for your guidance in improving the clarity and rigor of our work.

Thanks for your constructive comments.

Q1: Small Evaluation Datasets

A1: We followed several seminal works[r1,r2], and used widely accepted datasets such as Mt-bench, Vicuna-bench, and the WizardLM testset for our evaluations. These datasets are commonly utilized to assess the effectiveness of LLM's instruction tuning capabilities. To further substantiate the robustness and generalizability of our framework, we also conducted evaluations using datasets from the standard benchmark Open LLM Leaderboard. As shown in Table 1, despite the primary focus of our datasets on instruction tuning capabilities, our method demonstrated improvements across multiple test sets. This indicates that the Star-Agents framework not only excels in instruction tuning tasks but also performs well on a broader range of datasets.

Table 1. Performance on Open LLM Leaderboard

Q2: Multiple Teacher Models

A2: Compared to a single LLM, using multiple LLMs may increase the time complexity. However, we employed a model parallelism approach during the data generation process, which does not significantly increase the latency of our method while effectively improving the results. Additionally, we have proposed a selection strategy to reduce complexity. We introduced a dynamic selection strategy for LLMs, where during each data generation process, we sample based on the past performance of the LLMs. The sampled LLMs are responsible for generating diverse data. By limiting the number of activated LLMs each time, we can ensure the quality of data generation while significantly reducing the resources required for LLMs inference. In the future, we will continue to research simplified methods to enhance the efficiency of our approach.

Q3: Three-Stage Process

A3: Thank you for your comments. Firstly, Our three-stage method ensures diverse, high-quality data by leveraging multiple LLMs for varied data generation, using dual-model evaluation to identify high-quality, tailored data, and employing an evolution strategy to adjust sampling probabilities for efficient, task-specific data generation. Each stage serves a distinct purpose, and has a clear and justified function, and their integration ensures a streamlined process in practice. We acknowledge the need to simplify the process. Our future work will focus on optimizing data generation, reducing computational costs, and exploring automated tools to enhance usability. These improvements aim to make the Star-Agents framework easier to implement without compromising performance.

Q4: Lack of Human Evaluations

A4: In our paper, we conducted evaluations on three different test sets using the LLM-as-a-judge method based on GPT-4, as outlined in [r1]. According to [r1], the LLM-as-a-judge approach based on GPT-4 achieves an agreement rate of 85% with human evaluations, surpassing the inter-rater agreement rate among humans, which is approximately 80%. Moreover, numerous seminal works, such as [r3], and [r4], have successfully employed GPT-4 in LLM-as-a-judge to replace human evaluations. Additionally, we conducted a manual inspection of the evaluation results and found that the trends observed with GPT-4 evaluations align with those of human evaluations. Therefore, we adopted the LLM-as-a-judge method based on GPT-4 for our evaluations in this paper.

Q5：Standard Benchmarks:

A5：In response to this suggestion, we have conducted additional evaluations of the Star-Agents framework using the datasets available on the standard benchmark Open LLM Leaderboard. As shown in Table 1, our framework demonstrated improvements across various test sets, even though our chosen datasets were focused on dialogue performance. We believe that these additional benchmark results will provide a clearer and more robust comparison, demonstrating the effectiveness and versatility of the Star-Agents framework in various scenarios.

Q6：What will be the results if base models are larger, e.g. llama 13b or llama 70b?

A6：Due to time and computational constraints, we conducted preliminary experiments on the LLaMA2-13B model, as presented in Table 2 . Our method shows a significant improvement compared to the baseline. Additionally, in the MT-bench First-turn evaluation, Llama-2-13B-star_instruct achieved a score of 6.98, which is close to the Llama2-70B-Chat score of 6.99, a much larger model that has undergone SFT and DPO. While, the teacher models used in our experiments were mostly smaller than 13B. For larger models, it is essential to incorporate more powerful teacher models to further enhance their performance effectively. Table2. Performance of Llama2-13B

References:

[r1] Judging llm-as-a-judge with mt-bench and chatbot arena[J]. Advances in Neural Information Processing Systems, 2024, 36.

[r2] WizardLM: Empowering large pre-trained language models to follow complex instructions[C]//The Twelfth International Conference on Learning Representations. 2024.

[r3] Mixtral of experts[J]. arXiv preprint arXiv:2401.04088, 2024.

[r4] Minicpm: Unveiling the potential of small language models with scalable training strategies[J]. arXiv preprint arXiv:2404.06395, 2024.

Dear Reviewer piXU,

We sincerely appreciate the time and effort you invested in providing valuable feedback on our paper. In our rebuttal, we have thoroughly addressed all of your initial concerns and included the requested experimental results. If you have any further questions or concerns, we would be happy to discuss them with you. Additionally, we welcome any new suggestions or comments you might have!

Regards