Thank you for your review of our paper! We will address your questions/comments below:

Question 1: how does the effectiveness of the framework scale with the data?

Response: Thank you for your positive affirmation of our work! The question you raised, which has also been mentioned in our limitations, is indeed a direction for further exploration. In our main experiment, we used 10K data points. To show the benefits of scaling training data, we randomly subsampled 2K, 5K, and 8K data points and further expanded the data volume to 20K. The results are shown in the table below:

We found that when the data size is 5K, the performance of the student surpassed that of the student trained on 10K Self-Instruct data. As the data size increases, we can observe a continuous improvement in the performance of Alpaca Eval and MT-Bench, but the performance grows slower and slower. We believe this is due to two reasons:

1. As the student is trained, its data preferences will also change, so it is necessary to collect updated data influence to optimize the teacher in order to achieve sustained performance improvement.

2. As the synthesized data size increases, the ratio of similar data (identified by Rouge-L > 0.7) will increase, and the ratio of useful data will decrease. For example, when the size is 5K, the amount of similar data is ~2K; when the size is 20K, the amount of similar data is ~10K; and when the size is 50K, the amount of similar data reaches ~35K. We found this high data duplication rate in all the baselines, and we did not find papers studying this phenomenon. We believe this is a challenging issue and will raise this in the discussions of our paper's next version as a call for action for the community. In the future, we will explore diversifying the seed data to alleviate this phenomenon.

Dear Reviewer D2rt,

We have carefully addressed your feedback in our rebuttals and provided detailed responses to each of your comments, particularly regarding the scale of the experiments. We believe these clarifications will aid in assessing our work more comprehensively.

We would greatly appreciate it if you could review our rebuttals and provide any further feedback, given that the author-reviewer discussion will be closed on Nov. 26 at 11:59 p.m. AoE in no more than two days. We are willing to answer any further questions.

Thank you for your time and consideration. We look forward to your reply.

Best,

The Authors

Thank you for your review of our paper! We will address your questions/comments below:

Weakness: concerns about the additional cost

Response:

(1) 8xH100 is a very expensive overhead

We used the H100 to complete the experiments on the main table, which utilized the local data influence of 10K data to optimize the teacher, solely to achieve the best performance. As we mentioned in the general response, we only need about 1K local data influences to enable the student’s performance to surpass other baselines, which only require 8 A6000s or 4 A100-80GBs and can be finished in 3 hours. This actually provides researchers with diverse options: if resources are limited, generating approximately 1,000 data influences to optimize the teacher can yield significantly superior performance (2.40% for LC-WR, 5.85% for WR, and 0.304 for MT-Bench), all of which outperform the baselines. If resources are sufficient, the teacher can be continuously optimized to further raise the performance ceiling.

(2) whether we could achieve similar gains by investing these resources in using stronger teacher models

We believe that a strong teacher does not necessarily mean that it can generate equally high-quality synthetic data, as mentioned in this paper[1]. Thus, targeted optimization is an important condition for synthesizing high-quality data. We represent another dimension of improvement, one that not only enhances teacher’s capability but also supports student’s learning. Moreover, the construction of complex reward models is also very complicated and resource-intensive.

In fact, the resources we invest in optimizing the teacher represent a one-off cost. In Section 5.4, we fine-tuned four different models, including Qwen, Gemma, and Mistral, using the data synthesized by the same optimized teacher, all of which achieved better performance than the baseline. Therefore, our method allows the teacher to be trained once and fixed, without the need to retrain every time a student is changed.

(3) Discuss potential directions that could reduce the unit cost

We believe the most direct approach is to use a proxy model, as demonstrated in our experiments, where the local data influence obtained from a 1B small model can generalize well to an 8B model (54.13%WR, 53.61%LC-WR and 6.83 for MT-Bench). This can help reduce our computational load to a level comparable to the Self-Instruct method. Another future direction is to utilize classifier models, such as BERT-based models, to further accelerate the process of obtaining data influence[1]. Thank you for your suggestion.

(4) present the additional monetary costs for each method

Thank you for your suggestion! We calculated the FLOPs of each method in the General Response. According to the results, the FLOPs of our method is lower than both the Self-Reward and LLM2LLM baselines when using the proxy model, and it is already very close to Self-Instruct. However, our method demonstrates a significant improvement for the student, with increases of 3.61% for LC-WR, 4.13% for WR, and 0.34 for MT-Bench. Regarding the monetary costs, the A6000 is priced at \$1 per hour, the A100 at \\$1.50 per hour, and the H100 at \$2 per hour. In the Self-Instruct and LLM2LLM methods, we utilize GPT-4o to generate synthetic data. The API cost for GPT-4o is \\$2.50 per 1 million input tokens and \$10.00 per 1 million output tokens. Consequently, the average cost for synthesizing 10,000 data points (considering data waste) amounts to approximately \\$53.20, which is significantly more expensive than using GPUs. So in fact, our method is superior in terms of both computational load and cost.

Question 1: In lines 226-228, how were the 6,792 preference pairs collected from the 10K probing dataset?

Response: Thank you for pointing it out! As described in Section 3.3 (lines 196–199), we create preference pairs that satisfy the following conditions: 1) they share the same seed prompt, and 2) one has a positive influence while the other has a negative influence. To achieve this, we first aggregate the data in the 10K probing dataset by their seed prompts, dividing them into positive and negative groups. Every time we select one data point from the positive group and one from the negative group, if the two data points are generated from a common seed prompt, we combine them into a preference data pair. Since some seed prompts generate data with only positive or negative influences, we end up with 6,792 pairs from the 10K probing dataset.

Question 2: Will the general capabilities of the optimized teacher model deteriorate?

Response: Thanks for raising this invaluable question! We conducted comprehensive testing on the Llama3-8B-Instruct teacher (with Llama3-8B as the student) before and after DPO on MT-Bench, MMLU, GSM8K, GPQA, ARC-C, and HellaSwag. The results are as follows:

The results indicate that the teacher's ability, after specific optimization in data synthesis capability, is basically on par with the original model with minimal fluctuations, demonstrating that optimizing the teacher's data synthesis capability does not adversely affect performance on OOD tasks.

[1]: Yu, Z., Das, S., & Xiong, C. (2024). MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models. arXiv preprint arXiv:2406.06046.

Thank you for your review of our paper! We will address your questions/comments below:

Weakness 1: How to pick the reference dataset and determine its size?

Response: Thanks for raising this question! Regarding the choice of the reference dataset, we have two guiding principles: First and foremost, we select reference tasks that reflect the target capabilities we want the LLM to achieve. Following the practices of some previously accepted excellent works [1][2][3], we chose to use in-domain data that is the same as the seed data, specifically alpaca gpt4[4]. This is a dataset synthesized by GPT4 using prompts in the Alpaca format, aimed at improving the model's instruction-following ability. Second, we ensure that there is no data leakage to prevent potential overfitting. Our OOD experiments show good generalization ability of students, which demonstrates that we are not overfitting the targeted task. Regarding the size of the reference dataset, we selected the best quantity that balances performance and efficiency within the limits of our computational resources, which is 256.

We conducted ablation experiments on different reference datasets and different reference dataset sizes.

Regarding the size of the reference dataset, in addition to the original 256, we also chose 8, 32, and 128 for experimentation.

The experimental results show that changing the reference dataset size has a small impact on the student's performance. Although it performs well on Alpaca Eval when the size is 8, it performs poorly on MT-Bench, which may be due to the randomness of selecting the 8 data points. We also calculated the correlation coefficients between the data influence generated by different sizes and the data influence when the size is 256. All the correlation coefficients are greater than 0.9, demonstrating a very strong correlation among the reference datasets of different sizes. Therefore, the chosen size of 256 can achieve the best overall performance on both Alpaca Eval and MT-Bench metrics.

In addition to the original alpaca gpt4 as the reference dataset, we chose two other datasets: DOLLY[5] and Open Assistant[6]. These two open-ended generation datasets with human-written answers contain various forms of data, while the answers in the alpaca gpt4 dataset are synthesized by the model rather than written by humans. The results are shown below:

Using Dolly and Open Assistant as reference datasets will lower the scores of Alpaca Eval. We believe this decrease comes from that Dolly and Open Assistant are responses provided by humans, while Alpaca GPT4 generates answers using the GPT4 model. The latter has a smaller distribution difference from the model itself, making it easier to learn from. Additionally, we believe that the reason Open Assistant improves the MT-Bench score is that MT-Bench measures the model's ability in multi-turn dialogue, and only Open Assistant among these three reference datasets contains multi-turn dialogue data. Overall, our method demonstrates robustness on reasonable reference tasks.

Weakness 2: More details about preference data pairs

Response: We generate four instructions for each prompt, with each instruction assigned to a different local data influence. From these four instructions, we select one with positive influence and one with negative influence to form a preference data pair. If there are multiple positive/negative data influences, we will pair them together randomly, resulting in more than one preference data point for a single prompt. We will include more details in the next version of the paper.

Weakness 3: Ablation study on the sample size of the probing set and the final performance

Response: Thank you for raising this question. We actually conducted this experiment in the original paper and illustrated the relationship between student performance and teacher training steps in Figures 3(a) and 3(b), where the size of the probing dataset determines the teacher training steps since the teacher is trained with one epoch. To clarify, we used "probing dataset size" as the x-axis in Figures 3(a) and 3(b) instead of the teacher training steps. The discrete data points used to plot the relationship between probing dataset size and student performance are shown in the table below:

As shown, when the probing dataset size reaches 1K, the student's performance already surpasses the baseline. As mentioned in the general response, if researchers do not have sufficient resources, they can choose to use 1K probing data, which will ensure that the performance of the student model achieves the best results compared to other baselines while maintaining efficiency. We generated 6,792 probing data for the main experiments in our paper and achieved overall optimal performance within the resources available to us.

Weakness 4: some other additional costs as the proposed method involves DPO on the expert model, while most of the baselines do not.

Response: We want to clarify that the other two baselines—LLM2LLM and Self-Reward—also require additional resources: LLM2LLM relies on larger models via API calls to generate instructions, while Self-Reward introduces a separate 70B expert model dedicated solely to generating instructions for students. In contrast, our method requires only an additional DPO step for the teacher. The FLOPs of DPO can be estimated as 4 times the Policy Model Forward FLOPs plus 2 times the Reward Model Forward FLOPs, while the FLOPs of SFT can be estimated as 3 times the Policy Model Forward FLOPs. Although the FLOPs of DPO are higher than those of SFT on a per-unit data basis, DPO does not incur much additional consumption because it uses a smaller total amount of data. According to the Performance-FLOPs relationship in the General Response, even if we introduce DPO for the teacher, our overall FLOPs are still comparable to Self-Instruction.

This DPO process can be beneficial, as verified in a recent paper[7], which aligns with our experimental findings: a strong model does not necessarily excel at synthesizing high-quality data. Therefore, even with a strong teacher, producing high-quality data may still require investing additional resources to refine the teacher.

[1]: Xia, M., Malladi, S., Gururangan, S., Arora, S., & Chen, D. (2024). Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333.

[2]: Paul, M., Ganguli, S., & Dziugaite, G. (2021). Deep learning on a data diet: Finding important examples early in training. Advances in neural information processing systems, 34, 20596–20607.

[3]: Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, & Tongliang Liu (2023). Moderate Coreset: A Universal Method of Data Selection for Real-world Data-efficient Deep Learning. In The Eleventh International Conference on Learning Representations.

[4]: Peng, B., Li, C., He, P., Galley, M., & Gao, J. (2023). Instruction Tuning with GPT-4. arXiv preprint arXiv:2304.03277.

[5]: Conover, M., Hayes, M., Mathur, A., Xie, J., Wan, J., Shah, S., Ghodsi, A., Wendell, P., Zaharia, M., and Xin, R. Free. Dolly: Introducing the world’s first truly open instruction-tuned LLM, 2023.

[6]: Kopf, A., et al. "Openassistant conversations-democratizing large language model alignment," in Advances in Neural Information Processing Systems, vol. 36, 2024.

[7]: Xu, Z., Jiang, F., Niu, L., Lin, B. Y., & Poovendran, R. (2024). Stronger Models are NOT Stronger Teachers for Instruction Tuning. arXiv preprint arXiv:2411.07133.

Dear Reviewer 5Czb,

We have carefully addressed your feedback in our rebuttals and provided detailed responses to each of your comments, particularly regarding the ablation studies on the reference dataset, the size of the probing dataset, and a thorough analysis of our cost-performance relationship. We believe these clarifications will aid in assessing our work more comprehensively.

We would greatly appreciate it if you could review our rebuttals and provide any further feedback, given that the author-reviewer discussion will be closed on Nov. 26 at 11:59 p.m. AoE in no more than two days. We are willing to answer any further questions.

Thank you for your time and consideration. We look forward to your reply.

Best,

The Authors

Thank you for your review of our paper! We will address your questions/comments below:

Weakness 1: Include experiments with multiple runs and report the mean and the standard deviation.

Response: Thank you for pointing it out! We conducted three repeated experiments on the Llama3-8B group using different seeds. We report the mean and standard deviation of the evaluation results.

As shown in the table, our method demonstrates strong stability on both the Alpaca Eval and OOD benchmarks. Even in the case of $-3\sigma$, we are still better than the baseline. While the fluctuations in MT-Bench are slightly larger compared to the other benchmarks, we attribute this to the inherent instability of using LLM-as-a-judge in MT-Bench. We will repeat the other experiments in the following time and report the mean and standard deviation of all methods in the final version.

Weakness 2: The performance gain on OOD benchmarks appears minimal

Response: Thank you for raising this question! We want to clarify that our primary goal is to enhance the model’s instruction-following ability (evaluated using Alpaca Eval for in-domain and MT-Bench for OOD) while also preserving its generalization ability across other tasks (e.g., Math, QA). Synthetic data often exacerbate OOD issues, as evidenced by performance declines on OOD benchmarks for our baselines (e.g., a 3.73 point drop for Self-Reward-iter1 and a 2.8 point drop for LLM2LLM-iter1 on GPQA). However, our method effectively addresses this challenge by maintaining strong overall performance on OOD benchmarks, with notable improvements of 0.673 points on MT-Bench, 1.05 points on MMLU, 1.26 points on GSM8K, 2.99 points on ARC-C, and 1.05 points on HellaSwag (you can find more details in Table 1 in our paper). This strength was also highlighted by Reviewer 5Czb, who noted: “the method proposed is sound and can intuitively address the OOD issue with the synthetic data to enhance the generalizability of the student model.” We will emphasize this further in the next version of the paper.

Weakness 3: This reliance on existing techniques may reduce the perceived originality of the paper.

Response: We want to clarify that our focus is not on emphasizing local data influence as an innovation point. Influence functions are a well-established method in statistics and have a wide range of applications[1][2][3][4], except for guiding the generation of synthetic data. Our contribution lies in being the first to leverage data influence to guide the synthetic data generation process and demonstrate its superiority, as highlighted in lines 85-86 in the paper: “We incorporate influence functions to accurately capture the student’s data preferences and effectively guide the teacher’s optimization directions.” Also, in the ablation experiment titled “Effectiveness of Local Data Influence” in Section 5.3, we compared the performance of IF and an LLM-judger in evaluating data quality. The results clearly show that IF outperforms the LLM-as-a-judge, with a 1.50% improvement on LC-WR, a 3.66% improvement on WR, and a 0.172 point gain on MT-Bench. This shows the effect of introducing IF. We believe that incorporating IF in the generation of synthetic data has huge potential also in pre-training and post-training, given the importance and wide usage of synthetic data.

[1]: Sanford Weisberg and R Dennis Cook. Residuals and influence in regression. 1982.

[2]: Koh, P., & Liang, P. (2017). Understanding black-box predictions via influence functions. In International conference on machine learning (pp. 1885–1894).

[3]: Park, S., Georgiev, K., Ilyas, A., Leclerc, G., & Madry, A. (2023). Trak: Attributing model behavior at scale. arXiv preprint arXiv:2303.14186.

[4]: Grosse, R., Bae, J., Anil, C., Elhage, N., Tamkin, A., Tajdini, A., Steiner, B., Li, D., Durmus, E., Perez, E., & others (2023). Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296.

Dear Reviewer hB25,

We have carefully addressed your feedback in our rebuttals and provided detailed responses to each of your comments, particularly regarding the experiments involving multiple runs and the further explanation of our OOD performance gains. We believe these clarifications will enhance the comprehensive assessment of our work.

We would greatly appreciate it if you could review our rebuttals and provide any further feedback, given that the author-reviewer discussion will be closed on Nov. 26 at 11:59 p.m. AoE in no more than two days. We are willing to answer any further questions.

Thank you for your time and consideration. We look forward to your reply.

Best,

The Authors

Dear Reviewer hB25,

Given that Dec 2nd is the last day for reviewers to post messages, we would greatly appreciate it if you could review our rebuttals and provide any further feedback. We are willing to answer any further questions.

Thank you for your time and consideration.

Best,

The Authors

2 Cost-performance relation of all the methods

We further explain the reviewers' questions about the consumption of our method. We analyzed the Performance-FLOPs curve of four methods, with a particular focus on the changes in Self-Instruct's Alpaca Eval and MT-Bench Score as their FLOPs increase to levels comparable to those of Montessori-Instruct. We scale the FLOPs of Self-Instruct by synthesizing additional data. We also marked the Performance-FLOPs relationship of the two baselines, LLM2LLM and Self-Reward, in Figure 15 (a), Figure 15 (b) and Figure 15 (c) . We have also attached the PDF versions of these three figures in the uploaded files.

According to the figures, it can be seen that Self-Instruct quickly reached the upper bound during the scaling-up process, and even with more FLOPs, no better performance improvement can be achieved. The reason may be that the data generated by Self-Instruct is severely homogenized. In contrast, the upper bound of our method is significantly better and continuously grows when we invest more FLOPs into it.

Then we give a computational result of the FLOPs estimated for four methods, as well as the pretraining and test-time-scaling. The main FLOPs for Montessori-Instruct come from processing probing data. In the main table, we used 10K probing data to utilize the most resources to achieve the best performance, but as the Figure 3(a) and 3(b) in our paper suggests, using ~1K probing data can already achieve better performance than other baselines. To make a fair comparison, we calculate the FLOPs under 1K probing data. We estimate the FLOPs as follows (Llama3-8B-Instruct as the teacher, Llama3-8B as the student):

• Self-Instruct: $1.34\times10^{20}$ FLOPs
• Self-Reward: $2.11\times10^{21}$ FLOPs
• LLM2LLM: $2.3\times10^{20}$ FLOPs
• Montessori-Instruct: $6.43\times10^{20}$ FLOPs
• Pretrain Llama3-8B: $1.87\times10^{24}$ FLOPs
• Inference-Time Scaling : $1.60\times10^{23}$ FLOPs

We can see that Montessori-Instruct's FLOPs are 7 times less than Self-Reward, which is the current SOTA method. Furthermore, if we use the proxy model[1], such as a smaller-sized model (e.g., 1B parameters for assisting an 8B model) to process probing data, Montessori's FLOPs can further reduce to $1.92\times10^{20}$ FLOPs. This makes it comparable to Self-Instruct while still outperforming it. Using a proxy model has promising potential for enhancing both efficiency and performance, which we leave for future work. Regarding the pretraining, since the computational cost during the SFT phase is significantly lower than that during the pretraining phase ( $10^4$ times smaller), even if we increase resource investment in SFT, its overall consumption remains minimal. Recent work has focused on scaling inference time to achieve better performance [2]. However, the inference-time scaling FLOPs are also significantly larger than those of SFT, being approximately $10^3$ times greater, according to [3]. Nevertheless, our teacher training represents a one-time cost. As demonstrated in Section 5.4 of the paper, the optimized teacher can assist multiple students in improving their performance without the need for retraining from scratch.

The detailed derivation is provided in Section E.3 of the new version of the paper in the Appendix.

[1]: Yu, Z., Das, S., & Xiong, C. (2024). MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models. arXiv preprint arXiv:2406.06046.

[2]: Snell, C., Lee, J., Xu, K., & Kumar, A. (2024). Scaling llm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314.

[3]: Sardana, N., Portes, J., Doubov, S., & Frankle, J. (2023). Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. arXiv preprint arXiv:2401.00448.