How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition

Weakness2

We will make every effort to supplement the results of more models before the end of the rebuttal period !

Weakness3

In our paper, all the models, datasets, evaluation benchmarks, and training frameworks used are open-source. We will provide explicit details of the experimental setup in the implementation section, ensuring transparency and reproducibility. Furthermore, we are committed to following our company's open-source procedures to make the code publicly available, advocating for increased accessibility and collaboration within the research community.

Thanks for your suggestions, we have further added more SFT abilities and more evaluation benchmarks to verify our generalizability.

More abilities and datasets：

We are willing to supplement the results of different domain datasets in Appendix section titled "VALIDATION EXPERIMENTS IN MORE SFT ABILITIES" to verify the generalizability of our conclusions. We have selected several representative datasets to expand the evaluation of the capabilities of LLMs in different dimensions, including World knowledge (WebQuestionsSP), Language Understanding (CoNLL 2003), and Translation abilities (IWSLT14). The experimental results are as follows:

Individual domain: For the language understanding (NER) task, the performance (P, R, F1) of the model shows a positive correlation with the scaling curve of the data. These two abilities exhibit a similar performance scaling trend as the mathematical ability discussed in RQ1. On the other hand, for world knowledge (WebQSP) in terms of (F1, Hits@1), it also shows a positive correlation trend. However, when the data ratio decreases from 1/4 to 1/8, there is a significant fluctuation in performance. In the case of translation ability, the performance trends for German-English and English-German translations are not entirely consistent. These findings further support the core conclusion of RQ1 that different data exhibit different scaling curves.

Mixed domains: The findings align with the conclusions of RQ2 in mixed doamins, which states that abilities are improved with low-resource and decreased with high-resource compared to individual source abilities. This consistent observation holds true for world knowledge, language understanding, and translation abilities.

Question 1:

Thanks for your suggestion, we have supplemented all zero-shot performance of Llama7B to 33B in 3 benchmarks in rebuttal revised version in Table1 for reference.

Question 2:

For each dataset, we employed random selection by utilizing a random function with three distinct seeds for sampling. Subsequently, we conducted a comparative analysis of the results obtained from different subsets on the three benchmark tests. The specific details are presented as follows:

It can be observed that DMT maintains its superiority under three different random seed settings. The influence of different subsets on experimental results is not a key factor and does not affect the overall trend. More results can be found in Appendix section titled "RESULTS OF DIFFERENT RANDOM SEEDS"

Thank you for your suggestion. We have added the results of the mathematics benchmark MATH and the code benchmark MBPP in the Appendix "RESULTS ON MORE BENCHMARKS IN MATH AND CODE". Their test data belongs to the same field as GSM8k and Code Alpaca, but the distribution is relatively Out-of-distribution. It can be seen from the above results that our DMT strategy does have a certain degree of generalization. On the premise of ensuring excellent general ability performance, it has good results in the IND and OOD benchmarks of mathematics and code. The following are detailed results

We have made the following findings:

1. In individual domains, the performance of the Llama model in MATH and MBPP is positively correlated with the data volume, similar to the findings in RQ1.
2. When comparing individual and mixed domains, the llama-7B model exhibits the same pattern of conflicting high-resource performance and low-resource performance gains in MATH and MBPP, consistent with the findings in RQ2.
3. Combining the results of general abilities in Table 1, we observe that DMT maintains competitive results in MATH and MBPP while prioritizing general abilities. This further validates the effectiveness of DMT, consistent with the findings in RQ4.

The DMT strategy does have a certain degree of generalization. On the premise of ensuring excellent performance in general capabilities, it has good results in the IND and OOD benchmarks of mathematics and code. More detailed results we have added in Appendix section titled “RESULTS ON MORE BENCHMARKS IN MATH AND CODE”

Question 4:

Thank you for your suggestion. In this paper, we trained the models in the order of code -> math -> general abilities. However, to investigate the impact of training order on different SFT abilities, we have conducted additional experiments with six different training orders. The results and analysis of these experiments are provided below:

Based on our findings, we conclude the following:

1. The SFT ability trained in the final stage tend to retain relatively good performance.

2. If general and code abilities are trained in the first two stages, there is a noticeable performance decrease in code capability, while math capability does not show significant impact. One possible reason is that the task format of code generation and general ability exhibits similar data distributions (as discussed in RQ3 and Discussion1). This can result in a more severe catastrophic forgetting phenomenon during continuous fine-tuning.

Detailed results can be found in Appendix section titled “COMPARISON EXPERIMENT OF DIFFERENT TRAINING SEQUENCES”.

Question 5:

Based on your suggestions, we have also added the comparison results between the start layer and the end layer in the Appendix section titled “VISUALIZATION OF DIFFERENT LAYERS". The visualization result of the start layer are relatively chaotic, while the visualization results of the middle layer and the end layer are clearer. And the results of the middle layer and the last layer are consistent in pointing out that both base model and model with DMT strategy exhibit a certain level of separation in the mathematical data representations, there remains a certain degree of overlap between the representations of code and general samples.

For relevant support, please refer to Figure 18 in Section 6 in this paper [1]. It is also found that the Middle layer has a more stable visualization effect compared to the early layer.

[1] REPRESENTATION ENGINEERING: A TOP-DOWN APPROACH TO AI TRANSPARENCY

Data statistics:

Thank you for your suggestion. In fact, we have indicated the total number of samples for each dataset in the Appendix section titled "SFT Dataset" . We will also provide a comparative table in this section, showcasing the data size for different datasets and their respective proportions.

Equal Data Amount Experiment:

In a realistic SFT phrase for training general LLM, the data amount for different abilities is likely to differ. Therefore, instead of controlling the same amount of data, we select to mix datasets with the same proportion of subsets to better simulate real-world scenarios in all experiments. In addition, We are willing to further supplement the experimental results using different abilities mixed with the equal data amount and compare them with the results using the equla subset proportion.

Equal Data amount Setting: we utilize the data amount of GSM8k RFT as the baseline. We sampled data with proportions of {1/16, 1/64, 1/256}, and mixed samples of the same data amount from Code alpaca and ShareGPT.

Equal Proportion Setting: we sampled data with proportions of {1/16, 1/64, 1/256} according to the subset proportions of each dataset and mixed them, which is aligned with the setup in RQ2.

It can be observed that there is not a significant difference in the results of the three benchmark tests between the two settings. Therefore, these findings do not significantly impact the main experimental conclusions presented in the paper. Detailed results can be found in Appendix section titled “EQUAL DATA AMOUNT VS. EQUAL DATA PROPORTION”.

Thanks for your insightful comments.

W1: T-SNE graphs try to understand the task distribution and we find math reasoning dataset is very different from others and ShareGPT does have overlap with coding tasks which helps us understand the performance change under multi-task learning. We don't give a scaling law equation since we do not use the test losses to evaluate task performance. Chinchilla's law using losses versus model size and data amount cannot be easily transformed here since metric like MT-Bench is not a simple function can be determined by the test losses, it is unknown that the power law still exists using metric like MT-Bench. However, MT-Bench/Alpaca-Eval are useful for quickly understanding human alignment performance and should be researched.

W2: Our core finding is during multi-task learning, multiple abilities exhibits improvement in low-resource and decline in high-resource. Using our method can relief the decline in high-resource. Our main method should be defined as training routine instead of data mixture which trains on specialized tasks first and multi-task trains on human alignment data with a little specialized tasks. This method is emprically useful for LLM practioner to build a human-aligned chatbot with specialized abilities.

W3: Thank you for your suggestion, we will improve our written and update the manuscript later.

Q1: Our experiment setting is focus on aligning a pre-trained model to a chat model which mostly requires training samples from 1000 (less is more alignment) to 100,000 (ShareGPT). So our investigated fine-tuning budget is proper for our object.

Ethics Concern: Thank you for your suggestion, we will change our method name.

Thank you for your suggestion. We will incorporate the relevant literature on scaling laws into the appendix and include the calculation process for FLOPs，Training FLOPs budgets and GPU hours required for different datasets in the Appendix section titled "ESTIMATING FLOPS OF SFT".

Thank you for your suggestion. We have added the following experimental results：

Zeroshot results

In the revised version of Table 1, we have provided the zero-shot results of single domain SFT models on other datasets.

Results on more benchmarks

To validate the generalization of our findings on other benchmarks, we utilized GSM8K and Code Alpaca as the training sets. We further evaluated the results on the individual domain, mixed domain, and different training strategies on other specialized ability benchmark, including MATH and MBPP

We have made the following findings:

1. In individual domains, the performance of the Llama model in MATH and MBPP is positively correlated with the data volume, similar to the findings in RQ1.

2. When comparing individual and mixed domains, the llama-7B model exhibits the same pattern of conflicting high-resource performance and low-resource performance gains in MATH and MBPP, consistent with the findings in RQ2.

3. Combining the results of general abilities in Table 1, we observe that DMT maintains competitive results in MATH and MBPP while prioritizing general abilities. This further validates the effectiveness of DMT, consistent with the findings in RQ4.

More detailed results we have added in Appendix section titled “RESULTS ON MORE BENCHMARKS IN MATH AND CODE”.

Thank you for your suggestions. In fact, as mentioned in the first paragraph of our introduction, our paper primarily focuses on three outstanding abilities (mathematics, coding, and general human-aligned ability) that are of great interest to the LLM community. In addition, we are willing to supplement the results of different domain datasets in Appendix section titled "VALIDATION EXPERIMENTS IN MORE SFT ABILITIES" to verify the generalizability of our conclusions. We have selected several representative datasets to expand the evaluation of the capabilities of LLMs in different dimensions, including World knowledge (WebQuestionsSP [1]), Language Understanding (CoNLL 2003 [2]), and Translation abilities (IWSLT14 [3]). The experimental results are as follows:

Individual domain：For the language understanding (NER) task, the performance (P, R, F1) of the model shows a positive correlation with the scaling curve of the data. These two abilities exhibit a similar performance scaling trend as the mathematical ability discussed in RQ1. On the other hand, for world knowledge (WebQSP), it also shows a positive correlation trend. However, when the data ratio decreases from 1/4 to 1/8, there is a significant fluctuation in performance. In the case of translation, the performance trends for German-English and English-German translations are not entirely consistent. These findings further support the core conclusion of RQ1 that different data exhibit different scaling curves.

Mixed domains: The findings align with the conclusions of RQ2 in mixed domains, which states that abilities are improved with low-resource and decreased with high-resource compared to individual source abilities.

We further evaluated the bias of the models by assessing Truthful QA. We used the sharegpt data as the training set for SFT and explored the experimental setup of scaling the data from 100% to 1/256. We aligned the experimental setup with the evaluation criteria of TruthfulQA [4][5].

The results indicate that the performance of lama-7B on TruthfulQA did not show significant fluctuations. One possible reason for this is that the general ability data (sharegpt) may not enhance the abillity of debias. This will be a direction for future exploration.

We will make appropriate adjustments to the conclusion statements in my article, avoiding controversial points to prevent any misinterpretation of the claims. We will also continue to explore whether the current conclusions hold true in other domains in the future.

【1】WebQuestionsSP：A question answering dataset using Freebase as the knowledge base for qustion answering.

【2】CoNLL 2003：CoNLL-2003 is a named entity recognition dataset released as a part of CoNLL-2003 shared task: language-independent named entity recognition.

【3】IWSLT14: Evaluation Campaign featured three tracks: automatic speech recognition (ASR), spoken language translation (SLT), and machine translation (MT).

【4】Representation Engineering: A Top-Down Approach to AI Transparency

【5】Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Setting

Q1: The trend of k is indeed reliable. The inconsistency you mentioned seems to be referring to the increase of k from 0 to 1/256, which aligns with the conclusion of RQ2. As k increases from 0 to 1/256, a small amount of specialized data is mixed into the general ability data. This leads to a mutual benefit between general abilities and extremely low-resource specialized abilities, consistent with the conclusion of mutual benefit among different ability items under low-resource conditions in RQ2. However, as more specialized data is gradually introduced (k increasing from 1/256 to 1/1), the specialized abilities improve, but there is a gradual emergence of conflicting high-resource performance among different ability items (as observed in RQ2), ultimately resulting in a decline in general abilities.

Q2: We acknowledge that incorporating code can enhance the generalization ability in other domains, as seen in the previous work. However, the key difference lies in the amount of code data used for SFT. In those studies, SFT was performed using code data with token counts exceeding 10 billion, allowing the models to capture transferable structured knowledge. In contrast, the token count of code data in our experiments is even less than 0.01 billion, which makes it challenging for large models to learn transferable knowledge. This significant difference in experimental setup accounts for the contrasting conclusions.

In our experiments, we find that the factors influencing code mixing are more related to the style, format, and distribution of inputs from different domains (RQ3). We also acknowledge that investigating the effects of large-scale data with diverse abilities will be a valuable direction for future research.