Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs

We appreciate the reviewer’s careful reading of our paper and thoughtful comments!

---

Q1. Generalizability of the presented insights on new models.

A1. Thank you for highlighting this matter. In response, we have reproduced our experiments with the LLaMA 3B model, particularly focusing on phased v.s. stacked training and batch size impacts (TULU vs. LAB). Our findings align with those in our submission: larger batch sizes yield better benchmark performance (e.g., MMLU, MTBench, and Leaderboard), and stacked training is more sample-efficient and performs comparably to, or even better than, phased training.

We also note that the Granite model shares the same architecture as the LLaMA model. Hence we believe that the findings in this paper can generalize across the LLaMA model family. In the paper, however, we focus on the Granite and Mistral model families due to their Apache-2.0 licensing, which is significantly more permissive than the LLaMA license. Our primary objective is to encourage collaborative LLM advancements within the open-source AI community by providing a detailed guide for practitioners working on fine-tuning smaller LLMs, which is why we selected the Granite models over LLaMA. We did not conduct experiments on the Phi model as it only releases an instruction-tuned version, whereas our study centers on instruction tuning for model customization starting from base models.

---

Q2. Domain-specific datasets.

A.2. Indeed, this is a great suggestion! To evaluate the generalizability of our findings to domain-specific datasets, we conducted experiments using a Math, Reasoning, and Code (MRC) dataset. This dataset specializes in mathematical problem-solving, logical reasoning, and programming tasks.

We fine-tuned the LLaMA 3B model using both stacked and sequential phased training strategies with LAB hyperparameters, and we also compared the impact of batch size and the use of a constant learning rate (analyzed through TULU vs. LAB). For phased training, we partitioned the dataset into two phases based on response length, using length as a proxy for difficulty. We evaluated the models on several benchmarks, including GSM8K, ARC, MMLU, MTBench, and the Open LLM Leaderboard v2 benchmarks (MMLU-Pro, GPQA, MuSR, MATH, IFEval, and BBH). Our observation is consistent with our initial submission. For example, stacked training slightly outperforms phased training and is more sample-efficient across all benchmarks, and larger batch sizes (LAB) yield better benchmark performance (e.g., MMLU, MTBench, and Leaderboard).

Finally, we remark that adapting a general-purpose LLM base model to be a domain specific expert involves multiple factors (e.g., domain-specific data, converting such data into instruction tuning datasets, base model selection, and integrating the data into the model). This paper focuses exclusively on the final step, offering a detailed guide for practitioners to determine training configurations.

---

Q3. Main findings are not surprising.

A.3. Yes, we agree—many insights, such as the performance improvement from larger batch sizes, are expected. That said, we note that some findings might challenge existing literature. For instance, hyperparameter recommendations from TULU are suboptimal in our experiments, and phased training as suggested by Orca does not benefit model performance compared to stacked training.

Our objective is to share our empirical observations from fine-tuning LLMs. We support these insights through extensive experimentation, documenting effective hyperparameter settings and highlighting failure modes. We hope these results will offer valuable guidance to practitioners working with limited computational resources for hyperparameter tuning.

---

Q4. L37 needs citation for evidence.

A.4. Thanks for your suggestions! We added 3 citations to the revised paper.

---

Q5. L38 argues that fine-tune smaller model is good for researchers with limited infrastructure support. However, L82 argues that to train a small model well, one needs a larger batch size and specified resources. Is this a contradiction?

A.5. For researchers with limited computing resources, gradient accumulation offers a practical approach to achieve a large effective batch size. We confirm that gradient accumulation on a single node yields results comparable to those of multi-node distributed training; further details are provided in Appendix A.5.10.

We thank the reviewer for the thoughtful review and the kind comments!

---

Q1. Have a discussion on optimization techniques such as LoRA and the increasingly popular approximate optimization algorithms like GaLore.

A1. Indeed, this is a great suggestion. Our work primarily focused on supervised fine-tuning (SFT) as it aligns with our goal of customizing LLMs using knowledge and skills instruction tuning data. There is ongoing debate about whether parameter-efficient fine-tuning methods, such as LoRA, can help incorporate new knowledge into models [see e.g., Jiang etal., 2024]. Given this issue, we chose to focus exclusively on SFT, which is probably the most widely used way for instruction tuning and can maximize performance of small-sized LLMs. In the revised paper, we highlighted this limitation in Section 4 and clarified our setting (using SFT for instruction tuning) in Introduction.

Reference.

Jiang, Ting, et al. "MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning." arXiv preprint arXiv:2405.12130 (2024).

---

Q2. Evaluations on harder datasets.

A2. In response, we conducted additional evaluations on more benchmark datasets during the rebuttal period. Specifically, we evaluated our models on MMLU-Pro, GPQA, MuSR, MATH, IFEval, and BBH from the Open LLM Leaderboard v2, as well as GSM8K and ARC. These benchmarks cover a variety of tasks, including advanced reasoning, mathematical problem-solving, instruction following, and domain-specific knowledge. This broader evaluation helps to ensure that our findings are robust across different application areas and diverse contexts. For the specific experiment comparing the performance of the LAB setting against the TULU hyperparameter settings on the TULU dataset, we used the evaluation benchmarks from the TULU paper—MMLU, GSM8K, BBH, ToxiGen, and TruthfulQA. By using the same benchmarks as TULU, we can directly assess the impact of our hyperparameter choices relative to their results.

Finally, we maintained the limitations of this work in Section 4 and cautioned readers that our results are intended solely as a reference for practitioners fine-tuning LLMs who are uncertain about how to initialize their experiments. Achieving optimal model performance requires tailoring the approach to specific use cases, evaluation metrics, base models, and other factors.

---

Q3. The term "skill" is used quite loosely. What do you refer to as a skill - a specific domain like math or a reasoning like CoT?

A3. In our paper, skills refer to the specific capabilities that the LLM needs for customization for downstream use cases. Specifically, the dataset we used includes foundational skills (data from mathematics, coding, and reasoning, linguistic ability) and compositional skills (tasks that require a combination of knowledge and foundational skills).

In response to your question, we added the following paragraphs to the revised paper to elaborate on the curation process for the knowledge and skills data.

The datasets were curated using a taxonomy-driven approach to ensure comprehensive coverage of instruction-following, foundational knowledge, and compositional skills. The taxonomy hierarchically organizes tasks into three main branches—knowledge, foundational skills, and compositional skills—each further divided into granular subcategories. For each subcategory, manually written instruction-response pairs served as seed examples. These examples guided synthetic data generation using teacher models (e.g., Mixtral-7x8B) to expand the dataset while maintaining high quality and diversity. For knowledge data, reliable sources such as textbooks and technical manuals provided a grounding for synthetic questions and responses. Foundational skills data were drawn from public datasets covering essential areas like mathematics, coding, and reasoning. Compositional skills were synthesized using a taxonomy-guided approach to combine knowledge and foundational skills for complex tasks, such as writing detailed emails or generating logical arguments.

---

Q4. Phase 10 of complex skill development trains the model on tasks like poetry writing etc. - is this also organized as an instruction - answer task?

A4. Yes, it is organized as an instruction-answer task. An example is:

Instruction: "Write a haiku that captures the beauty of a single autumn leaf." Answer: "Crimson leaf adrift,\nDancing on a cool fall breeze,\nNature's art takes flight."

We thank the reviewer for the thoughtful comments and for appreciating the novelty of the work!

---

Q1. What order of experiments the authors ran and to what extent are the findings dependent on this ordering?

A1. Indeed, this is a great suggestion! In response, we clarified the specific experimental configurations in Section 2.3 (Line 216–236). In short, we used the LAB hyperparameter configuration as the default for all experiments where we varied a single factor (e.g., batch size, learning rate, learning rate schedule, training strategy) while keeping all other settings constant to isolate its effect. We started with a cross-product of learning rates and batch sizes (e.g., low and high values for each) to identify stable configurations. Using the best settings, we explored phased vs. stacked training by conducting a batch size sweep (128, 4K, 8K) and then assessed the effect of using a constant learning rate.

---

Q2. Why these choices of models?

A2. Our experiments focus on the Granite and Mistral model families. We select these models due to their Apache-2.0 license, which is one of the most permissive licenses available for off-the-shelf LLMs. This choice aligns with our goal to foster collaborative LLM development in the open-source AI community by providing a detailed guide for practitioners focused on fine-tuning smaller models. We note that the Granite model shares the same architecture as the LLaMA model. Hence we believe that the findings in this paper can generalize across the LLaMA model family. In the paper, however, we focus on the Granite and Mistral model families since Apache-2.0 licensing is significantly more permissive than the LLaMA license. During rebuttal, we have reproduced our experiments with the LLaMA 3B model, particularly focusing on phased v.s. stacked training and batch size impacts (TULU vs. LAB). Our findings align with those in our submission: larger batch sizes yield better benchmark performance (e.g., MMLU, MTBench, and Leaderboard), and stacked training is more sample-efficient and performs comparably to, or even better than, phased training.

---

Q3. How were the fine-tuning datasets curated? How might results vary depending on whether the data is more general or specific?

A3. The datasets were curated using a taxonomy-driven approach to ensure comprehensive coverage of instruction-following, foundational knowledge, and compositional skills. The taxonomy hierarchically organizes tasks into three main branches—knowledge, foundational skills, and compositional skills—each further divided into granular subcategories. For each subcategory, manually written instruction-response pairs served as seed examples. These examples guided synthetic data generation using teacher models (e.g., Mixtral-7x8B) to expand the dataset while maintaining high quality and diversity. For knowledge data, reliable sources such as textbooks and technical manuals provided a grounding for synthetic questions and responses. Foundational skills data were drawn from public datasets covering essential areas like mathematics, coding, and reasoning. Compositional skills were synthesized using a taxonomy-guided approach to combine knowledge and foundational skills for complex tasks, such as writing detailed emails or generating logical arguments. This structured curation ensures that this dataset is representative of real-world use cases for model customization.

This dataset represents a balance of general and domain-specific data, reflecting scenarios where models must generalize broadly while also adapting to specific tasks. This diversity ensures our findings—on the impact of batch size, learning rates, and training strategies—are applicable to other datasets with similar generality and specialization balance. For highly specific datasets, the observed trends might vary slightly, but the training dynamics we analyze, such as gradient norms and loss behaviors, provide a robust framework to identify optimal settings even in those cases. During the rebuttal period, to validate the generalizability of our findings, we included results on a dataset covering math, reasoning, and coding tasks, focusing on comparisons between phased and stacked training, as well as the impact of batch size and the use of a constant learning rate (analyzed through TULU vs. LAB).

We thank the reviewer for the thoughtful comments and for appreciating the novelty of the work!

---

Q4. Why are MMLU and MTBench the right datasets for evaluation?

A4. We selected MMLU and MTBench because they provide diverse coverage for general knowledge, conversational abilities, task-specific dialogue adaptations, and reasoning across a wide range of topics. These features are essential for model customization and closely align with the central question of our study: how to fine-tune a small-size LLM on large-scale instruction tuning datasets that cover diverse knowledge and skills.

During the rebuttal periods, we conducted additional evaluations on more benchmark datasets. Specifically, we evaluated our models on MMLU-Pro, GPQA, MuSR, MATH, IFEval, and BBH from the Open LLM Leaderboard v2, as well as GSM8K and ARC. These benchmarks cover a variety of tasks, including advanced reasoning, mathematical problem-solving, instruction following, and domain-specific knowledge. This broader evaluation helps to ensure that our findings are robust across different application areas and diverse contexts. For the specific experiment comparing the performance of the LAB setting against the TULU hyperparameter settings on the TULU dataset, we used the evaluation benchmarks from the TULU paper—MMLU, GSM8K, BBH, ToxiGen, and TruthfulQA. By using the same benchmarks as TULU, we can directly assess the impact of our hyperparameter choices relative to their results.

---

Q5. Why were TULU vs LAB used as primary comparisons for hyperparameter configurations?

A5. We selected TULU and LAB as primary comparisons for hyperparameter configurations due to their prominence and alignment with our study's focus. TULU is widely regarded as a gold standard for fine-tuning LLMs, leveraging high-quality instruction datasets and a two-stage training approach (instruction tuning followed by preference tuning). LAB, on the other hand, uses similar knowledge and skills data but introduces a multi-phase tuning framework aimed at reducing reliance on human annotations. LAB also advocates for phased training, which we found to be suboptimal compared to stacked training. By comparing these configurations, we systematically evaluated widely accepted practices, challenged their limitations, and provided actionable guidelines for improving performance and efficiency in fine-tuning small LLMs.

---

Q6. Showing individual training curves is quite noisy.

A6. We presented individual training curves due to computation constraints. We acknowledge the potential noise in individual training curves and added a discussion about this issue in the limitations section of the revised paper. Similar challenges have been noted in prior work on LLM fine-tuning, such as TULU and LAB, which also study hyperparameter tuning but do not consistently use multiple seeds due to computational constraints. Each phase of our training requires approximately 10 hours on 80 A100 GPUs, making repeated runs for multiple seeds prohibitively expensive. However, the stability observed in training dynamics (e.g., loss curves, gradient norms) across time suggests that the results are representative and unlikely to diverge significantly. This informed our decision to proceed with single-seed runs for the reported experiments.

---

Q7. Present most results in a table.

A7. Thanks for your suggestion! In response, we have replaced most figures with tables in the revised paper, relocating the original figures to the appendix for reference.

---

Q8. What are baseline performance of the various Granite and Mistral models on MMLU and MTBench?

A8. Thank you for the insightful suggestion. We agree that adding baseline scores for the pretrained base Granite and Mistral models on MMLU and MTBench improves clarity. These values are now included in the revised paper to better contextualize the fine-tuning improvements and highlight the significance of the reported metrics.

We thank the reviewer for the thoughtful review and for appreciating the merits of the work!

---

Q1. The study is focused on small models and may not generalize to larger models.

A1. Our focus is on small-sized LLMs (3B–7B parameters) as they are frequently used by individual researchers and smaller organizations as the backbone for model customization. We acknowledge that some findings may not extend to larger models and have discussed this limitation in Section 4. Nonetheless, we believe this paper can be a valuable resource for developers with limited resources, as small-sized LLMs are significantly more accessible in terms of both fine-tuning and inference costs. Additionally, we hope it fosters a collaborative environment within the open-source community, empowering more contributors to participate in LLM research and training, extending LLM development beyond large organizations.

---

Q2. The experiments mainly use Granite and Mistral model families, leaving out other architectures.

A2. Thank you for highlighting this matter. In response, we have reproduced our experiments with the LLaMA 3B model, particularly focusing on phased v.s. stacked training and batch size impacts (TULU vs. LAB). Our findings align with those in our submission: larger batch sizes yield better benchmark performance (e.g., MMLU, MTBench, and Leaderboard), and stacked training is more sample-efficient and performs comparably to, or even better than, phased training.

We also note that the Granite model shares the same architecture as the LLaMA model. Hence we believe that the findings in this paper can generalize across the LLaMA model family. In the paper, however, we focus on the Granite and Mistral model families due to their Apache-2.0 licensing, which is significantly more permissive than the LLaMA license. Our primary objective is to encourage collaborative LLM advancements within the open-source AI community by providing a detailed guide for practitioners working on fine-tuning smaller LLMs, which is why we selected the Granite models over LLaMA.

---

Q3. The evaluation relies primarily on MMLU and MTBench benchmarks, which, while broad, may not represent all potential application areas.

A3. We selected MMLU and MTBench because they provide diverse coverage for general knowledge, conversational abilities, task-specific dialogue adaptations, and reasoning across a wide range of topics. These benchmarks align well with the central question of our study: how to fine-tune a small-size LLM on large-scale instruction tuning datasets that cover diverse knowledge and skills.

In response to your concerns, we conducted additional evaluations on more benchmark datasets during the rebuttal period. Specifically, we evaluated our models on MMLU-Pro, GPQA, MuSR, MATH, IFEval, and BBH from the Open LLM Leaderboard v2, as well as GSM8K and ARC as you recommended. These benchmarks cover a variety of tasks, including advanced reasoning, mathematical problem-solving, instruction following, and domain-specific knowledge. This broader evaluation helps to ensure that our findings are robust across different application areas and diverse contexts. For the specific experiment comparing the performance of the LAB setting against the TULU hyperparameter settings on the TULU dataset, we used the evaluation benchmarks from the TULU paper—MMLU, GSM8K, BBH, ToxiGen, and TruthfulQA. By using the same benchmarks as TULU, we can directly assess the impact of our hyperparameter choices relative to their results. Finally, we have included a discussion on these limitations in Section 4.

---

Q4. There are some details in the paper that need further checking.

A4. Thanks for your careful reading of our paper! We corrected these citations in the revised paper.