Thank you wCXj for your thoughtful feedback. We’re encouraged you found our study comprehensive and useful, particularly the practical guidance provided by our distillation scaling law.

There are missing distillation and scaling law references from 2022-2024

We extensively cover distillation (App B.1) and scaling laws (App B.2). If specific key papers or topics are missing, please clarify, and we'll be happy to include them.

How is [formula 8] the Distillation Scaling Law derived?

The distillation scaling law was the simplest extrapolating function we found that:

1. Satisfied 3 properties (L243-250 right)
2. Extrapolated to unseen student-teacher combinations (orange scatter in grey region Fig 5).
3. Was a power law in student size $N_S$ and tokens $D_S$ (L187-189 right).

Is the Distillation Scaling Law universal?

Other expressions could work.

Even standard supervised scaling laws differ (footnote 9, L106-109), as well as which terms belong in the scaling law (Appendix H.1, footnote 7, L2965-2968).

We note this in our limitations, highlighting all scaling laws share this problem unless explicitly derived [1].

Practically what matters is correct limits and reliable extrapolation, both of which our law satisfies.

Are the coefficients universal?

The coefficients (Appendix F.3, Table 5, L2606-2619), are not universal, they’re dataset-dependent.

This limitation applies to all scaling law studies; data manifolds of different intrinsic dimensionality produce different scaling coefficients [2].

Broader conclusions from previous studies [3,4] have successfully generalized beyond their datasets. We expect our main conclusions and recommendations (Sec 1) to similarly generalize.

The paper needs more detailed ablation studies (e.g. distillation temperature, mixing coefficients)

We provide detailed studies in the appendix. The two you mention are:

1. Distillation temperature: App G.3
2. Mixing coefficients: App G.4

Additionally, we study learning rates (G.5), divergence measures (G.6), and truncation strategies (G.1-2). We believe these detailed analyses significantly expand the empirical understanding of distillation.

Can the findings be used in practice?

Yes. Our scaling law facilitates optimal experimental design (Sec 5, App D.2-4), assuming the scaling law is computed for your scenario (Sec 4). We'll clarify these practical application steps further in our revision.

Can the findings be used to select the teacher model for LLaMA?

Yes.

Given a computed scaling law (Sec 4) and candidate LLaMA teachers $\{\text{LLaMA}_i\}_{i=1}^n$:

1. Compute each teacher’s cross-entropy $L_T^{(i)}$ on target data
2. Given target student size ($N_S$) and token or compute budget, select the teacher yielding lowest student cross-entropy ($L_S$).

We'll clarify this selection method in the manuscript.

Is the statement “When teacher training is included in compute, the best student cross-entropy is always higher than in the supervised setting" on L385 correct? (if we want to improve the model's accuracy, we can only achieve this by distilling a larger teacher model through training, rather than relying solely on student cross-entropy)

The statement is about cross-entropy, not accuracy, and is correct. See Figure 8. Strategies involving training a teacher (red and green) produce student cross-entropies (y-axis) that are strictly above the cross-entropy of a non-distilled model (black line) for all compute budgets. This is expected, as to find the reverse would imply the existence of an algorithm more efficient than direct maximum likelihood optimization.

Predicting accuracy is an open problem in scaling studies. We observe some correspondence between downstream accuracy and cross-entropy (Appendix E.1), however, our primary claims throughout the work only relate to cross-entropy, as is standard practice in scaling law studies.

We thank you again for your valuable feedback which will help improve our work, and hope our response clarifies any misunderstandings regarding some points of the work that will be clarified in an updated version.

[1] 4+3 Phases of Compute-Optimal Neural Scaling Laws https://arxiv.org/abs/2405.15074

[2] A Neural Scaling Law from the Dimension of the Data Manifold https://arxiv.org/abs/2004.10802

[3] Scaling Laws for Neural Language Models https://arxiv.org/abs/2001.08361

[4] Training Compute-Optimal Large Language Models https://arxiv.org/abs/2203.15556

Thank you, 1KoJ for your valuable feedback. We’re happy you found our analysis useful, the compute-optimal recipes useful for practitioners, and appreciated our work in the context of the scaling literature.

Architectural diversity, a significant variable, was not investigated, limiting generalizability

Compared to scale, architectural diversity is not significant.

Although architectural modifications do influence performance, the effect of scale (model size, or data) is more significant. E.g. in [1], Fig 5, varying:

• depth/width by 40 alters cross-entropy by <3%
• number of heads by 100 alters cross-entropy by ~1%

In contrast, varying model size can alter cross-entropy by 10-50%.

How does the Distillation Scaling Law change when distilling across model families?

Architectural discrepancies should have very little impact distillation efficacy.

Student cross-entropy is influenced by teacher size only through teacher cross-entropy (L104-107). What matters is how well the teacher understands the data distribution, not how the teacher is parameterized. Combining these:

1. Properties of the teachers are summarized in its cross-entropy
2. Variations of the student impact its performance significantly less than the scale of the student (see answer to “architectural diversity”)

Evaluations are only on C4, limiting generalizability

The scientific conclusions (L99-81 right) should generalize, even though the coefficients are tied to C4.

We train on C4 (L193) and discuss limitations (L1107-1115).

Recent work [2] and prior studies [1,4] trained on similar datasets with generalizable conclusions.

Coefficients are tied to experiments, a limitation shared by all scaling studies; data manifolds of different intrinsic dimensionality produce different scaling coefficients [3].

Beyond C4, we evaluate on many downstream tasks (App E.1).

Techniques addressing the capacity gap problem could have been investigated

Early-stopping mitigation [5] is predicted by our Distillation Scaling Law, interchangeable with teacher size. We thoroughly analyze the capacity gap through scaling laws and kernel methods (App B.3, C.1-2) and agree exploring additional strategies is valuable.

Many studies have focused on improving distillation techniques to better calibrate student model

Appendix E.8 is dedicated to distillation calibration. Teachers and students are well-calibrated in the logit distillation setting, which follows from proper scoring rules.

The study is limited by its focus on logit-level distillation

Logit-level distillation is popular for training language models, e.g. Gemini, Gemma [6], Apple Foundation Models [7].

The other main technique is SeqKD [9] e.g. DeepSeek-R1 [10], and would be a good choice for extending our work (L1117-1126). We agree the other distillation techniques you mention are interesting.

A challenge in distillation is the many techniques, none of which are sufficiently well-understood for reliable language model training. Our work shows how one of the most popular methods behaves in scenarios of interest, and represents a step in bringing distillation practice at scale towards a science.

What happens if the teacher is trained on a different dataset to the student?

E.g. what happens when using LLaMA (trained on $p_{\text{LLaMA}}(x)$) for distilling a student into $p_{\text{new}}(x)$?

Two possibilities:

1. Properties of the teachers are summarized in its cross-entropy (see architectural diversity above)
2. It depends on how different $p_{\text{LLaMA}}(x)$ and $p_{\text{new}}(x)$ are.

Sketching 2. Assume LLaMA is well-trained its next-token distribution, $p_T(y∣x)$ reflects $p_{\text{LLaMA}}(y|x)$ not the new one. A student trained on these outputs will also approximate $p(y|x)_{\text{LLaMA}}$. However, in the case of disjoint support, the teacher is evaluated out-of-distribution, failing to provide a meaningful learning signal. 1. applies if the two distributions are sufficiently close; quantifying this closeness would be valuable.

Our setup intentionally uses the same data for distribution teacher and student, letting us isolate algorithmic effects. Exploring settings where the teacher is trained on unseen data would be valuable, though more complex, as it becomes a question of both data and algorithm. We also note that for LLM training in practice, it is quite common for teacher and to use the same data distribution [7, 8], or have the student distilled on a high quality filtered subset.

Thank you again for your insightful comments — they will contribute significantly an improved version of the paper.

[1] https://arxiv.org/abs/2001.08361

[2] https://arxiv.org/abs/2305.16264

[3] https://arxiv.org/abs/2004.10802

[4] https://arxiv.org/abs/2203.15556

[5] https://arxiv.org/abs/1910.01348

[6] https://arxiv.org/abs/2408.00118

[7] https://arxiv.org/abs/2407.21075

[9] https://arxiv.org/abs/1606.07947

[10] https://arxiv.org/abs/2501.12948

Dear hWGR. Thank you for your taking the time to review our paper and for your detailed feedback. We are encouraged that you found our extensive study useful for showcasing different aspects of distillation, and that you find the overall guidance provided by the Distillation Scaling Law to have utility for practical training scenarios. Finally, we are happy to see your overall enthusiasm for the work, and understanding of its place as a step towards more reliable distillation training, and extending the scaling laws field.

Figure 8 doesn't support the claim “Distillation outperforms supervised learning only if a teacher already exists or can be reused across multiple distillations [...] as we increase FLOPS there is no longer a meaningful difference”

Your understanding is correct, and is also the claim we make in the paper:

• (L020-022) If [...] a teacher already exists, distillation outperforms supervised pretraining until a compute level which grows predictably with student size.
• (L078-080, right) [...] distillation is only more efficient than supervised learning only if both of the following are true: i) the total compute or tokens used for the student is not larger than student size-dependent threshold given by our scaling law, and ii) a teacher already exists [...]
• (L427, right) distillation is only more efficient than supervised learning if: i) the total compute or tokens used for distillation is not larger than a student size-dependent threshold, and ii) a teacher already exists [...]

i.e. the right hand side of Figure 8 you point out is precisely the second part of our condition “compute or tokens is not too great, when supervised learning and distillation produce the same answer”. We provide further analysis in Appendix E.6 on the vanishing of the difference between methods.

The study focuses on logit-level distillation

This is true, we only make statements about logit-level distillation.

This is a popular distillation technique for training language models, as it is used by Gemini and Gemma [1], Apple Foundation Models [2], and Minitron [3].

The other primary distillation technique for language model training is SeqKD [4] employed by e.g. DeepSeek-R1 [5], and would be a good choice for an extension of our work, (discussed on L1117-1126).

While we do agree that other distillation techniques are also important to explore, in order to keep the scope of this paper (and compute) constrained we choose to focus on a commonly used form of distillation. We are happy we were able to make progress on one of the most popular techniques, but agree there is a lot more that can be done to bring remaining techniques to the same level of actionable understanding.

[1] Gemma 2: Improving Open Language Models at a Practical Size https://arxiv.org/abs/2408.00118

[2] Apple Intelligence Foundation Language Models https://arxiv.org/abs/2407.21075

[3] LLM Pruning and Distillation in Practice: The Minitron Approach https://arxiv.org/abs/2408.11796

[4] Sequence-Level Knowledge Distillation https://arxiv.org/abs/1606.07947

[5] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning https://arxiv.org/abs/2501.12948

What happens in the case where a more capable model is used for synthetic data generation (instead of logit distillation)

This is the case of SeqKD (see also answer above on logic-level distillation). In this case, the student would be trained on the modes of the distributions represented by the teacher, which should correspond to the modes of language. There is a qualitative and quantitative difference between SeqKD and token level logit distillation (see [4]). Studying the scaling properties of this would be a great next step.

[4] Sequence-Level Knowledge Distillation https://arxiv.org/abs/1606.07947

Are there known examples where teacher distillation is used in pretraining?

Yes (see also answer to logic-level query, as well as references at L073-080).

Gemini and Gemma [1], Apple Foundation Models [2], and Minitron [3] use token-level logit distillation, the subject of our investigation. DeepSeek-R1 [5], and the LLaMA family [6] use language model synthetic generation, i.e. a form of SeqKD [4].

[1] Gemma 2: Improving Open Language Models at a Practical Size https://arxiv.org/abs/2408.00118

[2] Apple Intelligence Foundation Language Models https://arxiv.org/abs/2407.21075

[3] LLM Pruning and Distillation in Practice: The Minitron Approach https://arxiv.org/abs/2408.11796

[4] Sequence-Level Knowledge Distillation https://arxiv.org/abs/1606.07947

[5] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning https://arxiv.org/abs/2501.12948

[6] The Llama 3 Herd of Models https://arxiv.org/abs/2407.21783

Again, thank you for reading our work and provide valuable feedback which will be incorporated in an improved version.