Why Knowledge Distillation Works in Generative Models: A Minimal Working Explanation

We would like to extend our sincerest thanks to the reviewer for their very positive and encouraging feedback. We are delighted that they found our paper to be "well-written," our perspective "novel and interesting," and our arguments and experiments "convincing enough." We are grateful for their strong support of our work.

We also appreciate the reviewer's thoughtful comment on the scope of our explanation, and we would like to briefly address it.

On the Scope of the Explanation (Weakness)

We thank the reviewer for their insightful comment: "While the theoretical and experimental evidence presented is compelling, the paper does not offer a fully rigorous explanation for the efficacy of Knowledge Distillation. Instead, it provides valuable insights and plausible explanations."

We agree entirely with this characterization. Our primary goal for this work was precisely to provide what the reviewer aptly describes: "valuable insights and plausible explanations" in the form of a "minimal working explanation". We aimed to establish the first foundational, theoretical explanation for why knowledge distillation is so effective in generative models, supported by rigorous empirical evidence.

We believe that establishing this fundamental understanding is a crucial prerequisite before a "fully rigorous" and all-encompassing theory can be developed. For instance, we agree that this foundational framework can now serve as a crucial base to explore other important phenomena, such as explaining the success of KD in model compression techniques or inspiring the design of novel distillation methods.

We see our work as building the essential groundwork upon which more complex and fully rigorous theories can be built in the future. We will revise our conclusion to better frame our contribution as this foundational first step and to more clearly articulate these exciting avenues for future research.

---

Once again, we thank the reviewer for their valuable feedback and their strong support of our work.

We are sincerely grateful to the reviewer for their exceptionally positive and encouraging feedback. We are thrilled that they found our work to be of "excellent" quality, clarity, and originality, and we deeply appreciate them highlighting our contributions, such as the use of GMMs as a "brilliant methodological choice" and our work providing the "first principled, distribution-level explanation for why KD works in this domain".

We also thank the reviewer for their thoughtful suggestions on areas for improvement and their insightful question. We believe these points will help us enrich the discussion in our final manuscript.

On Weaknesses and Future Directions

We thank the reviewer for their excellent points regarding the broader applications of our framework, such as explaining KD's success in model compression (e.g., quantization) and its potential for inspiring new distillation methods.

Our primary goal in this paper was to establish the first foundational, theoretical explanation for why knowledge distillation is so effective in generative models, supported by rigorous empirical evidence. We believe that establishing this core understanding is a crucial prerequisite for the very directions the reviewer suggests.

Our ground-truth → teacher → student framework provides a new analytical lens for the community. By demonstrating that introducing an intermediate teacher stage can fundamentally alter the learning outcome even with the same forward KL loss, we open up new avenues for investigation. We strongly agree that this framework could serve as a theoretical and empirical base to:

1. Analyze why KD is successful in model compression techniques like quantization.
2. Inspire the design of novel, more sophisticated distillation strategies or loss functions.

We see these as very promising and exciting directions for future research. We will update the "Limitations and Future Directions" section of our paper to explicitly mention these possibilities and frame our work as a foundational step toward them.

Response to Questions

This is a very insightful question regarding how the precision-recall trade-off might manifest across different downstream tasks, such as summarization, reasoning, and Chain-of-Thought (CoT) generation. Our framework provides a direct way to reason about this.

The modes in our Gaussian mixture model (Figure 1a of the manuscript) can be viewed as analogies for different capabilities or downstream tasks. In this view, each mode represents the data distribution for a specific skill (e.g., one mode for summarization, another for reasoning). A generalist model would need to cover all modes (high recall).

Our work demonstrates that distillation, especially with a selective teacher, guides the student to focus intensely on a subset of these modes (as shown in Figure 2c of the manuscript). This leads to a student model that achieves very high performance on the targeted tasks (high precision) but may lose some of its general capability across the other, non-targeted tasks (low recall).

This explains a widely observed empirical phenomenon: distilled student models often lose some of their generalist abilities but can match or even exceed teacher performance on the specific tasks they were distilled for. Our framework provides a principled, distribution-level explanation for this behavior. For a task like CoT, this implies that distillation could create a student that is highly proficient at generating a specific style of reasoning chain (high precision), but it might lose the ability to generate more diverse or exploratory reasoning paths (low recall).

---

Once again, we are deeply grateful for the reviewer's positive assessment. Your thoughtful feedback will help us improve the final version of the paper.

Dear Reviewer wtKg,

This is a gentle reminder that the discussion period will close in approximately 36 hours. We would be very grateful if you could take a moment to review our rebuttal and provide any final feedback.

Thank you again for your time and valuable contributions to our work.

Sincerely,

The Authors

After reviewing all the comments and rebuttals, I maintain my original positive rating. IMHO, this work differs from prior studies [1,2,3] in an important way: while existing works focus on proposing new techniques supported by specific observations or insights, this work provides a novel perspective for understanding the fundamentals of knowledge distillation (KD). I believe this new viewpoint will help researchers better understand the success and failure modes of KD techniques, particularly when applied to diverse task-specific scenarios.

The additional discussion the authors provided in the rebuttal regarding "how the precision-recall trade-off might manifest across different downstream tasks" would be valuable for readers if formally incorporated into the paper's discussion section or appendix.

We sincerely thank the reviewer for their time and for providing a detailed and thoughtful review. We are grateful for the positive feedback acknowledging the strengths of our work, particularly that it "bridges a simple theoretical model with a large-scale, practical application" and that our demonstration of the precision-recall trade-off as a "desirable feature, not a bug, is highly valuable".

We appreciate the opportunity to address the weaknesses and questions raised. We believe these points stem from a few key misunderstandings of our paper's core contributions and methodology. We will address each concern below and have committed to significant revisions to clarify these points in the manuscript.

1. On Originality and the Precision-Recall Trade-off (Weakness 1 & 4)

We thank the reviewer for pointing out several important references[1, 2, 3]. However, we respectfully disagree with the assessment that our main insight is "already demonstrated" in these works. Our contribution is fundamentally different in both its method and its analytical scope.

• A Novel Mechanism with Standard Forward KL: Our core contribution is demonstrating that a mode-seeking behavior can be induced using standard forward KL divergence—the default for many KD setups—simply by controlling the teacher's entropy. This is a distinct and novel mechanism compared to prior work like MiniLLM [1, 3], which proposes using a special objective, reverse KL divergence, or a sophisticated loss function [2], specifically to achieve mode-seeking. This is crucial because it provides a principled explanation for the effectiveness of the most common form of KD, rather than requiring specialized objectives. Our work explains the behavior of the tool that practitioners are already using.

• A More Comprehensive Analytical Framework: Prior works, including those cited, primarily analyze the relationship between the teacher and the student. Our paper introduces a more holistic three-stage analysis: ground truth → teacher → student. We are the first to theoretically and empirically show how a student model, trained with standard forward KD, exhibits mode-seeking behavior relative to the original ground-truth distribution. This broader perspective allows us to provide "a minimal working explanation" for why KD is effective in improving sample quality in generative models, a question that previous teacher-student analyses do not fully address.

In summary, while the goal of achieving precision is shared, our work offers a novel and complementary perspective to methods like MiniLLM. Where they modify the objective function, we demonstrate how to achieve similar desirable outcomes by controlling the teacher's properties within the standard KD framework. Our analysis of the three-stage framework provides a more general explanation for the effectiveness of KD. We thank the reviewer again for these valuable references and will add a new subsection to our Related Work section to discuss these papers and better contextualize our unique contributions.

2. On the Alleged Misalignment of Forward KL (Weakness 2)

We agree with the reviewer's correct assertion that forward KL divergence ($KL(P||Q)$) generally exhibits "mass-covering" or "mean-seeking" behavior. In fact, our paper explicitly demonstrates this very phenomenon. As shown in Figure 1 of the manuscript, when a simple student model (unimodal Gaussian) is trained directly on a multi-modal ground-truth distribution, it indeed averages the modes, resulting in a low-quality fit where probability mass is placed in low-density regions of the true distribution.

However, our paper's analysis does not stop at this simple two-way relationship. Our core contribution lies in analyzing the full ground-truth → teacher → student pipeline. As shown in Figure 2 of the manuscript, our process is as follows:

1. A teacher model with capacity $K' < K$ is trained on the ground-truth data, learning a subset of the modes (Fig. 2a).
2. This teacher is made more selective (low-entropy) via our $\beta$ parameter.
3. A student model with capacity $K'' < K'$ is then trained on this selective teacher using standard forward KL. The result is a student that focuses its mass on the single, dominant mode presented by the teacher (Fig. 2c).

Crucially, the final student model exhibits high precision and low recall with respect to the original ground-truth distribution. Therefore, our work shows how the standard forward KL objective, when used within this multi-stage framework, results in a desirable mode-seeking behavior in the final student model. There is no misalignment; rather, we provide a more complete picture that explains how this emergent behavior arises. This is precisely the "minimal working explanation" we aimed to provide—showing how a seemingly simple setup can explain a complex and valuable phenomenon in generative model distillation.

3. On the Alleged Contradiction in Simulation Results (Weakness 2 continued)

The reviewer correctly observes from our results that as the KD training loss $KL(p'\_{teacher}||p''\_{student})$ is minimized, the Recall metric (related to $KL(p*\_{ground-truth}||p''\_{student})$) worsens. The reviewer interprets this as a contradiction.

We respectfully clarify that this is not a contradiction, but the central finding of our paper. We explicitly define and demonstrate a trade-off where the student model intentionally sacrifices coverage of the full ground-truth distribution ($p^*$) to achieve higher precision by faithfully mimicking the selective, low-entropy teacher ($p'$).

Furthermore, the reviewer's hypothetical scenario for this contradiction is based on an invalid premise. The reviewer states: "When $K'=K$, the teacher model will exactly fit the real distribution $p^*$." This is incorrect for two reasons:

1. Practical Impossibility: In any real-world scenario, the number of modes in the ground-truth distribution, $K$, is unknown. Therefore, setting the teacher's capacity $K'$ to be equal to $K$ is not practically possible.
2. Inherent Learning Noise: As we explicitly state in our paper, even in the ideal case of $K′=K$, the teacher parameters are recovered only "up to some noise due to the finite number of samples within $D$" (see Line 113 of the manuscript). This inherent noise from statistical learning ensures that the teacher $p^′$ would never be a perfect replica of the ground-truth $p^∗$, even with perfectly matched capacity.
3. Methodological Design: Most importantly, even if we could assume $K'=K$, our method's application of the parameter $\beta > 1$ ensures that the teacher distribution $p'(x;\beta)$ is a lower-entropy modification and thus does not exactly fit the real distribution $p^*(x)$.

Therefore, the situation described by the reviewer cannot occur under our framework. Additionally, we believe that the core of this misunderstanding appears to be a conflation of the training objective ($KL(p'\_{teacher}||p''\_{student})$) with an evaluation metric (Recall, which is related to $KL(p*\_{ground-truth}||p''\_{student})$). Our method successfully trains the student to minimize its divergence from the selective teacher, which is the explicit goal of Equation (2). The fact that this simultaneously increases the student's divergence from the full ground-truth distribution is not a flaw in the optimization; it is the very definition of the precision-recall trade-off that our paper identifies and explains.

4. On the "Problematic Theory" (Weakness 3)

We appreciate the reviewer's close reading of our theoretical analysis. We would like to clarify two key points regarding the analysis following Equation (2), which we believe will resolve the reviewer's concerns.

1. Correction on the Bound Type: First, we would like to clarify a small but important technical point. The reviewer states that our analysis shows an "upper bound". However, because the logarithm is a concave function, the application of Jensen's inequality to the integral term results in a lower bound on that term, which our optimization seeks to maximize. This is equivalent to minimizing an upper bound on the KL divergence loss itself.
2. Clarification on the Analysis's Purpose: Second, and more importantly, this addresses the reviewer's core concern about the analysis's purpose. The purpose of this analysis was not to serve as a formal proof or to "sufficiently guarantee" the mode-seeking behavior. As we state in the paper, its goal is to "better understand this objective". The decomposition is intended purely as an illustration to provide intuition for how the optimization encourages the student to align with the teacher's high-density modes.

The mode-seeking behavior itself does not arise from this Jensen's inequality analysis. Instead, it is a direct consequence of our methodological setup: we intentionally construct a low-entropy, selective teacher distribution ($p^′$). The analysis via the bound simply helps to explain the mechanics of how the student model's optimization is guided by this pre-conditioned, selective teacher. Therefore, the theory is not problematic; the analysis serves its intended illustrative purpose, and the mode-seeking behavior is a result of the problem formulation itself. We will revise the text to make the illustrative purpose of this section even clearer.

---

We hope these clarifications address the reviewer's concerns. We are confident that by incorporating the suggested references and clarifying our methodology and contributions as outlined above, the revised manuscript will be significantly stronger. We thank the reviewer again for their valuable feedback.

Dear Reviewer sQ8t,

This is a gentle reminder that the discussion period will close in approximately 36 hours. We would be very grateful if you could take a moment to review our rebuttal and provide any final feedback.

Thank you again for your time and valuable contributions to our work.

Sincerely,

The Authors

Dear Reviewer sQ8t,

We hope this message finds you well. We noticed the author-reviewer discussion period has been extended, and we would be very grateful if you could take a moment to review our rebuttal and the ongoing discussion.

We also wanted to gently note that other reviewers have since shared their thoughts after reading our response. We believe their comments, alongside our rebuttal, might help clarify our contributions. We would sincerely appreciate any final feedback you may have.

Thank you again for your time and your valuable role in this process.

Sincerely,

The Authors