Multi-Granularity Semantic Revision for Large Language Model Distillation

Weakness3: More discussion for SCRG

We appreciate your feedback regarding the need for more experimental evidence to support our claims about the diversity improvements facilitated by our Sequence Correction and Re-Generation (SCRG) method. To address this, we conducted experiments to provide a robust comparison of SCRG against a leading data quality improvement approach by Kim et al. [1], which focuses on offline data pruning and selection.

[1] Kim M, Baek S. Measuring Sample Importance in Data Pruning for Training LLMs from a Data Compression Perspective[J]. arXiv preprint arXiv:2406.14124, 2024.

Experimental Results

Our results, summarized in Table below, demonstrate that SCRG outperforms the offline data enhancement method proposed by Kim et al. across multiple datasets:

These results show that SCRG not only outperforms the approach by Kim et al., but when combined with Kim et al.'s method, a slight improvement in performance is observed. While both SCRG and the method proposed by Kim et al. enhance data quality, the incremental gains from combining them are limited. This is likely due to the fact that both methods address similar underlying issues related to data quality, resulting in diminishing returns when applied together.

Qualitative Analysis

To further illustrate the impact of SCRG on output diversity, we present a comparative analysis of two sentences generated during the knowledge distillation training process:

• Sentence 1 (Without SCRG): "Men’s lacrosse has a limited amount of time to play play play as as as as as as as as as as as as as as as as as as as"

  • 1-grams: Total: 31, Unique: 12, Distinct-1: 0.387
  • 2-grams: Total: 30, Unique: 13, Distinct-2: 0.433
  • 3-grams: Total: 29, Unique: 14, Distinct-3: 0.483

• Sentence 2 (With SCRG): "Men’s lacrosse has a limited number of players and women’s lacrosse has a maximum number of players."

  • 1-grams: Total: 19, Unique: 12, Distinct-1: 0.632
  • 2-grams: Total: 18, Unique: 13, Distinct-2: 0.722
  • 3-grams: Total: 17, Unique: 14, Distinct-3: 0.824

The distinct n-gram statistics reveal significant improvements in output diversity when SCRG is applied. Sentence 2 exhibits higher distinct n-gram scores across all levels, demonstrating an increase in unique words and phrases. This not only highlights the effectiveness of SCRG in refining data but also emphasizes its role in enhancing the overall quality and diversity of the student model's generation process.

In summary, our experimental results and qualitative analysis provide substantial evidence to support our claim that SCRG improves the diversity of generated results. We believe that the combination of our method with existing approaches can lead to even more powerful outcomes, reinforcing the potential of SCRG in advancing model performance and output diversity.

Below is our point-by-point response to your main concerns. Please let us know if there's anything we can clarify further.

Weakness1: Efficiency Concerns

We believe there is a misunderstanding regarding Table 4(b). In fact, as shown in the table, the training efficiency of our method is higher than that of MiniLLM, achieving a significantly better batch/seconds ratio (0.18 compared to MiniLLM's 0.05). Additionally, our training efficiency is comparable to that of DistiLLM while demonstrating superior performance in terms of ROUGE-L scores. This indicates that the computational overhead of our approach is justifiable given the substantial performance improvements it delivers.

Even if we remove the SCRG module (as shown in the table below) to match the training efficiency of DistiLLM, our method still outperforms the baseline in terms of performance. This highlights the robustness and effectiveness of our proposed approach even under stricter efficiency constraints.

Weakness2: ExAccErr measures

The discrepancy in ExAccErr values was due to a formatting error in our initial submission. We apologize for any confusion this may have caused. The updated results are presented below:

These revised results align with the overall trend observed in our experiments and substantiate our claim in line 515 that the distilled student models exhibit lower exposure bias, which contributes to their ability to outperform teacher models. This reduction in ExAccErr with increasing generation length demonstrates the effectiveness of our proposed method in mitigating exposure bias relative to the baselines.

Weakness1: Improvement of Table and Figure Captions

Thank you for your valuable feedback. I fully accept your suggestion and will work on making the captions of images and tables more detailed and informative moving forward.

Question1: Using a much much more complex teacher than the student

We appreciate the reviewer's suggestion to explore the effects of using a much larger teacher model in distillation. To address this, we extended our experiments from the previous OPT 6.7B → 1.3B distillation setup by using a larger teacher model, OPT-13B, to distill the 1.3B student. The results, shown in the table below, demonstrate that while distilling with a much larger teacher does lead to smaller improvements in performance compared to the 6.7B → 1.3B case, our proposed distillation method still outperforms vanilla KD loss significantly, even when using a much more complex teacher.

As seen from the table, while the performance improvement decreases with the larger teacher (OPT-13B), our distillation method still provides a significant advantage over the vanilla KD approach, even when using a more complex and larger teacher model. This indicates that our method with DAC-KL loss helps mitigate the potential performance degradation seen when distilling with a much larger teacher.

Quesion2: Comparing the simple method with DAC Loss

To validate the effectiveness of DAC-KL, we have provided a detailed discussion in Appendix I. In Table 13, we compare DAC-KL with other logits-selective methods, including the Fixed Clipping Threshold approach, which is conceptually similar to the method described in [1], except that it uses a cumulative sum threshold of 95% as the clipping up. To address your concern, we have also included the baseline method from [1] in the updated table below. This allows for a direct comparison and demonstrates how DAC-KL, which operates at the token level, provides superior performance by effectively balancing information retention and noise reduction compared to simplistic logits pruning approaches. The results of this comparison are as follows:

Below is our point-by-point response to your main concerns. Please let us know if there's anything we can clarify further.

Weakness1: The intuition for correcting the distillation dataset

It is important to clarify that the purpose of our corrections is not to make the student model's outputs identical to the teacher's but to provide initial guidance that prevents severe errors, such as repetitive generation. In our experiments, even a single correction significantly improves output quality. While increasing the number of corrections may lead to outputs that resemble those of the teacher, this approach does not effectively address exposure bias. We have conducted a performance experiment on OpenLLAMA2-3B, which demonstrates how varying the number of corrections impacts the results, as shown below:

Weakness2: Detailed analyses about the clipping method

In response to your concern, we would like to clarify that detailed analyses of the clipping method have been provided in Appendix D. Specifically, we illustrate examples of the teacher's output probability distribution using kernel density estimation. The DAC-KL loss primarily focuses on capturing low-probability yet high-frequency regions of the distribution and combines these with the target class to form new logit vectors.

Additionally, further discussions are presented in Appendix I. Our approach is motivated by the goal of modulating the probability distribution to reduce the alignment difficulty between teacher and student distributions. This is conceptually similar to methods like DKD [1] and SKD [2]. However, the key distinction lies in how DAC-KL adaptively suppresses redundant information in the original distribution by a learnable sub-network. This suppression reduces the challenge of fitting the teacher's distribution when the student's capacity is limited.

The necessity of DAC-KL lies in its adaptability to complex probability distributions. While other clipping and sampling methods rely on manually defined thresholds, DAC-KL adjusts dynamically to the probability distribution of different tokens across samples during training. Fixed thresholds, though simpler, performed less effectively, as shown in the results below:

As demonstrated in the table, our approach significantly outperforms fixed clipping thresholds and other baseline methods across all metrics. DAC-KL's adaptive nature enables it to optimize the probability distribution modulation dynamically, which is crucial for effective distillation under limited student capacity.

We appreciate the reviewer’s suggestion and have expanded our discussions to clarify this aspect further. Thank you for highlighting this point.

[1]Zhao B, Cui Q, Song R, et al. Decoupled knowledge distillation[C]//Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. 2022: 11953-11962. [2] Yuan M, Lang B, Quan F. Student-friendly knowledge distillation[J]. Knowledge-Based Systems, 2024, 296: 111915.

Weakness3: Dailed analyses for the span loss

Human Evaluation

We compared our Span-Relation method with a random chunking approach (where the number of chunks is controlled to match that of span-relation) and a method that directly extracts relations between adjacent tokens without chunking.

To conduct a more comprehensive and reliable evaluation, we further employed GPT-4 to conduct a human-like evaluation of the models on the Dolly evaluation dataset. We sampled 100 test examples from both models—with and without span-level loss—and assessed their outputs based on the following criteria:

• Accuracy (Rate 1-5): Does the output correctly include all relevant details from the input?
• Completeness (Rate 1-5): Does the output provide a comprehensive list or description as required by the instruction?
• Fluency (Rate 1-5): Is the output natural, readable, and grammatically correct?
• Relevance (Rate 1-5): How well does the output align with the specific requirements of the instruction?

The evaluation results are summarized in the Table below:

These results illustrate that models utilizing span-level loss achieve higher average evaluations across all criteria compared to other configurations, highlighting the benefits of incorporating span-level signals to enhance model performance.

Example Outputs

To demonstrate the improved correlation among span-level tokens, we selected several typical examples where information extraction requires the output to repeat specific phrases from the input prompt. Below, we give examples of different models with and without span-level loss. Our model distilled with the span-level loss successfully and accurately outputs the fixed phrases from the prompt and context, showcasing the enhanced correlation among tokens within the span.

Example 1

• Instruction: Extract the essential tools for a mini survival kit from the following Wikipedia article.
• Input: Mini survival kits or "Altoids" tin survival kits are small kits that contain a few basic survival tools. These kits often include a small compass, waterproof matches, minimum fishing tackle, large plastic bag, small candle, jigsaw blade, craft knife or scalpel blade, and/or a safety pin/s. Pre-packaged survival kits may also include instructions in survival techniques such as fire-starting or first aid methods. In addition, paracord can be wrapped around the tin. The paracord can be used for setting up an emergency shelter or snaring small animals. They are designed to fit within a container roughly the size of a mint tin.
• Without Span-Level Loss: The essential tools include a small compass, waterproof matches, fishing tackle, plastic bag, candle, blade, knife, and safety pin.
• With Span-Level Loss: The essential tools include a small compass, waterproof matches, minimum fishing tackle, large plastic bag, small candle, jigsaw blade, craft knife or scalpel blade, and safety pin/s.

Example 2

• Instruction: Given the reference text below, what is Isthmohyla debilis?
• Input: Isthmohyla debilis is a rare species of frog in the family Hylidae. It occurs in the Atlantic slopes of the Cordillera Central and Cordillera Talamanca in Costa Rica and western Panama as well as on the Pacific slopes in southwestern Panama. Common name Isla Bonita treefrog has been suggested for it (the type locality is "Isla Bonita" in Costa Rica).
• Without Span-Level Loss: Isthmohyla debilis is a rare frog species in the family Hylidae. It is found in Costa Rica and Panama. The common name Isla Bonita treefrog is suggested for it.
• With Span-Level Loss: Isthmohyla debilis is a rare species of frog in the family Hylidae. It occurs in the Atlantic slopes of the Cordillera Central and Cordillera Talamanca in Costa Rica and western Panama as well as on the Pacific slopes in southwestern Panama. The common name Isla Bonita treefrog has been suggested for it.

The model distilled with span-level loss can better extract important adjectives in phrases like "minimum fishing tackle," "large plastic bag," and "small candle" in the first example, and output long phrases of complex location as in the second example.

Thank you for your continued engagement and for considering the additional results we provided. We understand your initial concern and appreciate your willingness to raise the soundness score.

We understand your concern regarding the 'Frequency of SCRG=0' experiment and the potential misunderstanding it may have caused. We wish to emphasize that this experiment was designed to demonstrate the scenario where we directly address exposure bias using student-generated data, which, while effective, can introduce additional generation errors. Our aim is not to ignore the exposure bias but to highlight the challenges inherent in this approach.

To address these challenges and to mitigate the introduction of generation errors, we have implemented SCRG. SCRG is not only about solving exposure bias but also about doing so in a way that avoids the propagation of erroneous data. It achieves this by refining the student model's outputs with the guidance of the teacher model, thus enhancing the quality of the distillation dataset without compromising the integrity of the data.

Furthermore, the 'Frequency of SCRG=10' experiment was included to illustrate the scenario where student-generated data closely resembles that of the teacher, which, as you correctly pointed out, could potentially undermine the effectiveness of addressing exposure bias. This experiment serves to demonstrate the balance that SCRG strikes between maintaining the teacher's guidance and the student's independence.

To provide a more intuitive understanding of SCRG and its impact on the distillation dataset, we have conducted additional analyses, which we detail below:

More analyses for correcting the distillation dataset (SCRG)

The role of SCRG is to mitigate the introduction of errors in the data produced by the student model during the initial phase of the knowledge distillation training. It achieves this by employing the teacher model's guidance to refine the generation process, thereby improving the overall quality of the output.

To provide empirical evidence, we present a comparison of two example sentences: one generated early in the distillation process without SCRG (Sentence 1) and another generated after applying SCRG (Sentence 2).

Sentence 1 (Without SCRG):

"Men’s lacrosse has a limited amount of time to play play play as as as as as as as as as as as as as as as as as as as"

• 1-grams:
  • Total: 31
  • Unique: 12
  • Distinct-1: 0.387
• 2-grams:
  • Total: 30
  • Unique: 13
  • Distinct-2: 0.433
• 3-grams:
  • Total: 29
  • Unique: 14
  • Distinct-3: 0.483

Sentence 2 (With SCRG):

"Men’s lacrosse has a limited number of players and women’s lacrosse has a maximum number of players."

• 1-grams:
  • Total: 19
  • Unique: 12
  • Distinct-1: 0.632
• 2-grams:
  • Total: 18
  • Unique: 13
  • Distinct-2: 0.722
• 3-grams:
  • Total: 17
  • Unique: 14
  • Distinct-3: 0.824

The distinct n-gram statistics reveal a significant improvement in generation diversity when SCRG is applied. Sentence 2 demonstrates higher distinct n-gram scores across all levels compared to Sentence 1. This increase in unique words and phrases highlights SCRG’s effectiveness in promoting diverse and meaningful outputs in the student model.

Furthermore, we conducted experiments to provide a robust comparison of SCRG against a leading data quality improvement approach by Kim et al. [1], which focuses on offline data pruning and selection.

[1] Kim M, Baek S. Measuring Sample Importance in Data Pruning for Training LLMs from a Data Compression Perspective[J]. arXiv preprint arXiv:2406.14124, 2024.

Experimental Results

Our results, summarized in the Table below, demonstrate that SCRG outperforms the offline data enhancement method proposed by Kim et al. across multiple datasets:

These results show that SCRG not only outperforms the approach by Kim et al., but when combined with Kim et al.'s method, a slight improvement in performance is observed. While both SCRG and the method proposed by Kim et al. enhance data quality, the incremental gains from combining them are limited. This is likely due to the fact that both methods address similar underlying issues related to data quality, resulting in diminishing returns when applied together.

Weakness2: Intuition of Span Loss

Our method emphasizes distilling correlation consistency among tokens within a span, rather than merely aligning semantics at the token level (as done in token-level KL divergence).

To further clarify the intuition behind our approach, we conducted the following analysis:

Human Evaluation

We compared our Span-Relation method with a random chunking approach (where the number of chunks is controlled to match that of span-relation) and a method that directly extracts relations between adjacent tokens without chunking.

To conduct a more comprehensive and reliable evaluation, we further employed GPT-4 to conduct a human-like evaluation of the models on the Dolly evaluation dataset. We sampled 100 test examples from both models—with and without span-level loss—and assessed their outputs based on the following criteria:

• Accuracy (Rate 1-5): Does the output correctly include all relevant details from the input?
• Completeness (Rate 1-5): Does the output provide a comprehensive list or description as required by the instruction?
• Fluency (Rate 1-5): Is the output natural, readable, and grammatically correct?
• Relevance (Rate 1-5): How well does the output align with the specific requirements of the instruction?

The evaluation results are summarized in the Table below:

These results illustrate that models utilizing span-level loss achieve higher average evaluations across all criteria compared to other configurations, highlighting the benefits of incorporating span-level signals to enhance model performance.

Example Outputs

To demonstrate the improved correlation among span-level tokens, we selected several typical examples where information extraction requires the output to repeat specific phrases from the input prompt. Below, we give examples of different models with and without span-level loss. Our model distilled with the span-level loss successfully and accurately outputs the fixed phrases from the prompt and context, showcasing the enhanced correlation among tokens within the span.

Example 1

• Instruction: Extract the essential tools for a mini survival kit from the following Wikipedia article.
• Input: Mini survival kits or "Altoids" tin survival kits are small kits that contain a few basic survival tools. These kits often include a small compass, waterproof matches, minimum fishing tackle, large plastic bag, small candle, jigsaw blade, craft knife or scalpel blade, and/or a safety pin/s. Pre-packaged survival kits may also include instructions in survival techniques such as fire-starting or first aid methods. In addition, paracord can be wrapped around the tin. The paracord can be used for setting up an emergency shelter or snaring small animals. They are designed to fit within a container roughly the size of a mint tin.
• Without Span-Level Loss: The essential tools include a small compass, waterproof matches, fishing tackle, plastic bag, candle, blade, knife, and safety pin.
• With Span-Level Loss: The essential tools include a small compass, waterproof matches, minimum fishing tackle, large plastic bag, small candle, jigsaw blade, craft knife or scalpel blade, and safety pin/s.

Example 2

• Instruction: Given the reference text below, what is Isthmohyla debilis?
• Input: Isthmohyla debilis is a rare species of frog in the family Hylidae. It occurs in the Atlantic slopes of the Cordillera Central and Cordillera Talamanca in Costa Rica and western Panama as well as on the Pacific slopes in southwestern Panama. Common name Isla Bonita treefrog has been suggested for it (the type locality is "Isla Bonita" in Costa Rica).
• Without Span-Level Loss: Isthmohyla debilis is a rare frog species in the family Hylidae. It is found in Costa Rica and Panama. The common name Isla Bonita treefrog is suggested for it.
• With Span-Level Loss: Isthmohyla debilis is a rare species of frog in the family Hylidae. It occurs in the Atlantic slopes of the Cordillera Central and Cordillera Talamanca in Costa Rica and western Panama as well as on the Pacific slopes in southwestern Panama. The common name Isla Bonita treefrog has been suggested for it.

The model distilled with span-level loss can better extract important adjectives in phrases like "minimum fishing tackle," "large plastic bag," and "small candle" in the first example, and output long phrases of complex location as in the second example.

Below is our point-by-point response to your main concerns. Please let us know if there's anything we can clarify further.

Weakness1: Lack of empirical evidence for SCRG

The role of SCRG is to mitigate the introduction of errors in the data produced by the student model during the initial phase of the knowledge distillation training. It achieves this by employing the teacher model's guidance to refine the generation process, thereby improving the overall quality of the output.

To provide empirical evidence, we present a comparison of two example sentences: one generated early in the distillation process without SCRG (Sentence 1) and another generated after applying SCRG (Sentence 2).

Sentence 1 (Without SCRG):

"Men’s lacrosse has a limited amount of time to play play play as as as as as as as as as as as as as as as as as as as"

• 1-grams:
  • Total: 31
  • Unique: 12
  • Distinct-1: 0.387
• 2-grams:
  • Total: 30
  • Unique: 13
  • Distinct-2: 0.433
• 3-grams:
  • Total: 29
  • Unique: 14
  • Distinct-3: 0.483

Sentence 2 (With SCRG):

"Men’s lacrosse has a limited number of players and women’s lacrosse has a maximum number of players."

• 1-grams:
  • Total: 19
  • Unique: 12
  • Distinct-1: 0.632
• 2-grams:
  • Total: 18
  • Unique: 13
  • Distinct-2: 0.722
• 3-grams:
  • Total: 17
  • Unique: 14
  • Distinct-3: 0.824

The distinct n-gram statistics reveal a significant improvement in generation diversity when SCRG is applied. Sentence 2 demonstrates higher distinct n-gram scores across all levels compared to Sentence 1. This increase in unique words and phrases highlights SCRG’s effectiveness in promoting diverse and meaningful outputs in the student model.

Furthermore, we conducted experiments to provide a robust comparison of SCRG against a leading data quality improvement approach by Kim et al. [1], which focuses on offline data pruning and selection.

[1] Kim M, Baek S. Measuring Sample Importance in Data Pruning for Training LLMs from a Data Compression Perspective[J]. arXiv preprint arXiv:2406.14124, 2024.

Experimental Results

Our results, summarized in the Table below, demonstrate that SCRG outperforms the offline data enhancement method proposed by Kim et al. across multiple datasets:

These results show that SCRG not only outperforms the approach by Kim et al., but when combined with Kim et al.'s method, a slight improvement in performance is observed. While both SCRG and the method proposed by Kim et al. enhance data quality, the incremental gains from combining them are limited. This is likely due to the fact that both methods address similar underlying issues related to data quality, resulting in diminishing returns when applied together.

Weakness3: External chunker to extract spans

We acknowledge the concern regarding the reliance on an external chunker for span extraction, particularly for low-resource languages. This requirement could limit the generalizability of extraing spans in such scenarios. However, for mainstream languages, there are well-established and robust NLP toolkits, such as SpaCy and NLTK, that provide reliable chunking capabilities. These tools have been extensively developed and optimized, making them highly effective and widely applicable to tasks like ours.

For low-resource languages, we believe our approach can be adapted by leveraging alternative methods for span extraction. For example, in the case of Chinese, the JieBa library provides an effective way to extract spans like noun and verb phrases. For smaller or low-resource languages, one possible solution is to utilize large pretrained models, such as GPT-4, for data preprocessing to generate spans. This unsupervised or weakly supervised approach could make our method more adaptable to diverse linguistic resources, and we plan to explore this avenue in future work.

Weakness4: DAC-KL Vs. selective distillation

To validate the effectiveness of DAC-KL, we have provided a detailed discussion in Appendix I. In Table 13, we compare DAC-KL with other logit selective methods, which, while not including the specific methods [1] and [2] you mentioned, belong to the same category of techniques. We appreciate your suggestion, and we have now added the baseline methods [1] and [2] to the comparison. The results of this updated comparison are as follows:

Weakness5: marginal improvement

Firstly, regarding the performance improvements presented in Table 1, we would like to highlight that most metrics show an improvement exceeding 1 ROUGE score, particularly for the LLAMA2 and OpenLLAMA2 models. This is especially evident in cases where baseline methods like DistiLLM and MiniLLM already achieve strong performance. In such high-performance contexts, improvements of around 1 ROUGE score might appear marginal at first glance. However, these relatively modest gains are actually significant, as they reflect a notable enhancement in models that are already performing at a high level.

Secondly, concerning the ablation study, we emphasize that we conducted a comprehensive evaluation across multiple test sets, including both the validation and zero-shot test sets. The contributions of individual modules vary across these different sets. While some modules have a smaller impact on specific test sets, their collective contribution is crucial in strengthening the overall performance of the proposed methods. Therefore, even though certain objectives might show a more modest effect in isolation, together they enhance the model's effectiveness.

Weakness6: Only use the ROUGE-L metric

While our primary evaluation relies on ROUGE-L for consistency and comparability with previous work, we have also conducted a small set of GPT-4 feedback experiments to assess the impact of Span loss, as detailed in Appendix B. Due to the high cost of large-scale GPT-4 evaluations, we provide a quantitative analysis of the distillation results of the teacher-student pair LLAMA2-13B to LLAMA2-7B, evaluated by a locally deployed LLAMA3.1-70B.

The use of LLAMA3.1-70B in our evaluation process is not only a pragmatic choice but also a strategic one. Our evaluation criteria for these experiments are informed by the methodologies employed in Distillm. These experiments provide additional insights into the quality of the model outputs, complementing the quantitative ROUGE-L results with human-like evaluation feedback.

Evaluation results by LLAMA3.1-70B feedback for the LLAMA2 teacher-student model pair