PROGRESSIVE KNOWLEDGE DISTILLATION (PKD): A MODULAR APPROACH FOR ARCHITECTURE-AGNOSTIC KNOWLEDGE DISTILLATION

---

Dear Reviewer, We sincerely appreciate the thoughtful comments and suggestions, which have been invaluable in improving our paper. Below, we provide detailed responses to address the concerns.

---

1. Novelty

On the topic of novelty and the references to [1,2], our approach is fundamentally different from the methods described in those works. Specifically:

• Our approach does not use CKA as a loss function. Instead, CKA is utilized to modularize both the teacher and student networks by defining module boundaries. This enables effective alignment and knowledge transfer, which is a novel use of CKA.
• Our contributions go beyond the use of CKA itself and include several key innovations:
  1. Architecture-Agnostic Design: The method generalizes across diverse architectures, including CNNs, ViTs, and hybrids, demonstrating broad applicability.
  2. CKA-Based Modularization: CKA is used to define the most appropriate module boundaries, ensuring that knowledge transfer focuses on the most critical features.
  3. Most Informative Knowledge Extraction: The approach prioritizes extracting representative features from each module for improved distillation efficiency.
  4. Progressive Alignment Strategy: By progressively aligning the teacher and student modules, the method achieves superior performance and generalizability compared to traditional static approaches.

These innovations set our work apart from [1,2], where CKA is used in static loss formulations, and highlight the originality of our method.

---

2. Training Overhead

With respect to training overhead and hyperparameter tuning, our method includes a warm-up phase to initialize module configurations. This phase is designed to stabilize the training process and improve convergence. The associated computational overhead is minimal compared to the significant performance gains achieved, as demonstrated in our experiments. Moreover, this warm-up phase results in more efficient training overall (fewer training epochs), as discussed in the "Optimization" section of the paper (lines 401–408).

To mitigate hyperparameter tuning complexity, our method leverages CKA to define module boundaries automatically. This eliminates the need for manual module partitioning and simplifies tuning. Furthermore, our ablation studies (detailed below) confirm the robustness of our method across a range of configurations, ensuring practical applicability with minimal tuning requirements.

---

3. Ablation Studies

For the request on additional ablation studies, we have already analyzed the impact of $\epsilon_{th}$ in Section B5 of the Appendix. In addition, comprehensive experiments detailed in Section B evaluate various aspects of our method, including:

• B1) Knowledge Transfer in Consistent Architectures: Evaluates performance when teacher and student share the same architecture, demonstrating the generalizability of our approach.
• B2) Consistent vs. Diverse Teacher Models: Explores scenarios with consistent and diverse teacher models, showcasing the adaptability of the method.
• B3) Knowledge Distillation Using Last Layer and Module-Specific Features: Compares the use of last-layer features versus module-specific features, highlighting the advantages of our modular approach.
• B4) Combining PKD with DKD: Shows that integrating PKD with DKD improves DKD’s performance, demonstrating the complementary nature of our method.
• B5) Impact of Threshold in Equation 5: Analyzes the sensitivity to the threshold parameter ($\epsilon_{th}$), demonstrating consistent performance across a wide range of values.

These studies provide detailed insights into the robustness and versatility of the method, validating its effectiveness under diverse settings.

---

4. Consistency in Equations and Figure Quality

On the matter of equation style and figure resolution, we will ensure that all equations are updated to follow a consistent style. Additionally, all figures will be replaced with high-resolution versions to enhance readability and presentation quality.

---

In summary, our contributions include a novel CKA-based modularization framework with a progressive alignment strategy, applicable to diverse architectures. The minimal training overhead is outweighed by the significant performance improvements. The extensive ablation studies and manuscript revisions address the concerns raised, ensuring clarity and comprehensiveness.

We hope this explanation addresses the concerns raised. If there are any additional questions or feedback, we would be happy to address them further.

---

---

Dear Reviewer,

Thank you for the constructive feedback and suggestions to enhance the clarity and rigor of our paper. Below, we address your concerns regarding the inclusion of related works, the necessity and effectiveness of CKA for modularization, the phrase in lines 56-57, and the clarification on $d_{SM}$ and $d_{DM}$.

---

Response to Related Literature:

We acknowledge the importance of including a broader discussion of related works and appreciate the references you provided. We will incorporate these into the revised manuscript to expand our literature review and better situate our work.

While we focused on recent state-of-the-art (SOTA) methods like OFAKD and its baselines, we recognize the value of incorporating these additional references for a more comprehensive view. Given the vast body of research in this area, our emphasis remains on capturing the most representative and impactful methods.

---

Response to Modularization Concerns:

1. Clarification on OFAKD as an Ablation:
   OFAKD itself serves as an ablation for modularization methods, dividing the network into four modules without any heuristic approach. Our experiments show that our CKA-based modularization outperforms OFAKD, highlighting its effectiveness.

2. Challenges in Cross-Architecture and Transformer Networks:
   Modularizing architectures like transformers is inherently tricky because all layers output feature maps with uniform dimensions. Simple methods like feature map resolution-based division are not applicable. CKA provides a meaningful alternative by comparing representations independently of their size.

3. Performance Comparison for CNN Teachers:
   To explore simpler modularization approaches, we conducted additional experiments using feature map resolution-based modularization (PKD-res). These results are summarized below:

Table 1. Teacher modularization based on feature map resolution indicated as PKD-res.

These results indicate that our CKA-based modularization (PKD) consistently outperforms the simpler feature map resolution-based approach (PKD-res). This demonstrates the robustness of CKA, especially in cross-architecture and transformer-based models. We will include this experiment in our revision.

---

Response to Unclear Phrase in Lines 56-57:

The rationale for starting with the shallowest module is grounded in the hierarchical structure of neural networks:

1. Generalized Low-Level Features:
   Shallow layers capture low-level, foundational features (e.g., edges, textures). These features are general and transferable, ensuring the student model begins with a strong foundation.

2. Progressive Complexity:
   Training shallow layers first aligns with the natural progression of feature extraction, minimizing the risk of noise or misalignment propagating from deeper, more complex layers.

3. Stability in Knowledge Transfer:
   Incremental knowledge transfer ensures stability. Aligning deeper layers first, without a solid base, risks incoherent representations and suboptimal performance.

4. Empirical Justification:
   Our experiments show that progressive modular training starting with the shallowest layer outperforms simultaneous or reverse-order approaches, supporting the effectiveness of this hierarchical training strategy.

We will revise the manuscript to clarify and emphasize these motivations.

---

Clarification on $d_{SM}$ and $d_{DM}$:

You are correct that $d_{SM}$ and $d_{DM}$ use the same computational formulation. They are named differently to highlight module boundaries. This distinction aids clarity in understanding our modularization approach. We will refine the manuscript to explain this more clearly.

---

Does this address your concerns? Please feel free to let us know if additional clarifications or results are needed. We appreciate your valuable feedback!

---

---

Dear Reviewer,

Thank you for your constructive feedback. Below, we provide detailed responses to address your concerns.

---

Related Works on Progressive Knowledge Distillation:

We acknowledge the importance of situating our work within the broader context of progressive knowledge distillation. The references you provided will be included and discussed in the revised manuscript.

While these works focus on progressive distillation, our approach differs significantly:

1. Architecture-Agnostic Design: Our method generalizes across diverse architectures, including CNNs, ViTs, and hybrids.
2. CKA-Based Modularization: We use CKA to define module boundaries, enabling effective alignment and transfer of knowledge.
3. Most Informative Knowledge Extraction: We prioritize extracting the most representative features from each module.
4. Progressive Alignment Strategy: Our method integrates these components into a unified framework, enhancing performance and generalizability.

Given this focus, we emphasized OFA as SOTA and its baselines in our experiments. We will expand the literature review to include the cited works and highlight these distinctions.

---

Complexity Analysis:

Our method’s complexity compares favorably to baselines:

1. Offline Feature Alignment: Steps like PCA and Gram matrix computation are performed offline, avoiding inference overhead.
2. Scalable Training: Modular alignment scales linearly with the number of modules. Iterative refinement in some baselines may introduce quadratic complexity.
3. Optimization Efficiency: As noted in the optimization section (lines 400-408), fewer training epochs are needed due to effective distillation, improving efficiency.

Distillation’s primary objective is to achieve significant downsizing with minimal performance degradation. Training is a one-time cost, while the distilled model is repeatedly deployed, where size, speed, and performance are critical. A detailed complexity comparison will be included in the revised manuscript.

---

Experimental Scale:

Our experiments on CIFAR and ImageNet datasets demonstrate versatility across architectures, following SOTA methods like OFAKD. Scaling to large models (e.g., LLMs) presents engineering challenges, but our modular, architecture-agnostic approach positions it well for such extensions. We will outline this as future work.

---

Block/Modular Correspondence:

The correspondence between teacher and student modules is determined using a binary vector-based strategy, as detailed in lines 266-310.

4. Misaligned Student modules:
   Misaligned modules result in suboptimal transfer. To explore this, we conducted additional experiments:
   • Misaligned Modules: In one experiment, we combined two middle student modules, creating misalignment. While results improved compared to OFAKD, they were worse than our aligned approach.
   • Shifted Modules: In another, we shifted one module from each group to the previous module (except the first). This caused no significant performance changes, showing robustness to minor shifts.

• ** Ablation study B4**, please also consider our ablation study reported in section B4 of Appendix. In this experiment, we leverage OFA modularization and the last layer most specific features. This can be considered as another misalignment where our experiment results presents the suboptimal performance.

---

Table 1. Comparison of teacher-student modularization methods.

---

We will elaborate on this process and its impact in the revised manuscript to ensure clarity.

---

Does this address your concerns? Please let us know if further clarification or additional results are required. Thank you again for your valuable feedback.

---

---

Dear Reviewer,

Thank you for your valuable feedback. We appreciate your thorough examination and the insights you've provided, especially regarding the scalability of our proposed pipeline and its conceptual overlap with existing progressive internal knowledge distillation methods for LLMs. We would like to address these concerns and clarify the unique aspects of our work.

Response to Scalability Concerns:

1. Overview of Current Approach:
   Our pipeline, which includes steps such as PCA and Gram matrix computation for feature discernment, was designed to showcase the method's theoretical robustness and empirical performance. However, we understand the potential computational bottlenecks as network sizes and image resolutions scale up.

2. Scalability Mitigation Strategies:

   • Incremental PCA: Utilizing incremental or batch-based PCA significantly reduces memory and computational overhead.
   • Approximate Gram Matrix Computation: Techniques like the Nystrom method or random feature mapping can efficiently approximate the Gram matrix.
   • Sparse Feature Sampling: Selecting the most informative features for alignment minimizes dimensionality and computational cost.
   • Parallel Processing: GPU acceleration and parallel processing mitigate scalability concerns, enabling practical implementation for larger models and high-resolution inputs.

3. Offline Processing:
   The feature discernment process is performed offline. Once trained, the final distilled network is optimized for deployment, offering a smaller size while maintaining or enhancing performance. This ensures that computationally intensive steps do not impact real-time inference.

4. Architecture Agnosticism:
   Unlike the referenced work, which is specific to transformer architectures, our method is architecture-agnostic. This flexibility allows application to a variety of model types, broadening its utility across domains and architectures.

Response to Conceptual Overlap Concerns:

1. Progressive Concept as Part of a Novel Package:
   While progressive internal knowledge distillation has been explored in LLMs with similar layer-wise learning curricula, in our work, the progressive concept is part of a broader package that collectively defines the novelty of our approach. This includes:

   • Architecture-Agnostic Distillation: Our approach is flexible and applicable across diverse model types, not limited to specific architectures like transformers.
   • CKA-Based Modularization: Using CKA ensures effective module boundaries, enabling meaningful knowledge transfer across architectures.
   • Most Informative Knowledge Extraction: The method emphasizes extracting and aligning the most significant features from each module.
   • Progressive Module Training: Modules are trained incrementally to stabilize knowledge transfer and enhance alignment across the network.

2. Distinctive Contributions:

   • General Applicability: The method integrates techniques to address challenges in handling representations across different architectures.
   • Innovative Training: The training procedure and progressive alignment strategies are designed to align representations efficiently and incrementally.

3. Empirical Validation:
   Extensive experiments validate that our method improves performance over state-of-the-art techniques, demonstrating its efficacy and adaptability across various domains and architectures.

Does this address your concerns? Please let us know if further clarification or additional results are required. Thank you again for your insightful comments.

---

---

Dear Reviewer,

We appreciate your detailed comments and the opportunity to address them. These clarifications and revisions will significantly improve the clarity and presentation of our paper. Thank you for the constructive feedback.

---

1. Main Optimization Problem and Equations

We clearly state in lines 154–157 that Eqn (2) is the loss function, which governs the alignment between teacher and student models during training. Eqn (7) is not an optimization problem but a method to determine the best binary vector assignment for student modules. For each binary vector permutation (e.g., [0,1,1], [1,1,0]), we calculate the average distance between selected student modules. The permutation with the highest average distance is chosen, and the corresponding modules are used for distillation. For instance, if [1,1,0] is selected, modules 0 and 1 are trained, while module 2 is frozen.

This approach is necessary because the structural differences between teacher and student architectures prevent a direct one-to-one correspondence of modules. By maximizing average distances, we effectively determine the optimal mapping. These distinctions will be clarified in the revised manuscript.

---

2. PCA and Feature Selection

Our PCA-based feature computation is described in lines 208–258, Algorithm 1, and Figure 3(a). Unlike standard PCA for dimensionality reduction, we use PCA to identify the indices of the most informative features. Non-important features are zeroed out while retaining the original matrix size.

Eqn (3): The mean variance is computed over samples ($ns$) for each feature ($m$). This is sorted in descending order, and the top-$k$ indices are selected. Using these indices, the representation matrix $R$ is revisited, and the non-important features are zeroed out. This process is illustrated in Figure 3(a), starting from the bottom-left representation matrix.

We acknowledge and will correct a typo in Eqn (3). These clarifications will be added to ensure the process is clearly understood.

---

3. Solving Eqn (7)

Eqn (7) is solved through a cross-validation-like approach:

1. Binary vector permutations are applied to assign student modules.
2. For each permutation, the average distance between consecutive student modules is calculated.
3. The permutation yielding the highest average distance is chosen, and the corresponding modules are used for distillation.

This approach avoids the complexity of solving a discrete optimization problem while ensuring effective module selection. For example, given a student with three modules, possible binary vectors include [0,1,1], [1,1,0], and [1,1,1]. If [1,1,0] yields the highest average distance, student modules 0 and 1 are selected for training, while module 2 is frozen. These details will be clarified in the revised manuscript.

---

4. Direct Correspondence

Directly assigning teacher-to-student module correspondence assumes a clear one-to-one mapping, which is unavailable for heterogeneous architectures. Our algorithm determines this mapping by evaluating all permutations of module assignments and selecting the one that maximizes average distances. This method ensures that only the most representative student modules are used, enhancing distillation efficiency.

---

5. Relation to Wang et al. (2024)

Our method is fundamentally different from Wang et al. (2024), which operates in the RAG domain. In contrast, our approach focuses on:

• Architecture-Agnostic Design: Applicable across CNNs, ViTs, and hybrids.
• CKA-Based Modularization: Defines module boundaries to optimize transfer.
• Informative Feature Extraction: Prioritizes representative features for distillation.
• Progressive Alignment: Dynamically aligns teacher and student modules, outperforming static methods.

We also reference baselines like OFA in all tables and never claim to be the first to explore cross-architecture KD. These distinctions will be emphasized in the related work section.

---

6. PCA Parameters and Sensitivity

We use 25 random samples per class for PCA computation (line 412). With a variance threshold of $10^{-4}$, feature dimensions are typically reduced by ~70–80%. Sensitivity to the number of samples and variance threshold is analyzed in ablation study B5 in the Appendix. For example, using fewer samples or a higher threshold results in less precise feature selection and reduced performance.

These details will be included in the revised manuscript for clarity.

---

7. Appendix

The Appendix is part of the supplementary material. It can be accessed by clicking "Show Details" on the submission page, where a clickable PDF link is available in the last row.

---

Please let us know if there are any additional concerns or further points requiring clarification.

---

Dear Reviewer,

Thank you for your detailed feedback and for giving us the opportunity to clarify our work. Below, we provide responses to your specific questions and concerns:

---

Main Optimization Problem
We appreciate your comment on Eqn (2), which represents our loss function combining KL divergence and cross-entropy losses. This loss optimizes the student model during training for knowledge distillation from the teacher model.

Our method goes beyond this baseline by incorporating targeted alignment between teacher and student representations. Specifically, we use modularization (Section 3.3.1) and establish direct correspondences between teacher and student modules (Section 3.3.2, lines 208–257). This selective transfer outperforms methods like OFA by focusing on the most informative teacher modules.

Additionally, we monitor the similarity score between the teacher’s final representation and the student’s corresponding module outputs. Once a predefined threshold is met, training halts for that module (lines 337–339). This mechanism ensures efficient and effective knowledge transfer, contributing to the superior performance of our approach.

---

PCA and Feature Selection
Regarding Eqn (3), the mean variance (mv) is not a scalar but a vector corresponding to the number of features ($m$). Figure 3(a) visually represents this vector, showing the mean variance of features across data samples. This approach helps identify and utilize informative features effectively.

---

Solving Eqn (7)
Yes, solving Eqn (7) requires evaluating $2^n - 2$ permutations, excluding cases where no modules or all modules are trained. However, since the number of modules is typically small (e.g., 3–4), the computational complexity remains manageable. For instance, with four modules, binary permutations include configurations like $[0,0,1]$, $[0,1,0]$, $[1,0,0]$, $[1,0,1]$, and $[1,1,0]$. This ensures practical exploration of configurations.

---

Direct Correspondence
Early experiments applied a strict one-to-one correspondence between teacher and student modules, inspired by OFAKD. However, this approach was suboptimal compared to our proposed modular strategy.

Below is an ablation study comparing our method (PKD) with one-to-one mapping (PKD-1to1), where teacher modules were identified using CKA scores, and student modules matched their number. All other phases of PKD were kept consistent:

These results, showing the superiority of our modular strategy, will be included in the final manuscript.

---

Relation to Wang et al. (2024)
Thank you for pointing this out. Section 2 (lines 90–104) provides a discussion of related works, including:

• Logit-based distillation for ViT models (Touvron et al., 2021; Huang et al., 2018),
• Hint-based distillation (Romero et al., 2015),
• Attention map mimicking (Zagoruyko & Komodakis, 2017), and
• Feature-based methods for ViT models, such as ViTKD (Yang et al., 2022) and VKD (Miles et al., 2024), which introduced orthogonal projection and task-specific normalization.

To further clarify, we now include a summary of Wang et al. (2024):
"KD-DETR (Wang et al., 2024), designed for DETR-based object detection, tackles distillation point inconsistencies through shared and specialized queries, enabling effective distillation for both homogeneous (DETR-to-DETR) and heterogeneous (DETR-to-CNN) setups."

This addition ensures our work is contextualized appropriately and highlights our novel contributions in modularity and selective knowledge transfer.

---

We hope these responses address your concerns comprehensively. Please let us know if further clarifications are required. Thank you once again for your valuable feedback!