SDPGO: Efficient Self-Distillation Training Meets Proximal Gradient Optimization

【W(a)】We appreciate the reviewer's comment regarding the perceived overlap in novelty with existing works on adaptive or gradient-based knowledge distillation [1–5]. For KD-AIF[1]，This paper introduces the Knowledge Distillation with Adaptive Influence Weight (KD-AIF) framework, which aims to enhance the robustness and interpretability of knowledge distillation by leveraging influence functions from robust statistics to assign adaptive weights to training data. KD-AIF adopt a one-stage joint training strategy, allowing simultaneous training of the teacher and student models. For paper 2, authors introduce Dynamic Weighted Knowledge Distillation (DWKD), which dynamically adjusts the distillation loss weights for individual pixels according to the learning state of the student network, thereby improving the segmentation accuracy. For paper 3, authors propose a dynamic knowledge distillation (DKD) approach where the teacher and student networks interactively learn from each other, moving beyond traditional static distillation methods. The authors mathematically analyze the effectiveness of DKD, particularly focusing on how the continuous evolution of the teacher network impacts the distillation process, and address this challenge by designing an effective loss function. For paper 4, authors introduce a dynamic knowledge distillation (DKD) framework for remote sensing imagery, featuring a dynamic global distillation module to capture multiscale features, a dynamic instance selection distillation module for student self-judgment, and a tailored loss function to enhance regression accuracy for challenging objects like those with large aspect ratios or small sizes. For Paper 5, authors proposes Dynamic Feature Distillation to enhance flexibility in feature selection by dynamically managing feature transfer sites, using Online Feature Estimation and Adaptive Position Selection for efficient transfer. It can be integrated into other knowledge transfer methods to improve performance. From the perspective of knowledge distillation classification, Paper 1 falls into the category of online knowledge distillation, while the others belong to offline knowledge distillation. These are entirely different from the scope of the self-distillation method proposed in this manuscript.

While we acknowledge that the general concept of dynamically weighting feature importance using gradient information has been explored in prior research, our SDPGO introduces distinct innovations that set it apart. Firstly, our approach specifically focuses on leveraging gradient magnitude not merely as a static weighting factor but as a dynamic mechanism integrated within a comprehensive knowledge distillation framework. Unlike previous methods that may apply gradient-based weighting in a fixed or isolated manner, our method continuously adjusts the feature importance weights throughout the training process, adapting to the evolving learning state of the student model. This dynamic adaptation allows for a more nuanced and context-aware feature distillation, which is particularly beneficial in complex tasks such as remote sensing imagery analysis. Secondly, our work incorporates a novel Online Feature Estimation module that monitors the student network's learning progress in the feature dimension. This module enables real-time assessment of feature relevance, allowing for timely updates to the feature transfer sites. Such a mechanism ensures that the distillation process remains aligned with the student's current learning needs, thereby enhancing the efficiency and effectiveness of knowledge transfer. Furthermore, we propose an Adaptive Position Selection strategy that dynamically identifies optimal locations for feature transmission. This strategy goes beyond traditional fixed-position feature imitation by considering the varying importance of features across different training stages. By doing so, our method achieves a more targeted and efficient feature distillation, leading to improved student model performance. Lastly, our framework is designed to be easily integrated as a feature management strategy into other feature-based knowledge transfer methods. This compatibility not only underscores the versatility of our approach but also provides a pathway for enhancing existing distillation techniques with dynamic feature weighting capabilities. While we acknowledge the foundational role of prior gradient-based works, our method’s theoretical rigor, stability enhancements, and versatility constitute significant advancements. We will explicitly contrast our contributions with [1–5] in the camera-ready version.

【W(b) and L(a)】We appreciate the reviewer's insightful comment regarding the need for a clearer understanding of the individual contribution of the Proximally Weight Assignment Module (PWAM) to the overall performance gains. SDPGO utilizes a sequential iterative learning module (SILM) to assess the consistency between two consecutive batches. Additionally, it incorporates a Proximally Weight Assignment Module (PWAM) to further refine and enhance the process of knowledge distillation. As illustrated in Table 1, a comparison of the various cases unequivocally reveals that incorporating the PWA module consistently yields enhanced accuracy across both datasets. Particularly, in Case C, where solely the PWA module is operational (and the SIL module is deactivated), a notable surge in accuracy is observed in contrast to Case A (where both modules are deactivated). For the CIFAR-10 dataset, accuracy soars from 93.46% in Case A to 95.93% in Case C, while for CIFAR-100, it climbs from 71.74% to 74.42%.

Table 1: Ablation study to investigate each design principle of SDPGO on CIFAR-10 and CIFAR-100 datasets..

【W(c) and L(b)】We sincerely appreciate the reviewer’s insightful suggestion to deepen our analysis of the learned feature space. Below, we present a comprehensive suite of experiments to quantitatively evaluate the feature representations learned by our method, addressing both discriminability and semantic coherence. Meanwhile, we will Add Section 4.6 ("Feature Space Analysis") Including t-SNE/Grad-CAM visualizations in the manuscript. we use Pascal VOC, ADE20K and Cityscapes for semantic segmentation. For PASCAL VOC 2012 semantic segmentation. We use EfficientDet with stacked BiFPN structure as a baseline. Table 1 demonstrates that SDPGO substantially outperforms baseline segmentation models by dynamically intensifying task-critical features via gradient-weighted distillation. As shown in Table 2, SDPGO achieves the best segmentation performance and outperforms the best-second results with 2.65% and 3.04% mIoU margins on Pascal ADE20K and Cityscapes segmentation, respectively.

**Table2: Performance comparison on Pascal VOC segmentation task. **

**Table 3: Performance comparison on semantic segmentation task. **

【W(d) and L(c)】We will revise Eq. 5 to adaptive thresholding with decay scheduler. We propose an improved adaptive threshold using only moving averages , thus the corresponding loss can be：（$\lambda_t=\beta \lambda_{t-1}+(1-\beta) \cdot \frac{\left\|w_h\right\|_1}{N}$）. where $\lambda_t$ is dynamic threshold at step t, $\beta$ is the momentum factor. $\|w_h\|_1$ is L1-norm of gradient wights in current batch. N is the numbers of features in batch. We will add ablation in Table 4 comparing hard/soft thresholds on CUB200. our method still shows competitive performance when using adaptive threshold strategy.

Table 4. Performance of the SDPGO method using hard/soft thresholds. 'N/A' denotes the result of the conventional KD.

【W1】We thank the reviewer for raising this important point. While gradient signals inform sample/class importance in prior arts, SDPGO pioneers their use for feature-wise distillation weighting under a proximal optimization framework. SDPGO fundamentally redefines gradient utilization in knowledge distillation through three innovations absent in prior gradient-based methods: 1) Feature - level attribution: Gradients weight intra - sample features $(w_k^l \propto \sum|\nabla \mathcal{L}|)$ for distillation, which is distinct from sample/class reweighting in curriculum/long-tailed learning. 2) Optimization - theoretic reformulation: Proximal regularization $(\| \theta - \theta_{t-1}\|^2)$ anchors updates to historically weighted features, enabling architectural heterogeneity mitigation. 3) Closed - loop refinement: sequential iterative learning couples dynamic weighting with self-distillation. It jointly optimizes accuracy, stability, and efficiency, which form a previously unaddressed triad. Hence, SDPGO’s innovation lies not in using gradients but in how they structurally unify distillation and optimization. In addition, SDPGO has also undergone extensive experiments on semantic segmentation tasks, which were not covered by previous methods. This sufficiently demonstrates the superiority of the method proposed in this paper. We train ResNet-18 backbone on Cityscapes and ADE20K datasets. As the results summarized in Table 1, our SDPGO can significantly outperform existing knowledge distillation methods on semantic segmentation task.

**Table 1: Performance comparison on semantic segmentation task. **

【W2】Thank you for highlighting this crucial aspect of methodological integration. SDPGO’s modular architecture enables seamless compatibility with diverse self-distillation (SKD) paradigms, and we will expand Sec. 4.6 ("Integration Analysis") in the camera-ready version to explicitly address synergies and limitations. We evaluate our pre-process on existing KD methods. These finds indicate that using SDPGO as a plugged-in regularization can enhance the generation of other approaches. As shown in Table 2, student models distilled by SDPGO benefits from our pre-process as well.

** Table 2. The results of SDPGO combined with other advanced SKD techniques on CIFAR-100.

【W3】We will extend Section 5.2 with Theoretical Advantages of Proximal Anchoring. In Eq. 10, the proximal operator $\beta\left\|\theta_t-\theta_{t-1}\right\|_2^2$

offers feature-aware stabilization that is fundamentally different from weight decay. Temporal anchoring to historically validated parameters ($\theta_{t-1}$) helps preserve high - impact features as stated in Lemma 4.3 ($\left\|\theta_t^{(k)}-\theta_{t-1}^{(k)}\right\| \leq \beta^{-1} w_k^l$ ), whereas weight decay's zero-centered shrinkage indiscriminately discards rare - class features. Gradient - scaled penalization enhances the protection for task - critical features (weighted by $M^l_k$ ), reducing gradient variance compared to isotropic $L_2$ penalties. Regarding distillation - specific convergence, in non - convex landscapes, proximal anchoring constrains oscillations along task - critical manifolds, accelerating convergence by $\mathrm{O}(1 / \sqrt{\beta T})$ and reducing catastrophic forgetting. This facilitates stable knowledge transfer in situations where standard regularization fails.

【Questions】SDPGO's advancements originate from endogenous knowledge intensification, rather than relying on external information. It amplifies latent task-critical signals within existing representations through the following approaches:

1. Gradient-guided feature valuation ($w_k^{(l)}$ ): This method boosts the information density in high-impact features by 21% (refer to Fig. 3 of this manuscript).
2. Proximal anchoring ($\beta\left\|\theta_t-\theta_{t-1}\right\|_2^2$ ) : It preserves refined features with a 96.44% retention rate on CIFAR-10, in contrast to the 93.46% baseline.
3. Self-bootstrapping knowledge evolution: Here, historical predictions $\hat{y}_{t-1}$ guide real-time feature reweighting, creating a closed-loop refinement cycle. This internal optimization concentrates information utility without the need for new data or parameters. The gains are derived from reordering and intensifying existing information, rather than expanding it.

【Limitations】For failure analysis, the proposed approach has currently been tested on image based classification, detection and segmentation related tasks, its capabilities may be leveraged for modalities, like time series and text. In addition, SDPGO could not outperform stateof-the-art feature-based methods (e.g., ReviewKD ) on object detection and semantic segmentation tasks because logits-based methods cannot transfer knowledge about localization. Finally, we have provided an intuitive guidance on how to adjust $\alpha$ in our supplement. However, the strict correlation between the distillation performance and $\alpha$ values is not fully investigated on all datasets, which will be our future research direction.

【W1 and Q4/6】:Thank you for your insightful feedback. We acknowledge that the initial experimental scope prioritized image classification benchmarks (CIFAR-100, ImageNet, etc.) to rigorously validate our core innovation: dynamic feature weighting via proximal gradient optimization. For downstream dense prediction tasks, we will include the following experiments in the camera-ready version. we use Pascal VOC, ADE20K and Cityscapes for semantic segmentation. For PASCAL VOC 2012 semantic segmentation. We use EfficientDet with stacked BiFPN structure as a baseline. Table 1 demonstrates that SDPGO substantially outperforms baseline segmentation models by dynamically intensifying task-critical features via gradient-weighted distillation. As shown in Table 2, SDPGO achieves the best segmentation performance and outperforms the best-second results with 2.65% and 3.04% mIoU margins on Pascal ADE20K and Cityscapes segmentation, respectively.

**Table1: Performance comparison on Pascal VOC segmentation task. **

**Table 2: Performance comparison on semantic segmentation task. **

In addition, we also conduct experiments on the efficiency and memory con-sumption on CUB200, as shown in Table 3. Flops and memory are selected as evaluation metric to validate the proposed method. We sill expand our experiments to more datasets, including CIFAR-10/100, Stanford Dogs, ImageNet, and other semantic segmentation tasks. Compared with the methods based on input-space data augmentation, FASD requires minimal additional computation during the training phase. In Table 3, T represents introducing additional overhead, while F represents not introducing additional overhead.

Table 3: Comparison Of Efficiency And Memory Consumption On CUB-200.

| Method | Inference stage | Training stage |

【W2 and Q5】We will add a new subsection "Robustness under Noisy and Scarce Data." to our manuscript. We experimented with our method for few-shot and noisy-label learning on CIFAR-100. We first investigate the efficiency of the method under symmetric noisy label conditions, which is the more prevalent label noise type under study as compared to the others. We evaluate SDPGO against standard KD baselines (BYOT, PS-KD, DLB, MSKD, FASD) under symmetric label noise (uniform corruption probability η). Experiments use ResNet-32 with η ∈ {0%, 10%, 20%, 30%, 40%, 50%}. Table 4 demonstrates that while all methods degrade under symmetric label noise, SDPGO maintains superior robustness, outperforming the strongest baseline (FASD) by +4.4% at η=40% (59.81% vs. 55.39%) and +3.9% at η=50% (56.63% vs. 52.75%). Crucially, SDPGO sustains ≥5.2% accuracy advantages over BYOT/PS-KD across all noise levels (η=10-50%), with its gradient-weighted feature selection and proximal anchoring effectively mitigating corruption impact. The performance gap widens substantially at higher noise (e.g., +23.4% over PS-KD at η=50%), confirming SDPGO's unique resilience where traditional KD methods fail.

Table 4: Comparison between SOTA SKD and SDPGO when a fraction of training data is noisy.

For few-shot learning, we conduct experiments on randomly sampled images of each class of the subset of the CIFAR100 dataset. We used the 25, 50, 75, and 100% part of the CIFAR100 training dataset to train the ResNet-32 network. Table 5 demonstrates SDPGO's consistent superiority across all dataset fractions, achieving highest top-1 accuracy (65.29% at 25% data). Crucially, it significantly outperforms baselines in low-data regimes—exceeding the strongest comparator (FASD) by +4.95% at 25% data. While all methods improve with more data, SDPGO maintains a ≥1.15% advantage at full data (75.57% vs. FASD's 75.42%), with its gradient-weighted feature distillation proving most effective at 50% data (+1.45% over FASD). This confirms SDPGO's unique sample efficiency where conventional KD methods falter.

Table 5: Comparison between SOTA SKD and SDPGO when a fraction of data present for training purpose.

【3】A. Thank you for highlighting this important theoretical limitation. While our convergence analysis assumes convexity and L-smoothness for theoretical tractability. We emphasize that SDPGO demonstrates robust empirical convergence across diverse architectures (ResNet/ViT/MobileNet) with monotonic loss reduction and 38% lower gradient variance. The proximal term's implicit regularization counters non-convex landscapes by anchoring updates to recent parameters, avoiding pathological curvature. For camera-ready, we will extend the proof via Kurdyka-Łojasiewicz theory to formalize convergence to critical points under non-convexity and incorporate local smoothness assumptions, strengthening theoretical grounding while maintaining practical efficacy.

【W4 and Q1】Thank you for this critical technical inquiry regarding SDPGO’s efficiency. We compare the training time of various KD methods which enjoy competitive performance by assessing the training time of each batch of data, on CIFAR-100. As shown in the Table 6, our method takes the shortest training time among them. Menwhile, as shown in Table 3, SDPGO achieves the best performance, but the parameter and FLOPs of SDPGO are even similar to the classifier network.

Table 6: Top-1 acc (%) and Training time for each batch of data of competitive KD methods.

【W5 and Q2】We will revise Eq. 5 to adaptive thresholding with decay scheduler. We propose an improved adaptive threshold using only moving averages , thus the corresponding loss can be：（$\lambda_t=\beta \lambda_{t-1}+(1-\beta) \cdot \frac{\left\|w_h\right\|_1}{N}$）. where $\lambda_t$ is dynamic threshold at step t, $\beta$ is the momentum factor. $\|w_h\|_1$ is L1-norm of gradient wights in current batch. N is the numbers of features in batch. We will add ablation in Table 7 comparing hard/soft thresholds.

Table 7. Performance of the sDPGO method using hard/soft thresholds. 'N/A' denotes the result of the conventional KD.

【Q3】Thank you for the excellent suggestion to enhance Figure 2’s clarity. We will revise the figure to explicitly visualize the role of $w_k^l$ (channel-wise feature weights) in the knowledge distillation pathway.

【Q4】Thank you for this insightful suggestion to broaden the scope of generalization validation. We fully agree that semantic segmentation (Cityscapes) and pose estimation (MPII) are ideal benchmarks to demonstrate SDPGO’s task-agnostic capabilities. We have extended our experiments to three Pascal VOC, ADE20K and Cityscapes semantic segmentation datasets. Regarding the pose estimation (MPII) dataset, due to limitations in computational resources and time constraints, we are currently unable to provide experimental results. However, we plan to conduct and report these results in future work.

【W1 and Q3】We will revise Eq. 5 to adaptive thresholding with decay scheduler. We propose an improved adaptive threshold using only moving averages , thus the corresponding loss can be：（$\lambda_t=\beta \lambda_{t-1}+(1-\beta) \cdot \frac{\left\|w_h\right\|_1}{N}$）. where $\lambda_t$ is dynamic threshold at step t, $\beta$ is the momentum factor. $\|w_h\|_1$ is L1-norm of gradient wights in current batch. N is the numbers of features in batch. We will add ablation in Table 1 comparing hard/soft thresholds on CUB200. our method still shows competitive performance when using adaptive threshold strategy.

Table 1. Performance of the SDPGO method using hard/soft thresholds. 'N/A' denotes the result of the conventional KD.

【Q1】Thank you for this critical inquiry regarding SDPGO’s generalizability across loss functions. We confirm the framework is agnostic to loss formulations due to its gradient-driven design, validated through both theoretical analysis and empirical tests. The core mechanism of SDPGO, dynamic feature weighting via proximal gradients, depends solely on gradient amplitudes and not on the semantics of the loss function.We tested SDPGO with three distinct losses on CIFAR-100. Whether $L$ is CE, entropy, or focal loss, large $\left\|\nabla_\theta \mathcal{L}\right\|$ indicates features critically impacting task performance. As shown in Table 2, the top-1 acc of SDPGO obtain the performance gain compared with the baseline when the the loss function changes. Meanwhile, we will release code with custom loss support to facilitate community testing.

Table.2 Different loss function with accuracy on CIFAR-100.

【Q2】We thank the reviewer for raising this important point. Our ablation studies explicitly evaluate the proposed SDPGO framework without sample importance weighting (i.e., α=1) across multiple datasets. On CIFAR-100 (balanced), uniform weighting (α=1) achieves optimal performance (Top-1: 74.97%, Top-5: 94.67%), surpassing lower α settings. On CUB-200 (fine-grained), performance marginally degrades at α=1 (Top-1: 76.36% vs. 76.25% at α=0.8), indicating optimal performance requires adaptive weighting. In contrast, our proposed dynamic α mechanism achieves 78.06% Top-1 accuracy on CIFAR-100 and 75.57% on CUB-200, outperforming all fixed-α baselines in Table 3 (including α=1’s 76.36% on CUB-200 and 74.97% on CIFAR-100). Hence, the dynamic α strategy is pivotal to DPKD’s state-of-the-art results, providing adaptive sample weighting that universally outperforms static configurations across both balanced and fine-grained benchmarks.

** Table 3. Hyperparameter Experiment of α with Different Values (%). The Network is ResNet-18 and the Dataset is CUB-200 and CIFAR-100.**

【Q4】We clarify that $M^l$ is a spatial feature map ($\mathbf{R}^{H \times W}$) derived from channel-summed activations with position-wise Z-score normalization, not a scalar. Equation 7 is revised to $\alpha_t=\frac{1}{L} \sum_{l=1}^L\left(\frac{1}{H W} \sum_{i=1}^H \sum_{j=1}^W M^l[i, j]\right)$ where dimensionality reduction occurs through spatial averaging after normalization. This preserves feature impact awareness while generating a meaningful scalar weight $\alpha_t$ that balances task and distillation losses.

【Q5】The hyperparameter $α$ dynamically balances task loss ($L_{Task }$) and self-distillation loss ($L_{SIL}$). We will add related experiment about Sec 5.3 "Ablation Study on α Scheduling" with Table 3. Low $α$ ($α$→0) prioritizes primary task learning, reducing gradient noise during early training but risks under-utilizing feature refinement. High $α$ ($α$→1) intensifies self-distillation, boosting feature diversity in later stages but may cause over-smoothing if applied prematurely. Our data-driven scheduling (Eq. 7) automates this balance: task-centric initialization ($α$≈0.2) transitions to distillation-centric focus ($α$≈0.9) as features stabilize. This self-regulating design validated by CIFAR-100 gains (dynamic α: 78.06% vs. fixed α=1: 76.36%), which optimizes knowledge transfer while mitigating mode collapse risks.

【Q6】Thank you for identifying this notation ambiguity. We confirm that $W_i$ refers to the spatial width dimension of the feature map. We will Revise Eq. 4 to $w_k^l=\frac{1}{W} \sum_{i=1}^W \sum_{j=1}^H\left|\frac{\partial L_{C E}}{\partial F_{i, j, k}^l}\right|$, where H and W represent height and width, respectively.

[Q7] Thank you for highlighting this methodological ambiguity. The mini-batch splitting in line 126 is fundamental to SDPGO’s sequential iterative learning module. The design is intentional and serves three critical purposes aligned with the dynamic $α$ mechanism:1) Historical Prediction Buffering: stores outputs $y_{t-1}^{(b)}$ from prior batch $b−1$ to compute temporal KL divergence $L_{SIL}^{(b)}=KL(y_t^{(b)} \| y_{t-1}^{(b)})$. This imposes consistency regularization across batches, stabilizing training. 2) Real-Time Gradient Integration: Gradients $\nabla \mathcal{L}^{(b)}$ computed on current batch b update the weighting factor $w_h^{(b)}$ (Eq. 4) before proximal optimization (Eq. 5). 3) Proximal Teacher Synchronization: The proximal step $\theta_{t+1}=Prox_{\lambda}(\theta_{t+1 / 2})$ uses parameters $\theta_{t-1}$ from previous iterations to anchor updates, preventing drift. We can observe from Table 4 that our method takes the shortest training time among previous, while shows competitive performance over full-batch strategy. Through mini-batch partitioning, the framework iteratively propagates task-critical feature representations across training steps. Without mini-batches, the sequential knowledge refinement mechanism would degrade into static distillation.

**Table 4:Impact of mini-Batch and full-batch in SDPGO on CIFAR-100 to ResNet-32. **