Rethinking Self-Distillation: Label Averaging and Enhanced Soft Label Refinement with Partial Labels

W4. When comparing existing PLL methods, the authors claim, "Our method differs by directly employing the refined partial labels derived from the teacher’s outputs, achieving the same benefits as multi-round distillation in just one round." Does this imply that other methods incorporating the teacher's output could achieve similar or even better results? Are there corresponding experiments to support this?

Partial Label Learning (PLL) is a type of weakly supervised learning where training instances are annotated with a set of candidate labels rather than a single ground-truth label. The goal of PLL is to train a model capable of predicting the ground-truth label for unseen data using this partially labeled dataset. Some existing PLL methods incorporate a teacher-student framework, leveraging the teacher model to refine or guide the candidate label distribution to improve the student model's ability to generalize [1, 2, 3, 4, 5, 6, 7]. For instance, the teacher model may provide soft pseudo-labels or confidence scores to weight the candidate labels, enabling the student model to focus more on the most plausible label. This interaction helps mitigate ambiguity in candidate sets by progressively narrowing down the label distribution during training, ultimately enhancing the accuracy and robustness of the PLL framework.

Our method differs from these existing PLL approaches in a key aspect. While traditional PLL studies focus on utilizing the given partially labeled dataset to train a model that predicts the ground-truth label precisely, our research emphasizes constructing the candidate label set itself from the teacher’s output, particularly in the context of noisy supervised training. We demonstrate that a noisy supervised training problem can be reformulated as a PLL problem by leveraging a pretrained feature extractor and linear probing.

In our experiments (as shown in Figure 7 in Appendix D.2), we observe that features extracted by a pretrained ResNet34 exhibit high feature correlation among instances of the same class and low feature correlation between instances of different classes. In Section 3, we show that the principle behind self-distillation lies in prediction averaging based on feature correlation. Thus, even for noisy instances—where the ground-truth label differs from the given label—the outputs are influenced by the average predictions of other instances within the same ground-truth class. This effect causes the output probability at the ground-truth label position to become larger, significantly increasing the likelihood that the ground-truth label is included in the size-2 candidate sets derived from the teacher's outputs.

By directly employing the refined partial labels from the teacher's outputs, our method effectively achieves the benefits of multi-round self-distillation in just one round. This approach is specifically designed to correct label noise by constructing candidate label sets that are more likely to include the true label, without relying on additional PLL techniques that adjust the candidate labels during training.

Regarding the reviewer’s question, while other PLL methods incorporating the teacher's output aim to improve performance by refining the candidate label distribution, they typically focus on disambiguating given candidate labels rather than constructing the candidate set itself from the teacher's predictions. Our approach is unique in that it leverages the teacher's outputs to build the candidate label sets, directly addressing the issue of label noise in supervised learning.

We did not conduct experiments comparing our method with other PLL approaches that utilize the teacher's output because the objectives and problem settings differ. Our research contributes a novel perspective by demonstrating how to construct candidate label sets from the teacher's outputs to enhance robustness to label noise in linear probing with self-distillation.

[1] Xia, Shiyu, et al. "Towards effective visual representations for partial-label learning." CVPR 2023.

[2] Li, Beibei, et al. "AsyCo: An Asymmetric Dual-task Co-training Model for Partial-label Learning." arXiv preprint, 2024.

[3] Xu, Ning, et al. "Aligned Objective for Soft-Pseudo-Label Generation in Supervised Learning." ICML 2024.

[4] Wang, Guangtai, et al. "Dealing with partial labels by knowledge distillation." Pattern Recognition, 2025.

[5] Wang, Haobo, et al. "Pico: Contrastive label disambiguation for partial label learning." ICLR 2022.

[6] Wu, Dong-Dong, Deng-Bao Wang, and Min-Ling Zhang. "Distilling Reliable Knowledge for Instance-Dependent Partial Label Learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 14. 2024.

[7] Li, Wei, et al. "Generalized Contrastive Partial Label Learning for Cross-Subject EEG-Based Emotion Recognition." IEEE Transactions on Instrumentation and Measurement, 2024.

W3. The experiments use ResNet34 as the backbone for linear probe experiments; however, training ResNet34 with all parameters is not particularly time-consuming or resource-intensive. Should a backbone with a larger parameter count be chosen for the experiments?

Thank the reviewer for the insightful suggestion. We acknowledge that ResNet34 is not a particularly large model, and training it with all parameters is not highly resource-intensive. However, we chose ResNet34 for our experiments to effectively observe the progressive gains of multi-round self-distillation.

To address the reviewer’s concern, we also conducted additional experiments using a larger backbone—a pretrained ViT-B (Vision Transformer) model—as a fixed feature extractor. The test accuracies (%) on the CIFAR-100 dataset across different label corruption rates are presented in the table below:

Using a larger backbone like ViT-B enhances feature extraction capabilities, resulting in a greater disparity between intra-class and inter-class feature correlations. For example, when calculating feature correlations on the CIFAR-100 dataset using the pretrained ViT-B model, we observed that the average intra-class feature correlation increased to 0.35, compared to 0.25 with ResNet34. This higher intra-class correlation amplifies the clustering effect of self-distillation on model predictions, allowing significant performance improvements to be achieved with fewer distillation steps, as implied by our Theorem 4.1.

In experiments with the larger backbone (ViT-B), we found that most of the distillation gains occur within the first few rounds. As shown in the table above, nearly all performance improvements are observed in the first and second distillation steps when using ViT-B, although additional distillation steps still bring slight gains in high noise rate regimes. The PLL student model also effectively achieves the gains of multi-round self-distillation in a single round in high label-noise regimes for the ViT-B backbone. These additional experiments confirm that our approach is effective with larger models, further validating the versatility and robustness of our method.

We sincerely thank the reviewer for the constructive feedback. Please see our responses to the reviewer’s questions.

W1. What are the clear insights or significance of Sections 2.2, 3, and 4 in relation to the proposed method in Section 5? Could the authors summarize the roles and conclusions of Sections 2.2, 3, and 4, and their implications for Section 5? These sections are not sufficiently clear.

W2. In Section 5, the authors state, "selecting the top two labels with the highest values and assigning a weight of 1/2 to each, setting all the other entries to zero." What is the rationale behind this approach?

Sections 2.2, 3, and 4 are crucial for understanding the development and significance of the proposed method in Section 5. We introduced the PLL (Partial Label Learning) student model in Section 5 based on the intuition gained from our closed-form analysis of multi-round self-distillation effects in Section 3 and the resulting robustness to label noise observed in Section 4. Specifically, the PLL student model is designed to replicate the benefits of multi-round self-distillation—particularly its robustness to label noise—in a single round.

In Section 3, we analyzed the effect of multi-round self-distillation and provided a quantified closed-form solution for the output of the $t$-th student model. This analysis revealed that multi-round self-distillation induces label averaging effects among instances with high feature correlations when generating predictions on the training data. This means that as the number of distillation rounds increases, the model's output predictions gradually form clusters corresponding to each ground-truth label based on input feature correlations. This clustering effect is illustrated in Figure 2, where the outputs become more similar for instances sharing the same true label as $t$ increases.

In Section 4, we explored how this clustering effect enhances robustness to label noise. The label averaging makes the model's predictions less overfitted to the given (possibly noisy) labels and more influenced by the average labels among highly correlated instances. Recognizing that each distillation round gradually increases the weight on the average labels while decreasing the weight on the individual labels, we analyzed how many distillation rounds are sufficient for the student model to make correct predictions for all training instances, including those with noisy labels. This is summarized in Theorem 4.1, which captures the relationship between label corruption rates and the effectiveness of the label averaging process over multiple distillation rounds.

Building on these insights, the PLL student model proposed in Section 5 aims to replicate the behavior of multi-round self-distillation through refinement of the teacher's output and a single step of self-distillation. The label averaging effect from multi-round self-distillation shifts the softmax output away from a one-hot vector of the given label toward the average labels among instances sharing the same ground-truth label. For clean samples, this averaging effect slightly reduces the confidence in the correct label but maintains the correct prediction as $t$ increases. For noisy samples, it decreases the confidence in the incorrect (noisy) label and moves the prediction toward the correct label.

To facilitate this process efficiently, our PLL student model refines the teacher's output by selecting the top two labels with the highest values, assigning a weight of 1/2 to each, and setting all other entries to zero. Under mild conditions on the label corruption matrix—as stated in Theorem 5.1—we show that the top-two list obtained from the teacher's output always includes the ground-truth label: it ranks first for clean samples and second for noisy samples. Assigning equal weights to the top two labels balances the influence of the teacher's confidence and mitigates overconfidence in potentially incorrect labels due to noise.

Using these refined targets (two-hot vectors), a single round of self-distillation—through label averaging among highly correlated inputs—ensures that the model's softmax outputs assign the highest probability to the ground-truth label for all training instances, achieving 100% accuracy. This is possible because the average of the two-hot vectors from instances of the same ground-truth class has its highest value at the ground-truth label, since the two-hot vectors always include the true label for both clean and noisy samples.

Thus, the PLL student model effectively replicates the advantages of multi-round self-distillation in just a single round. This approach not only enhances computational efficiency but also maintains or improves performance, especially in high label-noise regimes.

We sincerely thank the reviewer for the constructive feedback. Please see our responses to the reviewer’s questions.

W1. In the experiments, Generalized Cross Entropy loss is applied for PLL which differs from the setting of CE loss in the main theory. The authors need to explain whether applying GCE loss is fair for comparison or provide experimental results of CE loss for PLL.

Thank the reviewer for the insightful comment. We clarify our rationale for using the Generalized Cross Entropy (GCE) loss in our experiments with the PLL student model, and provide additional experimental results using the Cross Entropy (CE) loss below.

The GCE loss is designed to balance the trade-off between the robustness of the Mean Absolute Error (MAE) loss and the fast convergence of the CE loss. In a $K$-class classification problem, for a given data pair $(\boldsymbol{x}\_i, y\_i)$, the CE loss $\mathcal{L}\_\mathsf{CE}$ and the MAE loss $\mathcal{L}\_\mathsf{MAE}$ are defined as

\mathcal{L}\_\mathsf{CE}(\mathbf{e}(y\_i), f(\boldsymbol{x}\_i;\theta)) = -\sum\_{k=1}^K [\mathbf{e}(y\_i)]\_k \log[f(\boldsymbol{x}\_i;\theta)]\_k, \quad \mathcal{L}\_\mathsf{MAE}(\mathbf{e}(y\_i), f(\boldsymbol{x}\_i;\theta)) = \sum\_{k=1}^K |[\mathbf{e}(y_i)]\_k - [f(\boldsymbol{x}\_i;\theta)]\_k|. $$ The gradients for each loss are given as

\nabla_\theta \mathcal{L}_\mathsf{CE}(\mathbf{e}(y_i), f(\boldsymbol{x}_i;\theta)) = -\frac{1}{[f(\boldsymbol{x}_i;\theta)]_{y_i}} \nabla_\theta [f(\boldsymbol{x}_i;\theta)]_{y_i}, \quad \nabla_\theta \mathcal{L}_\mathsf{MAE}(\mathbf{e}(y_i), f(\boldsymbol{x}_i;\theta)) = -2\nabla_\theta [f(\boldsymbol{x}_i;\theta)]_{y_i}.

The GCE loss introduces a hyperparameter $q$ to interpolate between CE and MAE losses: $$\mathcal{L}\_\mathsf{GCE}(\mathbf{e}(y\_i), f(\boldsymbol{x}\_i;\theta)) = \frac{1 - ([f(\boldsymbol{x}\_i;\theta)]\_{y_i})^q}{q}, $$with gradient:

\nabla_\theta \mathcal{L}_\mathsf{GCE}(\mathbf{e}(y_i), f(\boldsymbol{x}_i;\theta)) = -[f(\boldsymbol{x}_i;\theta)]_{y_i}^{q-1} \nabla_\theta [f(\boldsymbol{x}_i;\theta)]_{y_i}. $$By adjusting $q$, GCE loss balances robustness to label noise and convergence speed.

In our experiments, we observed that using CE loss with the PLL student model often led to instability during training. The PLL student model trains with a set of candidate labels for each sample with equal weights—in our case, the top two labels with weights of $1/2$ each. Using CE loss with equally weighted candidate labels can cause instability because the model may converge incorrectly when the candidate set includes incorrect labels.

Various PLL approaches address this instability by refining the candidate label set during training or adjusting label weights. However, our focus is to demonstrate the effectiveness of one-step self-distillation using PLL without incorporating additional PLL techniques. Therefore, we utilized the GCE loss, which behaves similarly to the CE loss but offers greater stability during training.

We also conducted additional experiments using CE loss with the PLL student model. The test accuracies on the CIFAR-100 dataset under varying label corruption rates ($\eta$) are presented below (corresponding to Figure 4 and Table 5 in the main text):

Across three repeated experiments, the PLL student model still generally outperforms the multi-round self-distillation model in high corruption regimes ($\eta\geq 0.6$), even with CE loss. However, we observed greater variability and occasional drops in accuracy when using CE loss. GCE loss provided more consistent and stable performance across different corruption rates. We will include these explanations in the revised manuscript to clarify our experimental setup.

W2. The proposed theory assumes a fixed feature extractor and therefore cannot be applied to trainable feature extractors, which are more commonly used in task adaptation and transfer learning.

While our main analysis assumes a fixed feature extractor and focuses on training a linear classifier on top of it, our theory can be generalized to scenarios where the feature extractor is updated during self-distillation, as we explain in Appendix C. Our framework decouples the gains from self-distillation into two components: feature learning (which modifies the feature map assumed in Equation (7)) and feature selection (which occurs during training the classifier, i.e., linear probing).

To illustrate this point, consider the quantified gains of self-distillation in label-noise scenarios presented in Theorem 4.1. This theorem provides the sufficient number of distillation rounds required for the student's softmax output to assign the highest value to the ground-truth label for both clean and noisy samples. The condition depends not only on the class-wise label corruption rates but also on the relative gap in feature correlations between samples of the same ground-truth class, parameterized by $c$, and those of different ground-truth classes, parameterized by $d$.

We can generalize our theory by allowing these correlations to evolve over distillation rounds, denoted by $c^{(i)}$ and $d^{(i)}$, due to feature updates during self-distillation. Recent work by Allen-Zhu and Li (2022) demonstrates that self-distillation's effectiveness arises from an implicit ensemble of teacher and student models, enabling the student to learn more diverse features when using the teacher's softmax outputs as targets. This suggests that the intra-class feature correlation $c^{(i)}$ may increase with each distillation round $i$, enhancing the separation between classes.

Assuming the class-wise feature map defined in Equation (7), and allowing $c$ and $d$ to change over distillation steps, our parameters $p$ and $q$ in Equation (14), which govern the label averaging effect, also become functions of $i$: $$ p^{(i)}:=(1-c^{(i)})/(K^2 n\lambda+1-c^{(i)}); \quad q^{(i)}:=(1-c^{(i)}+n(c^{(i)}-d^{(i)}))/(K^2 n\lambda+1-c^{(i)}+n(c^{(i)}-d^{(i)})).

Under the extended class-wise feature correlation assumption, our Theorem 4.1 can be generalized to: **Theorem C.1 (extended version.)** *Under the evolving feature correlation model, the $t$-th distilled model achieves 100% population accuracy if* $$[\mathbf{C}]\_{k, k}>[\mathbf{C}]\_{k, k'}+\frac{1}{\prod\_{i=1}^t ({q^{(i)}}/{p^{(i)}})- 1},\quad\forall k,k'(\neq k)\in[K].$$ If the student model learns more diverse features than the teacher, resulting in an increase of $c^{(i)}$ over distillation rounds, the ratio $q^{(i)}/p^{(i)}$ also increases. This makes it easier for the student model to meet the condition for achieving 100% population accuracy. Consequently, the regime where the student model achieves perfect accuracy expands, leading to performance gains from self-distillation. In summary, while our primary focus was on understanding self-distillation's benefits with a fixed feature extractor, our analysis can be extended to incorporate feature learning dynamics. By integrating our findings with existing feature learning approaches, we demonstrate that self-distillation enhances performance through both label averaging and the evolution of feature representations. Our extended theory quantifies the gains from both aspects, as shown in Theorem C.1. This extended analysis is detailed in Appendix C of our manuscript. [1] Allen-Zhu, Z. and Li, Y. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. In The Eleventh International Conference on Learning Representations, 2022.

We sincerely thank the reviewer for the constructive feedback. Please see our responses to the reviewer’s questions.

W1. This paper focuses on self-distillation with linear probing and provides a theoretical analysis in this context. I believe that both self-distillation and linear probing are valuable techniques, but I am unclear about the purpose of combining self-distillation with linear probing. As far as I know, linear probing is widely used in self-supervised learning as a method to evaluate learned features. Why should we combine linear probing with self-distillation, especially in scenarios involving label noise?

Combining self-distillation with linear probing is particularly beneficial when adapting large pre-trained neural networks (foundation models) to downstream tasks. Fine-tuning an entire pre-trained model is computationally intensive and may risk overfitting, especially with limited labeled data or noisy labels. Linear probing, which trains only a linear classifier on top of a fixed feature extractor, offers a more efficient way to utilize the rich features of pre-trained models. While linear probing is efficient, the linear classifier can still overfit to noisy or corrupted labels. Self-distillation improves linear probing by reducing overfitting and enhancing generalization. By training the linear classifier over multiple rounds, self-distillation shifts the model's focus from fitting noisy labels to leveraging feature correlations among instances. This process helps the classifier make more robust predictions based on the underlying data structure rather than the potentially noisy labels.

We also propose a new self-distillation scheme that achieves these benefits in a single round using partial labels from the teacher's top predictions. This approach further reduces computational complexity while maintaining or even improving performance. It is especially useful when computational resources are limited or when quick adaptation to new tasks is required without extensive retraining.

In summary, combining self-distillation with linear probing enhances the effectiveness of linear probing by improving generalization and robustness, especially in domains with noisy labels (e.g., crowdsourced data). This combination is valuable in applications where computational efficiency and model robustness are critical.

Q2. Could you explain the intuition behind the condition (Eq. 10) in Theorem 4.1? At first glance, it seems almost implausible to reduce training losses to zero in the presence of label noise, even with multiple rounds of self-distillation.

(Eq. 10) in Theorem 4.1 provides a condition under which the $t$-th self-distilled model can achieve 100% population accuracy, even in the presence of label noise in the training dataset. The key intuition behind this condition lies in the label averaging effect of multi-round self-distillation.

Each round of self-distillation causes the model's output predictions to gradually shift away from the one-hot encoded given labels—which may be noisy—toward the average label vectors of instances with high feature correlations (i.e., instances from the same ground-truth class). This process is described in (Eq. 9) of Theorem 3.1. Essentially, at each distillation round, the student model is trained to fit new targets that increasingly reflect the collective information from highly correlated instances, rather than fitting the noisy labels. As the number of distillation rounds $t$ increases, the influence of individual noisy labels diminishes, and the student model relies more on the averaged labels derived from other instances of the same class. This shift reduces the impact of label noise because the averaging process amplifies the true signal (correct labels) while diluting the noise (incorrect labels).

(Eq. 10) specifies the sufficient number of distillation rounds $t$ needed for this label averaging effect to ensure that the model's softmax outputs assign the highest probability to the ground-truth label for all training instances—including those with noisy labels. To understand the condition in (Eq. 10) more intuitively, consider (Eq. 13), which shows that the output prediction for a particular training instance can be expressed as a weighted combination of: 1) The given (possibly noisy) label, 2) The average label vector among instances from the same ground-truth class, 3) The average label vector among instances from the same superclass, 4) The uniform distribution vector.

The weights assigned to these components depend on the number of distillation rounds $t$ and shift away from the individual noisy label toward the average label as $t$ increases. As long as the proportion of correctly labeled instances in each class is higher than the proportion of mislabeled ones (i.e., $[C]\_{k,k}>[C]\_{k,k’}$ for all $k\neq k’$), the average label vector will have its highest value at the ground-truth label position. Therefore, after sufficient distillation rounds, the model's predictions will correctly identify the ground-truth label, even for samples that were initially mislabeled.

In summary, the condition in (Eq. 10) captures the relationship between the label corruption rates and the effectiveness of the label averaging process over multiple distillation rounds. It provides the minimum number of rounds needed for the model to overcome label noise by leveraging the collective information from correlated instances, ultimately achieving perfect accuracy despite the presence of noisy labels. We will add this high-level intuition in the main text.

Q3. It seems that the terms “true label” and “given label” are not properly defined. As I understand it, the “given label” refers to the target label, which may include noise, while the “true label” represents the oracle.

In our paper, the true label $y(\mathbf{x})$ refers to the ground-truth class of each sample $\mathbf{x}$. This represents the actual class to which the sample inherently belongs, as determined by the underlying data distribution. The given label $\hat{y}$ is the label provided during training. This is the target label used by the teacher model, and it may include label noise. We will revise our paper to clarify these terms.

We sincerely thank the reviewer for the constructive feedback. Please see our responses to the reviewer’s questions.

Q1. The second contribution of this work is to determine the number of distillation rounds required to achieve 100% population accuracy in the presence of label noise. On first reading, the term "100% population accuracy" may not be immediately clear. Introducing a brief explanation of this concept in the first place would be beneficial for readers who may be less familiar with the subject.

Thank you for your feedback. We will clarify the term “100% population accuracy” where it first appears in the manuscript. In our problem setup, we consider a ground-truth distribution of input-label pairs $(\mathbf{x},y(\mathbf{x}))\sim\mathcal{P}$. The population accuracy of a classifier $\boldsymbol{\theta}\in\mathbb{R}^{d\times K}$ is defined as $\mathbb{E}\_{(\mathbf{x},y(\mathbf{x})) \sim\mathcal{P}}\left[\mathbb{1}\left(\arg\max\_{k\in[K]}[\sigma(\boldsymbol{\theta}^{\top} \phi(\mathbf{x}))]\_k =y(\mathbf{x})\right)\right],$ i.e., the probability that the classifier (the softmax output) correctly predicts the ground-truth label $y$ for an input $\mathbf{x}$, drawn from $\mathcal{P}$.

Achieving 100% population accuracy means that the classifier correctly identifies the ground-truth class for all possible inputs from the distribution $\mathcal{P}$. In other words, despite the presence of label noise in the training data, the classifier perfectly generalizes to the true underlying data distribution. Assuming a sufficiently large number $n$ of training instances per class, this indicates that the trained model can correctly classify the (potentially noisy) training instances into their true ground-truth classes rather than overfitting to the given noisy labels. We will incorporate this explanation into the manuscript to ensure clarity for all readers.