Improving Resistance to Noisy Label Fitting by Reweighting Gradient in SAM

Regarding Figure 3, since Group A also contributes to SAM’s resistance to noisy fitting, why not reweight Group A as well?

While Group A contributes to SAM's resistance to noisy fitting by upweighting clean samples (Baek et al., 2024), this effect implicitly addresses the mechanism of slowing down the fitting of noisy samples. Instead, our work focuses on the specific term that explicitly mitigates noisy fitting.

To this end, we begin by exploring the gradient behavior of SAM. Our empirical study reveals that Group B inherently slows down the convergence rate and represents a comparable proportion to Group A. This aligns more closely with our objective of addressing noisy fitting explicitly, which is analyzed in Section 4. Moreover, as illustrated in Figure 3, Group B exhibits a greater influence on noisy fitting, establishing it as the primary target for intervention.

While reweighting Group A could potentially improve performance, such an analysis requires deeper investigation and is beyond the scope of this work.

Experimental results for high label noise settings (e.g., 80% symmetric label noise) are absent. Given that this manuscript primarily addresses label noise learning, evaluating the proposed method under high label noise conditions is essential, as it is a common practice in current label noise learning literature.*

We conducted experiments using ResNet-32 and CIFAR-10 with noise levels of 20%, 40%, 60%, and 80%, following Xie et al. (2024). SANER consistently outperformed SAM across various noise ratios, as shown in Table.

Since SAM enhances model performance in clean label scenarios, would SANER perform worse than SAM in such cases?

Actually, SANER even has minor performance improvements over SAM in clean label scenarios. We conducted experiments on ResNet18 using the clean CIFAR-10 and CIFAR-100 datasets, as shown in Table. The hyperparameters were configured as described in Appendix A.1 (Training Details) for reference.

Are there any improvements when combined with other label noise learning algorithms?

We conducted experiments to compare the test accuracy of SAM and SANER trained on ResNet-18 with CIFAR-10 and CIFAR-100, integrated with hard bootstrapping (Reed et al., 2014), as used in Foret et al. (2021). The results demonstrate that SANER consistently outperformed Bootstrap + SAM, as shown in Table.

References:

• Christina Baek, Zico Kolter, and Aditi Raghunathan. Why is sam robust to label noise? ICLR, 2024. URL https://arxiv.org/abs/2405.03676.
• Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. ICLR, 2021.

Conclusion

We thank you for your valuable feedback. We believe we have addressed all your concerns and have added experiments related to various noise rates (including noise-free setup) in Appendix D.1 and D.2. As a result, we would really appreciate it if you consider increasing your score.

Dear Reviewer,

Thank you once again for reviewing our manuscript. We have done our best to address your comments and have revised the paper based on suggestions from all reviewers.

We understand today is the weekend and do not wish to disturb you. However, with the approaching deadline, we hope you could review the revisions and provide your feedback at your earliest convenience.

Thank you for your valuable insights and support. Please let us know if you have any additional questions or need further clarification.

How do different values of $\alpha$ affect performance across varying noise ratios and network architectures?

Based on our analysis of $\alpha$ in Section 5, $\alpha$ is related to the model's fitting ability. If the model shows signs of overfitting, lowering $\alpha$ can help mitigate this issue. However, an extremely low $\alpha$ can lead to underfitting (e.g. 50% label noise). To further support this analysis, we provide a Table evaluating $\alpha \in \{0.25, 0.5, 0.75\}$ and $k \in \{25, 50, 75\}$, trained on ResNet34 and WideResNet40-2 with CIFAR-100 under 25% and 50% noisy labels.

References:

• Bingcong Li and Georgios Giannakis. Enhancing sharpness-aware optimization through variance suppression. NeurIPS, 2023.
• Xie, Wanyun, Thomas Pethick, and Volkan Cevher. SAMPa: Sharpness-aware Minimization Parallelized. NeurIPS, 2024.
• Christina Baek, Zico Kolter, and Aditi Raghunathan. Why is sam robust to label noise? ICLR, 2024.

Conclusion

We thank you for your valuable feedback. We believe we have addressed all your concerns and have added experiments related to SANER's hyperparameters in Appendix C. As a result, we would really appreciate it if you consider increasing your score.

Regarding the division of gradient components into Group A, B, and C, what specific criteria or methodology did you use for this grouping? Could you provide pseudocode for this process to make the grouping mechanism clearer?

The way to divide these groups is carefully specified in Section 3 and Algorithm 1. We summarize it for you as follows: In the SAM algorithm, we have two gradients, $g^{SAM}$ and $g^{SGD}$. We then calculate the ratio $r = g^{SAM} / g^{SGD}$ (component-wise operator). Finally, we identify the $i$-th parameter as belonging to Group A, Group B, or Group C if $r_i \geq 1$, $0 \leq r_i < 1$, or $r_i < 0$, respectively.

Why these particular components were chosen for reweighting, and include systematic mathematical derivations to show how these components effectively suppress noisy label fitting?

The motivation for selecting Group B for reweighting is detailed in Section 3, with its role in suppressing noisy label fitting further discussed in Section 4. Here is a summary: Baek et al. (2024) have thoroughly discussed how the gradients in Group A implicitly prevent noisy fitting. Meanwhile, based on our observations, Group B includes a comparable proportion of parameters as Group A but naturally slows down the convergence rate. This leads us to hypothesize that Group B explicitly controls noisy fitting. Hence, our analysis focuses on this group within the SAM gradient vector.

Through our empirical study, we find that SAM loses its ability to resist noisy fitting when the magnitudes in Group B are replaced with those from SGD (detailed in Section 3). Furthermore, our analysis reveals that Group B contains a much larger proportion of "noise-dominated components" (as defined in Section 4) compared to "clean-dominated components." This suggests that reducing the magnitudes of these components can enhance resistance to noisy fitting without significantly affecting the clean fitting rate. As a result, our method consistently outperforms SAM across all cases even with clean data.

The paper includes comparisons with VaSSO; however, the results under different noise rates show considerable gaps. Could you explain why such significant differences occur under identical experimental settings?

These differences occur under identical experimental settings, as specified in Appendix A.1 Training Details. The hyperparameters we used for VaSSO are $\rho=0.1$ and $\theta=0.2$, as recommended by Li & Giannakis, (2024). We applied the same hyperparameters when integrating our method, SANER, into VaSSO. The hyperparameters for SANER are $\alpha=0.5$ and $k=50$, which are the default values recommended in our paper.

Could you conduct additional experiments covering more diverse noise ratios to provide a balanced comparison, and clarify whether these differences are due to inherent limitations of the methods or specific experimental setups?

We conducted experiments using ResNet-32 and CIFAR-10 with noise levels of 20%, 40%, 60%, and 80%, following Xie et al. (2024). SANER consistently outperformed SAM across various noise ratios, as shown in Table.

But there is a lack of detailed experimental analysis on the selection of α values (e.g., from 0 to 1 or values greater than 1)

We would like to clarify that a detailed experimental analysis of $\alpha$ selection is provided in Section 5. Figures 5a and 5b demonstrate that $\alpha \in (0, 1)$ effectively slows down the fitting of noisy labels, while $\alpha > 1$ accelerates it. Furthermore, we recommend $\alpha=0.5$ as the default value for all our experiments, as specified in Appendix A1: Training Details. Please refer to these sections for more details.

Dear Reviewer,

Thank you once again for reviewing our manuscript. We have done our best to address your comments and have revised the paper based on suggestions from all reviewers.

We understand today is the weekend and do not wish to disturb you. However, with the approaching deadline, we hope you could review the revisions and provide your feedback at your earliest convenience.

Thank you for your valuable insights and support. Please let us know if you have any additional questions or need further clarification.

The authors can also analyze the effect of the proposed reweighting methods combined with existing robust training methods, as is done in the original SAM paper.

We conducted experiments to compare the test accuracy of SAM and SANER trained on ResNet-18 with CIFAR-10 and CIFAR-100, integrated with hard bootstrapping (Reed et al., 2014), as shown in Table. The results showed that SANER consistently outperformed Bootstrap + SAM.

References:

• Reed, Scott, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. ICLR, 2015.

Conclusion

We thank you for your valuable feedback. We believe we have addressed all your concerns and have added experiments in Appendix B, C, and D.2. As a result, we would really appreciate it if you consider increasing your score.

What is the effect of datasets and architectures on the observed phenomena? What's the effect of the label noise level on the observed phenomena?

The main phenomena observed in our paper are as follows:

1. The experiment reveals that SAM not only up-weights gradients but also involves a significant portion of down-weighted gradients.

• We demonstrate in this Figure that Groups A and B account for a larger proportion than Group C during the early stages of training across different architectures.

2. Group B primarily contributes to noisy fitting (Section 3).

• We demonstrate that Group B mainly contribute to prevent noisy fitting in ResNet18, ResNet34, ResNet18 with doubled width in each layer. Group B contribute on par with Group A to prevent noisy fitting in WideResNet40-2 and DenseNet121. Our experiments, conducted on CIFAR-10 with 25% and 50% label noise, are shown in this Figure.
• We did not present this phenomenon on CIFAR-100 because it is challenging to illustrate the difference in noisy fitting between SAM and SGD, as indicated by the noisy training accuracy in Figure 17 of our revised manuscript. However, this figure also highlights that the reweighting of Group B (our method) remains effective in slowing down the noisy fitting rate.

3. Group B has a greater impact on slowing down noisy fitting compared to clean fitting (Section 4).

• We also show that Group B consistently has a greater impact on slowing down noisy fitting compared to clean fitting across different architectures and datasets, including ResNet, WideResNet, and DenseNet trained on CIFAR-10 and CIFAR-100 with 25% and 50% label noise, as shown in this Figure.

On Cifar-10, the improvement is not significant. While on Cifar-100 the proposed method works much better. Nevertheless, there is no explanation or deeper analysis provided in the paper.

The improvement of our method on CIFAR-10 is significant. Without noisy labels, SAM achieves a best performance of approximately 96% with ResNet18 as shown in Table. However, its performance drops to 93% when noisy labels are present, resulting in a 3% performance gap. In contrast, our method, SANER, achieves 94%, reducing the performance gap to 2%. This represents around 33% improvement in closing the gap compared to SAM trained on clean CIFAR-10.

The performance on CIFAR-100 exhibits a similar pattern to the improvements observed on CIFAR-10. The performance of SAM without noisy labels is 79% as we ran and shown in Table. However, with 25% noisy labels, SAM's performance drops to 70%, resulting in a 9% gap. In comparison, SANER achieves 73%, reducing the gap to 6%. This also represents around 33% improvement.

Our proposed method is based on the behavior of SAM and is not tailored to any specific dataset or architecture. We demonstrated the consistent improvement of SANER over SAM across CIFAR-10, CIFAR-100, and Mini-Webvision, as shown in our paper, as well as on Tiny-ImageNet, which we provide for additional reference.

Both the reweighting factor and k needs to be tuned and I didn't find an ablation study.

We used $\alpha=0.5$ and $k=50$ for all experiments across various architectures and datasets, demonstrating that achieving superior performance compared to SAM does not require extensive tuning. We ran experiments for $\alpha \in \{0.25, 0.5, 0.75\}$ and $k \in \{25, 50, 75\}$, trained on ResNet34 and WideResNet40-2 with CIFAR-100 under 25% and 50% noisy labels, as shown in Table.

Key Observations:

1. Sensitivity to $k$:

• Performance is relatively stable across different values of $k$ when $\alpha$ is moderate (0.5 or 0.75). This indicates that our method is not highly sensitive to $k$ under well-performing settings.
• In underfitting scenarios (e.g., when $\alpha$ is small or noise is high), the choice of $k$ becomes critical. Higher values of $k$ (e.g., $k=75$) improve gradient reweighting, enhancing the model’s ability to fit clean data effectively and preventing severe underfitting.

2. Sensitivity to $\alpha$:

• When $\alpha$ is too small (e.g., 0.25), the model struggles to fit clean data effectively, especially under higher noise levels (50%).
• Moderate values of $\alpha$ (0.5 or 0.75) provide a good balance, achieving strong clean fitting and noise resistance without requiring extensive tuning.

I didn't find any insight on how to set these parameters for different datasets/architectures. Table 5, 6 in the appendix only compares using or not using this scheduler (k=0 and k=50). How should one set the parameters of the proposed method?

We believe that tuning hyperparameters based on training dynamics is a more robust and common practice compared to relying on datasets or architectures. To make the tuning process deterministic, the effect of hyperparameters on training dynamics must remain consistent across different setups.

In our work, we introduced two hyperparameters, $\alpha$ and $k$, and analyzed their impact on training dynamics, as illustrated in Figure 5. To further verify their consistency, we conducted experiments varying $\alpha \in \{0.25, 0.5, 0.75\}$ and $k \in \{25, 50, 75\}$, evaluating them across different architectures. Noisy train accuracy across different $\alpha$ is shown in this Figure and clean accuracy across different $k$ is shown in this Figure. Our experiments indicate that the characteristics of $\alpha$ and $k$ remain consistent across various configurations.

Based on these observations, we provide guidelines for tuning these parameters according to training dynamics:

• $\alpha$ (Reweighting Factor): The $\alpha$ parameter is directly proportional to the noisy fitting rate, as demonstrated in Figure 5a. Lower $\alpha$ values increase resistance to noisy label fitting, making it advantageous to reduce $\alpha$ when the model shows signs of overfitting.
• $k$ (Linear Scheduler): The $k$ parameter governs the clean fitting rate during the early stages of training. If the model struggles to fit clean samples effectively—resulting in underfitting where training accuracy is low—increasing $k$ can help prevent this issue by improving clean sample fitting.

Dear Reviewer,

Thank you once again for reviewing our manuscript. We have done our best to address your comments and have revised the paper based on suggestions from all reviewers.

We understand today is the weekend and do not wish to disturb you. However, with the approaching deadline, we hope you could review the revisions and provide your feedback at your earliest convenience.

Thank you for your valuable insights and support. Please let us know if you have any additional questions or need further clarification.

I thank the authors for taking the time to addressing my and other reviewers concern. I've augmented my review and increased my score, but I still believe that the paper needs a major revision to be ready for publication.

I noticed that Figure 2 in Section 3 is based on training a ResNet-18 model on CIFAR-10. Shouldn't testing be conducted on more models and datasets to confirm that the three types of parameters, Group A, B, and C, indeed exhibit the proportions mentioned in the paper? Perhaps conducting experiments on a larger ResNet model (ResNet-50) and a larger dataset like CIFAR-100 or ImageNet (if conditions permit) would provide more convincing results.

Yes, the trends of Groups A, B, and C are consistent with our observed phenomenon when training ResNet50 on CIFAR-100, as shown in this Figure. During the early training, Groups A and B represent a larger proportion compared to Group C. The parameter distribution of Group C increases in the later stages of training, after the model has fit the noisy labels as shown in this Figure.

Based on the author's analysis, in fact, at epoch 100, there was a noticeable increase in the parameter ratio of Group C, which maintained around 10% during the early training stages. Would it be more meaningful to directly use the reverse gradients for noisy label data?

We mentioned it in lines 195–202 of our paper. "Analyzing Group C is particularly challenging due to the inconsistency between the objectives of SAM and SGD. The divergence in gradient component directions complicates the learning process, as reversed gradients may hinder effective learning from the data. Moreover, our study focuses on the “memorization” phase (Arpit et al., 2017), where the transition from fitting clean examples to overfitting noisy examples occurs. This typically happens during the middle stage of training, when most clean examples have already been learned. During this period, Groups A and B are still dominant compared to Group C. Therefore, in this section, we focus on comparing the effects of Group A and Group B of SAM on noisy fitting, leaving the analysis of Group C for future work."

We would like to emphasize that the "noticeable increase in the parameter ratio of Group C, which maintained around 10% during the early training stages" is not a special phenomenon that occurs due to fitting to noisy labels. Even in clean data, Group C exhibited a similar pattern, as shown in this Figure. This suggests that the increase in Group C might not be directly related to noisy label data.

The motivation for focusing on Group B is grounded in two key points:

• Group A, which comprises the largest subset of parameters, has been extensively discussed by Baek et al. (2024) in how upweighting gradients can help mitigate noisy fitting implicitly.
• Group B represents the next significant subset of parameters and is characterized by its nature of slowing convergence. This observation leads to the hypothesis that this slower convergence may be linked to a reduced rate of learning noisy labels. Consequently, our analysis prioritizes Group B to investigate this relationship further.
• To strengthen our finding in Figure 3 by including Group C, we ran the SGD-GrC experiment, which replaces the gradient of Group C in SAM with those from SGD, as shown in this Figure. Group B still mainly contributes to noisy fitting.

Also, I noticed that the ratios of Group B and Group C are quite similar in the early stages of training, the values for Group A are mostly around 1.0 while those for Group B are around 0.99. If this is the case, I don't believe that SGD and SAM have different perspectives on the parameters in these two groups.

We illustrate the distribution of the ratios of the SAM gradient to the SGD gradient at the 60th, 120th, and 180th epochs in this Figure for your reference. The range of values for the ratios is wide and not concentrated mostly around 0.99 and 1.0, as you believe, indicating that Group A, Group B are very different from each other during training.

References:

• Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. ICLR, 2021.
• Christina Baek, Zico Kolter, and Aditi Raghunathan. Why is sam robust to label noise? ICLR, 2024.

Conclusion

We thank you for your valuable feedback. We believe we have addressed all your concerns and have added experiments related to the proportion of each group in Appendix B.1. As a result, we would really appreciate it if you consider increasing your score.

Dear Reviewer,

Thank you once again for reviewing our manuscript. We have done our best to address your comments and have revised the paper based on suggestions from all reviewers.

We understand today is the weekend and do not wish to disturb you. However, with the approaching deadline, we hope you could review the revisions and provide your feedback at your earliest convenience.

Thank you for your valuable insights and support. Please let us know if you have any additional questions or need further clarification.

Please explain how you consider the role of early stopping, and what regime does the analysis and SANER experiments consider.

In our experiments, we report the best test accuracy checkpoint across all epochs and this approach is better than early stopping, this is because it effectively captures the model’s performance without carefully tuning hyperparameter patience of early stopping and matches training default in practice when we do not know training dataset containing noisy labels or not.

Does SANER improve over SAM even accounting for early stopping, i.e. is the best early-stopped checkpoint of SANER better than best early-stopped checkpoint with SAM?

Yes, SANER performs better than SAM when using the best early-stopped checkpoint setting. You can refer to the test accuracy in Figures 1c, and 17 of our revised manuscript. The test accuracy of SANER is consistently higher than that of SAM across most epochs, which can include the best early-stopped checkpoint.

How do you set the SAM perturbation radius hyperparameter?

We set the SAM perturbation radius hyperparameter $\rho = 0.1$ as the recommendation in the original SAM paper by Foret et al. (2021) for label noise settings.

How does perturbation radius interact with $\alpha$?

We provide the Table compares the test accuracy of SAM and SANER under different radii $\rho = \{ 0.05, 0.1, 0.15, 0.2 \}$ for your reference. Tuning $\rho$ can slightly improve the performance of SANER, but with the default settings of $\alpha=0.5$ and $k=50$, SANER consistently outperforms SAM without being highly sensitive to the perturbation radius.

Do the trends of gradients across different groups hold for different radii as well?

Yes, these trends are consistent across different radii. We ran ResNet18 on CIFAR-10 with $\rho = \{ 0.05, 0.1, 0.15, 0.2 \}$ and show distribution of these groups in this Figure. Groups A and B account for a larger proportion compared to Group C. The parameter distribution of Group C increases during the later stages of training when the model has already fit the noisy labels.

References:

• Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. ICLR, 2021.

Conclusion

We thank you for your valuable feedback. We believe we have addressed all your concerns and have added experiments related to the perturbation radius in Appendix B.1 and C.1. As a result, we would really appreciate it if you consider increasing your score.

Dear Reviewer,

Thank you once again for reviewing our manuscript. We have done our best to address your comments and have revised the paper based on suggestions from all reviewers.

We understand today is the weekend and do not wish to disturb you. However, with the approaching deadline, we hope you could review the revisions and provide your feedback at your earliest convenience.

Thank you for your valuable insights and support. Please let us know if you have any additional questions or need further clarification.