Equally Critical: Samples, Targets, and Their Mappings in Datasets

[W1] For scaling law research, the experimental scale is too small, even with the so-called “large-scale” Tiny-ImageNet dataset mentioned in the paper.

Thank you for your advice. But we want to emphasize that the scaling law mainly reveals the pattern that model performance improves predictably as a power-law behavior of model size, data size, or compute, and there are lots of work to reveal that scaling laws are not restricted to large datasets but are also applicable to small datasets. For example:

• [1] showes in their work that even when dataset sizes are reduced, the relationship between model scale, data size, and performance remains consistent.
• [2] shows that Vision Transformers adhere to scaling laws on small datasets such as CIFAR10, with performance improvements following similar trends to larger datasets.
• [3] also take CIFAR10 as a case study and reveal that the pattern observed in CIFAR10 is consistent in ImageNet.

Building on this perspective, our work aligns with these prior studies by showing that meaningful scaling patterns can be derived even from smaller datasets, and here we would like to emphasize that the primary innovative contributions of this work lie in:

• Improving the efficiency of traditional training paradigms by redefining the mapping between samples and targets, thereby introducing a novel training framework that, to the best of our knowledge, has not been proposed before.
• Scaling laws primarily emphasize a power-law relationship, and case studies on smaller datasets can be meaningfully analyzed and are sufficient to reveal such patterns.

It is worth noting that in real-world scenarios, abundant data is often scarce. From this perspective, the novel Strategy C we proposed, and the pattern observed in Section 5 provides an effective way to enhance model performance in data-scarce conditions.

To address concerns about scalability, we have extended our experiments to ImageNet, and we also use ResNet50 and ViT as the backbone for model training, as shown in Figure 15 and Table 6, 7 for numerical results, Under different backbone settings, we also find that:

• Strategy A shows long-term advantages, while Strategy B exhibits short-term benefits.
• Our proposed Strategy C effectively addresses the short-term limitations of Strategy A and the long-term deficiencies of Strategy B.
• Moreover, the advantages of Strategy C become increasingly prominent when applied to weaker teacher models.

We believe this further supports the validity of our findings and their applicability under more settings.

[1] Scaling Laws for Neural Language Models.

[2] Scaling Vision Transformers. CVPR 2022.

[3] Beyond neural scaling laws: beating power law scaling via data pruning. NeurIPS 2022.

Dear Reviewer trqj,

Thank you for your detailed feedback and comprehensive review of our paper. Your observations have provided valuable guidance in improving our work, and we have made substantial revisions to address your concerns. Below, we summarize your key points and outline how we have responded:

1. Experimental Scale and Applicability of Scaling Laws:
   • Your concern: The experimental scale is too small for meaningful scaling law insights.
   • Our response: We have extended our experiments to include ImageNet with ResNet50 and ViT backbones, as shown in Figure 15 and Tables 6 and 7, to validate our findings in larger-scale settings. These results demonstrate consistent early-stage acceleration and final accuracy improvements for Strategy C, particularly with weaker teacher models, aligning with scaling law patterns observed in prior studies.
2. Depth of Insights and Practical Implications:
   • Your concern: Observations lack deeper insights or actionable implications.
   • Our response: We have provided intuitive explanations above for the observed training dynamics, Strategy C balances short-term acceleration (via more targets) and long-term stability (by reducing noise from incorrect targets). Additionally, we discuss practical applications for selecting target types, teacher models, training strategies and augmentation strategies, in real-world, data-scarce conditions.
3. Accuracy of Results and Experimental Setups:
   • Your concern: ResNet-18 results on CIFAR-10 are subpar, and Figure 1 lacks experimental details.
   • Our response: We clarified the setup for Figure 1 in the revised version(line 715). The observed ResNet-18 accuracy aligns well with our goal of analyzing target mappings. As shown in Appendix B (Figure 10), trends remain consistent across teacher models of varying accuracies.
4. Novelty of Key Findings:
   • Your concern: Observations such as "soft targets expedite early-stage training" are trivial.
   • Our response: While soft targets accelerating convergence is known, our study uniquely frames this through the lens of sample-to-target mappings, and there are more non-trivial observations like weaker teacher models can significantly enhance student model performance.
5. Relevance of targets to Neural Scaling Laws:
   • Your concern: The claim of ”the first to emphasize the critical role of targets in breaking scaling laws“ is overstated.
   • Our response: Our claim focuses on systematically exploring sample-to-target mappings and demonstrating their role in improving training efficiency across scales. Strategy C, in particular, mitigates the inefficiencies of traditional approaches, which has NEVER BEEN PROPOSED before.
6. Additional Literature on Data Selection:
   • Your concern: Relevant works on offline and online data selection are not discussed.
   • Our response: We acknowledge these studies and clarified that our focus is FUNDAMENTALLY DIFFERENT. While data selection methods optimize sample importance, our work emphasizes the role of TARGETS AND THEIR MAPPINGS, presenting a complementary perspective that broadens the understanding of efficient learning.

We hope that our clarifications and revisions adequately address your concerns. If you find the updates align with your expectations, we would be grateful if you could consider revising your score, as it would significantly support the progression of our work.

Please let us know if you have any further feedback or questions. Thank you again for your thoughtful review and for engaging with our submission.

Best regards,

The Authors of Submission 10518

Dear Reviewer trqj,

Thank you for your thoughtful response and for carefully reviewing the other reviewers’ comments and our rebuttal. We appreciate your decision to adjust your evaluation and your careful consideration of our work.

Thank you again for your time and feedback.

Best regards,

The Authors of Submission 10518

[Q1] Can the authors explain why the results/insights from the paper are relevant to neural scaling laws at the practically relevant scales discussed in the introduction?

Thank you for your question. In addition to systematically introducing the three distinct target-mapping strategies, we would also like to emphasize that:

• Both STRATEGY B and STRATEGY C can break the scaling law limitations of the traditional STRATEGY A within a short training period, leading to faster convergence.

• Furthermore, STRATEGY C can overcome the scaling law limitations of STRATEGY B in the long term, resulting in higher final accuracy.

  Our proposed Strategic C effectively addresses the short-term limitations of Strategy A and the long-term deficiencies of Strategy B.

Additionally, our paper is the first to:

• systematically investigate the factors influencing the scaling law from the perspectives of samples (Section 4) and targets (Section 5).
• highlight the important role that targets generated by a weaker teacher model can play in guiding the training of the student model throughout the training process in the context of knowledge distillation.

In Section 6, we briefly highlight the applicability of our findings:

• We can extend our findings to larger-scale datasets like text-based tasks. This paper uses image classification as a case study to provide a concrete and focused demonstration of the proposed methods, with broader implications left for future exploration.
• We believe this work is a step toward understanding how target mappings impact training dynamics across domains.

And we have also extended our experiments to ImageNet and we also use ResNet50 and ViT as backbone for model training to verify the effectiveness of our proposed Strategy. Thus, we believe this work is highly relevant to neural scaling laws at the practically relevant scales.

[W3] Claim 3 on the efficacy of the different strategies is misleading. In particular, the best overall performance is in fact obtained by Strategy B with a 90% accurate teacher model (Table 2 in the appendix).

We acknowledge that the description of Claim 3 could have been clearer. However, as shown in Table 2 in the appendix, our results indicate that:

• When the teacher model's accuracy is fixed, the gain from Strategy C continues to increase as training progresses. For example, with a 90% accurate teacher model, the gain from Strategy C progresses from 0.94 → 0.96 → 0.99, showing a steady improvement over time.
• Additionally, Figure 8 presents the accuracy of the student model trained with an 80% accurate teacher model over different training steps. From the plot, we observe that Strategy C does not show a significant convergence bottleneck, and the gap between Strategy C and Strategy B grows as training progresses.

Moreover, we also would like to give an intuitive explanation to why the performance improvement of Strategy C over Strategy B is positively correlated with the number of training steps and Strategy C can achieve higher final accuracy than Strategy B. As noted in line 344 of the paper, Strategy C generates fewer distinct targets during training compared to Strategy B. For instance, on CIFAR10 with 100 training epochs, the ratios of sample-to-target mappings are as follows:

• Strategy A: 50,000 × 100:10 = 500,000:1
• Strategy B: 50,000 × 100:50,000 = 1:1
• Strategy C: 50,000 × 100:50,000 = 100:1

Strategy C establishes a mapping between samples and targets that scales proportionally with the number of training epochs. Consequently, when the number of training steps is sufficiently large, Strategy C will asymptotically approach the performance of Strategy A.

Based on these observations, we have strong reason to believe that, with further training, Strategy C would achieve the highest final accuracy.

[W4] The choice to separate the model training into the "backbone" (feature extractor) and classifier is motivated by the claim that the cross-entropy loss cannot handle soft labels, but this is not true. The cross-entropy can be computed between any two discrete distributions with the same support (https://en.wikipedia.org/wiki/Cross-entropy). In fact, the KL divergence and cross-entropy differ by a quantity which is constant w.r.t. the trained model, so the gradients for the proposed training strategy for the backbone are the same as the standard CE loss. It should also be noted that previous data augmentation methods which use soft labels (such as MixUp) also apply CE with the soft labels directly.

The primary reason for this decision is not solely due to the claim that cross-entropy loss cannot handle soft labels, as you pointed out. Instead, the motivation is twofold:

• In traditional knowledge distillation, where the teacher model’s backbone and classifier are trained jointly, there can be cases where the backbone extracts strong features, but the classifier’s performance does not align well with the teacher’s output. This can lead to suboptimal training outcomes.
• By decoupling the backbone from the classifier, which aligns with approaches in unsupervised learning where feature extraction is performed independently of specific downstream tasks [1,2], we can better assess and refine the feature extraction capacity of the model, ensuring that both components perform optimally.

We hope this clarification better conveys our rationale for this approach. Thank you for highlighting this.

[1] A Simple Framework for Contrastive Learning of Visual Representations. ICML 2020.

[2] Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning. NeurIPS 2020.

[W5] The description of the results in Fig. 5a seems to be incorrect. This paragraph states that MixUp > standard augmentation > no augmentation, but the plot has standard augmentation with the highest performance with MixUp and no augmentation approximately equal for the relevant purple and blue lines.

We acknowledge the mistake in our description in line 404. As you correctly pointed out, the results should be interpreted as: standard augmentation > MixUp > no augmentation. However, we would also like to emphasis that

• For high-accuracy teacher models, applying MixUp augmentation to the teacher does not significantly benefit the student model’s training and may even perform worse than using a NoAug-trained teacher model.

which is also an intriguing and novel observation😊 we have corrected this misstatement in the revised version.

[W2] On the other hand, the insights the authors provide showing the advantage of augmented targets all occur in low data regimes. As they return to the size of the full CIFAR-10 dataset, regular 1-hot labels have the best performance. Thus, it's unclear what relevance the insights in the paper have to the stated practical problem of interest, i.e., datasets at a scale far beyond that of CIFAR-10.

Thank you for your concern about our insights. Here we would like to emphasize that:

• In most real-world deep learning scenarios, abundant data is often scarce. Modern deep learning research increasingly focuses on achieving better performance under limited data conditions [1,2], and our findings are highly relevant to practical applications.
• Take Section 5.2 for example, different data augmentation strategies affect model performance differently. Under limited samples, specific data augmentation strategies help the model converge more effectively, breaking the traditional power-law relationship between data size and model performance.

Regarding your observation that "regular 1-hot labels have the best performance with the full CIFAR-10 dataset," we provide two clarifications:

• In practical scenarios, it is often difficult to determine whether the available dataset is "sufficient." Many real-world applications involve insufficient data, where different training strategies can offer significant advantages [3,4].
• The "No Free Lunch" theorem [5] reminds us that no single machine learning method is optimal for all situations🧐. In real-world settings, the choice of method depends on the specific conditions. In the context of our findings:
  • When data is limited, Strategy B is preferable.
  • When data is abundant, Strategy A performs best.

To address concerns about scalability, we have extended our experiments to ImageNet, and we also use ResNet50 and ViT as the backbone for model training, as shown in Figure 15 and Table 6,7 for numerical results, Under different backbone settings, we also find that

• Strategy A shows long-term advantages, while Strategy B exhibits short-term benefits.
• Our proposed Strategy C effectively addresses the short-term limitations of Strategy A and the long-term deficiencies of Strategy B.
• Moreover, the advantages of Strategy C become increasingly prominent when applied to weaker teacher models.

We believe this further supports the validity of our findings and their applicability under more settings.

[1] Beyond neural scaling laws: beating power law scaling via data pruning. NeurIPS 2022.

[2] Deep learning on a data diet: Finding important examples early in training. NeurIPS 2021.

[3] Scaling laws of synthetic images for model training... for now. IEEE 2024.

[4] How much data are augmentations worth? an investigation into scaling laws, invariance, and implicit regularization. ICLR 2023.

[5] No Free Lunch Theorems for Optimization. IEEE 1997.

[W1] The main thrust of the paper is that, in the context of improving neural scaling laws, their "finding underscores the significance of the exploration the target component, a frequently overlooked aspect in the deep learning community." The discussion of neural scaling laws centers of the fact that exponentially larger datasets are needed to achieve only marginal performance improvements; in particular, these scaling laws are a problem only once we have reached the "extremely large dataset regime."

We appreciate your observations and would like to clarify that:

• Neural scaling laws mainly describe how model performance follows a power-law relationship with compute, data size, and model scale, emphasizing the inefficiency of performance improvements as these factors scale.
• The main goal of our paper is to explore methods to mitigate this inefficiency during training.

And we also would like to emphasize that the primary innovative contributions of this work lie in:

• Improving the efficiency of traditional training paradigms by redefining the mapping between samples and targets, thereby introducing a meaningful and novel training framework that, to the best of our knowledge, has never been proposed before.
• Scaling laws primarily emphasize a power-law relationship, and case studies on relatively larger datasets (like TinyImageNet) can be meaningfully analyzed and are sufficient to reveal such patterns.

The research focus on targets presented in our paper is not contradictory to neural scaling laws. To provide a quick recap, we summarize how modifying the mapping between samples and targets can influence model training efficiency:

• Strategy B accelerates early convergence, achieving better model performance with fewer computational resources compared to traditional methods (Figure 3a).
• Weaker teacher models can also aid early convergence, providing better performance with less computing than traditional methods (Figure 3b).
• Our proposed Strategy C effectively addresses the short-term limitations of Strategy A and the long-term deficiencies of Strategy B, achieving higher model performance improvements for the same computational resources (Figure 4).

Thank you for your thoughtful comments and for carefully reviewing our revised paper. We have further updated the paper to include additional results, specifically adding Table 7, which provides a detailed comparison of early-stage numerical results between Strategies A and C. And we address your concerns as follows:

1. On the starting accuracy discrepancy in Figure 15 and other plots:

Here, we provide clarifications for the observed differences:

• Validation frequency: For large-scale datasets like ImageNet, performing validation after every training step is computationally infeasible. Instead, we evaluate every 500 steps. This does not affect the trends observed, as shown in Table 6,7, where we provide further comparisons that confirm consistency in observed patterns.

• LOWESS smoothing: As noted in line 734 of the paper, we used the LOWESS method for local smoothing to reduce noise and better illustrate overall trends. Because this approach incorporates data from neighboring points, and since Strategies B and C accelerate early-stage convergence significantly, the smoothed curves may visually imply that these methods achieve high accuracy almost immediately. And we believe this visualization highlights the early advantage of Strategies B and C compared to the slower convergence of Strategy A.

  Recognizing that smoothing may influence the interpretation of early-stage results, we addressed this concern in line 736 by providing concrete numerical results in Tables 2, 5, 6, 7.

In Table 7, we further provide detailed comparisons between Strategies A and C in early training steps. These results show that Strategy C significantly accelerates convergence within the first 2k steps, regardless of the teacher model used. For comparisons between Strategies B and C, detailed results are already provided in Table 6.

2. On Strategy C’s effectiveness and the interpretation of results in Figure 15:

We respectfully disagree with your interpretation that Strategy C "retains the downsides" of Strategies A and B. Allow us to clarify:

• Early-stage acceleration: As mentioned above and supported by Table 7, Strategy C demonstrates substantial early-stage acceleration across all teacher models.

• Final accuracy: Contrary to your claim, the results in Figure 15 and Table 6 clearly show that Strategy C consistently achieves higher final accuracy than Strategy B across all teacher models. The numerical gains (in Table 6) for Strategy C compared to Strategy B, particularly with weaker teacher models, are evident and significant.

• Why Strategy C works: To aid understanding, we have previously provided an intuitive explanation: Strategy C represents a novel compromise between Strategies A and B. By reducing the sample-to-target ratio compared to Strategy A, Strategy C naturally accelerates early convergence. Additionally, by increasing sample-to-target ratio, Strategy C effectively decreases the noise inherent in Strategy B and improves final accuracy.

  We believe this dual benefit distinguishes Strategy C and validates its utility as a meaningful contribution compare to the traditional training paradigm.

3. On the relevance of findings to large-scale datasets and neural scaling laws:

As for the practical deep learning scenarios, we clarify that:

• Insights remain valid: Even with a large dataset like ImageNet, as training steps increase, we observe consistent patterns that align with our conclusions in the paper from relatively larger scale experiments (like TinyImageNet). Strategy C continues to demonstrate early-stage acceleration and final accuracy gains, particularly with weaker teacher models.
• Core contribution: The main contribution of our work is introducing a novel perspective on the sample-target mapping and systematically exploring its impact on training dynamics. This perspective offers a meaningful approach to addressing inefficiencies in neural network training, which is relevant across data and compute scales in neural scaling law.

---

Apart from that, as you correctly noted in your Strengths section:

• "Finding ways to improve the neural scaling laws that have been observed until now will be essential for continuing to improve model capabilities. Thus, the stated problem under study is relevant to the ICLR community."

We firmly believe our work is a meaningful step toward this goal and merits further exploration. By introducing new insights into how target mappings influence training dynamics, we propose practical methods for deep learning community on how to select target types, teacher models, and augmentation strategies to enhance training efficiency.

Dear Reviewer pm2n,

Thank you for your thorough review and detailed feedback on our paper. Your insights have been instrumental in refining our work, and we sincerely appreciate the time and effort you’ve dedicated to assessing our submission. Below, we summarize your key concerns and outline how we’ve addressed them in the revised version:

1. Effectiveness of Strategy C:

   • Your concern: Strategy C seems to combine the drawbacks of Strategies A and B, without retaining their respective advantages.

   • Our response: We want to emphasis that:

     • The main purpose of our paper is NOT to propose a universal SOTA or even an OMNISCIENT strategy that consistently achieves the best all the time, but to examine traditional training paradigms through the lens of the MAPPING relationship between samples and targets.
     • It is UNREALISTIC IN PRACTICAL SCENARIOS to expect a single algorithm to perform OPTIMALLY THROUGHOUT THE ENTIRE TRAINING PROCESS.

     We believe that Strategy C provides improvements over traditional Strategy A in the short term and outperforms Strategy B in the long term, offering a NOVEL PERSPECTIVE for addressing training inefficiencies, and we also emphasis the important role a weaker model can play. Additionally, we provided an intuitive explanation of Strategy C's scaling advantage through its sample-to-target ratio adjustments.

2. Relevance to Neural Scaling Laws:
   • Your concern: The paper's findings appear relevant only in low-data regimes, with diminishing applicability as data size or compute increases, making them less impactful for practical neural scaling scenarios.
   • Our response: We clarified that our insights provide actionable insights for deep learning community on how to select target types, teacher models, and augmentation strategies to mitigate inefficiencies in training dynamics which are prevalent in real-world applications, we extended our experiments to ImageNet using ResNet50 and ViT backbones, providing new evidence supporting Strategy C’s utility in both early-stage acceleration and final accuracy gains.

We hope that these updates adequately address your concerns. If our clarifications and revisions meet your expectations, we kindly request you to consider revising your score, as this would greatly support the progression of our work.

Should you have any additional feedback or questions, we remain available to further discuss. Thank you again for your invaluable contributions and for engaging deeply with our submission.

Best regards,

The Authors of Submission 10518

Dear Reviewer pm2n,

Thank you again for your feedback on our paper. We have carefully addressed your concerns about the effectiveness of Strategy C in practical large-scale settings and we wish to re-emphasize that our primary objective is to 

• SYSTEMATICALLY analyse the CRITICAL and often OVERLOOKED role of TARGETS AND THEIR MAPPINGS in a dataset, thus propose and validate a NOVEL, MEANINGFUL and IMPACTFUL training paradigm that benefits EFFICIENT LEARNING.
• Provide ACTIONABLE INSIGHTS for deep learning community on how to select TARGET TYPES, TEACHER MODELS, and AUGMENTATION STRATEGIES to enhance training efficiency and address key challenges in neural scaling laws.

We respect your thoughtful suggestions, but directly reject our paper may potentially overlook the substantial and novel contributions that it brings to the field. We firmly believe that the significance of understanding target mappings is a fundamental topic that deserves attention and further exploration, and our paper is a vital step in this direction.

We believe our paper delivers these insights while addressing all points raised in your review and respectfully request that you reconsider your score based on our clear clarification. If there are any remaining concerns, we are ready to provide further clarifications promptly.

Thank you again for your time and for contributing to this process.

Best regards,

The Authors of Submission 10518

Dear Reviewer pm2n,

Thank you for your thorough review and for engaging deeply with our work throughout the discussion phase. We greatly appreciate your thoughtful feedback and understand your perspective.

From the strategies discussed in our paper, we think it is possible to hypothesize a hybrid approach: employing Strategy B during the early stages and then switching to Strategy A in the later stages to benefit from its superior final accuracy, may obtain improvement at some point in the training process and could potentially yield a more efficient training paradigm.

HOWEVER, we deliberately chose not to pursue this direction because it does not align with our primary research objective. Our paper is NOT intended to propose a SOTA method. Instead, as highlighted in our title, we focus on the critical role of samples, targets, and their mappings. Our insights extend far beyond Strategy C itself, but more importantly, we offer ACTIONABLE guidance on selecting targets, teacher models, and augmentations to enhance training efficiency.

Please allow us to kindly suggest that a decision to directly reject our work without a more comprehensive consideration of its novelty and practicality might overlook its broader value and insights it provides to the community. While we respect your final decision, we hope you will reconsider it in light of the fundamental motivations and contributions of our paper.

Thank you again for your time, thoughtful engagement, and constructive feedback throughout this process.

Best regards,

The Authors of Submission 10518

[W1] it is possible that strategy A in the results from figure 3a) simply needs a different learning rate. While the experiments are repeated at least five times, no uncertainty quantification (such as standard error) is included in the plots or the analysis

Thank you for your advice. As for your concern in Strategy A from Figure 3a, in our revised version, we further show that tuning a different learning rates and batch size cannot help, as shown in Figure12 (line 955, page 18), all results support our finding in Claim 1 that:

• Strategy A shows long-term advantages, while Strategy B exhibits short-term benefits.
• The important role that targets generated by a weaker teacher model can play in guiding the training of the student model throughout the training process.
• For simplicity, we compare Strategy A and B as a case study and provide numerical results in Table 4 (line 95 on page 18) where we experiment five times and record the final accuracy with standard error using different teacher models. The experimental results demonstrate that our error margins are sufficiently small, ensuring the reliability of our reported findings.

[W2] Research data: No code is provided Experiment results are not included, e.g. as csv files

We appreciate your concern about the research data, but we would like to highlight that,

• Appendix B contains extensive experimental results, mainly including comparisons of teacher models with varying accuracies and more numerical results, which we believe is clear and sufficient enough to support the key findings.
• We also commit to releasing the complete code upon acceptance of the paper.

[W3] While interesting experiments are designed and phenomena are observed, little explanation is offered as to why these patterns are being observed. For example, in the context of Figure 7 it would be interesting to discuss the effects the different augmentation strategies have on the class probabilities, which might explain why some augmentation methods perform worse with strategy C than with strategy B. When an augmentation method actually changes the class probability of an image (such as random cropping), using strategy C to train the network does not reduce noise but instead feeds the network wrong soft labels.

In Figure 7, we agree with your opinion that for augmentation methods that significantly affect class probabilities (e.g., random cropping), STRATEGY C may provide incorrect soft targets. And we appreciate your suggestion to provide a deeper theoretical explanation for the observed patterns.

However, we would like to emphasize that our current paper primarily focuses on uncovering key factors influencing scaling laws that were previously unaddressed or insufficiently discussed. While theoretical explanation is undoubtedly valuable, it is often the case in the deep learning community that empirical observations precede and inspire theoretical studies. For example:

• In He et al.'s ResNet work [1], skip connections were empirically shown to mitigate gradient vanishing issues prior to detailed theoretical analysis.
• Contrastive learning methods, such as SimCLR [2], were initially evaluated through empirical results before rigorous theoretical frameworks were developed.
• The GPT-3 work [3], where scaling phenomena were identified and validated empirically, lacked a complete theoretical understanding at the time of publication.

Thus, we believe our results similarly highlight a highly non-trivial open problem that merits further exploration. We identify this as an important direction for future work.

[1] Deep Residual Learning for Image Recognition. CVPR 2016.

[2] A Simple Framework for Contrastive Learning of Visual Representations. ICML 2020.

[3] Language Models Are Few-Shot Learners. NeurIPS 2020.

[W1] Theoretical Analysis: It would be ideal to provide a theoretical framework or intuition to explain the empirical observations, especially concerning why weaker teacher models can aid early learning and why STRATEGY C effectively reduces noise.

Thank you for your insightful feedback and interest in the theoretical aspects of our work, and we would also like to emphasize that the primary objective of this paper is to propose the novel Strategy C and systematically reveal its significance. While theoretical explanation is undoubtedly valuable, it is common in the deep learning community for empirical observations to precede and inspire theoretical studies, for example:

• Similar to the empirical findings in He's ResNet work in [1], where skip connections were observed to mitigate gradient vanishing issues prior to a detailed theoretical explanation.
• Contrastive learning methods, such as [2], were initially evaluated based on empirical results without rigorous theoretical justification.
• GPT-3 work[3], where the authors identified and empirically validated scaling phenomena without yet providing a complete theoretical understanding.

Thus, we believe our results similarly highlight an important open problem that merits further exploration. However, we are willing to provide some intuitive explanations to address the two key questions raised:

• Why weaker teacher models can aid early learning?

  • Compared to a highly accurate teacher model, a weaker teacher model typically outputs higher-entropy probability distributions than a stronger teacher. This increases the KL divergence between the teacher and student models at the beginning of training, leading to larger gradients and faster updates of the student model parameters.
  • However, as the teacher's accuracy decreases excessively (e.g., nearing uniform random predictions), the output target of the teacher model approaches a uniform distribution, reducing its informational content.

  Consequently, relatively weaker teacher models strike a balance: they provide sufficient signal to guide early learning while avoiding over-saturating the student model with overly confident predictions.

• Why does STRATEGY C effectively reduce noise?

  As noted in line 344 of the paper, STRATEGY C generates fewer distinct targets during training compared to STRATEGY B. For example, on CIFAR-10 with 100 training epochs, the ratios of sample-to-target mappings are:

  • STRATEGY A: 50,000×100:10=500,000:1,
  • STRATEGY B: 50,000×100:50,000=1:1,
  • STRATEGY C: 50,000×100:50,000=100:1.

  By reducing the number of distinct targets, STRATEGY C effectively minimizes the uncertainty introduced by noisier (more) targets. Our proposed Strategic C effectively addresses the short-term limitations of Strategy A and the long-term deficiencies of Strategy B, striking a balance between short-term acceleration and long-term convergence.

In a word, we believe this work is a step toward understanding how target mappings impact training dynamics across domains.

[1] Deep Residual Learning for Image Recognition. CVPR 2016.

[2] A Simple Framework for Contrastive Learning of Visual Representations. ICML 2020.

[3] Language Models Are Few-Shot Learners. NeurIPS 2020.

[Q1] "Have you explored applying STRATEGY C to larger datasets like ImageNet? What computational or methodological challenges do you anticipate in scaling up this approach?" Teacher Model Selection: How does the choice of teacher model architecture impact the student model's performance under STRATEGY B and STRATEGY C? $\searrow$ "Have you considered comparing the impact of different teacher model architectures (e.g., ResNet vs. Vision Transformer) on student performance under STRATEGY B and STRATEGY C?

Yes, we have explored applying STRATEGY B and STRATEGY C to larger datasets on ImageNet, and we also use ResNet50 and ViT as the backbone for model training, as shown in Figure 15 and Table 6, 7 for numerical results, Under different backbone settings, we also find that:

• Strategy A shows long-term advantages, while Strategy B exhibits short-term benefits.
• Our proposed Strategy C effectively addresses the short-term limitations of Strategy A and the long-term deficiencies of Strategy B.
• Moreover, the advantages of Strategy C become increasingly prominent when applied to weaker teacher models.

Regarding the computational challenges:

• Within the 250k training steps we tested, we have already found the performance improvements of STRATEGY C over Strategy B under different teachers.

• As discussed in [R1], the efficacy of STRATEGY C is highly dependent on the number of training steps, since STRATEGY C establishes a mapping between samples and targets proportional to the number of training epochs.

  Since we can find in the table that the Gain Strategy C over Strategy B consistently increases as the training process, we believe that the noise-reduction effect will become more significant with longer training steps.

Regarding the impact of different model architectures:

• The numerical results show that, due to the larger number of parameters in ViT compared to ResNet50, using ViT as the backbone to train the student model under Strategy C leads to more significant performance improvements within the same computational budget (i.e., the same number of training steps).

Regarding the scalability:

• In Section 6, we briefly highlight the applicability of our findings

  • We can extend our findings to larger-scale datasets like text-based tasks. This paper uses image classification as a case study to provide a concrete and focused demonstration of the proposed methods, with broader implications left for future exploration.
  • We would also like to emphasize that this work is the first to systematically emphasize that a weak teacher model can significantly contribute to the training of student models in the context of knowledge distillation.

  We believe this further supports the validity of our findings and their applicability under more settings.