LeRaC: Learning Rate Curriculum

• In Figure 2, the authors present a straightforward example illustrating the relationship between shallow and deep features. This example is intuitive and easy to grasp; however, there are some points open for discussion. For instance, the statement "as the information in x is lost" might not necessarily hold when utilizing random convolutional kernels or random transformations, as seen in the popular diffusion model's noise injection process. Therefore, I suggest rephrasing this part.

Reply: We agree. We wanted to express the fact the signal-to-noise ratio drops from one layer to the other. In the revised manuscript, we rephrased the statement accordingly.

• Regarding the example mentioned earlier, I believe the network's representation should be considered holistically. Is it meaningful to discuss the representations of individual layers separately? Will merely increasing the learning rate for shallow layer parameters lead to faster convergence of shallow network parameters? The first theorem seems insufficient to address this issue.

Reply: In principle, a larger learning rate implies a larger update. However, we do agree that if the learning rate is too high, the model can actually diverge. This is because the gradient describes the loss function in the vicinity of the current location, providing no guarantee for the value of the loss outside this vicinity. Our implementation actually takes this into account. Instead of increasing the learning rate for earlier layers, it reduces the learning rate for the deeper layer to avoid divergence. More precisely, we set the learning rate for the first layer $\eta_1^{(0)}$ to the original learning rate $\eta^{(0)}$ and the other initial learning rates are gradually reduced with each layer i. During training, the lower learning rates are gradually increased, until epoch k. Hence, LeRaC actually slows down the learning for deeper layers, until the earlier layers have learned “good enough” features. We clarified this point in the revised supplementary.

• In Equation (9), the author mentions, "we empirically observed that an exponential scheduler is better." It would be beneficial for the author to provide an insightful explanation as to why the exponential scheduler is superior to linear or logarithmic schedulers. Are there any other potentially better scheduling methods, such as cosine learning rate variations?

Reply: We believe that a suitable scheduler is one that adjusts the learning rate at layer i proportionally to the estimated signal-to-noise drop at layer i. To demonstrate this, we have plotted the average SNR / entropy of randomly initialized LeNet layers for CIFAR-100 images. The new plot shown in Figure 3 added to the revised supplementary shows that the SNR / entropy drops exponentially, from one layer to the next. This can explain why the exponential scheduler is a more suitable choice. We thank the reviewer for this comment, which lead us to discovering a better explanation for our design choices.

• It is difficult to understand why LeRaC is effective. The motivation is questionable. The random parameter initialization resulting in a propagation of noise can be well addressed with a standard warm-up strategy.

Reply: In the supplementary, we provide a theoretical proof (Section 1) and an empirical proof (Section 2) of our motivation, i.e. the fact that “a random parameter initialization results in a propagation of noise”. It is therefore not clear why our motivation is “questionable”. If the reviewer considers that the theoretical proof is wrong, we kindly ask the reviewer to be more specific and indicate the mistake. If the reviewer considers that the empirical results are not credible, we kindly ask the reviewer to try out the code provided with the supplementary. How can we further convince the reviewer, aside from giving a theoretical and an empirical proof? Regarding the warm-up comment, we underline that all the CvT-13 results based on the “conventional” regime reported in Table 2 use Linear Warmup with Cosine Annealing, this being the recommended scheduler for CvT (Wu et al., 2021). When we introduce LeRaC, we simply deactivate Linear Warmup with Cosine Annealing between epochs 0 and k. Hence, our experiments already include a direct comparison between a widely-used warm-up strategy and LeRaC. The results show that LeRaC leads to significantly better results, according to McNemar testing at a p-value of 0.001. This aspect was too already clarified in the supplementary.

• More theoretical analysis on the effectiveness of LeRaC will make this paper more convincing.

Reply: We provided a theoretical proof of the underlying assumption of our method in the supplementary. In general, the reviewer raised many issues that were already solved in the supplementary, which makes us believe that the reviewer did not read the supplementary material. Aside from the theoretical proof (Section 1 from the supplementary), we also present several experiments to empirically demonstrate the underlying hypothesis / motivation of LeRaC in Section 2 from the supplementary. Moreover, the comprehensive evaluation on 10 data sets from different domains based on a large variety of neural architectures clearly indicate that LeRaC is consistent in surpassing the standard training regime, as well as other curriculum-based methods. Although we understand that reviewers have limited time to review, we kindly ask the reviewer to browse through the supplementary material. We strongly believe that the (revised) supplementary material addresses all the concerns raised by the reviewer.

• The results on full-ImageNet are absent, which I think is necessary.

Reply: We have limited computational resources and we considered that it is more important to present results on 10 data sets from multiple domains (vision, language, audio) and test multiple architectures. Hence, we sampled 200 classes from ImageNet-1K and performed the experiments on this subset (see results on ImageNet-200 shown in Table 2). Moreover, we observe that not all the references indicated by the reviewer, e.g. [*1], report results on ImageNet, confirming that full ImageNet experiments are not always a necessity. Still, in the limited time given for the rebuttal, we were able to run ResNet-18 based on LeRaC on the full ImageNet. The results are shown in the newly added Table 10 from the updated supplementary. Our improvements on ImageNet-1K over CBS and [*4] are close to 1%, further confirming the consistency of LeRaC, which we already observed on 10 data sets from multiple domains. We thank the reviewer for this suggestion, which greatly enhances our empirical validation.

• The authors may consider citing [*1-*4] and comparing with them.

Reply: First, please note that we already have a comparison with [*3] in the supplementary (see Table 7 from the supplementary), showing that our method achieves superior results. Moreover, we note that [*4] should be regarded as concurrent work, since ICCV 2023 took place after the submission deadline for ICLR 2024. Nevertheless, we compare with [*4] in the newly added Tables 10 and 11 from the updated supplementary. Table 10 shows that our improvement on ImageNet-1K over [*4] for ResNet-18 is 0.96%. Table 11 indicates that we obtain similar improvements on CIFAR-10 and CIFAR-100, where LeRaC surpasses EfficientTrain [*4] in 3 out of 4 cases. In summary, the added results are favorable to LeRaC. We also note that the code for [*1] is not publicly available. We asked the authors to share the code, but they were not able to share it because it depends on a private package. The code for [*2] is available, but it does not seem to work, i.e. the loss becomes NaN after a few epochs, in spite of our efforts to repair the code and make it work. We thank the reviewer for this recommendation, which strengthens our empirical validation.

• Marginal empirical gain. As shown in the Table 2 and 3, most of the gain over CBS, which is the direct competitors, is less than 1 or 2%.

Reply: To determine if the reported accuracy gains observed for LeRaC with respect to the baseline are significant, we applied McNemar significance testing to the results reported in the main article on all 10 data sets. There are 26 test cases presented in Tables 2, 3 and 4 from the main paper. In 20 of 26 cases, we found that our results are significantly better than the corresponding baseline, at a p-value of 0.001. This confirms that our gains are statistically significant in the majority of cases. The results of the statistical tests were already included in the supplementary. Moreover, we underline that CBS leads to performance drops with respect to the conventional regime in 12 out 26 cases, while LeRaC improves the performance in each and every case. In other words, using CBS, one can expect performance drops in almost half of the test cases. In the experiments presented on 10 data sets, this does not happen with LeRaC. Hence, there is clear evidence showing that our improvements are not marginal, at least in about 50% of the cases, i.e. in the cases when CBS leads to performance drops.

• Method is too simple without intuitive ground that it should work better than others.

Reply: In the supplementary, we provided a theoretical proof (Section 1) and an empirical proof (Section 2) of our intuitive ground presented in the introduction. The reviewer raised many issues (including this one) that were already solved in the supplementary, which makes us believe that the reviewer did not read the supplementary material. Although we understand that reviewers have limited time to review, we kindly ask the reviewer to browse through the supplementary material, which strengthens our intuition presented in the main paper with comprehensive evidence.

• Although the analysis is intuitively sensible, the simplicity of the method brings marginal gain over CBS, even with the quite thorough ablation study. As authors mentioned, the empirically chosen exponential based method may not be the best choice or the intuition of noise amplification may not be a serious problem. Given the results, it is difficult to judge the main reason for the unsatisfactory performance.

Reply: In the revised supplementary, we added new evidence showing that the SNR from one layer to the next drops exponentially (see Fig. 3 from the revised supplementary). This justifies why the exponential scheduler performs better. As mentioned above, we conducted statistical tests that clearly indicate that our performance gains are statistically significant, in the majority of cases (20 out of 26). Moreover, LeRaC surpasses CBS in 24 out of 26 test cases. we kindly ask the reviewer to read the supplementary, which contains convincing evidence against regarding the performance “unsatisfactory”.

• Can you elaborate why the proposed layer-wise LR learns better than the previous work?

Reply: We believe that both CBS and LeRaC share the same intuition and they are both simple methods. However, LeRaC is based on a more direct implementation, operating directly over the learning rates of neural layers, while CBS requires inserting additional architectural components, i.e. convolutions with Gaussian blur kernels. This reduces the applicability of CBS to neural architectures for which the application of blur kernels actually makes sense. The blur kernels also increase the training time of CBS (as per Fig. 1 from the supplementary). In summary, LeRaC has several advantages, which are demonstrated across a wide range of neural architectures and domains.

• Contrast the difference of the proposed method to the existing CBS.

Reply: Unlike CBS, LeRaC does not introduce new operations, such as smoothing with Gaussian kernels, during training. As such, our approach does not increase the training time with respect to the conventional training regime, as shown in Fig. 1 from the supplementary. Another clear difference is the fact that LeRaC works for a wider variety of architectures and domains. This is confirmed by our experiments, where LeRaC brings performance gains across 26 test cases, while CBS leads to performance drops in 12 test cases.

• How much of empirical significance is there for choosing exponential based method?

Reply: As suggested, we performed additional statistical tests to compare the results of the exponential scheduler with the linear and the log schedulers. The tests show that the exponential is significantly better, at a p-value of 0.05. New results shown in Fig. 4 and 5 from the revised supplementary further support the exponential scheduler. Moreover, we added new evidence showing that the SNR from one layer to the next drops exponentially (see Fig. 3 in the revised supplementary). This further justifies why the exponential scheduler performs better.

• The experiments focus on architectures and tasks and some ablation studies, with little analysis provided to empirically demonstrate the claims in the paper. A convincing argument to demonstrate the properties of the learning process compared against a number of learning rates, schedules, etc, must be provided.

Reply: We thank the reviewer for reading the supplementary material. As recommended, we included additional graphs in the revised supplementary to show the effect of various learning rates and schedulers on the learning progress. More specifically, we present results with various schedulers in Figures 4 and 5 from the revised supplementary, and results with various learning rates in Figures 6, 7 and 8. The new results further confirm the benefits of the exponential scheduler and the use of gradually decreasing initial learning rates. We thank the reviewer for this observation, which improves our validation.

• The interplay between this learning rate scheduler and initialisation/optimisers has not been studied.

Reply: Indeed, this point was not well highlighted in the paper. We employed LeRaC on a high variety of architectures, using the most suitable optimizer for each architecture. Hence, LeRaC was tested in combination with SGD, Adam, AdamW and AdaMax (as per Table 1). LeRaC brings performance improvements, regardless of the selected optimizer. This demonstrates that LeRaC is compatible with a variety of widely-used optimizers. Regarding initialization, please note that some of the architectures are trained from scratch (and use Glorot init), while others come with pre-trained weights (e.g. CvT-13 pre-trained, BERT pre-trained). Once again, LeRaC brings performance improvements, regardless of the weight initialization. This confirms that LeRaC is suitable for both fine-tuning and training from scratch. Moreover, we underline that all the CvT-13 results based on the “conventional” regime reported in Table 2 use Linear Warmup with Cosine Annealing, this being the recommended scheduler for CvT (Wu et al., 2021). When we introduce LeRaC, we simply deactivate Linear Warmup with Cosine Annealing between epochs 0 and k. Hence, our experiments already include a direct comparison between a widely-used warm-up strategy and LeRaC. The results show that LeRaC leads to significantly better results, according to McNemar testing at a p-value of 0.001. This indicates that LeRaC can successfully replace widely-used initialization schemes.

• Can the improvement in learning dynamics be demonstrated empirically in a wide variety of settings as critiqued above? How does LeRaC interface with different initialisers/optimisers?

Reply: The questions correspond to the identified weak points, which we already addressed above.