To Reviewer tuob

We greatly appreciate your insightful review, which has highlighted key areas for enhancement in our work. We are addressing your concerns and suggestions as detailed below.

Quick Guide:

1. Regarding Weakness points 1 and 2: We would like to discuss how SoftMax is much more efficient than LGL2O in terms of computational cost and memory usage.
   1. Memory usage： Our method, utilizing SoftMax, does not add notable memory overhead compared to standard optimization methods. SoftMax operates on already computed gradient matrices during backpropagation, and thus, doesn't require additional memory for storing new matrices or tensors. LGL2O, on the other hand, may require more memory to store intermediate results, particularly the gradients, and parameters updated by both optimizers, which could significantly increase the memory requirement, especially for large models.
   2. Computational cost： The HUB approach introduces minimal computational overhead through the SoftMax function. Each layer's time complexity is $O(N)$ (with $N$ denoting the number of parameters per layer), leading to a total complexity of $O(L⋅N)$ for $L$ layers. Since $N$ is substantially smaller than the total operations involved in training (encompassing forward and backward passes, along with other computations like batch normalization), the additional cost of HUB is akin to that of layer normalization, which also normalizes on a layer-by-layer basis. Conversely, LGL2O demands a more resource-intensive process. It requires separate model updates using both a learned optimizer and a hand-designed optimizer before each model update, followed by loss computation for the subsequent step. The choice of optimizer is then based on which one results in a lower forthcoming loss. This method essentially doubles the update steps and significantly heightens computational demands, especially in comparison to the relatively straightforward SoftMax calculation in HUB.
2. Regarding Weakness point 3: Our method is inspired by the concept of prompt tuning, wherein both strategies aim to maximize the latent potential of pre-trained models without necessitating extensive additional training. Prompt tuning typically guides a pre-trained model to adapt to new tasks or data by incorporating human-understandable rules and templates into the input, thereby unlocking the model's inherent capabilities. Similarly, the HUB strategy enhances a learned optimizer's performance by directing it with hand-designed rules. This approach aligns with our overarching goal of optimizing the utilization of existing models and optimizers, efficiently and effectively, without demanding significant extra resources.

Your feedback is invaluable in refining our work, and we hope this response adequately addresses your points, offering enhanced clarity and depth to our submission.

References

[A] Pr'emont-Schwarz, Isabeau et al. “A Simple Guard for Learned Optimizers.” ArXiv abs/2201.12426 (2022): n. pag.

To Reviewer (i8fH)

Thanks for the feedback, we appreciate your time and effort to evaluate our work. We will address your concerns and suggestions in the following sections.

Quick Guide:

1. Regarding Weakness point 1: In our opinion, the primary objective of training is always to achieve optimal performance within the allocated time constraints. Therefore, there should be no prohibition on utilizing either LO or hand-designed optimizers, or even a combination of both, as long as they contribute towards achieving this goal. Prior works such as GL2O and LGL2O have been accepted by the community, indicating a recognition of the value in leveraging a combination of approaches. Importantly, the incorporation of human-guided rules into deep neural networks is a common practice in various domains, including NLP and CV (e.g., prompt tuning). Our proposed method provides a plug-and-play solution for effectively integrating human rules to guide learned optimizers. Furthermore, based on experimental results in Sections 3.2 and 4.1, we believe that we provide strategies enabling LO to exhibit stable convergence, as previously mentioned. Notably, when dealing with tasks in distribution settings, VeLO demonstrates significantly faster convergence compared to tuned Adam and AdamW (refer to Figure 2(c), Figure 4(b), Figure 5, and Figure 12).
2. Regarding Weakness point 2: As you correctly stated, 'learned optimizer can be unstable near optima', it also implies some times learned optimizer is stable. In fact, we have demonstrated in Section 3.2 and 4.2 that VeLO can achieve fast and stable convergence in in-distribution settings without any guidance. So the rationale behind our methods is to give chance for VeLO to explore the parameter space and then use the hand-designed optimizer to stabilize the training process once it deviates too far. This process may face fluctuations, as shown in Section 3.2, but compared to other competitors, we have shown that HUB can better facilitate convergence.
3. Regarding Weakness point 3 and 4: The work you provided is not directly comparable to ours, as they discuss how to train a more powerful learned optimizer from scratch. In contrast, our method aims to improve the stability and generalization ability of a pre-trained learned optimizer in a plug-and-play manner, without additional training. Furthermore, our baseline VeLO already adopts a layer-wise LSTM structure, which is similar to the coordinate-wise RNN structure proposed in this paper. The stability results on ADAM have already been proven in the original paper on Adam, and here, we provide a concise proof in Sec B.2 in the Appendix. Our primary purpose in proving this is to offer a better understanding of the stability of hand-designed optimizers.

In conclusion, we believe that our method is a simple yet effective solution for enhancing the stability and generalization ability of learned optimizers. We hope that our response has addressed your concerns.

References

[A] Metz, Luke et al. “VeLO: Training Versatile Learned Optimizers by Scaling Up.” ArXiv abs/2211.09760 (2022): n. pag.

[B] Chen, Reddi et al. "Learning to learn with better convergence"

[C] Kingma, Diederik P. and Jimmy Ba. “Adam: A Method for Stochastic Optimization.” CoRR abs/1412.6980 (2014): n. pag.

To Reviewer (BBSt)

Thank you for your meticulous and invaluable feedback. We genuinely appreciate the time and effort you dedicated to evaluating our work, and we welcome this opportunity to clarify and enhance our study based on your input.

Quick Guide:

1. Regarding Weakness Point 1: While we acknowledge the effectiveness of alternative approaches in controlling learned optimizers, our methodology, as extensively outlined in the entire second paragraph of our introduction, primarily focuses on addressing the specific challenges associated with fine-tuning or training a learned optimizer. Our method and baselines are designed to be easily integrated and offer simplicity and effectiveness at a lower implementation cost. This provides us with a distinct advantage over some of the methods you've mentioned, which will be further elaborated on below in Detailed Discussion.
2. Regarding Weakness Point 2: With regards to your suggestion of utilizing early stopping, it is indeed a widely employed technique in practical scenarios. However, for all the experiments conducted in Section 4, we have already compared against the historical best evaluation checkpoint, which surpasses the effectiveness of simple early stopping. Moreover, Section 3.2 primarily focuses on employing a setup function to demonstrate how HUB's negative feedback attribute offers enhanced control over learned optimizers even during extended training steps. Our ultimate objective for this experiment renders it unsuitable for implementing early stopping.
3. Regarding Weakness Points 3, 4, and 5: We would like to apologize for the omission of a full stop sign after Section 4.2, paragraph 1. We assure you that we will thoroughly review this issue along with other typographical errors, inappropriate notations, and spacing inconsistencies in order to rectify them before the camera-ready version is submitted. Furthermore, we would like to clarify that Sections 4.1 and 4.2 primarily aim to provide a comprehensive overview and empirical evidence of the effectiveness of our methodology at a high level. As mentioned at the beginning of Section 4, for an extensive and detailed analysis of the experimental setup, methodology, and results, please refer to Section A in the appendix.
4. Regarding Question Point 1: Yes, we can certainly provide some examples. In the technique report of VeLO, specifically in Section 4.4 titled "Limitations and Failure Cases," the author enumerates four prevalent failure scenarios. By analyzing the loss curve presented in these cases, it becomes evident that failures often manifest during the middle of training with abrupt fluctuations in loss values. Subsequently, this initial fluctuation leads to increased instability and eventual failure of the training process. Remarkably similar to our demonstrated failed case in Section 4.1 paragraph 2 when dealing with unseen LTC architecture, a sudden loss fluctuation occurs followed by NAN.

To Reviewer (dXMr)

First, we greatly appreciate receiving such a valuable comment with a clear and specific description of your concerns. It gives us a sense of accomplishment and we are committed to addressing the weak point you have pointed out in order to make a more solid contribution to the community.

Quick Guide:

1. Regarding Weakness point 1: In brief, We would offer a more theoretical elucidation of the computational complexity of HUB in Part 1 of our response. Additionally, we apologize for seeking clarification, but it is imperative for us to ensure that we comprehensively understand and accurately explain your concern. Based on our best knowledge, neither a hand-designed optimizer nor a learned optimizer can guarantee unconditional convergence. Besides, in LGL2O, they already proved conditional convergence can be guaranteed with a hybrid scheme. Hence, within Part 1 of our response, we intend to dissect this question into various scenarios and provide an elaborate explanation.
2. Regarding Weakness point 2: We apologize for the compact table arrangement due to page limitations, which may cause confusion. However, we made efforts to ensure that the tables are placed near their corresponding paragraphs for easy reference. Additionally, we are pleased to note that two other reviewers have given positive ratings on the presentation of our paper. Nevertheless, we understand that subjective opinions can vary and would greatly appreciate more specific suggestions on improving any unclear aspects of our presentation.
3. Regarding Weakness point 3: In Sec. C of our supplementary material, we have discussed three types of variations of HUB, and one of the variations - the Clipping HUB sets a bound for the hybrid ratio to avoid allocating very small weights to one of the optimizers. We find that this variation can be useful when the network is extremely wide/narrow. However, the improvement is marginal because the weight allocation is dynamically adjusted during training, we would update the figure in the rebuttal PDF to show the result of the variations of HUB.
4. Regarding Question point 1: As you correctly stated, in order to reduce cost, HUB uses a shared gradient matrix for computing the hybrid ratio. However, this is impossible in LGL2O, because the main idea for LGL2O is to refer to future loss. In other words, LGL2O needs to compute the gradient matrix for both optimizers and then compute the loss for both optimizers, and then compare the loss to decide which optimizer to use. If we try to share the gradient matrix, the loss will be the same for both optimizers, and the comparison will be meaningless.
5. Regarding Question point 2: As mentioned in Sec. A of our supplementary material, we used the pre-trained model Xception and scratch model ResNet-50 for the JAX framework from our codebase https://github.com/abarcel/haikumodels. This codebase gives a basic implementation of the models and the results are matched to the original Deep Residual Learning for Image Recognition paper (https://paperswithcode.com/paper/deep-residual-learning-for-image-recognition). The >95% Top1 accuracy you mentioned is probably using the data-augmentation technique shown in ResNet strikes back: An improved training procedure in timm(https://paperswithcode.com/paper/resnet-strikes-back-an-improved-training).