Dear Reviewer iCAR,

We sincerely appreciate your thorough feedback and constructive comments, which we believe will significantly enhance the clarity and readability of our paper. We are deeply encouraged that you found our work novel and interesting. Moving forward, we are committed to addressing your inquiries and concerns in detail.

Your observation regarding numerical experiments comparing SANIA to other tuning-free methods is highly relevant and aligns with one of the directions we are actively pursuing in our ongoing research.

We apologize for not providing a clearer definition of the “interpolation condition.” To address this, we have added a concise paragraph that offers additional details and intuition about this assumption on page 3.

Reviewer: "In Page 2, SPS has been derived, why not just cite it?"

The main reason for including the derivation of SPS is to provide intuition for readers who may not be familiar with this method. We believe that presenting this compact and concise derivation enhances the coherence and accessibility of our paper without detracting from the key ideas discussed later.

Lastly, we have corrected minor grammatical errors and addressed the missed citations you kindly pointed out.

Thank you once again for your valuable feedback.

Dear Reviewer iCAR,

Thank you again for your thoughtful feedback on our paper. We have addressed all the questions and concerns you raised in our responses. We wanted to kindly follow up to see if you have any additional questions or require further clarifications. If our responses address your concerns, we would greatly appreciate it if you could consider updating your score to reflect this.

Thank you for your time and consideration.

Reviewer: "From the numerical results, it seems that the test accuracy of SANIA is slightly worse than existing methods. The author should also compare the results with SGD, since it usually has competitive generalization ability."

We agree that SGD is a strong baseline with well-established generalization capabilities. To address this, we have conducted additional experiments comparing SANIA to manually fine-tuned SGD on real datasets. These comparisons have been included in the Appendix E (Figure 5) . The results indicate that SANIA performs competitively with SGD while eliminating the need for a tuned learning rate schedule, highlighting its practical utility. Due to time constraints, we were unable to include large-scale experiments; however, we plan to explore a broader range of problems with SANIA in future work. If we obtain the results in time, we will add large-scale experiments by the end of the discussion deadline and upload the updated findings.

Thank you for this valuable suggestion.

Reviewer: "Due to the anisotropic property of the loss landscape, the estimate f∗ may not be a good guess for the local optimization. How will this affect the performance of the new method?"

Could the reviewer kindly elaborate on this comment?

We are grateful for your feedback and will incorporate detailed responses and additional experiments to address your concerns in the revised manuscript. Thank you for your valuable insights and suggestions.

Reviewer: "In Line 209, what is the meaning of 'Otherwise, we replaced step-size parameter γt to parameter fi∗?' The authors should explain more clearly."

We recognize that this statement required clarification. It has been revised, and we trust it is now clearer. Please refer to the end of page 4 for the updated wording.

Reviewer: "Instability of the Euler method and noises are believed to be helpful for generalization... This may be a severe problem for the new framework."

It is true that injecting noise can play a regularizing role in training for stochastic gradient descent methods. Please note that our method can, and does, take large step sizes, which allows it to jump out of local minima. Regarding the reviewer’s remark about our step size spontaneously inhibiting desired instabilities, we are unsure of the intended meaning. Could the reviewer kindly clarify their comment? If the concern pertains to whether SANIA’s step-size determination process prevents the addition of controlled noise or perturbations to the optimization process, the answer is no. In fact, it would be possible to explore how such modifications might enhance generalization performance, and we view this as an interesting avenue for future research.

We thank the reviewer for bringing the additional reference to our attention, and we will include it in the revised version of the paper.

Reviewer: "From the numerical methods, the new method does not show improvements compared to previous methods."

We acknowledge that SANIA’s performance, as presented, is comparable to existing methods and does not show significant improvements. One key difference, however, is that SANIA operates without requiring hyperparameter tuning, unlike traditional optimizers such as Adam or Adagrad, which often require careful tuning to achieve their best performance. This feature of SANIA can potentially help to save substantial amounts of resources. To further illustrate this, we have added additional experiments to Appendix E (Figures 3, 5, 6, 7) comparing SANIA’s performance against Adam, Adagrad, SGD and KATE under suboptimal tuning conditions and their best-tuned step sizes. These results highlight that SANIA’s performance remains robust across a wider range of scenarios.

Dear Reviewer zkYN,

We sincerely thank you for your thoughtful review of our work and for highlighting both the strengths and weaknesses of our proposed method. We deeply appreciate your acknowledgment of the soundness, ease of realization, and presentation quality of our approach. Your constructive feedback and insightful questions have guided us to identify areas for improvement and further experimentation. Below, we provide detailed responses to the points raised under the weaknesses and questions sections.

Dear Reviewer zkYN,

Thank you again for your thoughtful feedback on our paper. We have addressed all the questions and concerns you raised in our responses. We wanted to kindly follow up to see if you have any additional questions or require further clarifications. If our responses address your concerns, we would greatly appreciate it if you could consider updating your score to reflect this.

Thank you for your time and consideration.

We have included updated experiments with KATE, however due to time constraints we are still running experiments with Sophia. Nevertheless, we have found both of them to be highly relevant to our research and included in Lemma 5 in Appendix as special cases of SANIA General Framework

Thank you again for your valuable feedback and thoughtful suggestions.

Please let us know if some concerns are still unanswered.

Question: What are your thoughts on how learning rate schedules such as cosine decay, etc., compose with the $\lambda_t$ schedule defined in (19)? How does $\lambda_t$ change during training?

It is true that learning rate schedules such as exponential decay, cosine annealing, and other procedures are widely used in practice. However, one problem persists – different schedules are often suited to different scenarios. For example:

1. Exponential decay works best for problems where a steady reduction in the learning rate is advantageous.
2. Reduce-on-plateau is ideal for training pipelines where the convergence speed varies significantly.
3. Cosine annealing is well-suited for tasks that benefit from smooth reductions in the learning rate, such as fine-tuning.

In contrast, the $\lambda_t$ schedule has an update rule directly influenced by the function landscape, which may alleviate the need for manually selecting a specific learning rate schedule. To provide insights into the behavior of $\lambda_t$, we have included numerical experiments demonstrating the evolution of SANIA’s step size compared to fine-tuned methods and methods employing learning rate schedules.

Due to time constraints, we present these experiments using a synthetic dataset. Nonetheless, we are eager to investigate the behavior of $\lambda_t$ in large-scale problems as part of future work. Please find the aforementioned experiments in Figure 10 of the Appendix in the revised version of our paper.

Dear Reviewer zTra,

We are extremely delighted to learn that you found our work easy to understand, and we sincerely thank you for appreciating our theoretical analysis. We have carefully studied the two papers you kindly shared with us and found both to be highly relevant to our work. We have now referenced and discussed both of these works in the revised version of our paper.

Dear Reviewer zTra,

Thank you again for your thoughtful feedback on our paper. We have addressed all the questions and concerns you raised in our responses. We wanted to kindly follow up to see if you have any additional questions or require further clarifications. If our responses address your concerns, we would greatly appreciate it if you could consider updating your score to reflect this.

Thank you for your time and consideration.

Finally, we have updated Figure 1 based on your feedback and added training loss to our plots (Figure 5, 6 and 7). We hope you find the revised version to be an improvement

Thank you once again for your insightful comments and suggestions.

Question: Could the authors provide a formal link between scale invariance and using SANIA? It seems that there is some overlap between the two, but SANIA does not ensure scale invariance by itself. It would be interesting for potential users of SANIA to provide conditions for their algorithm to be scale-invariant.

Response: It is accurate that SANIA does not guarantee scale invariance by itself and to design an algorithm that enjoys this property one must ensure the following conditions are satisfied:

1. Diagonal Transformations: The algorithm should behave identically under transformations of the form $w \to Vw$, where $V$ is a diagonal, non-degenerate matrix. This ensures it is independent of feature scaling.

2. Invariant Preconditioning: Any preconditioning applied, such as in adaptive gradient methods, must adapt only to scales (diagonal transformations) and not to rotations. This implies avoiding operations like taking square roots of preconditioners if they break scale invariance.

3. Scale-Invariant Updates: The step size or learning rate, denoted as $\gamma_t$, should remain unaffected by scaling transformations. This requires careful design of step-size rules.

4. Consistent Function Values: The function values and gradients evaluated during optimization must remain consistent after scaling, ensuring identical convergence paths.

5. Bijective Mapping: There must be a one-to-one mapping between the scaled and original parameter spaces, ensuring equivalent updates in both spaces.

These principles are critical for algorithms like SANIA AdaGrad-SQR and SANIA Adam-SQR, which maintain identical performance across scaled datasets. For more proofs and details please refer to Appendix D.1 and D.2.

Reviewer: " At the end of Section 2, the authors use the notations: "SANIA $I_d$", "SANIA $(V^{-1})^2$" and "SANIA $\mathrm{diag}(H^{-1})$", while writing that there is no preconditioning. Thus, I understand that "SANIA $A$" refers to Eqn. (6) with $D_t = A$. But there is no formal definition of "SANIA $A$". It should appear somewhere. "

We apologize for any misunderstanding regarding the notation used to describe different preconditioning matrices for SANIA. In Section 2.4 (Affine and Scale Invariance), we discuss several matrices used for preconditioning in SANIA to illustrate scale invariance (note that choice of $I$ as preconditioning will not lead to invariance). Before detailing them, we would like to remind you that the update rule for these methods is described in Eq. (18):

In this update rule, the selection of $B_t$ plays a crucial role in the algorithm, and Section 2.4 proposes choices for $B_t$ that ensure SANIA remains scale invariant.

1. SANIA $diag((V^{-1})^2)$: One option is to utilize the squared inverse of the vector $V$ used to scale the dataset, which simulates badly scaled data.
2. SANIA $diag(H^{-1})$: Another approach is to calculate the diagonal of the Hessian matrix.
3. SANIA $\mathit{I}_d$: As a baseline ablation study, we include SANIA $\mathit{I}_d$, where $B_t = \mathit{I}_d$, effectively using no preconditioner.

Since Section 2.4 does not mention SANIA $A$, could you please clarify what you are referring to as SANIA $A$?

Dear Reviewer jmo1,

We sincerely thank you for your thorough review, positive comments on the originality and clarity of our paper, and for recognizing its significance to the optimization community. We have carefully considered your remarks, concerns, and questions, and we will do our utmost to address them effectively.

Dear Reviewer jmo1,

Thank you again for your thoughtful feedback on our paper. We have addressed all the questions and concerns you raised in our responses. We wanted to kindly follow up to see if you have any additional questions or require further clarifications. If our responses address your concerns, we would greatly appreciate it if you could consider updating your score to reflect this.

Thank you for your time and consideration.