AdaFisher: Adaptive Second Order Optimization via Fisher Information

Dear Reviewer K3HY,

Thank you for your positive feedback and for highlighting the strengths of our work. We appreciate your recognition of the novel diagonal block-Kronecker approximation for balancing efficiency and curvature use, AdaFisher’s strong empirical performance across benchmarks, its stability in varying hyperparameters, and the rigorous theoretical convergence analysis. Your insights validate our contributions and encourage further improvements.

In response to the comments in the Weaknesses and Questions sections:

[R-K3HY-C1] Regarding the non-convex optimization: Thank you for raising this important point. We acknowledge that many DNN architectures introduce nonsmoothness due to operations like ReLU and max pooling. While our theoretical analysis primarily focuses on regret bounds for convex and nonconvex optimization, nonsmoothness presents additional challenges that are not explicitly addressed in the current version of our work. That said, nonsmoothness is common in modern deep learning, and optimizers like AdaFisher are designed to handle such challenges in practice, even if theoretical guarantees are harder to establish. For example, subgradient methods have been shown to work well with nonsmooth objectives, and many nonsmooth functions encountered in DNNs (e.g., ReLU) are piecewise linear, which can still be effectively optimized using standard techniques [1]. As the reviewer suggested, we have included a discussion in Section 3.4 in the revised manuscript (highlighted in blue text) to clarify the impact of nonsmoothness and its implications for our theoretical framework. Additionally, we emphasize that our empirical results demonstrate the robustness and effectiveness of AdaFisher, even in the presence of nonsmooth operations in DNNs.

[R-K3HY-C2] Hyperparameter Tuning: We thank the reviewer for highlighting the missing information on the number of epochs in Appendix D. We used 50 epochs for each fine-tuning experiment and have now clarified this in detail in the revised manuscript to enhance reproducibility and transparency.

[R-K3HY-C3] Comparison with AdamW: We appreciate the reviewer’s suggestion to clarify the comparison with AdamW. This is a typo—Adam/AdamW and AdaFisher/AdaFisherW are interchangeable when it comes to CNNs and ViTs. Hence, we compared AdaFisherW and AdamW across all ViTs and transformer-based models. In Table 2, we now specify that all ViT experiments (including Tiny Swin, FocalNet, and CCT-2/3x2) used AdamW and AdaFisherW, given its well-known performance with transformer architectures, while Adam is more effective in CNN-based architectures [2]. This detail has been added in the revised manuscript. Additionally, Figures 16–19 in Section D.2.4 illustrate that AdamW is indeed represented in the ViT experiments, showing the corresponding loss and test error.

[R-K3HY-C4] Regarding Figure 1: We thank the reviewer for highlighting the variability in convergence across multiple runs. Sensitivity to initialization is a known phenomenon that impacts all optimizers, including AdaFisher, as parameter initialization plays a critical role in shaping training dynamics and the resulting local minima [3,4]. To address this, we have added the following sentence to the revised manuscript in Appendix C: “Note that while the results may vary due to the stochastic nature of parameter initialization, AdaFisher typically converges to a better local minimum compared to its counterparts.” This statement reflects the observed behavior of AdaFisher in our experiments.

[R-K3HY-C5] Regarding Algorithm 1: We thank the reviewer for their question regarding the choice of values for $\gamma_1$ and $\gamma_2$. As detailed in Appendix D.2.1 of the revised manuscript, the current best-performing values for these decay factors in AdaFisher are $\gamma_1 = 0.92$ and $\gamma_2 = 0.08$, which naturally sum to $1$. As also raised by Reviewer jye4, [R-jye4-C2], we acknowledge that defining both $\gamma_1$ and $\gamma_2$ as independent parameters is not strictly necessary as $\gamma_2$ is derived directly as $1 - \gamma_1$. For transparency and potential flexibility in future applications, we initially presented them as separate parameters to allow for scenarios where independent tuning might prove beneficial. In light of this, we have revised the manuscript to clarify that $\gamma$ (previously $\gamma_1$) alone is sufficient, with $\gamma_2$ being implicitly defined.

[R-K3HY-C6] Related Work: We thank the reviewer for highlighting the omission of recent second-order optimizers, such as Sophia and INNAProp, from the related work section. We have revised this section to include these and other relevant optimizers, providing a more comprehensive overview of advancements in adaptive second-order methods.

[R-jye4-C4] Comparison with consistent epoch counts: We thank the reviewer for the insightful question. In our initial experiments, we evaluated the performance of the optimizers using the Wall-Clock-Time (WCT) technique, as it ensures a fair comparison by allocating the same computational budget for each optimizer. This approach aligns with widely accepted practices for evaluating optimizers under realistic time constraints [3,4]. To address the reviewer’s concern, we have additionally compared all optimizer baselines, including EVA and AdaFactor (as suggested by Reviewer jcNA), under the same number of epochs. Specifically, we conducted experiments on ResNet-18, ResNet-50 and MobileNet-V3 on CIFAR100 for 200 epochs. The results, now included in the revised manuscript under Appendix D.2.6, demonstrate that while Shampoo achieved slightly better performance than AdaFisher on the ResNet-18 network, its training time was approximately eight times longer than that of AdaFisher. Conversely, AdaFisher outperformed all other optimizers, including Shampoo, in the MobileNet-V3 and ResNet-50 experiments, delivering superior results with significantly better efficiency compared to the other second order optimizers. Additionally, we have summarized these findings in the following table 1 which provides a comprehensive overview of the highest test accuracies and training efficiencies for all methods discussed.

Table 1: Performance comparison of AdaFisher and other optimizers using (a) ResNet-18, (b) MobileNet-V3, and (c) ResNet-50 on CIFAR100 for 200 epochs.

(a) ResNet-18

(b) MobileNet-V3

(c) ResNet-50

[R-jye4-C5] Gamma Fine Tuning: We thank the reviewer for raising this point. As clarified in our previous response, [R-jye4-C2], only one parameter ($\gamma$, previously $\gamma_1$) is necessary for the AdaFisher optimizer, with $\gamma_2$ implicitly defined as $1 - \gamma_1$. For all experiments in our work, $\gamma$ was fixed at $0.92$, and no extensive tuning was required. This consistency across tasks demonstrates that AdaFisher is not only effective but also user-friendly, as it avoids the need for expensive hyperparameter tuning. This simplicity makes AdaFisher a practical and efficient optimizer for a wide range of applications.

[R-jye4-Concluding Remarks] We thank the reviewer for their time and effort in reviewing our work and we hope the reviewer would kindly consider a fresh evaluation of our work given the main clarifying points outlined above.

References

[1] Zou, Difan, et al. "Understanding the generalization of adam in learning neural networks with proper regularization." arXiv preprint arXiv:2108.11371 (2021).

[2] Zhang, Yushun, et al. "Why transformers need adam: A hessian perspective." arXiv preprint arXiv:2402.16788 (2024).

[3] Kaddour, Jean, et al. "No train no gain: Revisiting efficient training algorithms for transformer-based language models." Advances in Neural Information Processing Systems 36 (2024).

[4] Ye, Huigen, Hua Xu, and Hongyan Wang. "Light-MILPopt: Solving Large-scale Mixed Integer Linear Programs with Lightweight Optimizer and Small-scale Training Dataset." The Twelfth International Conference on Learning Representations.

Dear Reviewer jye4,

We sincerely thank the reviewer for their thorough evaluation and insightful questions regarding our work to be innovative, providing competitive results and well-organized. Their constructive feedback has helped us alleviate the quality of our manuscript and clarify key points about the AdaFisher optimizer. To address the reviewer’s concerns comprehensively, we have conducted additional experiments and provided detailed explanations for each query. For the ease of the reviewer, all new content and revisions in the manuscript are highlighted in blue text.

Below, we provide responses to each reviewer’s questions.

[R-jye4-C1] Pseudo-code implementation: We thank the reviewer for pointing out the typo in the indexing of the Kronecker factor $\mathcal{H}$. This has been corrected in the updated pseudo-code to ensure consistency. As for clarity, we recall that $\mathcal{H}\_{i-1}$ is the output of layer $i−1$ which also corresponds to the input activation of layer $i$. Specifically, $\mathcal{H}\_0$ corresponds to the batched input data ($x$) for the first layer ($i=1$). This was purely a typographical error, and the updated manuscript reflects the correct implementation. Thank you for bringing this to our attention.

[R-jye4-C2] Gamma parameters: We appreciate the reviewer’s keen observation regarding the non-necessity of using two separate parameters for the EMA of the Kronecker factors $\mathcal{H}$ and $\mathcal{S}$ in the AdaFisher optimizer. Upon the reviewer’s suggestion, we acknowledge that introducing both $\gamma_1$ and $\gamma_2$ is redundant, as $\gamma_2 = 1 - \gamma_1$ in our experiments. Specifically, we used $\gamma_1 = 0.92$ and $\gamma_2 = 0.08$, which naturally sum to $1$. However, for transparency and clarity in implementation, we initially opted to define them as separate parameters to allow for potential flexibility in future extensions or scenarios where independent tuning may provide benefits. In light of the reviewer’s comment, we revised the manuscript to reflect that only one parameter, $\gamma$, is strictly necessary.

[R-jye4-C3] SGD Experiments: We thank the reviewer for emphasizing the importance of including SGD experiments. In response, we have added SGD results for image classification tasks in the revised manuscript. We fine-tuned the learning rate and the momentum hyperparameters as described in Appendix D.2.1. Prior work [1] highlights that while SGD often outperforms Adam on CNN architectures, it underperforms on Transformers [2]. Nevertheless, as shown in Table 2, AdaFisher consistently surpasses SGD, even in CNN based architectures, demonstrating its superior effectiveness across these settings.

Thanks for the clarification, but I still have some questions.

• Regarding $\gamma$ parameter. In your original manuscript, $H_i = \gamma_1 H_{i-1} + (1-\gamma_2) \hat{H}_{i}$. If $\gamma_1 = 1 - \gamma_2$, we have $H_i = \gamma_1 H_{i-1} + \gamma_1 \hat{H}_{i}$. But in your revised manuscript, I see $H_i = \gamma H_{i-1} + (1-\gamma) \hat{H}_{i}$. So I guess you used $\gamma_1 = \gamma_2$, right? Moreover, in your revised manuscript, you mentioned that AdaFisher Decay Factors: The decay factor γ for AdaFisher was tuned within {0.1, 0.2, . . . , 0.9, 0.99}. The optimal value is: γ = 0.92.. However, I noticed that 0.92 is not within the range you specified.
• Confusion about SGD's performance. Regarding the performance of SGD on the ResNet18 model with the CIFAR-100 dataset, I saw that the value in Table 2 of Reference [1] is 78.0 and in Reference [2] is 79.49. However, in your experiment, it is only 76. Reference [1] use a batch size of 128 and train 200 epochs; Reference [2] use a batch size of 128 and train 400 epochs. I am not sure if the difference between the author's results and others' results is caused by the batch size. Could you run an experiment of SGD with a batch size of 128 in your benchmark?

[1] Liu, Y.; Mai, S.; Cheng, M.; Chen, X.; Hsieh, C.-J.; You, Y. Random Sharpness-Aware Minimization. In NeurIPS 2022; 2022.

[2] Yue, Y.; Jiang, J.; Ye, Z.; Gao, N.; Liu, Y.; Zhang, K. Sharpness-Aware Minimization Revisited: Weighted Sharpness as a Regularization Term. In KDD 2023; 2023.

Dear Reviewer 5WtX,

We thank the reviewer for their feedback, highlighting the strength of the paper of using a novel FIM for preconditioning the gradient and mentioning the related work. Please note, in addition to our official comments in this section, we have modified parts in the revised manuscript in blue color.

Below are the responses to the comments in Weakness and Question sections:

[R-5WtX-C1] Comparison between FOOF and AdaFisher: Please note we have cited the paper suggested by the reviewer “Kronecker-Factored Second-Order Optimizers Perform First-Order Descent on Neurons” from NeurIPS2021-OPT Workshop by Frederik Benzing in Section 5 (Related Work) from the revised manuscript. Moreover, we provide the following evidence that Fast First-Order Optimizer (FOOF), the optimizer introduced by Benzing, and AdaFisher are fundamentally different.

FOOF is a first-order optimizer that updates weights based on neuron outputs, utilizing a preconditioning matrix of the form: $\mathcal{H}\_{i-1} \otimes \mathbf{I}$ for a given layer $i$. This approach requires constructing and inverting only the Kronecker factor $\mathcal{H}\_{i-1}$, which captures limited curvature information inherent to its first-order nature. In other words, it does not incorporate curvature information from the FIM, which is essential for capturing the complex geometry of the loss surface in neural networks. In contrast, AdaFisher is a second-order optimizer leveraging a novel approximation of the FIM, $\tilde{F}\_{D_i} = \mathcal{H}\_{D_{i-1}} \otimes \mathcal{S}\_{D_i}$, to adaptively modulate learning dynamics. This formulation enables AdaFisher to integrate richer curvature information compared to FOOF while only using block diagonal Kronecker factors (please refer to section 3.1). Furthermore, AdaFisher introduces a novel computation of the FIM for normalization layers, enhancing the training performance on architectures where such layers are prevalent.

These distinctions underline AdaFisher’s second-order nature and advanced adaptivity, setting it apart from FOOF’s first-order approach. Furthermore, the scope of the experiments in Benzing’s manuscript is also limited as compared to our work. Only some preliminary experiments are shown using MLPs on MNIST and Fashion-MNIST reporting only training loss as compared to our work which uses extensive experiments on SOTA models and CIFAR10/100, TinyImageNet and Imagenet datasets.

[R-5WtX-C2] Computational Cost and Acceleration: It is true that second order optimizers are generally more computationally expensive than first-order methods like SGD for each iteration. However, our method, AdaFisher, is designed to mitigate this by employing a diagonal block Kronecker approximation of the FIM, significantly reducing the overhead compared to full-matrix or Kronecker-sum-based methods such as K-FAC or Shampoo. We recall that the main objective of our paper is to bridge the gap between enhanced convergence capabilities and computational efficiency in second-order optimization framework for training DNNs.

[R-5WtX-C3] Enhanced Generalization: AdaFisher improves generalization by leveraging the FIM to better capture the loss landscape’s curvature and parameter sensitivity. As shown in Figure 11, AdaFisher’s FIM distribution narrows to smaller values with less variation compared to Adam, indicating convergence toward flatter minima, which are strongly associated with better generalization [1,2]. The diagonal block Kronecker approximation ensures AdaFisher captures key curvature information while avoiding overfitting to noise (please refer to Section 3.1), which is a common issue in some second-order methods. This combination allows AdaFisher to achieve both improved generalization and computational efficiency. We refer the reviewer to Appendix B.1 (Convergence Efficiency) for more information.

Dear Reviewer jcNA,

We sincerely thank the reviewer for their valuable feedback, which highlights the novelty of our manuscript through the introduction of a new approximation of the FIM and the demonstration of AdaFisher’s performance via thorough experimentation. Your insightful comments and questions have been instrumental in improving the clarity and quality of our work.

Below are the responses to the comments in Weakness and Question sections:

[R-jcNA-C1] Feasibility of AdaFisher for large training systems: Please note adopting AdaFisher for training large models (e.g. LLMs) does not require significantly more computational resources than Adam, as AdaFisher’s computational overhead is comparable to Adam’s (please refer to Figure 24 in Appendix D.2.7 for memory usage and Figure 6 in Section 4.4 for time comparisons). We recognize that our computational resources are limited compared to the extensive infrastructure available to large corporations like Meta and Google. Despite our constraints, we are committed to achieving the highest possible standards in our work. We are hopeful that access to additional resources in the future will enable us to further expand the scope and impact of our research.

[R-jcNA-C2] AdaFactor and EVA comparison: The reviewer highlights two important and relevant optimizer methods: AdaFactor and EVA. In response, we have included a detailed comparison of AdaFisher against these baselines in the revised manuscript (please refer to Appendix D.2.5 and D.2.6). Our results demonstrate that while EVA offers improved efficiency and slightly better performance compared to K-FAC, AdaFisher consistently achieves superior results under WCT (Appendix D.2.5) and with the same amount of training epochs (Appendix D.2.6). Moreover, while AdaFactor is slightly more efficient than both Adam and AdaFisher, AdaFisher achieves significantly better performance, underscoring its effectiveness. Additionally, we have summarized these findings in the following tables (Table 1 shows WCT results, and Table 2 illustrates performances with the same amount of epochs), which provide a comprehensive overview of the highest test accuracies and training efficiencies for all methods discussed.

Table 1: Performance comparison of AdaFisher and other optimizers using ResNet-18 (CIFAR100) and MobileNet-V3 (CIFAR10). Results are reported using WCT of 200 AdaFisher training epochs as the cutoff.

Table 2: Performance comparison of AdaFisher and other optimizers using (a) ResNet-18, (b) MobileNet-V3, and (c) ResNet-50 on CIFAR100 for 200 epochs.

(a) ResNet-18

(b) MobileNet-V3

(c) ResNet-50