Understanding Adam Requires Better Rotation Dependent Assumptions

We thank the referee for their consideration of our work. We would like to bring up some crucial points in response to the referee’s concerns:

---

W1: Concerns about rotations degrading Adam’s performance:

W1.1 Impact on validation:

We thank the reviewer for making us realize our mistake with the figures: we made an error in the labels used for Figure 13, which led to the wrong Figures being loaded. Although Figure 13 (left) is correct, Figure 13 (mid) representing the validation loss is wrong, and Figure 13 (right) is just a duplicate of Figure 14 (right), which is for ResNet50, not ViT.

We sincerely apologize for that mistake, which wrongly indicates that the differences in training loss were not extended to validation. Though openreview does not allow sharing figures, we will fix this error. The final validation accuracy for ViT is about 69% for original Adam, against 64% for Adam with global rotations. The original Adam reaches 64% validation accuracy in only 20/100 epochs, 5 times faster than with global rotations, an even wider gap than for training loss.

To summarize the validation accuracy after fixing the error in Figure 13:

• For GPT, we train for much less than a full epoch, and training samples are never seen multiple times. Therefore, the training loss has the same distribution as the validation loss and shows a clear difference in performance.
• For ViT, we observe a significant difference in training and validation loss, and an even greater difference in validation accuracy.
• For ResNet, we see no significant difference with rotations. This supports our claim that Adam’s advantages over SGD are tied to Adam leveraging the properties of the canonical basis. Indeed, Adam is not superior to SGD for training ResNet, and precisely does not seem to be affected by the basis in this setting. Adam is sensitive to the basis in the settings where it performs better than SGD (ViT and GPT).

W1.2 Parallel training curves, non-convergence at the end, small loss delta:

We provide evidence that these concerns raised by the referee are natural artifacts of empirical training of transformer models at this scale, well documented in this type of setting, and do not contradict a large degradation of performance under global rotations: In fact, they make optimizer performances appear visually smaller than they are.

The training curves reported in the original blog post (Muon: an Optimizer for Hidden Layers in Neural Networks) by Keller Jordan (Figures 1 and 2) display the same ‘parallel curves’ that the referee is concerned about. Similarly, they also seem to have been stopped before converging, which, as explained below, is only in appearance. More strikingly, we invite the referee to consult Figure 2 of [2], which shows almost perfectly parallel lines, with a final difference between Muon and AdamW training losses of around 0.02. Given the clear success of Muon and the striking similarities with our experiment results, we hope the referee will reconsider the significance of these observations. We detail below the reasons for the artifacts noted by the referee.

The apparent non-convergence after 100k iterations (the x-axis is in thousands of iterations, our training does not complete a single epoch of the dataset) is a typical artifact of learning rate decay in LLM training. A large portion of the loss improvement is driven by the decay of the LR (allowing more fine-grained representation), but the LR reaches 0 by the end of training. There is no way to pursue training any further, and keeping the LR constant before it reaches 0 would lead to a near-stagnation of the loss. A striking example can be seen in Figure 10 of [1]: although the training curves appear not to have converged at 200k steps, increasing the training duration to 400k steps leads to a near identical final training loss. The referee can also refer to Karpathy’s NanoGPT Github repository, which our experiments are based on, as it provides some insightful comparison, including a training curve of GPT2 (almost identical setting to ours – except we improved hyperparameter tuning) trained over 400k steps. The range of the y-axis in this baseline training is the same as ours, and the referee will notice that the final loss is 2.905, which is worse than ours, even though we trained for ‘only’ 100k steps. As such, our final training loss after 100k steps indicates a strong final loss that would be hard to significantly improve simply by training longer. The same phenomenon also leads to training curves having near-parallel lines due to the similar impact of the LR annealing.

Additionally, evaluating the impact of a fixed shift in training loss can be misleading for LLMs, due to cross entropy and the naturally high noise in the data. For instance, from iterations 120k to 400k, Kartpathy’s GPT2 goes from loss ~3.0 to ~2.9 over 280k steps, and even this shift is largely driven by LR annealing. A shift of .02 at the same point of the LR schedule is thus significant. We refer to Figure 10 of [1] in which the optimizer introduced by the authors as superior to Adam achieves a .01 loss improvement (in an almost identical setting!) over AdamW, over 200k steps. We observe a degradation twice as large in half the training steps compared to the sizable gain claimed by the authors.

As a side note, Xie et al. [3] concurrently made the same observation as us: “Adam on the original loss converges the fastest as expected. But convergence of Adam on a rotated loss is much slower [...]”

[1] Sophia: A Scalable Stochastic Second Order Optimizer For Language Model Pretraining, H. Liu et al., 2024, arXiv 2305.14342 v4
[2] Muon is Scalable for LLM Training, J. Liu et al., 2025. arXiv 2502.16982 v1
[3] Shuo Xie, Mohamad Amin Mohamadi, and Zhiyuan Li. Adam exploits $L_\infty$-geometry of loss landscape via coordinate-wise adaptivity. In ICLR, 2025

W2: Logical framework of Section 4:

The referee brings an important point. The goal of Section 4 is not to reject theoretical assumptions or their usefulness. These assumptions have their purpose in theoretical frameworks. The goal of Section 4 is instead to assess whether existing assumptions that are rotation-dependent (and thus capable of discriminating bases) can explain the variations in performance of Adam observed across bases. The minimal requirement is that when varying only the basis, a basis with the worst property should lead to the worst performances and vice-versa.

Interestingly, the paper cited by the referee, Xie et al. [3], precisely uses the same reasoning as us to support the validity of the $(1,1)$ norm, and, by proxy, of the $L_\infty$ smoothness that it lower bounds. See Table 1 of [3], which they use to justify the effectiveness of their quantity by observing that its values align with Adam’s performance in the canonical and randomly rotated bases. While we also observe this alignment for the canonical basis and global rotation, it breaks under a type of basis that [3] had not considered, obtained via output-wise rotations. For the same reason, they supported the effectiveness of their assumption by observing this alignment; our observation that this alignment does not generalize contradicts its effectiveness.

Very importantly, this does not imply that metrics such as $L_\infty$ smoothness are not useful. It simply invalidates its ability to capture the properties of the basis that Adam leverages to achieve its advantages over SGD.

W3: Scaled (semi-)orthogonality as a beneficial theoretical assumption:

We agree with the referee that this aspect should be explicitly discussed in the main paper. We discuss the connection with Muon, but do not explicitly remind or point to Muon’s theoretical justification, which directly justifies scaled orthogonality as a powerful theoretical tool for studying the optimization of neural networks. We will update our submission to explicitly refer to the theoretical origins of Muon (and therefore the justification for scaled orthogonality), and remind them in an Appendix section. For a detailed overview of the theoretical justifications of Muon, we refer to Jeremy Bernstein’s blog post “Deriving the Muon optimizer”. To the best of our knowledge, there is no published paper explicitly on this topic, but we will cite the blog post accordingly.

For a summary of the importance of scaled orthogonality in theoretical frameworks, we refer to our reply to reviewer z9Dc Q2, who raised a similar concern. We will also add this to the appendix and refer to it in the main paper.

Q1: Difference between $R$ and $R’$:

The reason for the distinction is to be able to distinguish the basis that is used for measuring a quantity, and the basis that is used during training. For instance, if $R’=$None and $R$=Global, we would use the checkpoints from a normal Adam training run, and measure, such as the $(1,1)$-norm of the Hessian in a globally rotated basis. The initial purpose was to separate the impact of training on a basis from the impact of measuring on a basis. However, we found that the significance of the results obtained with $R!=R’$ was not well justified. Reporting results that have no clear interpretation or meaning was confusing, so we removed them. We thank the referee for this remark, and we will remove this separated notation by a single dependency on R since we only report with $R=R’$.

Typo in $r_i$:

The reviewer is correct, and we will fix this for clarity, though amusingly, with the Hessian being symmetric, its rows and columns are the same anyway (up to transpose).

---

We thank the reviewer for helping us improve the paper. We kindly ask the reviewers to consider increasing their scores if they find our responses satisfactory. We remain available and happy to address any further questions during the discussion period.

import matplotlib.pyplot as plt

def calculate_y_iterative(n_steps, rate, noise):
    n_values = np.arange(n_steps + 1)
    z_values = np.zeros(n_steps + 1, dtype=float)
    z_values[0] = 1.0
    for n in range(1, n_steps + 1):
        lr = (1 + np.cos(np.pi * n / n_steps))
        z_values[n] = (1 - rate * lr) * z_values[n-1] + (noise * lr)**2
    return n_values, z_values

n, z = calculate_y_iterative(2000, 0.0005, 0.006)
_, z2 = calculate_y_iterative(2000, 0.00055, 0.006)
plt.figure(figsize=(10, 6))
plt.plot(n, z, z2)
plt.grid(True)
plt.legend()
plt.show()

Thank you again for staying engaged in the discussion.

We respect the referee's disagreement and will continue the discussion by providing additional evidence, including the one requested by the referee, below. But before that, we would like to temporarily set aside our disagreement on the takeaways of our training curves, and make an argument at a higher level.

The core goal of our research is to enable theoretical frameworks that are more relevant to practice, and that will be suitable to guide algorithmic design and understanding of real-world behaviour. To this end, we argue that the criterion for determining whether the degradation of Adam performance by random rotations is significant should be the criterion of practitioners. Regardless of what theory considers significant or not, the differences in the training curves that we reported is considered significant by the practitioners community, and is very impactful in real-world scenarios. We provided numerous evidence of that, including the fact that in a near-identical setting, the difference that prior works reported betweem Muon and Adam is very similar to the one we report between Adam and full-rot-adam -- and the significance of Muon advantages over Adam in practice is now clearly established. The Lion optimizer has become a popular choice and was published based on much smaller reported difference of performance compared to our curves, see e.g., Figure 11 of [2] for transformer training on text, which are eerily similar to ours, except the gap we observe is larger. Our work lies at the intersection of theory and practice: our goal is to build empirically relevant theoretical frameworks, and this naturally requires to acknowledge what practice considers relevant. How can we hope to build theoretical frameworks that are useful, if we choose in the process to ignore real-world impact in favor of what theory considers important?

To be clear, we have absolute confidence that the claim that the degradation we observe from random rotations in transformer training is empirically significant is not controversial, and is supported by extensive prior work and evidence. Besides all the prior work we referred to that observed similar training patterns, and claimed significant differences even for smaller gaps, as additional anecdotical evidence of the community's opinion on the matter, the concurrent work of [Xie et al., 2025] which made a similar claim based on similar experiments, received an spotlight paper award at ICLR this year.

With that said, we now address directly the referee's argument and provide requested information:

Could you please also report the offset of these curves at 20 epochs, 40 epochs, etc.

Absolutely, here is the gap between standard Adam and fully rotated Adam, at various number of steps (we remind the referee that there are no epochs here, we do not consume all of the tokens in OpenWebText, the x-axis is in thousands of iterations):

40k: 0.03594

60k: 0.02980

80k: 0.01844

100k: 0.01534

As the referee can see, the curves are not quite parallel, the gap between them slowly decreases over time. This is expected in optimization: consider any two optimizers, no matter how different their rate of convergence are, if they both converge to a minima of same loss, the difference between their loss will necessarily go to zero asymptotically, and it does not contradict the difference in their performance: otherwise, we would only be interested in whether an optimizer converge, and the rate would be irrelevant.

The impact of the lr decay is that it hides that the actual learning has mostly flattened, by making constant progress through noise reduction at the end of training. It makes the curves look like the model is still making steady progress and was interrupted early, but if we were training for twice the number of steps (and updating the lr schedule accordingly), we would get very small improvement of the final loss. The large 'slope' at the end is driven by lr decay. For a very solid evidence of this claim, please see e.g., Figure 6 (left) of [1]. Our cosine schedule corresponds to the red curve, which seems to be making consistent progress and has a fairly straight slope. However, the blue curve, which uses a constant lr for most of training, and concentrates the lr decay (and corresponding benefit of noise reduction) at the end of training, reveals what happens when we separate the decay artifact: the curve is progressively flattening and converging as we would expect.

[1] Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations, A. Hagele et al., 2024

[2] Symbolic Discovery of Optimization Algorithms, X. Chen, 2023

We thank the reviewer for recognizing our contributions and relevance to recent new optimizers, and we address their concerns below.

---

W: Significance of contribution:

Regarding the significance of impact, we claim the success of Muon and SOAP justifies more work studying how the choice of basis affects the performance of various optimization algorithms. Many optimizers are proposed and justified theoretically (for example, see Table 2 of [1]), but are rarely used in practice. We believe this is partially because the typical assumptions that are used in theory are not predictive of what makes a good deep learning optimizer. We find that the basis does matter, and therefore, many works basing their derivations on rotationally invariant assumptions are unlikely to predict good performance. In a more practical sense, recent works show that choosing a good basis can enhance performance in practice. SOAP [2], for example, “is equivalent to running Adam in a rotated space” as per the authors. We additionally note that the basis SOAP uses is different from that of the SVD rotation, while both improve performance, meaning there is likely room for further research into other bases that could lead to even superior performance.

The experiment on Hessian rows doesn't directly explain why the SVD-rotated space performs better.

The reviewer is correct that the Hessian row experiments do not explain the improved performance of SVD rotation. Our purpose with the experiment on Hessian rows is to 1) verify the block-diagonal dominance of Hessian, a rotation-dependent characteristic that is mentioned in existing literature; and 2) identify whether simplifying the Hessian to a strict diagonal is a sufficient representation. Our finding shows that even though the Hessian is block-diagonal dominant, the off-block diagonal elements still play an important role and should not be ignored, hence arguing this assumption is inadequate in explaining Adam’s behaviour under rotation, including under SVD rotation.

Figure 6 regarding the second-moment distributions is presented without sufficient analysis or discussion, or conclusion

Figure 6 acts as a complementary observation that global rotations concentrate Adam’s second moments, leading to less variation in effective learning rates across dimensions. This loss of adaptivity directly accounts for the performance drop observed with global rotations. While we cannot directly interpret why the second moment values are much more widely distributed under the SVD rotation, we can see that it leads to a much more varied effective step size across dimensions. We suspect this basis is allowing the algorithm to be “more adaptive,” but leave the underlying mechanism for future work.

Q1: Intuition on SVD rotation enhances performance, and update orthogonality is potentially the correct metric:

A full explanation for SVD rotations enhancing performance is an active area of research, but we appeal to SVD Rotated Adam’s connections to Muon as discussed in Appendix E. Given this connection, an intuition for explaining Muon’s effectiveness should extend here. For a summary of the importance of scaled orthogonality in theoretical frameworks, which we will add to the appendix and refer to in the main paper, we refer to our reply to reviewer z9Dc Q2, who raised a similar concern. For a detailed overview of the theoretical justifications of Muon, we refer to Jeremy Bernstein’s blog post “Deriving the Muon optimizer”. To the best of our knowledge, there is no published paper explicitly on this topic, but we will cite the blog post accordingly.

Q2: Add Muon to the performance comparison:

We are also interested in the performance of Muon as compared to the various Adam rotations, but our initial runs have shown subpar performance relative to Adam. We suspect a more thorough hyperparameter search is necessary. Hyperparameter choices often used in GPT speedrunning ([3]) are optimized for a slightly different model (modded-nanogpt) and dataset (fineweb10B), in contrast to our use of NanoGPT and Openwebtext. We are currently conducting this hyperparameter search and hope to be able to report results during the discussion period.

[1] Robin M. Schmidt, Frank Schneider, and Philipp Hennig (2021). “Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers”. In: International Conference on Machine Learning (ICML). Vol. 139, pp. 9367–9376.
[2] Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham M. Kakade. 2024. Soap: Improving and stabilizing shampoo using adam. ArXiv, abs/2409.11321.
[3] https://github.com/KellerJordan/modded-nanogpt/tree/master

We thank the referee for their consideration of our work. We would like to bring up some crucial points in response to the referee’s concerns:

---

W1: Novelty with respect to [Xie et al., 2025]:

We have communicated to the PCs evidence showing that our work could be treated as concurrent work to the ICLR 2025 reference (and their previous arXiv versions), and thus our novelty evaluation should not be negatively impacted by their paper. To stay compliant with the anonymity requirements, we leave it to the PCs to continue communicating on the topic with the reviewer and AC.

Besides the above, the referee is concerned that our core findings overlap with [Xie et al., 2025]. On the contrary, we would like to point out that our work directly contradicts the findings from [Xie et al., 2025]. Indeed, their main claim is “the exploitation of nice $L_\infty$-geometry is the key advantage of Adam over SGD”. They support their claim empirically by showing in [Table 1] that Adam’s performance is better when the (1,1) norm is lower. Although we find their work very valuable, we show experimentally that this correlation doesn’t generalize to other bases than the ones they considered, with random rotations by output neurons increasing the (1,1) norm while simultaneously improving Adam’s performance. Our work investigates an alternative quantity to link Adam’s performance to the properties of the basis.

The novelty of our work includes not only the study of Adam’s behavior under SVD-based rotations, but also the study of Adam’s behavior under different scopes of random rotation (layer wise, input wise, output wise), which brings significant differences (described in great details in Figures 18 to 33), and also the evaluation of several existing theoretical quantities under each of these rotations, leading to different conclusions from prior works.

Q1: Link between Adam in SVD-based rotations and Muon:

The closest existing method to running Adam in SVD-based rotations like we do is GaLore, but without rank compression. Although there are indeed some links between Muon and our approach in that they both tend to orthogonalize the update (similarly to SOAP and Shampoo), there are significant differences: Muon orthogonalizes the first moment at each step, while our approach updates SVD every T steps Muon uses Newton-Schulz (an approximative method) to compute UV, whereas we use SVD to compute U and V separately so that we can run Adam in the latent space. Muon orthogonalizes the first moment, whereas we compute SVD and apply it to the gradients Using Adam in the SVD rotated space only orthogonalizes the update if you assume Adam behaves exactly like sign descent

W2/Q2: Theoretical motivation for scaled orthogonality:

Regarding the theoretical motivation for scaled orthogonality, we agree with the referee that it should be explicitly mentioned in the main paper. We discuss the connection with Muon, but do not explicitly remind or point to Muon’s theoretical justification, which directly justifies scaled orthogonality as a powerful theoretical tool for studying the optimization of neural networks. We will update our submission to explicitly refer to the theoretical origins of Muon (and therefore the justification for scaled orthogonality), and remind them in an appendix section. For a detailed overview of the theoretical justifications of Muon, we refer to Jeremy Bernstein’s blog post “Deriving the Muon optimizer”. To the best of our knowledge, there is no published paper explicitly on this topic, but we will cite the blog post accordingly. Here is a quick summary of the justifications:

• Due to the sequential nature of operations in Deep Learning, the function value and the gradient with respect to higher layers only depend on the output of layer $L$, not directly on the weights of layer $L$. Thus, when updating the weights $W$ of layer $L$, one should control how much the update $dW$ impacts the possible outputs, rather than the weights themselves. In other words, instead of constraining the Frobenius norm of dW (like SGD does), we should seek to control the $L_2$ norm of the output shift $dy=(W+dW)x - Wx = dWx$. This is achieved by constraining the operator norm of dW instead of its Frobenius norm. It turns out that if the gradient $G$ has SVD decomposition USV, minimizing $dW^\top G$ (i.e., the linear approximation of the loss) under a constraint on the operator norm of $dW$ has a closed form solution: $-\eta UV$, which is the scaled orthogonalization of the gradient. It is the update that maximizes the reduction of the loss (under linear approximation) while constraining the worst-case shift of output.

This gives a theoretical justification to orthogonalizing the gradient like Muon does, and directly relates the scaled orthogonality of the update to optimization performance. Since Adam’s update orthogonality depends on the basis, it could be an explanatory factor of Adam’s variation of performance across bases, which we find to be empirically supported.

---

In light of the above clarifications and changes, we hope the referee will reconsider their assessment of our work’s significance and novelty.

We thank the reviewer for recognizing our novelty, and we address their concerns below.

---

W1.1 Scale of experiments

While we agree that the models we train do not reach the scale used in industry, this work was conducted within the computational constraints of an academic institution. However, we argue that the scale of the models we train is sufficiently representative. Notably, the GPT2-Small (124M parameters) style model we use (NanoGPT) has been the basis for several previous and concurrent works throughout the literature on explaining the performance of Adam and related optimization methods. In particular, [1] and [2] use similar GPT2-style models while using a smaller dataset (Wikitext-103), and [3] uses the slightly smaller GPT-Neo variant. The same model is also used in [4], where the authors additionally conduct a wide array of experiments on other models that show similar results to their GPT2-Small model. The same is true of [5], as mentioned by Reviewers z9Dc and Jnyt. Given this trend of prior and concurrent work, we feel that the largest model we train is sufficient, but if the reviewer is interested in seeing any particular experimental setting, we would be happy to try to accommodate.

[1] Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie (2020a). “Why Gradient Clipping Accelerates Training: A Theoretical Justification for Adaptivity”. In: International Conference on Learning Representations (ICLR).
[2] Michael Crawshaw, Mingrui Liu, Francesco Orabona, Wei Zhang, and Zhenxun Zhuang (2022). “Robustness to Unbounded Smoothness of Generalized SignSGD”. In: Neural Information Processing Systems (NeurIPS).
[3] Yan Pan and Yuanzhi Li (2023). Toward Understanding Why Adam Converges Faster Than SGD for Transformers. NeurIPS 2022 Workshop on Optimization for Machine Learning. arXiv/2306.00204.
[4] Frederik Kunstner, Alan Milligan, Robin Yadav, Mark Schmidt, and Alberto Bietti. Heavy-tailed class imbalance and why Adam outperforms gradient descent on language models. Advances in Neural Information Processing Systems, 2024.
[5] Shuo Xie, Mohamad Amin Mohamadi, and Zhiyuan Li. Adam exploits $\ell_\infty$-geometry of loss landscape via coordinate-wise adaptivity. ICLR 2025\

W1.2 Hyperparameters:

Within the rebuttal budget, we conducted a small set of hyperparameter sensitivity tests on the choice of $\beta_2$ as suggested by reviewer c8DY (See “W1: Hyperparameter robustness” response). We regret not being able to visualize these results in the rebuttal due to the NeurIPS policy of not allowing images and PDF links. Overall, the results are consistent with what we observe in the original experiments. We will add these additional experiments to the next revision.

W.2 Significance of contribution:

..it is unclear how it contributes new insights into motivating the design of other optimization algorithms.

Theoretical assumptions are often central to the design and analysis of new algorithms in both classical and deep learning optimization. In this work, we argue that the assumptions made in several prominent theoretical arguments on the superior performance of Adam do not correlate with that superior performance. These works often conduct theoretical analysis to justify the effectiveness of a new algorithm. This is the case in roughly half of the works we mention in Table 2, and a much more extensive list of proposed (and largely forgotten) optimization algorithms can be found in Table 2 of [1]. It is often the case that algorithms are developed based on vague intuitions of what may work in a specific setting, but lack performance guarantees and fail to generalize to other settings (for example, consider the 160 optimizers in Table 2 of [1]). Given the current scale and cost of training large models, we argue that rather than continue this “brute force” approach, newer theories and assumptions are needed.

We aim to remind the community that for their theory to predict the effectiveness of a new optimization algorithm, the assumptions they use in their analysis should be able to capture the differences between gradient descent and Adam. Our findings show that many common assumptions do not capture this difference, or do not correlate with downstream performance. Our first contribution to the development of new methods is a reminder of the insufficiency of these common assumptions when conducting analysis and development, and the second is to look for some problem characteristics that correlate with performance. We conducted a preliminary investigation into update orthogonality as a characteristic that we find to correlate. We additionally note that it is still often the case that theoretical analysis leads to new practical optimizers, for example, Muon builds off ideas introduced in previous work, such as [2] and [3].

Based on the development of SOAP and Muon, it is evident that some type of orthogonalization helps speed up convergence. … analyzing this for Adam doesn’t seem to give new insights…

While “orthogonalization” has been shown to help in very recent optimizers like Muon, there are many important questions left unresolved. There are limited theoretical explanations for why orthogonalization is beneficial (which is itself an interesting question to theoreticians), how it improves upon Adam, whether it is the best quantity to optimize for, or in which parts of the network it is helpful, and so on. The Muon optimizer specifically suggests still using Adam’s update in embedding, prediction, and layer-norm layers; it is unclear what the correct method is for regrouping and orthogonalizing tensors beyond two dimensions, such as in convolutional networks.

As Adam has been proven over time to be effective and widely used in practice, we contend there is still significant value in the study of its performance and specifically its dependency on the basis, and make the connection to the idea behind new methods like Muon. Our work highlights that the coordinate basis being used (represented through different rotations) impacts how “close” to an orthogonal update. In Appendix E, we additionally note that under simplification, AdamW with the SVD rotation is equivalent to Muon. We argue that insights like this can directly lead to the development and proper usage of new optimization methods. For example, SOAP [4] is equivalent to Adam running on a rotated basis, highlighting the importance of studying the impact of rotations on Adam.

[1] Robin M. Schmidt, Frank Schneider, and Philipp Hennig (2021). “Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers”. In: International Conference on Machine Learning (ICML). Vol. 139, pp. 9367–9376.
[2] Jeremy Bernstein and Laker Newhouse. “Old optimizer, new norm: An anthology.” arXiv preprint arXiv:2409.20325 (2024).
[3] Tim Large, Yang Liu, Minyoung Huh, Hyojin Bahng, Phillip Isola, and Jeremy Bernstein. Scalable optimization in the modular norm. arXiv:2405.14813, 2024.
[4] Nikhil Vyas, Depen Morwani, Rosie Zhao, Itai Shapira, David Brandfonbrener, Lucas Janson, and Sham M. Kakade. 2024. Soap: Improving and stabilizing shampoo using adam. ArXiv, abs/2409.11321.

---

We kindly ask the reviewers to consider increasing their scores if they find our responses satisfactory. We remain available and happy to address any further questions during the discussion period.

We thank the reviewer for recognizing our contributions, and we address their concerns below.

---

W1: Hyperparameter robustness:

Regarding robustness to the $\beta_2$ hyperparameter, we have conducted an additional sensitivity analysis on the base AdamW run and under global rotation. We fixed all other hyperparameters but $\beta_2$, with the additional runs using $\beta_2=0.9$ and $\beta_2 = 0.99$ to compare to the original $\beta_2= 0.95$.

In the case of AdamW under global rotation, both increasing and decreasing $\beta_2$ lead to inferior performance of the optimizer. This indicates that our original choice of $\beta_2$ was near optimal with all other hyperparameters fixed for the global rotation. Specifically, the globally rotated AdamW with $\beta_2=0.95$ completed training with training loss 2.912, whereas with $\beta_2$ being set to $0.9$ and $0.99$, the training completed with loss 2.9165 and 2.9167, respectively. For the base AdamW, we found that decreasing $\beta_2$ led to worse performance than the original run, while increasing it led to superior performance. While this does mean that base AdamW could have been tuned more, in all configurations of $\beta_2$, the base AdamW showed superior performance to all configurations of the globally rotated AdamW. From our experimentation, we would expect further hyperparameter tuning to only widen this gap. Specifically, our original run with $\beta_2=0.95$ ended with a loss $2.896$, whereas $\beta_2=0.9$ and $\beta_2=0.99$ led to a loss of $2.905$ and $2.888$, respectively. We regret not being able to visualize these results in the rebuttal due to Neurips' policy of not allowing images and PDF links, but the ordering of these loss values remains consistent throughout the 100 epochs. We plan to add these visualizations in future revisions.

For the rebuttal period, we prioritize budget for searching over $\beta_2$ as we felt it would be more likely to alter performance. Regarding batch size, we refer the reviewer to other recent works [1] and [2] that primarily study the impact of batch size on the performance gap between Adam and SGD. For our focus on Adam’s performance under different coordinate bases, we do not expect the change of gradient variance to have a strong interaction with the rotations. However, it is possible that a larger batch size beyond a certain threshold would better recover to full batch properties that are basis dependent. We will investigate the impact of batch size in future revisions.

[1] Teodora Srećković, Jonas Geiping, Antonio Orvieto. (2025) Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling. arxiv/2506.12543
[2] Martin Marek, Sanae Lotfi, Aditya Somasundaram, Andrew Gordon Wilson, Micah Goldblum. (2025) Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful. arxiv/2507.07101

W2: Comparison metrics

Our aim in comparing Adam under various rotations was strictly to understand Adam’s dependence on the basis. While our SVD Rotated Adam algorithm does offer some improvements as a function of iteration number, we are not proposing this as an effective replacement for Adam or a competitor to Muon (although the two are connected, as we note in Appendix E). As we mentioned in our response to the reviewer wWFQ, we are planning to add a direct comparison to Muon for further clarity, but again are focusing on comparing training speed rather than wall clock time. As a quick note on the distinction between training and validation loss, the GPT experiments are conducted by doing a single pass over the training set, so the validation and training loss are equivalent in theory. That said, for completeness, we do observe the same patterns in the validation loss, where the final validation loss of global rotation is $2.929$ and the base run is $2.909$.

W3: Re: ..more evidence is needed to claim that better typical performance of SGD compared to Adam when training ResNets is explained by reduced sensitivity to rotations…

We agree that more evidence would be required if one were to make the strong claim that the only factor leading to a gap or lack of gap in optimization performance is sensitivity to the basis being used. Instead, we claim that the basis can have a significant impact, as shown in our experimentation. While it is certainly possible and likely that other factors, such as properties of the datasets and architectures, would also impact optimization performance differences, we feel our results showing that the choice of basis can both degrade and improve performance make a meaningful contribution.

Mistakes/edit suggestions:

We thank the reviewer for their careful read. We will fix the following in the next revision:

• W4: The zero matrix should not be considered as semi-orthogonal,
• W4: There is a mistake in Figure 13/14 where we uploaded the wrong figures for ViT validation loss and accuracy. Although Figure 13 (left) is correct, Figure 13 (mid) representing the validation loss is wrong, and Figure 13 (right) is just a duplicate of Figure 14 (right), which is for ResNet50, not ViT. Additionally, the "Epoch" replaced with "Iteration (k)" in Fig. 13 is also part of this mistake. We invite the reviewer to read our response to Reviewer Jnyt for a more detailed summary correcting this mistake, and we will certainly replace the incorrect figures in future revisions.
• Limitation 1: We will also move the limitation discussion to the main text and further discuss the possibility of other factors contributing to optimization performance differences.

Q2: Re: Is A assumed to be wide and full-rank?

We don’t make any assumptions on the matrix shape or rank, given that the parameter matrices vary in shape and the training process cannot guarantee any rank structure.

Thank you for these answers. I appreciate the additional hyperparameter sweeps.

I have read the discussion with other reviewers and found it useful. I agree with the authors that the gap in the training loss is significant, and I take their point that for the GPT experiment validation and training loss are equivalent. I understand that the contribution is somewhat incremental compared to Xie et al. [3], although this empirical study is more extensive, and using the criterion the AC gave, I would say it would in fact be interesting to see both papers side by side. I also agree with the authors that comparing training speed is informative and is standard in the literature; in general, I find that the authors defended their paper well in the discussion with Reviewer Jnyt. I would suggest replicating some of the large-scale experiments more times with different seeds, to alleviate the concerns about the difference being an accident.

Taking into account the authors' response to my and other reviews, I will raise my score to 5.