Thank you for the thoughtful comments. We appreciate your recognition of the value of our study! We address specific points below.

The results for other claims are less thoroughly evaluated, e.g. the combination with Shampoo is only evaluated on one dataset. Comparision of WA with LR schedules (Fig 7) seems to be only based on one model.

We acknowledge that the mentioned experiments were missing some data, and we are happy to share additional results:

• We performed additional experiments on Shampoo, considering more workloads and including EMA. Shampoo consistently hits the target faster across tasks when equipped with LAWA or EMA.
• WA vs Short LR schedule We report results on additional workloads and on EMA. LAWA and EMA yield better performance throughout training, closely matching the results from shorter schedules.
• We also included EMA in the previous analysis on Librispeech Conformer, and find results to be consistent with the one on LAWA. When little or no annealing is applied, EMA provides substantial improvements on the final validation performance; however, when the LR is fully annealed to zero, EMA converges closely to the annealed model.
• LR sweep on EMA: We add results on EMA when testing WA at different learning rates, observing that EMA efficiency is preserved across LR's.

For the recommendation of averaging window size, I think there are a couple of factors that could be important: batch size, weight decay, learning rate warmup.

We agree that batch size and weight decay might affect the optimal averaging configuration of LAWA and EMA. We appreciate you highlighting this, and we will include this clarification in the revised version of the paper. However, we would like to emphasize that:

1. Due to the high computational cost of AlgoPerf and limited resources, we were unable to sweep all hyperparameters and instead focused on the best-performing combinations for each workload.
2. These configurations account for a variety of hyperparameters, with batch size, weight decay, and learning rate warmup varying across workloads. As such, our recipe is relatively robust to these choices, though some variability may still occur.
3. Finally, as suggested by Reviewer #egdu, we included additional results on the effect of $\beta$ and $\nu$ on EMA performance. We refer to the answer to Reviewer #egdu for a discussion on these additional experiments.

The type of learning rate scheduling considered is limited only to cosine schedule, more recent advances such as WSD or learning rate schedule with warmup would be interesting.

We would like to clarify that the employed schedule is Cosine Decay with warmup. We apologize if this was not clear and will ensure it is explicitly stated in the revision. Regarding recent advances like WSD, we find this to be a very interesting question! We would be particularly interested in whether averaging can accelerate the decay phase of WSD, potentially reducing the computational cost of cooldowns. We believe this is a promising area for future work, and hope that our study inspires further research on this topic.

It would be much better if the averaging strategies considered can come with some formulas to help readers understand better.

Thank you for pointing this out! We had included the algorithms in an earlier version of the paper but excluded them for the submission. We display them at this URL and will include them in the Appendix of the revised manuscript.

Can the authors elaborate more on the computational overhead part? I don't see why it is the case

You are correct in noting that averaging does not incur additional overhead when implemented efficiently. In the quoted sentence, we were referring to a naive implementation of WA, where the average is offloaded to CPU memory and then transferred back to CUDA memory before evaluating the model. We appreciate the question and will revise the sentence to make this distinction clearer. For more details on this topic and data about the overhead, please refer to our response to Reviewer #egdu and this analysis.

To conclude, we sincerely thank you for your review and thoughtful questions. The discussion has been valuable, and we believe that addressing these points and providing additional evidence has strengthened our work by broadening the scope of our analysis and better supporting our claims. We hope to have clarified your concerns, and we would greatly appreciate if these improvements could be reflected in a higher evaluation. We are happy to address any further questions or doubts.

Thank you for the thorough review! We appreciate the detailed feedback.

I think only studying two WA methods is a bit limited.

We agree that broader benchmarking would be valuable and hope this works encourages further study at scale. We focus on LAWA and EMA due to their success and adoption. Influential works such as Szegedy et al. (2016), Vaswani et al. (2017), and Merity et al. (2017) used one of these two averaging flavors. We briefly tested Polyak averaging, but found it ineffective, consistent with Defazio et al. (2024), and omitted it. Finally, AlgoPerf's high computational cost limits extensive experimentation, so we prioritized exploring LAWA and EMA in depth.

I'm surprised the dependence on the EMA coefficient and update frequency was not studied.

Thank you for mentioning this! We recognize its importance and are happy to share additional results: https://anonymous.4open.science/r/ICMLE1C2/EMA.pdf

1. Small values of $\nu$ require a large $\beta$, and viceversa.
2. Update frequency around 0.003 with $\beta=0.9$ work well across tasks.
3. $\nu=1$ (first group in each plot) performs poorly for $\beta\leq0.95$

Many experiments only consider LAWA and not EMA.

We acknowledge this and would like to share additional experiments:

• Shampoo We include more workloads and EMA. Shampoo consistently hits the target faster across tasks when equipped with LAWA or EMA.
• LR schedule. Results on more workloads and EMA confirm that WA closely matches a short LR schedule.
• We test EMA with different annealing, and find consistent results with LAWA.
• EMA efficiency is preserved across different LR's, similar to LAWA.

Can the authors detail the link between WA and LR decay that they find in Thm 5.3. in Garrigos & Gower?

Thm 5.3 provides convergence results for SGD under decreasing stepsizes, requiring an averaging scheme adapted to the stepsize schedule. A further connection to averaging stems from comparing this result to Lemma 7.3 where the gradient is computed at the running average of the iterates (see also Thm 7.4).

Following Anonymous, 2025, WA perform worse than NadamW. What is the reason for this discrepancy? Could the authors give the additional time that averaging adds to training?

This is a subtle point, thank you for raising it.

1. The AlgoPerf competition API did not allow switching model parameters to the WA buffer before evaluation, resulting to a suboptimal WA implementation. This plot shows the overhead of EMA. On Criteo it's 10x slower.
2. AlgoPerf does not allow asynchronous CPU-GPU transfers and has limited VRAM, requiring WA buffers to be stored in CPU memory and transferred frequently over bandwidth-constrained hardware.

However, WA can be implemented efficiently using asynchronous non-blocking transfers, as shown by Deepseek AI et al. (2024). We used offline averaging to reduce computational costs, but an efficient implementation would have negligible overhead.

The difference between LAWA and EMA is not really analyzed in the article.

We do not find significant difference between the two (Fig 2). Moreover, our goal was not to find the best averaging scheme, so we omitted a thorough comparison. However, we agree this could be better discussed and will update the paper.

What is the answer to the title of the paper?

WA should always be used with a LR schedule to produce better models for free during training. How to access the optimal model at any time during training remains an open fascinating question. Our study presents WA as a step in this direction, moving the model closer to Pareto optimality of loss vs training time. This is precisely the reason behind the observed efficiency gains. We will clarify this point in the article.

Additional comments

• We exclude ResNet because neither NadamW nor Shampoo reach the target (Kasimbeg et al, 2025), leaving no baseline for comparison.
• Fig 7 is Librispeech Conformer, $\eta_{max}=0.00175$, which is the best configuration from Dahl et al. (2023).
• Fig 3: the y-axis shows the proportion of step wrt training horizon. Runs with $y>1$ do not reach the target and are hence excluded

Finally, while some aspects of our work were explored in prior studies, our contribution expands their scale and scope. We believe our paper helps unify existing findings, evaluating them across tasks, and providing thoroughly validated claims.

Conclusion

We appreciate the insightful questions. We think that these additional analyses will significantly enhance our study and reinforce its contributions. We hope to have addressed your concerns, and if so, we would appreciate your endorsement for acceptance.

Thank you for your review and for your insightful comments. We appreciate your recognition of the value of our study! We respond below to specific comments.

It would be useful to discuss existing application of weight averaging a little bit, especially in relation to robust finetuning [1] and OOD generalization [2].

Thank you for the additional references, we will include them in the revised manuscript.

It would be nice to clarify which dataset the figures are associated with in all cases. Eg. Figure 7.

Thank you for spotting this! We will correct the labeling. Figure 7 is obtained by training a Conformer on Librispeech. We note that a similar effect can be observed on other workloads as well, and provide additional results at this URL for comparing Weight Averaging against shorter LR schedules, and at this URL for testing the effect of EMA and LAWA when the learning rate is not annealed to zero. As suggested by Reviewers #egdu and #fFcZ, we will incorporate these additional experiments and considerations in the revised article to provide a broader view and strengthen our findings.

I like the idea that learning rate schedules impede continual learning, since you decay to 0 at some point and restarting could be problematic from an optimization standpoint. Do the authors have any evidence that WA can be used for improved continual or transfer learning?

This is a very interesting question! Resuming training from a checkpoint obtained with standard cosine annealing can indeed present challenges, such as potential forgetting during the re-warming phase (Singh, V. et al., 2025). Therefore, it looks appealing to avoid annealing, and instead resume from the average model. We have not experimented with this point yet, but we think it would be an interesting direction for future work. An interesting question is how to optimally resume training from the averaged model, especially how to adapt the learning rate value in this scenario. Since averaging acts as a proxy of implicit learning rate decay, we hypothesize that using a smaller learning rate could be more beneficial for continued training from the average rather than restarting from the previous top learning rate. We believe this fits into the broader question of how to better utilize the average for training (Kaddour et al., 2022; Defazio et al., 2024), a topic that remains underexplored but could offer significant benefits to the community.

Why, in Figure 8, does annealing + weight averaging help whereas in figure 7b LAWA adds no benfit relative to the annealed model? Is it a different dataset, or difference in some other setting?

Figure 7b shows results on Librispeech Conformer, which, as reported in Figure 8, benefits only minimally when combined with LAWA. This small gains are not visible at the scale of the plot in Figure 7b.

References

Singh, V., et al. (2025). Beyond Cosine Decay: On the effectiveness of Infinite Learning Rate Schedule for Continual Pre-training. arXiv preprint arXiv:2503.02844.

Kaddour, J. (2022). Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging. arXiv preprint arXiv:2209.14981.

Defazio, A., et al. (2024). The Road Less Scheduled. arXiv preprint arXiv:2405.15682.

Thank you for your comments! We respond to specific comments below, but please let us know if you have any additional questions.

I do not agree that the paper shows how to optimally combine the two. What the paper does show is that combing standard weight averaging with standard learning rate schedules [...] is better than just using one of them.

You are right—we do not explore optimal combinations of the two, but rather demonstrate that combining weight averaging with learning rate schedules outperforms using either alone. The broad scope of the benchmark makes a full optimality study expensive, but we believe this is still a useful result for practitioners. We will revise the manuscript to clarify this distinction. Thank you for pointing this out!

I somewhat disagree that this is a new finding. Practical works that use e.g. EMA on top of a learning rate schedule do so exactly because it works better than not using EMA.

You are right, this is not a completely novel claim. Previous works indeed show that combining WA with learning rate schedules is beneficial. We will adjust the revised paper and clarify this point. We believe the impact of our work comes from demonstrating this at scale, unifying insights from previous work in a single competitive benchmark, and showing that WA alone does not replace learning rate decay across a wide range of settings.

Only relative savings matter, 100s of GPU hours can range from very significant to very insignificant.

We agree that relative savings are important, and we will rephrase this for clarity. We did report the 15% saving in GPU hours, which we believe to be significant for this benchmark, and will ensure this is emphasized more clearly in the revision.

I think calling Shampoo a second order optimizer is a bit of a stretch.

We will clarify that while Shampoo is inspired by second-order principles, it may not strictly be a second-order optimizer.

"coupling WA with learning rate annealing": I think "combining" would be clearer than "coupling" here.

Thank you for the useful suggestion. We agree that "combining" would be clearer than "coupling" and will make that adjustment in the revision.

On related work: I think some types of weight averaging were used earlier than you mention in deep learning.

Thank you for the additional references. We are aware of this and will update the manuscript accordingly. We will add a paragraph in the Related Work section, citing earlier studies that predate SWA. Apart from the Transformer paper (Vaswani et al., 2017), we found that other influential studies such as Szegedy et al. (2016) and Merity et al. (2017) have also employed averaging schemes to improve model performance and reduce overfitting.