Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training

We appreciate your thoughtful feedback. We believe the core of the concern arises from a slight misalignment between the primary focus of our paper and the review’s emphasis on Section 5, which treats the work as if its main goal were to propose a new optimizer.

As reflected in our title, “Through the River: Understanding the Benefit …”, our central contribution is a principled analysis of SF dynamics, grounded in the river-valley loss landscape framework. This focus spans the majority of the paper (Sections 2–4), where we present both theoretical and empirical insights into the behavior, strengths, and limitations of SF methods. The refined method introduced in Section 5 is intended as a concrete illustration of this framework—a proof-of-concept that addresses a diagnosed failure mode—rather than the core contribution of the work.

We acknowledge that the refinement could appear preliminary if evaluated in isolation as a standalone optimizer. However, its purpose is to demonstrate how insights from our analysis can inform practical improvements. With this context in mind, we respond to the reviewer’s specific points below.

Contribution and Evaluation of Refined SF method

the section evaluating this refined version doesn't really provide enough details

The refined SF method was introduced to empirically test a key prediction from our analysis: that the coupling between averaging and momentum in the vanilla SF update leads to degraded performance of x_t relative to y_t under small β1 (Figure 5). By introducing the decoupling parameter C, we isolate these effects, and the observed improvement of x_t over y_t (Figure 8) provides empirical support for our hypothesis. While the primary goal of the refined method is diagnostic, these results show how our analysis informs a practical path to improving SF in regimes where they currently struggle.

The main disadvantage of introducing a new HP is ... tuning costs

To address this, we conducted sensitivity analysis of C, reported in Tables 2 and 3 of our response to Reviewer kaR1, Q4. The results show that refined SF-AdamW is robust to the choice of C: it consistently outperforms vanilla SF across a broad range of values, reducing the practical cost of tuning.

Does the optimal C change systematically with batch size? Is having x_t track y_t sufficient for good models, or only necessary?

A systematic batch size ablation is an important future direction. Our hypothesis is that larger batch sizes, which reduce gradient noise, would favor larger optimal C values that prioritize recent iterates. However, we refrain from making strong claims without direct evidence. Due to computational constraints—and since the refined method is not the paper’s main focus—we did not perform these scaling experiments. We believe it would be valuable to conduct a study akin to Fig. 2 in [1], measuring the critical batch size of the refined SF method.

Having x_t track or outperform y_t is a necessary but not sufficient condition for good performance. A good averaging scheme may improve x_t, but the quality of the underlying momentum iterates y_t is also important.

It’s a little hard to compare the results in Fig.8. I'm left wondering if C really does "fix" SF-AdamW?

Our intention is not to claim that C universally “fixes” SF-AdamW, but rather that it mitigates a specific failure mode where x_t lags behind y_t due to entangled averaging. The differing y-axis scales were chosen to highlight the x_t vs. y_t gap within each subplot, not to compare across momentum values.

Other empirical concerns

But how are you evaluating the results across the sweep? On the validation set?

Yes, our evaluation was conducted on the validation set, which is standard in LM pretraining where, under the single-pass setup, prior work has shown there is no generalization gap (see [2]).

To rigorously address this concern, we evaluated on a held-out test set (9M-token subset of SlimPajama) with HP sweep of refined method (see Tables 2 and 3). The performance trends on the test set closely match those on the validation set, confirming that our conclusions are not due to overfitting.

Cosine-10x decay is a bit of a “straw man”

We conducted an HP sweep for AdamW with cosine decay to 0, evaluated on a held-out test set:

Table 4: AdamW with Cosine Decay to 0 (Batch Size 2M, Test Perplexity)

These results show that well-tuned AdamW with cosine decay to 0 slightly outperforms refined SF-AdamW (Table 3). Nonetheless, our refinement significantly narrows the gap between vanilla SF-AdamW and AdamW with optimal cosine decay in the large-batch regime. We appreciate your insightful comment and will clearly discuss this result in the revised manuscript.

Regarding the paper's main contributions

"We begin by revisiting two widely used strategies" – is this really a main contribution?

Section 2 is indeed a review, not a novel contribution. Its purpose is to contextualize existing strategies through the lens of scalable training and river‑valley framework, setting the stage for subsequent analysis. We will revise main contributions list to reflect this.

"Reveal" that SF performs a form of weight averaging – you need to be a lot more precise here

The original SF method [3] indeed frames x_t as a uniform average of the z iterates. Our contribution is to show an alternative and more informative reformulation: x_t can be viewed as a weighted average of y iterates, where y follows a momentum‑like update.

This perspective aligns with [4], who interpret SF‑SGD as a weighted average of accelerated SGD, but our work extends it in several ways:

1. We analyze SF-AdamW, which has not been studied in this context.
2. We connect the reformulation to the river-valley perspective, showing that y tracks the valley floor while x acts as a smoothed average.
3. We observe that EoS arises specifically at y iterates, not at x or z.

These observations support our proposal that y should be viewed as the primary optimization path, with x serving as its weighted average. We will revise the text to clearly distinguish our contributions from prior work.

Regarding prior work

weight averaging does not necessarily need to consume extra memory, e.g., in DeepSeek-V3

We clarify as follows:

• Even if DeepSeek‑V3’s implementation is fully optimized, tracking EWA requires extra CPU memory, so it is not strictly correct to say it “consumes no extra memory” unless the claim is limited to GPU memory.
• While techniques exist to reduce GPU memory use, in practice they demand non‑trivial engineering efforts, depend on training setup, and still incur overhead from CPU allocations, extra compute, and CPU–GPU synchronization. The degree of overhead also varies with hardware.

Therefore, we believe our original statement that weight averaging often incurs non-negligible practical overhead is a fair characterization of the trade-offs involved.

Paper Organization

Thank you for the suggestions. We will revise the paper accordingly.

Writing

Section 5: “if large β has all these benefits in terms of keeping us in the valley, why not just use a large β all the time”?

Our analysis shows that larger β reduces oscillation variance, keeping y_t closer to the river floor, but does not imply that larger β always improves performance.

• For y_t, performance depends on both oscillation magnitude and effective "speed" along the river, requiring a balance when tuning β.
• For x_t, β also controls averaging weights, and the optimal weighting is task‑dependent.

Thus, β must be tuned to jointly account for its dual effects on x_t and y_t.

“large β slows yt updates,..." – isn’t the point that (1-β) becomes small and we pay less attention to recent gradients, but if batch sizes are larger, we actually want to pay more attention to recent gradients?

Our original argument highlighted that as $β\approx 1$ and $t\to\infty$, both $β c_{t+1}$ and $(1-β)$ in (SFy) become small, slowing updates of y_t. We also agree with your perspective: large β places less emphasis on recent gradients, potentially degrading y_t performance in large-batch regimes. Conversely, a larger β is beneficial for x_t due to its narrower averaging window. We will clarify this trade-off in the revised manuscript, incorporating your insights.

Minor

WSD “allows dynamic adjustment of HPs across phases” But doesn’t Cosine as well?

WSD decouples the constant-LR "trunk" from temporary "decay" branches, enabling one to experiment with phase-specific HPs (e.g., decay shape) on new branches without altering the main run. In contrast, a monolithic cosine schedule binds LR to the entire training horizon, so testing a different decay profile requires a full restart.

Line 37-when you mention the "workaround" of weight averaging ... forward pointer to your related work

We will add a pointer to Section 2.3.

124M model trained on 6B tokens, so that's 48.4 TPP

Our pre-training runs were set to match the Chinchilla-optimal compute scale. For our main experiments, 124M model was trained for 5k iterations on 2.5B tokens, consistent with this scale (see Appendix B).

Typos: Line 263, Line 276

Thank you for catching these. They will be corrected.

Limitations

I feel the paper could have benefited from a detailed limitations section

We will add a "Limitations" section outlining theoretical assumptions and the scope of our experiments.

---

[1] Zhang et al., How Does Critical Batch Size Scale in Pre-training?, ICLR 2025.

[2] D’Angelo et al., Why Do We Need Weight Decay in Modern Deep Learning?, NeurIPS 2024.

[3] Defazio et al., The road less scheduled, NeurIPS 2024.

[4] Morwani et al., Connections between schedule-free optimizers, ademamix, and accelerated sgd variants, 2025.

I had a look at the other reviews, my original review, the rebuttals, and again at the paper. Overall, experimentally, this is good, careful work, while the overall impact is more a matter of opinion. The authors have provided new experiments, which should strengthen the paper. In particular, they have addressed some of my own concerns. I will revise my score.

Regarding tuning hyperparameters on the validation set:

• You say, this "is standard in LM pretraining" and "prior work has shown there is no generalization gap"
• This isn't very compelling to me: there's no generalization gap until there is a generalization gap. And one way we might create a generalization gap is introducing new hyperparameters and specifically tuning them on the validation set!
• So, there's no "presumption of innocence" here: I mean, we should at least establish that new hyperparameters do generalize before relying on that fact, you know what I mean? And even then, I don't think I would rely on it :)

Thank you for your valuable feedback! We appreciate that you found our work significantly contributes to our understanding of SF optimization. We would like to address your questions as below.

W1: Implications of the Large Batch Results

While the authors aim to address the large batch size limitation of SF, they do not perform an analysis on the effect of large batch size.

We acknowledge that while our paper aims to address the large-batch limitation of SF, our experimental focus is primarily on analyzing the effect of momentum rather than batch size. This choice was motivated by the fact that the role of momentum in SF methods remains relatively underexplored, and our goal was to address this gap. In contrast, the impact of batch size on SF performance has already been studied—most notably by [1], who conducted scaling experiments and identified the critical batch size at which vanilla SF methods begin to degrade.

Rather than replicating those experiments, we sought to build on their findings by providing intuition for why vanilla SF fails in large-batch regimes. Our analysis suggests that the failure stems from the entanglement between momentum and the weight averaging window, which motivated our refinement to decouple them.

We agree with the reviewer that our current large-batch results for the refined SF method are preliminary. Due to computational constraints, and because proposing the refined method is not the main focus of our paper, we did not conduct a full batch size ablation. We will revise the text to avoid overclaiming the effectiveness of the refined method in large-batch settings.

[1] Zhang et al., How Does Critical Batch Size Scale in Pre-training?, ICLR 2025.

W2: Analysis of HP C

Introduces a tunable HP C. Its unclear how sensitive SF is wrt C in different training regimes beyond pre-training.

We agree that introducing a new HP entails tuning cost. To assess this, we conducted an additional sweep over C values, reported in Tables 2 and 3 of our response to Reviewer kaR1, Q4. We observe that performance remains stable across a wide range of C.

W3: Assuming River Valley Landscape

I believe interpolation between different iterates along the valley and river directions would be helpful.

We appreciate the suggestion and would like to clarify our position. Our work builds upon the river-valley landscape as a modeling framework that has been established in recent literature. This assumption is not specific to any particular optimizer, as the river-valley refers to the geometry of the loss landscape itself, rather than the dynamics of a specific optimization method.

Interpolation-based visualizations have already been presented in prior works—for example, Fig. 7 in [2] and Fig.2 in [3]—demonstrating that modern deep learning loss landscapes often exhibit a convex, valley-shaped structure. In our own setup, we independently verified this landscape structure through interpolation between iterates and observed the same trend (see Table 1 of our response to Reviewer kaR1, Q1&Q2).

In addition, we would like to emphasize that the river-valley landscape is further supported by several well-established empirical signatures:

• The sharp loss drop during LR decay.
• The EoS phenomenon, where iterates bounce between the steep walls of the valley.
• The Central Flow, which captures the trajectory along the valley floor.

Further supporting evidence comes from [4], who show that optimization updates projected onto the river direction (i.e., high-curvature hill directions removed) preserve learning efficacy. This implies that motion in the hill directions is often unnecessary once the iterates are near the valley floor.

We will revise the text to include these clarifications and highlight that we also validated the river-valley structure in our setting.

[2] Wen et al., Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape View, ICLR 2025.

[3] Belloni et al., Universal Dynamics of Warmup Stable Decay: understanding WSD beyond Transformers, ICML 2025 Workshop on HiLD.

[4] Song et al., Does SGD really happen in tiny subspaces?, ICLR 2025.

Q1: Is Central Flow Analysis of x_t Possible?

I think the Central Flow can be used to explain why the iterate x_t deviates from the river direction at suboptimal β.

We believe that applying central flow analysis to the x iterate is not appropriate. Central flow is intended to model the time-averaged optimization trajectory at the Edge of Stability. Our observations indicate that only the y iterates operate at the EoS, exhibiting oscillations centered around the river. Therefore, we model the central flow based on the y iterates, which allows us to capture the oscillation variance in the hill direction away from the valley floor.

In contrast, the x iterates do not oscillate around the river and do not operate at the EoS. Consequently, central flow analysis does not provide a suitable explanation for the deviation of x_t from the river at suboptimal β. We appreciate the reviewer’s insight and would welcome suggestions on alternative frameworks for modeling this behavior.

Q2: x_t vs y_t in SF method

If x_t deviates from the river direction and y_t does not ... one can always evaluate the loss at the iterate y_t, and it should perform well.

It is not the case that the y iterates always outperform the x iterates. For optimal values of momentum (e.g., β = 0.95), we observe that x_t achieves better performance than y_t. This is because x_t corresponds to an "appropriate" weighted average of the y iterates, which can enhance performance when the averaging is well-balanced.

In contrast, when β is small, the weighting scheme places too much emphasis on early y iterates, leading to degraded performance of x_t. This issue motivated our introduction of the refined SF method, which decouples the averaging mechanism from the momentum parameter β. This decoupling allows for improved averaging regardless of the momentum setting. As shown in Figure 8 (left), the refined method enables x_t to consistently outperform y_t, even at smaller β values.

Returning to the original question: no, evaluating the loss at y_t alone does not always yield optimal performance. A well-designed weighted average over the y iterates is often necessary for achieving the best results.

Q3: The Toy Objective

What's the motivation behind the toy model?

Our design of the toy model is intended to construct a minimal model that captures the river-valley geometry while replicating key behaviors observed in language model training.

The first term, $(w_1w_2-1)^2/2$, can indeed be viewed as the loss of a two-layer network on a single example. More importantly, it defines a river-shaped loss landscape, where the constraint $w_1w_2 = 1$ defines the valley floor, and the curvature in the hill direction becomes sharper as $w_1$ increases. This structure is central to capturing the loss geometry observed in modern training settings.

The second term, $\log(1+\exp(-w_1))$, is included to create a directional bias along the valley floor so that the optimizer progresses in a consistent direction. The specific form of this term is not critical to the main phenomena we study and could be replaced by other forms (e.g., a linear term like $-w_1$) without qualitatively changing the behavior. Its role is simply to ensure that the loss decreases along the river as training progresses.

While the exact form of the toy objective may not correspond to a practical loss function, that is not its purpose. Instead, it serves as a minimal model that exhibits rich dynamics consistent with those observed in realistic settings. Notably, it replicates:

• How SF dynamics remain near the valley floor and how this behavior depends on the momentum parameter β.
• The emergence of the EoS, specifically at the y iterates, which mirrors our findings in large-scale experiments.

The fact that this toy model reproduces such nontrivial behaviors confirms its utility as a theoretical tool. It provides insight into the mechanisms behind SF optimization that would be difficult to extract from large-scale models alone.

Q4: Memory Overload in SF-AdamW

How does SF use the same memory as AdamW? It has 4 variables x, y, z, v. Is it because of the reformulation in Section 4.4?

SF-AdamW does not require storing all four variables explicitly. In particular, y is computed on the fly from x and z, and therefore does not need to be stored as a separate tensor. This implementation detail is not specific to our reformulation and is already described in the original SF method (see Section 4.4 in Defazio et al.).

To clarify:

• AdamW stores the model parameters x_t, the first-moment vector m_t, and the second-moment vector v_t.
• SF-AdamW stores the model parameters x_t, the auxiliary parameters z_t, and the second-moment vector v_t.

SF-AdamW can be interpreted as SF method applied over RMSProp, just as AdamW can be viewed as momentum applied over RMSProp. In both cases, the optimizer maintains three parameter-sized tensors, and SF-AdamW does not incur any additional memory overhead compared to AdamW.

Limitations

Experimental results are limited to two pre-training setups at ~100M scale trained up to Chinchilla optimal.

We agree and will add a "Limitation" section to acknowledge this. Our experiment scale was intentionally chosen to make the training dynamics of SF methods more interpretable within our compute constraints, as our primary goal is to understand the underlying mechanisms of SF.

We acknowledge that validating these findings at larger scales and across more diverse architectures would further strengthen their generality. We view this as an important direction for future work, though it is beyond the scope of the current paper.

The visualizations can be improved

Thank you for the suggestion. We will revise the figures for better readability.

Thank you for your efforts and valuable feedback. Let us address your concerns and questions below. We look forward to discussing further if anything remains unclear.

Q1 & Q2: Justification of the River-Valley Landscape

The river-valley landscape lacks mathematical justification and its connection to the experiments in Section 3 is unclear.

We would like to clarify that our paper does not introduce the “river-valley” landscape as a novel proposal. Rather, we build upon an emerging framework established in recent peer-reviewed literature to analyze and interpret our findings.

The river-valley model was introduced by Wen et al. (2025) and further supported by Belloni et al. (2025). It is grounded in empirical observations, including the characteristic sharp loss drop during learning rate (LR) decay and direct measurements of a convex, valley-shaped loss profile (see Fig.7 in Wen et al and Fig.2 in Belloni et al). This framework also aligns with established theoretical concepts such as the “Edge of Stability,” where iterates bounce between valley walls, and the “Central Flow,” which corresponds to the low-curvature valley floor that the optimization trajectory follows.

Independent work by Song et al. (2025) provides complementary evidence. They explicitly measured Hessian eigenvalues and eigenvectors during training to identify high-curvature (“hill”) and low-curvature (“river”) directions. They showed that projecting optimizer updates onto the river direction—by removing components along hill directions—preserves learning efficacy. Their results suggest that once the iterate reaches the valley floor, motion along hill directions is unnecessary, while progress along the river remains essential.

We use this established framework to interpret the experiments in Section 3. The reviewer’s convex quadratic intuition is consistent with this view: under the river-valley lens, the sharp loss drop after LR decay arises because the initial LR is too large for the steep "valley walls"; decay enables convergence to the "valley floor".

Our key empirical finding is that SF-AdamW does not exhibit such a sharp drop upon LR decay. Within the river-valley framework, this suggests that SF iterates remain already close to the valley floor throughout training, with the SF mechanism effectively keeping the trajectory in the river.

To provide direct justification in our own experimental setting, we measured the loss along linear interpolations between training checkpoints under the setting of Fig.2 (Slimpajama, 0.5M batch size). We compared (1) AdamW with constant LR, (2) AdamW with linear LR decay to 0, and (3) SF-AdamW with constant LR, using checkpoints after 2B and 2.5B tokens. Results are reported in Table 1 as below.

Table 1: Test loss evaluated at linear interploation $\alpha w_0 + (1-\alpha) w_1$ between two checkpoints (2B and 2.5B tokens)

These new results clearly demonstrate three distinct dynamics, replicating the findings of prior work and supporting our hypothesis:

• AdamW (constant LR) shows a convex, valley-shaped loss profile between checkpoints.
• AdamW (linear decay to 0) shows a sharp, monotonic decline, consistent with an iterate transitioning from the valley wall down to the valley floor.
• SF-AdamW (constant LR) shows a flat, slow decline, consistent with an iterate already moving along the low-curvature valley floor.

Notably, (1) and (2) replicate the loss profiles reported by Fig.7 in Wen et al. and Fig.2 in Belloni et al., while (3) further supports our claim that SF-AdamW closely tracks the river.

This combination of an established literary framework and our new, direct empirical evidence provides a solid foundation for our analysis using the river-valley landscape.

References

Wen et al. (2025), Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape View, ICLR 2025.

Belloni et al. (2025), Universal Dynamics of Warmup Stable Decay: understanding WSD beyond Transformers, ICML 2025 Workshop on High-dimensional Learning Dynamics.

Song et al. (2025), Does SGD really happen in tiny subspaces?, ICLR 2025.

Q3: The Toy Objective

The observation is on a simple 2D toy objective... Can this carry over to large scale?

The toy objective in Section 4 is deterministic and simplified by design. This intentional simplification allows us to isolate and study how the optimizer interacts with the key geometric properties of the river-valley landscape without the confounding effects of stochasticity.

What validates this approach is our finding that this minimal model reproduces surprisingly rich phenomena that are observed in deep learning experiments. For instance, the toy objective accurately captures:

• How schedule-free dynamics track the valley floor and how this behavior depends directly on the momentum parameter.
• The emergence of the Edge of Stability phenomenon, specifically occurring at the $y_t$ iterates, which is a highly non-trivial dynamic.

The fact that such a minimal setup can replicate these complex behaviors makes it a powerful and insightful tool for understanding the fundamental mechanisms of schedule-free optimization, confirming its relevance beyond the simplified setting.

Q4: The Refined Optimizer and Hyperparameter C

We have a new hyperparameter C to tune. It might still be sensitive to C?

We appreciate this concern and directly address it with new sensitivity experiments conducted during the rebuttal period. Building on our main experiments on Slimpajama with 0.5M and 2M batch sizes, we swept over a range of values for C and evaluated test perplexity on 9M held-out tokens. The results are shown in Tables 2 and 3 below.

Table 2: Refined SF-AdamW, Batch Size 0.5M, $\beta_2=0.99$, LR 2e-3, Test Perplexity

Table 3: Refined SF-AdamW, Batch Size 2M, $\beta_2=0.99$, LR 2e-3, Test Perplexity

Our findings show that refined SF-AdamW is robust to the choice of C: it consistently outperforms vanilla SF across a broad range of values. For example, in Table 2, settings such as $C=50,100,200,500$ all improve over vanilla SF for $\beta_1=0.9, 0.95$, and $C=50,100$ improve over vanilla SF across all momentum configurations. Similarly, in Table 3, refined SF outperforms vanilla SF over a wide range of $C$, with improvements persisting up to very large values.

These results indicate that refined SF-AdamW remains robust and effective across a wide range of C, reducing the cost of tuning this hyperparameter.

Q5: Memory Overhead

How can you claim no additional memory overhead compared to AdamW when SF methods require three sequences (x, y, z)?

We would like to clarify that our claim about SF-AdamW having no additional memory overhead compared to AdamW is accurate. The key point is that SF-AdamW can be interpreted as the Schedule-Free method applied over RMSProp, just as AdamW is momentum applied over RMSProp.

To be specific:

• AdamW: Requires storing the model parameter (x_t), the 1st moment vector (m_t), and the 2nd moment vector (v_t).
• SF-AdamW: Requires storing the model parameter (x_t), the auxillary parameter (z_t), and the 2nd moment vector (v_t). Note that y_t is computed on the fly from x_t and z_t, and does not need to be stored separately.

Thus, both optimizers require storage for three full copies of the model parameters, and SF-AdamW does not introduce any additional memory overhead compared to AdamW. We will add this clarification in the next revision.

Thank you for your efforts and for appreciating our work. We address your concerns and questions below.

Response to Weaknesses

The experiments are limited to relatively small-scale models (e.g., 124M parameter transformers decoder) due to computational resource constraints. / The paper admits the lack of error bars report and analysis due to computational constraints.

We agree and will explicitly acknowledge this in the Limitations section of the revised manuscript. Our experiments were conducted on 124M-scale models trained up to the Chinchilla-optimal compute budget. This scale was intentionally chosen to balance interpretability of Schedule-Free dynamics with available computational resources.

The primary goal of our paper is to understand the mechanisms underlying Schedule-Free optimization, rather than to demonstrate state-of-the-art performance. Accordingly, we prioritized controlled ablations and mechanistic analysis over repeated trials or larger-scale benchmarks. In Sections 3 and 4, which focus on analyzing training dynamics rather than making performance claims, we believe error bars would provide limited additional insight. In Section 5, however—where we introduce the refined SF method and claim improvements over vanilla SF—we agree that error bars would strengthen the statistical significance of the results.

Due to the limited time and resources during the rebuttal period, we could not provide these additional runs. We will, however, include error bars for the refined SF experiments in Section 5 in the revised version.

Response to Questions

The authors have demonstrated the improved validation loss of their refined SF-AdamW optimizer compared to the original SF-AdamW. Can the refined SF optimizer lead to measurable performance gains on downstream tasks based on different models and benchmarks?

Evaluating downstream task performance is beyond the scope of this work. Our focus is on developing efficient and scalable pretraining algorithms. Assessing downstream performance would require fine-tuning models that have been pretrained using our method, which in turn necessitates pretraining very large models on massive datasets—a level of computation that is currently beyond our resources.

While we acknowledge the differences between pretraining and fine-tuning loss landscapes, we believe that our refinement may, to some extent, generalize to fine-tuning tasks. Verifying the effects of decoupling and other findings from pretraining dynamics in the context of fine-tuning tasks remains a promising direction for future work.