On the Provable Separation of Scales in Maximal Update Parameterization

1. Comparison with Earlier Wide-Network Theory Frameworks:

We appreciate the suggestion to clarify how our work provides a novel perspective complementing earlier wide‐network theories like the NTK and mean-field SDE frameworks. In our paper, we emphasize an explicit separation of scales that distinguishes between global (macro) and local (micro) dynamics in neural network training—a perspective that is not directly addressed by NTK or conventional mean-field approaches.

NTK theory primarily examines the behavior of networks in the infinite-width limit, effectively linearizing the training dynamics. While this yields valuable insights into the global macroscopic behavior of the network function via a constant kernel, it largely overlooks the finite-width nuances and the evolution of individual weight updates. In contrast, mean-field SDE methods focus on modeling the microscopic evolution of weights, capturing the stochastic fluctuations of individual parameters. However, these methods do not inherently highlight how aggregated statistics, like loss landscapes or activation norms, stabilize rapidly.

Our approach under µP rigorously demonstrates that macro-variables converge at an O(1/n) rate, while the micro-variables evolve more slowly at an O(1/√n) rate, building upon the SDE method used in the Xie et al. (2024), as discussed in Section 2.3 as a recent work. Thus, our work unifies these perspectives: it extends NTK analysis by accounting for finite-width effects and complements mean-field approaches by clearly delineating the roles of global and local dynamics. This multi-scale perspective not only deepens our theoretical understanding in addition to the two frameworks but also has practical implications for the effectiveness of early-stage HP tuning: even when individual weights have not yet converged, the stable, global signals are sufficiently reliable to guide HP selection. .

2. Experimental results:

As proof of concept, we take a two-layer MLP where the input layer is 3072 units (flattened 3x32x32 CIFAR-10 images), the hidden layer is n=10000 units with ReLU activation, and the output layer is 10 units (for class probabilities), the SGD optimizer with LR 0.01 - 0.30 with 0.01 increments, CE loss, batch size 128, and n=300 training epochs with a sampling interval of checkpoint 10. Our experiment confirms the macro-micro scale separation we observe theoretically. In particular:

i. Fast convergence of macro-variables: The relative loss differences between learning rates are very small in early epochs (at epoch 10 already, as an example, between LRs 0.07 and 0.08, the loss difference is 0.0005), indicating that the loss landscape has a consistent structure across different learning rates, as predicted by our O(1/n) convergence rate for macro-variables. We observe effective early hyperparameter tuning: in low LR (0.01-0.1), medium LR (0.1-0.15), and high LR (0.15-0.3) settings, the respective top learning rates emerge early (at epochs 60, 60, 100), indicating the loss landscape stabilizes quickly.

ii. Slow convergence of individual weights: Despite early identification of optimal LRs, the loss continues to decrease and finetune throughout all 300 epochs for LR < 0.1 and until 180 epochs (LR = 0.3) to 250 epochs (LR = 0.15) for larger LRs. We also see a dramatic increase in the loss gap in later epochs (the ratio between the worst and the best losses, across lrs, is 1.10 at epoch 10 and 238.33 at epoch 120), indicating that small initial differences are being amplified. This phenomenon is consistent with the idea that while the macro structure stabilizes quickly, the micro-scale adjustments (individual weight evolutions) continue refining the network, progressively enhancing the performance gap.

3. Proof Completeness: We promise to include expanded proofs with explicit epsilon-delta arguments and tighter bounding constants to enhance rigor in the appendix.

4. Limitations Under SP and NTK: μP provides the cleanest setting for demonstrating scale separation due to two properties: (1) consistent gradient/activation scaling preserving feature learning as width increases, and (2) self-similar behavior of global statistics that stabilize quickly.

For standard parameterization (SP), gradients shrink with width, leading to "lazy" training without clear timescale decoupling. Similarly, NTK parameterization preserves a limit but keeps features near initialization, yielding kernel machine behavior without pronounced scale separation. We acknowledge this limitation and view it as an exciting direction for future research to determine whether analogous separation phenomena can be rigorously established in alternative parameterization regimes. For example, one could consider the α-scaled framework (α=1 for μP, α=1/2 for NTK), and hypothesize that values closer to 1 likely maintain scale separation; however, a comprehensive characterization requires further investigation. We will add the discussion.

1. Def. of “Separation”: Thanks for suggesting the formal use of “separation” in learning theory. Here, we use “separation of scales” to describe the difference in convergence rates between macro- and micro-variables under μP. We show that loss landscape descriptors converge at rate O(1/n), while weights evolve at Θ(1/√n). This holds in practice: coarse statistics stabilize early even as parameters continue evolving. We agree that we shall clarify our use of Θ(1/√n) instead of O(1/√n). What we establish is a quantitative gap in convergence rates with meaningful implications for early-stage hyperparameter (HP) transfer.
2. Initialization in Assumption 3.3 and Lemma 4.6: We thank the reviewer for catching a typo in A3.3 (due to accidental copy-paste). The input weights were misstated as N(0,1/n^2); they should be N(0,1). Importantly, this typo does not affect our derivation despite the text misstating it. L4.6 already assumes that input weights are N(0,1), ensuring pre-activations stay O(1). All weight initializations after L4.6 are correct as they concern hidden-layer weight N(0,1/n). Thus, the typo does not make our result incorrect. We made typos in A3.3 re the scaling of learning rate (LR). In subsequent derivations, though we wrote that η=O(1/n) for output-layer parameters under μP, we clarify that this refers to the LR per parameter, not the total per-layer update. In μP, LR and gradient norms are co-scaled so that each layer’s update remains O(1), preserving stability as width grows. For example, the gradient on output weights scales as O(1), and with η=O(1/n), the resulting update is also O(1/n), matching the intended scale.
3. Lemma 4.6 and Gradient Norms: Our argument that O(1/n) LR results in O(1) weight updates can indeed be expanded. Under μP, the LR and gradient magnitudes are co-scaled to ensure stable updates: gradients scale as O(1) for output layer weights (due to downstream fan-in), LR scale as O(1/n) for output weights; thus, their product (weight update) remains O(1/n), matching the expected per-step update scale. This is consistent with μP’s core principle that all layers receive comparably scaled updates across widths. We will clarify this interaction between LR and gradient norm.
4. Assumption 4.5: Width Regularity: The assumption is standard to ensure the convergence of macro-variables under increasing width. Similar assumptions are used in the NTK/SDE literature and μP papers: e.g., Yang (2021) assumes that activations are smooth (or smoothed ReLU), and many proofs invoke pseudo-Lipschitz or bounded-derivative conditions to enable LLN-type arguments. A4.5 ensures that descriptors like average activation variance or loss stabilize as width increases—empirically supported and analytically tractable. This assumption holds for standard activations (e.g., ReLU, GELU) either directly or via minor smoothing. It does not weaken the generality or correctness of our results.
5. Memory Kernel in Theorem 5.1: q(τ) is not an ad hoc invention: the term arises when modeling the effect of time-varying LR momentum in a continuous limit. A prominent example is the Generalized Langevin Equation, which includes a memory kernel to capture how past states influence the current update. The recent DMFT for SGD yields integral equations with memory kernels too. Here, it serves as a continuous-time analog of LR memory effects studied in training delay. In Tissue et al. (2024), the delayed response of validation loss to changing LRs is modeled by cumulative functions like S1(t) and S2(t). Our notation makes explicit the role of memory weight q(τ), enabling clear continuous-time modeling of past LRs. We will add citations and clarify the role.
6. Gradient Norm O(1) in Section 5.2: The claim that gradient norms remain O(1) with time-varying LR η(t) relies on Assumption 5.2, not just μP scaling. Under μP, at any time t, the per-layer LR should still scale with width to ensure bounded updates. Our use of time-varying η(t) assumes that such width scaling is preserved pointwise in t, as is common in practical μP. The theoretical foundation for this substitution comes from NTK/μP literature, where smooth η(t) can be inserted into gradient flow ODEs without breaking convergence. We will make it explicit.
7. Tensor Programs Faithfulness: While we build on the μP works, our approach uses SDE to distill the key phenomena of macro vs micro scales. We do not replicate the exact Gaussian Process construction as our focus is on connecting HP transfer and early‐stage alignment under μP. But our theoretical lens is compatible with the same scaling laws, and we cite those results to justify certain variance and regularity assumptions. Nowhere do we claim to reproduce the entire measure-theoretic formalism from the ground up. Our vantage point is narrower—yet consistent—focusing on the rates at which global vs local descriptors converge.

Minor Issues: We will correct the typo at the end of Section 4.2 and fix the ICML template.

We thank the reviewer for their careful reading and thoughtful comments. Below we address each point raised:

(1) Lemma 4.7 and the asymptotic notation: The reviewer correctly identified that if L is treated as a constant independent of n, this would typically yield O(n^(3/2)). However, in our analysis, the Lipschitz constant L in our context decays with the network width, specifically L=O(1/√n). As d(n)=O(n^2), \sqrt{d(n)}=O(n). Therefore, L√d(n)⋅O(√n) = O(n).

(2) Applicability of μP to phenomena in Sections 5 and 6: in addition to providing theoretical justifications as laid out in the submission, we have conducted small experiments on PreResNet-101 + CIFAR-10 (aligned with original You et. al. 2020 setting), and confirm that using μP, we can still replicate Table 2 accuracy results at p = 0.3, 0.5, 0.7. Due to 1-week limit, we were unable to fully replicate the LLM experiments in Tissue et al., 2024) using μP, but we will continue afterwards and updates results to our paper too once we finish.

(3) Intuition on Magnitude-based vs Hessian-based pruning: While the original early-bird work focuses on iterative magnitude pruning (IMP) in the context of the lottery ticket hypothesis, subsequent research has introduced non-IMP “early ticket” methods—such as ProsPr, EarlyBird, and EarlyCroP—that incorporate additional training signals to enhance mask quality and speed up early convergence. However, to the best of our knowledge, no prior work has specifically explored Hessian-based pruning in the early-bird setting (they do exist in another related setting of “pruning at initialization” or PaI). Moreover, our paper’s analysis relies on modeling the network’s training dynamics with a gradient-flow SDE (i.e., approximating discrete SGD updates by a continuous-time stochastic differential equation), and thus does not readily apply to Hessian-based approaches. Intuitively, Hessian-based methods typically exhibit slower initial convergence but can achieve better long-term performance by using curvature information to prune weights that minimally affect the loss landscape, even if those weights do not have the smallest magnitudes. We will leave the more comprehensive study as future work.

Two other quick fixes:

(1) We have fixed the assumption and theorem numbering in each section.

(2) We revised the notation in Theorem 3.1 to η*(t₀), so that in general η*(t) represents the optimal hyperparameter value at time t, while η*(∞) represents the global optimum.

We are grateful to the reviewer for their valuable feedback, which has helped us improve the clarity of our manuscript.

1. Width-Dominance

Our analysis explicitly relies on the width-dominance regime (Assumption 3.2)​ ensuring as width grows, certain terms dominate the learning dynamics. We will revise to explicitly mention it in the abs/intro. Note that it is a standard condition formalizing the intuition that we operate in the large-width limit where 𝜇P theory is intended to apply. Prior 𝜇P literature also shows partial validity beyond purely “infinite-width” conditions. Besides, our additional experiments in reply to Reviewer g228 show that scale separation behaviors indeed happen in training MLPs.

2. Justifying Assumption 3.4

Wide-network studies provide strong empirical and theoretical support for A3.4. Prior 𝜇P work (e.g., Yang et al. 2021; Lingle 2024) shows that layer-wise statistics, e.g., activation variances and gradient norms, stabilize early in training and remain nearly invariant across widths. This is consistent with infinite-width theories where macro-level descriptors concentrate, as common in NTK and mean-field analyses. Additionally, recent large-scale experiments (e.g., Vyas et al., 2023; Cerebras “Practitioner’s Guide to 𝜇P”) also confirm that optimal HPs stay stable across model scales, indirectly verifying the training dynamics exhibit self-similarity.

3. Convex Settings & Strong Convexity of 𝑀(𝜂;𝑛)?

We clarify that our main results do not require global convexity of the loss. Rather, we assume 𝐿-Lipschitz gradients in the standard sense of gradient-based methods (A4.2), typically used in nonconvex analyses to control the norm of the gradients and Hessians locally. Nowhere do we require the global objective to be strictly convex.

In Lemma 4.8, we consider 𝑀(𝜂;𝑛) as the effective scalar function describing the expected loss after training steps 𝑇 under a macro-level parameter 𝜂. For wide enough networks, macro-level fluctuations vanish, and local expansions around 𝜂*(𝑛) ensures a strongly convex local basin in 𝜂. This is akin to standard “strict local minimum” conditions in practical large-scale NNs near an optimal LR. We will clarify in the revision that this does not imply the full model’s loss surface is globally convex.

4. Novelty and Paper Structure

Our main innovation is the macro–micro separation framework under 𝜇P. Specifically, we derive a continuous-time SDE approximation (Sec. 4) that rigorously separates the convergence of macro-level variables at O(1/𝑛) from the slower micro-level O(1/√𝑛) behavior. This interplay had not been previously formalized; we will state more explicitly.

We will add a clear roadmap of the paper by end of Sec. 2:

Sec. 3 formulates the assumptions and “early-stage coarse-graining” argument.

Sec. 4 establishes the main SDE-based theorems (macro/micro separation).

Secs 5 and 6 apply our unified theoretical framework to explain two empirical phenomenon, on HP and parameter early stability respectively:

Sec. 5 treats the ‘learning rate delay’ using macro-level integrated LR descriptors to explain the lag law;

Sec. 6 extends to the early stabilization of “critical parameter subset”.

5. Comparison to Xie et al. 2024

Ours and Xie et al. (2024) both leverage SDE to study training dynamics, but from different perspectives. Xie et al predict training loss and optimal LR schedules, deriving convergence rates and escape probabilities under time‑inhomogeneous SDEs, providing an empirical scaling law for HPs. In contrast, we rigorously develop a width scaling theory within μP. We prove a separation of scales between macro-level dynamics (which govern optimal HP transfer) and micro-level fluctuations, thereby explaining why HPs become width‑invariant as the network grows. In short, while Xie et al. offer complementary SDE-based insights on HP sensitivity in non-convex regimes, our work uniquely addresses the underlying mechanism of HP transfer via width-dominance—a question left open in their work.

Technically, Xie et al. is formulated for general non-convex optimization at the cost of ignoring width-specific dynamics​. Our analysis is based on conditions that allow us to rigorously prove separations, typically invoking local convexity or smoothness in the effective (meta-)objective governing HP selection. Empirically, the HP transfer phenomena appears robust even in non-convex NNs. Nonetheless, to extend our rigorous analysis fully to the general non-convex case, additional work would be needed to overcome challenges from multiple local minima and non-unique optimal HPs. We will clarify this comparison/limitation.

6. Miscellaneous

Citing arXivs: We will update our reference list to cite arXiv's accepted versions whenever available. Our result does not depend on Lingle (2024); the citation was meant to give context of validating and extending 𝜇P in practice.

Generality of Theorem 3.1: You are right. These assumptions are more about how the model is parameterized and how global descriptors converge rather than about specific architectures.