4+3 Phases of Compute-Optimal Neural Scaling Laws

We thank the reviewer for their insightful comments and suggestions which were very helpful. We address these below.

1. Novelty of Techniques.
   • Analysis of learning rates via Volterra equations in concert with random matrix theory has appeared before (say $10$ / $11$ in the paper). Volterra analysis is not widespread, and it remains to be seen how general purpose they are. For precise risk curve analysis of stochastic algorithms, they seem useful, including momentum methods $1$ . A more general point of view is needed for analysis of non-quadratic problems, but there is an equivalent system of ODEs reminiscent of mean field theory (this is equivalent to the Volterra description in the least-squares case) which has been used as well $2$ . Generalizing the type of results in this paper to a non-linear setting would require understanding how these systems of ODEs evolve without appealing to convolution-Volterra-theory; these look quite similar for the linear and nonlinear cases, so I would be cautiously optimistic that the risk curve analysis could be adapted to nonlinear PLRF

$1$ Paquette et al. Dynamics of stochastic momentum methods. 2021

$2$ Collins-Woodfin et al. Hitting the high-dimensional notes: An ODE for SGD learning dynamics on GLMs and multi-index models. 2023.

• Random matrix theory analyses of generalization errors in random features models are now pretty well-developed, and this paper certainly fits within that tradition. The majority of the technical work and mathematical novelty in this paper is the analysis of the PLRF resolvent, which is a pure random matrix theory question. After that, there’s a fair bit of asymptotic analysis which is needed, which will probably be common to all analyses of power-law scaling law problems.

2. Regarding applications to realistic models. We’ve also been asking ourselves this question. Probably the quantitative predictions of this model will be impossible to fit to anything like Chinchilla; one would need to have a way to estimate both $\\alpha$ and $\\beta$, which is approaching science fiction.

   Now on the other hand, it is possible (as in Chinchilla) to measure the resulting scaling laws. One may attempt to vary parameters that could influence $(\\alpha,\\beta)$, and see if the scaling laws respond in the way one would expect from this paper. One might also look for a phase plane of LLM parameters in which the optimal scaling d^\* \= \\sqrt{f} with $f =$flops changes; we saw that this is possible in some parts of the $(\\alpha, \\beta)$ plane.

   Another possible complexity is that adaptive and preconditioned methods are all but necessary for training LLMs, and so some effort should be made to establish to what extent the phase diagram we described is affected by different optimizers.

Responses to Questions:

1. (Deeper pattern to where d^\* \= \\sqrt{f}). This is a great question and perhaps one of the biggest mysteries of the phase diagram, given that it appears in 3 distinct regimes. I can share some possible speculations, but I don’t really know.
   • One possible answer is that real-life is closest to $\\alpha=\\beta$ (or perhaps autoregressive problems are always somewhere close to $\\alpha=\\beta$), and along this line it just so happens $d^\*= \\sqrt{f}$.
   • Potentially, regimes where d^\* \= \\sqrt{f} are not optimal are a reflection of the algorithm ‘failing’ in some sense, and one should be looking to improve it. Conversely, existing work suggests that Phase Ia might be algorithmically unimprovable – see the discussion above.
2. (Extensions to more complex models). Of course the proof will be ``in the pudding,’’ but we would guess that the answer is yes – nonlinear power-law-random-features, anisotropic weight matrices, models with some level of feature learning in mean-field scaling. Existing work on random features regression and 2-layer networks strongly suggests that the extension to the nonlinear case should be solvable. The case of anisotropic weight matrices should be quite similar mathematically to this work (it mostly comes to mind as a way to better understand the effect of using preconditioned algorithms on scaling laws).
3. (Practical lessons for LLM scaling). We have a few possible lessons:
   • We give some further evidence that d^{\\star} \= f^{½} is justified.
   • The ‘functional form’ of the scaling laws of the risk curves which is empirically fit to scaling laws should be updated to be Risk(n,d) \= c \+ a\_1 n^{-b\_1} \+ a\_2 d^{-b\_2} \+ a\_3 d^{-b\_2} n^{-b\_3}.. The last term looks like $F\_{ac}$, and is usually dropped. But it is needed even in this simple setting to get the correct scaling law behavior (in our setting $b\_2$ is always $1$).
   • Finite-dimensional effects in size of parameters (d) exist and potentially have a huge impact on the compute optimal exponents. We see even in our simple model (see Fig. 4). If one is doing empirical scaling laws, then one can easily see measured exponents that change by something like $0.05$ as one increases the FLOP-budget by a factor of $10^9$. In LLM scaling, this $0.05$ is the same order of magnitude as the observed scaling law itself, and it is not (per se) related to a breakdown of the scaling laws.
   • Speculatively: with some hypotheses about which phase one is in, one may be able to determine what is the dominant feature of the loss curve ($F\_{ac}$ dominated, $F\_{pp}$ dominated, $K\_{pp}$ dominated). This could allow one to determine if in future training runs, how to increase parameter counts or compute. (Roughly $F\_{ac}$ is expensive from a compute point of view, and it is cost efficient to increase scale – or reserve compute for other parts of the model when one enters this regime). Of course, this heuristic is just a hypothesis, based on the work here and needs to be tested.

We thank the reviewer for their insightful comments and suggestions. We address below questions and concerns raised by the reviewer. Because there was a lot of depth in the questions raised, we needed additional characters to respond adequately so below is the first part of the response. There will be an additional “Official Comment” with the remaining response.

Responses to Weaknesses:

1. (Related Work) We are very happy to add discussion regarding random feature generalization (especially the relations to generalization error bounds which were developed for source/capacity conditions) and SGD complexity on power-law problems, which are absolutely related to this work. See response to the ‘General Reviewers’ and the attached pdf for comparison of our notation to source/capacity conditions.

2. (Accessibility)

   • We will add an introduction to the appendix to guide how the proofs are done.

   • The contour is `what works.’ We do not believe the contour is absolutely necessary to proving the result; it is a nice technical tool, but not essential. Indeed – we heuristically derived all the arguments long before we found the contour in the process of formulating the proof.

     The origin of the contour is solving the fixed point equation for $m$. The ‘kink’ in the contour occurs at real-part $x \\approx d^{-\\alpha}$. This corresponds to a point of transition in the spectrum of the PLRF matrix: for smaller scales, the self-consistent spectrum forms a spectral bulk. For larger scales, the self-consistent spectra forms outliers, roughly located at $j^{-2\\alpha}$ for integer $j$. In the self-consistent spectra, these outliers actually have densities of width approximately $O(1/sqrt(d))$.

     Now the contour is basically chosen as the closest contour to this axis that makes this transition invisible, which means the imaginary part of the contour is just a bit larger than the inter-eigenvalue spacing, and it means we see a clean power-law everywhere along the contour.

     One could instead not put the kink, but then would have an absolutely continuous part and an ``approximately’’ pure point part of the spectrum, which is more annoying than anything (mostly because the m function undergoes some relatively high-frequency changes near each outlier, and all this high-frequency excitement turns out to be completely invisible and unimportant to the larger picture). The method we use to approximate the m is fundamentally perturbative, so matching all the high-frequency changes in m is super-hard; so it’s easier to change the contour and then have a good, smooth ansatz for m.

3. (Comparison with Bordelon et al. $1$ )
   We highlight below the main differences between our work and Bordelon et. al $1$ :

• Bordelon et. al assumes the functional form for the loss is $n^{-\\tau} \+ d^{-\\sigma}$ (see Appendix N, Eq. (153)). Then they use DMFT to find the exponents $\\tau$ and $\\sigma$. We prove that the functional form for the loss of power-law random features is

  $P(n,d) \\asymp \\underbrace{n^{-\sigma\_1}}\_{F\_{pp}} \+ \\underbrace{n^{-\sigma\_2}}\_{K\_{pp}} \+ \\underbrace{d^{-\tau\_1}}\_{F\_0} \+ \\underbrace{ d^{-\tau\_2} n^{-\sigma\_3}}\_{F\_{ac}}$

  Note the cross-term $\\underbrace{ d^{-\tau\_2} n^{-\sigma\_3}}\_{F\_{ac}}$ which is missing from the functional form for Bordelon and plays an important role in Phase II and III for the compute-optimal curves. Additionally, Bordelon et. al. do not consider phase III, and the impact of the SGD algorithm through the term $K\_{pp}$.

• Our work holds for any $\\alpha$ and $\\beta$ and Bordelon et. al. only work for the trace class setting, i.e., $2 \\alpha \> 1$. Moreover Bordelon et. al. compute-optimal result agrees with our compute-optimal exponents when $2\\beta \< 1$. This is consistent since the loss curves in Phase Ia only depend on $F\_{pp}$ and $F\_0$. The cross term $F\_{ac}$ and the impact of SGD $K\_{pp}$ do not appear until Phase II-IV.

  Comparison of $(a,b)$ and $(\\alpha, \\beta)$ between the two papers

  ---

    Bordelon, et. al. |   This paper
  

  ---

        b             |    $2\alpha$ 
  
        a             |    $2\beta + 2\alpha$ 
  

  We will add a comparison to the text (see also Table 1 in attached pdf of revision).

$1$ B. Bordelon, A. Atanasov, and C. Pehlevan. A Dynamical Model of Neural Scaling Laws. arXiv preprint arXiv:2402.01092, 2024.

I thank the reviewers for compiling a more proper bibliography for this quality of work.

I also thank them for explaining in detail the differences between this work and the prior works, especially of Bordelon et al. I looked at the supplementary note, comparing notations and especially Table 3 comparing this work with others. I think this is a very nice table, that will be important and useful for the community, and I encourage them to either start the appendix with this table or to even consider putting it in the main text. I definitely hope that the related work section on page two can be expanded beyond just five sentences to highlight some these facts and point the reader to the table. It makes it much more clear what this paper contributes over prior work.

Lastly, given that the contour is indeed just "what works" I strongly encourage the authors to state that clearly in the appendix. I found the appendix relatively readable until this contour came out of nowhere. Even just an idea of what motivated the authors to consider it would go a long way.

As promised, I have raised my score.

Responses to Weaknesses and questions: Thank you for your comments. We will definitely add a more thorough evaluation of related works. Great catch about Footnote number 6 – it’s actually not true – we’ve removed it. We’d be very happy to hear any other comments about the content of the paper – time allowing, we will add some discussion in the appendix about batch size, which was a not-fully-explored thread of this paper.

Regarding the questions (square loss, lack of non-linearity in PLRF, deterministic teacher vector $b$, need for $v \> d$). ``All these assumptions are reasonable, but must be compared to related works on the subject and why it is difficult to lift them.’’

1. Extending to square loss and adding a nonlinearity are both very interesting directions of future research, and it would certainly be reasonable to do them after considering the square loss – linear case!
2. The deterministic teacher vector could be replaced by a random one with neither an increase in complexity, but also no change in phenomenology (if it has the same behavior as $b$). Perhaps we can add: the goal here isn’t really theory-building, in the sense of covering as wide a class of kernel regression problems as possible. The goal is to map out as much phenomenology as possible. So generalizations of the problem-setup which are not expected to change this phenomenology were not prioritized.
3. The case of $v\>d$ is for simplicity and also because we broadly believe it aligns with what one sees in neural scaling laws. Indeed you can consider another phase when ($2\\alpha \+ 2\\beta \< 1$) with $v \< d$, but we suspect it is another ‘Phase I’. The same comment goes for negative $\\beta$.

We agree with the reviewer that more references and a more thorough discussion of background and related work would improve the accessibility of this paper. For that reason we have included 32 references and proposed changes to the related work and background sections (see the general response). In light of these changes, we hope the reviewer will reconsider their score.

Hi reviewer, in the response you wrote that you would like to increase their score from a 4$\rightarrow$6. However seems the system not actually change their score. If you want to increase your score, may you implement this to the openreview system?

Thanks.

Your AC

We thank the reviewer for their comments which were very helpful. We address these below. Because there was a lot of depth in the questions raised, we needed additional characters to respond adequately so below is the first part of the response. There will be an additional “Official Comment” with the remaining response.

Response to weakness: Yes we agree, the comparison to existing work should be expanded. We will include a more detailed discussion of existing work and how it compares. We will also include a section detailing how these SGD-generalization bounds compare to optimized non-algorithmic bounds (especially to optimal ridge regression). The suggestions for references are super helpful – thank you – and we have made an attempt to integrate them. We’ll outline the results of our attempt to answer your question in the points below. We’re happy to have feedback if you believe the works considered are still inappropriate.

Regarding the specific suggestions, thanks, many are new to us. We will formulate our response partly based upon the concurrent article $1$ , which features a sharp analysis of ridge regression on a problem with the capacity/source conditions and which improves over Rudi-Rosasco. Note the source/capacity conditions are not meaningful for $2\\alpha \< 1$, so we restrict the discussion to phases Ia/II/III ($2\\alpha \> 1$).

Directly addressing the points you have made below:

1. Ridge regression comparison. The bounds for ridge regression agree with SGD in phases Ia/II, which is to say that if one takes ridge regression with r-samples, computes the generalization error with the ridge estimator with a O(1) ridge parameter; then the SGD risk curve we compute agrees with the generalization error and the ridge error agree up to constants. (Using Corollary 4.1 of $1$ ).

To match notation with the source/capacity literature, we'll use our $\alpha$ parameter (so the capacity is $2\alpha$) and we'll use the source parameter $r = (\\alpha+\\beta-0.5)/(2\\alpha)$; see also Table 1 in the pdf attached to OpenReview. We note that phases Ia/II/III correspond to previously observed hardness regimes ($r \\in (0,0.5), (0.5,1.0), (1.0, \\infty)$)

A few comments about this: from $1$ , one recovers that the ridge-generalization-error equals the SGD-generalization-error in Phase 1a/II, at all points in the risk curve. In Phase III, ridge regularization with O(1) ridge has better generalization error than SGD. The comparison to the Rudi-Rosasco bounds can be seen in (48) of $1$ . They match the generalization errors in Phase Ia, but are not optimal in Phase II/Phase III.

The Carratino et al. bounds (Corollary 1 of that paper) match the minimax-optimal rates attained by kernel ridge regression, and they effectively show that there are parameter choices for one-pass SGD which match those rates. We discuss below minimax optimal rates.

2. Regarding non-parametric regression: thanks for these references. Bahri et al. and Maloney et al are indeed not mathematics papers. We still believe they are quite valuable scientifically to this paper. We’re happy to clarify how we view their contributions.

• Regarding your main question Figure 2, it is not possible to get this figure by identifying steps with samples. It’s super important to plot the x-axis of the curve in FLOPs, or there is no compute-optimal frontier (i.e. each iteration has a cost which is d-dependent). So without choosing an algorithm which attains the regression bounds in the papers you list and assigning them a computational cost, the problem is not well-posed.

  Now if we are talking about ridge regression with $O(1)$ ridge, then we could assign a computational cost of ($n \\times d$), for example using conjugate gradient (CG) (up to log factors). One could also do this by considering vanishing ridge parameter, but then (since the condition numbers of these matrices grow like $d^{2\\alpha}$), the computational cost of computing using CG will increase to $n \\times d \\times (\\min\\{ d^{\\alpha}, \\sqrt{1/ridge}\\})$.

  Let’s suppose we continue with ridge parameter $O(1)$, which is morally comparable to one-pass SGD, in that they use the same amount of compute – it’s also comparable in the sense that a non-optimized algorithm like gradient descent on the ell2-regularized-least-square objective will be similar in complexity (and vanilla SGD is surely a non-optimized algorithm).

  In Phase Ia/II, the optimal ridge estimates in $1$ (which agree with Rudi-Rosasco in Phase 1a but not in II) would indeed yield the same curves, treating computational complexity as $n \times d$ (with $n$ the number of samples and $d$ the dimension). In Phase Ia, you get the same loss curves as SGD with O(1) ridge regularization, and so you get the same compute-optimal frontiers for CG+O(1)-ridge as SGD. Incidentally, here you gain nothing by increasing the amount of ridge regularization.

  In Phase III, CG+O(1)-ridge performs better than SGD for small sample counts. However, if you run SGD to the stopping criteria for which it is compute optimal, it performs the same as CG+O(1)-ridge with the same choices of parameters/sample counts. Now, on the other hand, it could be (and should be) that using CG+O(1)-ridge there is a different compute-optimal frontier curve.

  We’re very happy to add the details (and some of the computations) of the comparison of SGD and ridge regression in an appendix, showing how these curves agree/do not agree. We will also improve the discussion surrounding how compute plays a role, and link it to kernel regression literature.

$1$ Defillips, Loureiro, Misiakiewicz. Dimension-free deterministic equivalents for random feature regression. arXiv: May, 2024

We thank the reviewer for their report, for their comments on directions of improvement, and for the questions.

Response to weakness: Yes we agree, the comparison to existing work should be expanded. We will include a more detailed discussion of existing work and how it compares (see comments to all reviewers). The suggestions for references are helpful – thank you – and we have made an attempt to integrate them (see the proposed related works section). We’re happy to have feedback if you believe the works considered are still inappropriate. We provided a comparison with the concurrent work $2$ (see Table attached in pdf) which was released after NeurIPS submission.

Response to question (2-layer Neural Network): With regards to the question, the reference $3$ is very interesting. It is the first time this author is reading it, but it’s very close to my interests and I will dig back into it after NeurIPS.

In $3$ , if one takes the single-index model to have a suitable decay rate of its Hermite expansion (which I believe is power-law decay of its Hermite coefficients), the gradient flow loss curve would have power-law decay. That seems like it should open the door to an analysis of one-pass SGD that exhibits similar phenomenology to what we’ve done. I’m not bold enough to speculate how similar it would be to what we’ve done, because I don’t have a good picture of what the SGD-noise looks like in this setup. If the SGD noise behaves similarly to the quadratic problem we study, then it could fit naturally as a line through our phase diagram parameterized by the exponent $\\beta$ (which would come from the decay rate of the target function).

That would be a beautiful theorem!

$2$ Lin, L., Wu, J., Kakade, S. M., Bartlett, P.L., & Lee, J.D. (2024) Scaling laws for learning with real and surrogate data. arXiv preprint arXiv:2402.04376.

$3$ Berthier, R., Montanari, A., & Zhou, K. (2023) Learning time-scales in two-layers neural networks. arXiv preprint arXiv:2303.00055.

Response to limitation: A deeper empirical investigation of LLMs would also interest this author, who hopes it can be done. This is certainly a limitation; also estimating the alpha (and worse) estimating beta are serious obstructions in using the results of this paper quantitatively to do optimal model-parameter decision making. However, as a matter of better understanding the training and optimization of LLMS, we agree this is an important direction of investigation.

Related work section to be added to our article.

Random features and random matrices

This paper uses random matrix theory to analyze a random features problem, which in statistical language would be the generalization error of the one-pass SGD estimator. Random matrix theory has played an increasingly large role in machine learning (see for $11$ for a modern introduction).

The input we need for our random matrix analysis is for sample covariance matrices with power-law population covariance (i.e. linear random features). Of course, the analysis of sample covariance matrices precedes their usage in machine learning (see e.g. $5$ ), but to our knowledge, a detailed study of all parts of the spectrum of sample covariance matrices with power-law population covariances has not appeared before. The narrower study of ridge regression has been extensively investigated (see for e.g. $4,9$ ), and the concurrent work $13$ provides a complete analysis of the ridge regression problem when $2\\alpha \> 1$. However, while (roughly speaking) ridge regression requires analysis of the resolvent $(A-z\\text{Id})^{-1}$ statistics for negative spectral parameter $z$ (which might be very close to $0$), the analysis in this work requires resolvent statistics for essentially all $z$.

There is a larger theory of nonlinear random features regression, mostly in the case of isotropic random features. Including nonlinearities in this model is a natural future direction; for isotropic random features with *proportional dimension asymptotics* this has been explored in works such as $21$ and for some classes of anisotropic random features in $22,31,32$ (we mention that lots of the complexity of the analysis of power-law random features arises from the analysis of the self-consistent equations -- indeed the self-consistent equations we use date to $28$ , but the analysis of these equations may still be novel). This strongly motivates non-proportional scalings (which would be inevitable in power-law random features with nonlinearities); in the isotropic case, the state of the art is $18$ .

Random features regression, ``source/capacity'' conditions, and SGD.

A large body of kernel regression and random features literature is formulated for ``source/capacity'' conditions, which are power-law type assumptions that contain the problem setup here, when $2 \\alpha \> 1$ (the low-dimensional regime). For convenience, we record the parameters

\\alpha\_{\\text{source}} \= 2\\alpha \\quad\\text{and}\\quad r \= \\frac{2\\alpha \+ 2\\beta \- 1}{4\\alpha}.

Here we have taken $r$ as the limit of those $r$'s for which the source/capacity conditions hold (see Table in attached pdf). We note that in this language $r$ is often interpreted as 'hardness' (lower is harder), and that $r \\in (0,0.5)$, $r \\in (0.5,1.0)$ and $r \\in (1.0,\\infty)$ correspond to $3$ regimes of difficulty which have appeared previously (see the citations below); they are also precisely the 3 phases Ia, II, and III.

The authors of $26$ establish generalization bounds for random feature regression with power-law structures in $2\\alpha \> 1$ case. These bounds were sharpened and extended in $13$ (see also the earlier $7$ which shows kernel ridge regression is `minimax optimal' under various `source-capacity conditions'); we give a comparison to these bounds in Appendix (..), but we note that the problem setup we have is not captured by `minimax optimality' (in particular minimax optimality is worst-case behavior over a problem class, and our problem setup is not worst-case for the traditional source/capacity conditions)

We note that this paper is fundamentally about computation, but the novel mathematical contributions could also be recast in terms of generalization bounds of one-pass SGD, some of which are new. The work of $8$ compares SGD to kernel ridge regression, showing that one-pass SGD can attain the same bounds as kernel ridge regression and hence is another minimax optimal method (again under `source-capacity' conditions). See also $15$ which considers similar statements for SGD with iterate averaging and $25$ for similar statements for multipass SGD; see also $14, 27$ which also prove the single-batch versions of these. These bounds attain the minimax-optimal rate, which are worse than the rates attained in this paper (see Appendix (..) for a comparison).

References

$1$ Luca Arnaboldi, Ludovic Stephan, Florent Krzakala, and Bruno Loureiro. From high-
dimensional & mean-field dynamics to dimensionless odes: A unifying approach to sgd
in two-layers networks. In The Thirty Sixth Annual Conference on Learning Theory,
pages 1199–1227. PMLR, 2023.

$2$ Søren Asmussen. Applied probability and queues. Springer, 2003.

$3$ Krishna B Athreya, Peter E Ney. Branching processes. Courier Corporation,
2004.

$4$ Francis Bach. High-dimensional analysis of double descent for linear regression with random
projections. SIAM Journal on Mathematics of Data Science, 6(1):26–50, 2024.

$5$ Zhidong Bai and Jack W Silverstein. Spectral analysis of large dimensional random matrices,
volume 20. Springer, 2010.

$6$ Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural
scaling laws. arXiv preprint arXiv:2402.01092, 2024.

$7$ Andrea Caponnetto and Ernesto De Vito. Optimal rates for the regularized least-squares
algorithm. Foundations of Computational Mathematics, 7:331–368, 2007.

$8$ Luigi Carratino, Alessandro Rudi, and Lorenzo Rosasco. Learning with sgd and random
features. Advances in neural information processing systems, 31, 2018.

$9$ Chen Cheng and Andrea Montanari. Dimension free ridge regression. Arxiv 2022.

$10$ Elizabeth Collins-Woodfin, Courtney Paquette, Elliot Paquette, and Inbar Seroussi. Hitting
the high-dimensional notes: An ode for sgd learning dynamics on glms and multi-index models. Arxiv 2023.

$11$ Romain Couillet and Zhenyu Liao. Random matrix methods for machine learning. Cambridge
University Press, 2022.

$12$ Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborov´a. Generalization error
rates in kernel regression: The crossover from the noiseless to noisy regime. Advances in
Neural Information Processing Systems, 2021.

$13$ Leonardo Defilippis, Bruno Loureiro, and Theodor Misiakiewicz. Dimension-free deterministic equivalents for random feature regression. arXiv preprint arXiv:2405.15699, 2024.

$14$ Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, and Lin Xiao. Optimal distributed online
prediction using mini-batches. Journal of Machine Learning Research, 13(1), 2012.

$15$ Aymeric Dieuleveut and Francis Bach. Nonparametric stochastic approximation with large
step-sizes. The Annals of Statistics, 44(4):1363 – 1399, 2016.

$16$ Cedric Gerbelot, Emanuele Troiani, Francesca Mignacco, Florent Krzakala, and Lenka Zdeborova. Rigorous dynamical mean-field theory for stochastic gradient descent methods.
SIAM Journal on Mathematics of Data Science, 6(2):400–427, 2024.

$17$ Walid Hachem, Philippe Loubaton, and Jamal Najim. Deterministic equivalents for certain
functionals of large random matrices. Ann. Appl. Probab., 17(3):875–930, 2007.

$18$ Hong Hu, Yue M. Lu, and Theodor Misiakiewicz. Asymptotics of random feature regression
beyond the linear scaling regime, 2024. arXiv

$19$ Kiwon Lee, Andrew Cheng, Elliot Paquette, and Courtney Paquette. Trajectory of mini-
batch momentum: batch size saturation and convergence in high dimensions. Advances in
Neural Information Processing Systems, 35:36944–36957, 2022.

$20$ Licong Lin, Jingfeng Wu, Sham M. Kakade, Peter L. Bartlett, and Jason D. Lee. Scaling
laws in linear regression: Compute, parameters, and data. arXiv: 2024

$21$ Song Mei and Andrea Montanari. The generalization error of random features regression:
Precise asymptotics and the double descent curve. Communications on Pure and Applied
Mathematics, 75(4):667–766, 2022.

$22$ Gabriel Mel and Jeffrey Pennington. Anisotropic random feature regression in high dimen-
sions. In International Conference on Learning Representations, 2021.

$23$ Courtney Paquette and Elliot Paquette. Dynamics of stochastic momentum methods on
large-scale, quadratic models. Advances in Neural Information Processing Systems, 34:
9229–9240, 2021.

$24$ Courtney Paquette, Kiwon Lee, Fabian Pedregosa, and Elliot Paquette. Sgd in the large:
Average-case analysis, asymptotics, and stepsize criticality. In Conference on Learning
Theory, pages 3548–3626. PMLR, 2021.

$25$ Loucas Pillaud-Vivien, Alessandro Rudi, and Francis Bach. Statistical optimality of stochas-
tic gradient descent on hard learning problems through multiple passes. Advances in Neural
Information Processing Systems, 31, 2018.