The $\varphi$ Curve: The Shape of Generalization through the Lens of Norm-based Capacity Control

We thank the reviewer for acknowledging the novelty of our contribution. Below, we provide our point-by-point responses to the comments.

Q1: The writing is not good enough, too much basic settings is places in the appendix, which is not good, if only read the main text, it will be hard to understand the paper.

A1: We thank the reviewer for checking the appendix. According to the reviewer's suggestions, we will include more preliminaries and proof sketches from the appendix into the main text for better readability, while keeping technical details in the appendix. We also plan to enhance the overview in the introduction to better guide the reader through the paper's structure.

---

Q2: The setting is too simple. From the author's perspective, this is essentially a variant of a linear problem. The author needs to discuss how this differs from the classical linear problems?

A2: The results for RFMs differ from those of linear models in two key aspects: i) the structure of the deterministic equivalents; ii) the nature of the risk-norm relationship.

Firstly, different from classical linear models only involving with data randomness, the random feature model (RFM) involves the random nonlinear feature mapping, leading to two sources of randomness—data sampling and random feature initialization. This makes deriving deterministic equivalents substantially more challenging with two self-consistency equations, leading to two effective regularization ($\nu_1$, $\nu_2$) for each source of randomness in the derivation of deterministic equivalence.

For comparison, we have provided theoretical results for linear regression in Appendix D. To illustrate the differences in structure and complexity, we take the bias term of the risk and the norm as an example. To be specific,

$B^{RFM}\sim\frac{\nu_2^2}{1-\Upsilon(\nu_1,\nu_2)}[\langle\theta_*,(\Lambda+\nu_2I)^{-2}\theta_*\rangle+\chi(\nu_2)\langle\theta_*,\Lambda(\Lambda+\nu_2I)^{-2}\theta_*\rangle],$

$B^{LS}\sim\frac{\lambda_*^2\langle\beta_*,\Sigma(\Sigma+\lambda_* I)^{-2}\beta_*\rangle}{1-n^{-1}Tr(\Sigma^2(\Sigma+\lambda_*I)^{-2})},$

Similarly, for the bias term of the estimator norm, we have:

$B^{RFM}\sim\frac{p\langle\theta_*,\Lambda(\Lambda+\nu_2I)^{-2}\theta_*\rangle}{p-Tr(\Lambda^2(\Lambda+\nu_2I)^{-2})}+\frac{p\chi(\nu_2)}{n}\cdot\frac{\nu_2^2\left[\langle\theta_*,(\Lambda+\nu_2I)^{-2}\theta_*\rangle+\chi(\nu_2)\langle\theta_*,\Lambda(\Lambda+\nu_2I)^{-2}\theta_*\rangle\right]}{1-\Upsilon(\nu_1,\nu_2)},$

$B^{LS}\sim\langle \beta_*,\Sigma^2(\Sigma+\lambda_* I)^{-2}\beta_*\rangle+\frac{\lambda_*^2\cdot\langle\beta_*,\Sigma(\Sigma+\lambda_* I)^{-2}\beta_*\rangle\cdot Tr(\Sigma(\Sigma+\lambda_* I)^{-2})}{ n-Tr(\Sigma^2(\Sigma+\lambda_* I)^{-2})}.$

In the $p\to\infty$ limit, $\nu_1$ and $\nu_2$ in the RFMs converge to a single value $\nu_\kappa$. Consequently, $\Upsilon \to n^{-1} Tr\left( \Lambda^2(\Lambda + \nu_\kappa I)^{-2} \right)$, and $\chi \to 0$. Under this limit, the expressions for both the risk and the norm in RFMs reduce to those of linear regression, with the feature covariance matrix $\Sigma$ effectively replaced by $\Lambda$. This highlights that despite the structural differences, RFMs reduce to linear regression in the kernel limit, revealing a fundamental connection between the two models.

Secondly, both RFMs and linear regression exhibit U-shaped trends in their risk–norm curves, but their behaviors in different regimes are exactly reversed: RFMs show a U-shape in the under-parameterized regime and monotonic increase in the over-parameterized regime, whereas linear regression shows the opposite.

To illustrate this contrast, we refer to Proposition 4.1 for RFMs, which shows a linear relationship

$R=\lambda_n N + C$

for min-norm estimator within the over-parameterized regime (fully characterizing the curve in the under-parameterized regime remains analytically intractable). And for linear regression, Proposition D.8 (Appendix D.3.2) provides a closed-form expression for the risk–norm curve:

$R=\begin{cases}N-{||\beta_*||}^2,&\text{if} d<n\\\\ \sqrt{[N-({||\beta_*||}^2-\sigma^2)]^2+4{||\beta_*||}^2\sigma^2}-\sigma^2,&\text{otherwise}\end{cases},$

which is linear in the under-parameterized regime and takes the form of a hyperbola in the over-parameterized regime.

Despite these differences, the core findings remain consistent across both models: phase transitions are present, double descent does not occur, and the learning curves exhibit a U-shaped trend. This suggests that our conclusions are not limited to linear settings, but extend to more complex models as well.

---

Q3: Need more references, such as:《On the Double Descent of Random Features Models Trained with SGD》,nips2022.

A3: We thank for the reviewer for pointing out this paper that provides a nice upper bound for RFF with SGD. Our current work does not tackle the optimization dynamics. Providing a precise charecterization of the test risk under (S)GD via deterministic equivalence, is highly nontrivial and often requires significantly more complex techniques. For instance, recent work [S1,S2] precisely charecterize the test risk under SGD in linear regression settings. We will cite these references and leave the SGD analysis as future work.

---

Q4: The network and data used in the paper is simple, even close to a linear model. However, commonly used neural networks and data are highly nonlinear, does your conclusion help us understand the phenomenon of double decline in general situation?

A4: To validate our theory, apart from the existing experimental validation for two-layer neural networks in Appendix H.3, we also conducted comprehensive experiments on ResNet18, a three-layer MLP, and a CNN, respectively.

Firstly, we reproduced OpenAI's results [S3] for deep double descent using ResNet18 on CIFAR-10 with 15% label noise, and computed the path norm following the definition in [S4]. The table below summarizes the results for ResNet:

Based on this, we can plot test error vs. path norm (we cannot put the figure here due to the NeurIPS rebuttal policy). We can find that, in the sufficiently over-parameterized regime, the test risk and norm decrease together, ultimately aligning with the $\varphi$ curve. This suggests that double descent is a transient phenomenon, whereas the phase transition and the $\varphi$-shaped trend reflect more fundamental behavior.

Besides, we also conducted experiments on a 3-layer MLP and a 3-layer CNN. These results demonstrate similar $\varphi$-shaped trends in test error versus path norm and further confirm the generality of our findings. For detailed results, please refer to our answer to Reviewer miNz’s Q1.

Conclusion: These results concide with our theory by demonstrating the existence of phase transitions, while double descent does not always occur—particularly under sufficient over-parameterization. Notably, the $\varphi$ curve exhibits a U-shaped trend, aligning with our theoretical predictions. These findings will be included in the updated version. Besides, we will release the code, model parameters, and visualizations to foster further research in the community, which would be helpful to understand the interplay between data, model complexity, and computational resources in modern ML. We will include these results in our updated version.

---

Q5: Is RFM being used in some field nowadays? Some examples?

A5: Random Feature Models (RFMs) were originally proposed to accelerate large-scale kernel methods by approximating kernels via low-rank random projections [S5]. This idea was later adopted in linear attention mechanisms for Transformers, where the softmax function is closely related to an exponential kernel; see [S6, S7] for details. A survey [S8] provides an overview of this connection.

More recently, the concept of low-rank random projection has also been applied to parameter-efficient fine-tuning techniques such as LoRA [S9] and its variants [S10]. For instance, [S10] leverages random features and learns their combination coefficients for fine-tuning. Notably, these random features can be generated on-the-fly using random number generators, significantly reducing memory overhead.

We contend that RFMs retain both theoretical significance and practical relevance, even in the current era dominated by large language models.

---

Reference

[S1] Courtney Paquette, Elliot Paquette, et al. "Homogenization of SGD in high-dimensions: Exact dynamics and generalization properties." Mathematical Programming, 2024.

[S2] Elliot Paquette, Courtney Paquette, et al. "4+3 Phases of Compute-Optimal Neural Scaling Laws." NeruIPS'24.

[S3] Preetum Nakkiran, Gal Kaplun, et al. "Deep Double Descent: Where Bigger Models and More Data Hurt." ICLR'20

[S4] Yiding Jiang, Behnam Neyshabur, et al. "Fantastic generalization measures and where to find them." ICLR'20.

[S5] Ali Rahimi, Benjamin Recht. "Random features for large-scale kernel machines." NeurIPS'07

[S6] Krzysztof Choromanski, Valerii Likhosherstov, et al. "Rethinking attention with performers." ICLR'21

[S7] Hao Peng, Nikolaos Pappas, et al. "Random feature attention." ICLR'21

[S8] Fanghui Liu, Xiaolin Huang, et al. "Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond." IEEE Transactions on Pattern Analysis and Machine Intelligence 44.10 (2021): 7128-7148.

[S9] Edward J. Hu, Yelong Shen. "LoRA: Low-Rank Adaptation of Large Language Models." ICLR'22.

[S10] Koohpayegani, Soroush Abbasi, et al. "NOLA: Compressing LoRA using Linear Combination of Random Basis." ICLR'24.

We thank the reviewer for the positive evaluation and are pleased that the clarity, theoretical rigor, and empirical validation of our work were well received. Below, we provide our point-by-point responses to the comments.

Q1: Further explain as to why the bias follows a U-shaped curve, as opposed to the monotonic decrease shown in the classical bias-variance trade-off explanation? Specifically, what is the mechanism that causes a model's attempt to fit training data more closely to result in higher bias?

A1: We thank the reviewer for the insightful question. The U-shaped behavior of the bias in the under-parameterized regime arises due to the interaction between model capacity and the conditioning of the feature representation. To be specific,

• In the low-capacity regime ($p \ll n$), increasing the number of random features improves the model’s ability to approximate the target function $f^\star$. Each additional feature offers a new direction in function space, leading to better alignment with $f^\star$ and reducing bias.
• Around $p = O(\sqrt{n} \log n)$, this number of random features is theoretically proved to be sufficient for appromating $f^\star$ well [S1], yielding the lowest bias.
• As $p \to n$ (high-capacity regime), the kernel matrix becomes ill-conditioned. The model starts amplifying high-frequency components and fitting noise in the feature space, which degrades the alignment with $f^\star$ and increases bias.

This intuition can be also formalized in our deterministic equivalence analysis. Under Assumption 2 (power-law spectrum) and the scaling $p = n^q$, $\lambda = n^{-(\ell - 1)}$ with $q, \ell > 1$, we derive the following asymptotic expressions (Appendix E.3.3):

• When $p \ll n$, $\text{Bias} \approx C_3 \left( \frac{p}{C_1} \right)^{-2\alpha r}$, which decreases with $p$, as expected.

• When $p \to n$, $\text{Bias} \approx \frac{n}{n - p} \cdot C_3 \left( \frac{p}{C_1} \right)^{-\alpha(2r \wedge 1)}$, with a positive derivative w.r.t. $p$ near $p = n$, indicating a monotonic increase in bias as the model becomes nearly interpolating.

Thus, while classical bias-variance trade-offs assume monotonic bias decay, our setting reveals a bias increase near interpolation threshold, due to kernel ill-conditioning and projection distortion—resulting in a U-shaped curve in the under-parameterized regime.

---

Q2: Under what conditions does the test error curve in the overparameterized regime transition from being consistently below its underparameterized counterpart to above it ("$\varphi$"-shaped curve) and vice versa?

A2: This is an interesting question! As shown in Appendix G.1, the transition to a “$\varphi$”-shaped curve depends on the data distribution, regularization, and to a lesser extent, the activation function.

For Gaussian inputs (synthetic data), the over-parameterized test error consistently stays below the under-parameterized regime, aligning with our theory. On real data like FashionMNIST, we observe a $\varphi$-shaped curve in the ridgeless case, where test error initially rises before decreasing. With sufficient ridge regularization, the over-parameterized curve remains below throughout.

These results suggest that deviations from our theoretical assumptions (e.g., independent feature inputs) play a key role in shaping the test error curve. Extension to the setting of correlated features requires additional efforts and new technical tools, leaving as a future work.

---

Reference

[S1] Alessandro Rudi, Lorenzo Rosasco. "Generalization Properties of Learning with Random Features." NeurIPS'17.

We thank the reviewer for the constructive feedback and appreciate the recognition of the theoretical depth, clarity of insight, and empirical validation. Below, we provide our point-by-point responses to the comments.

Q1: The analysis is restricted to linear models and RFMs, focusing only on the second layer and lacking applicability to deep neural networks.

A1: We acknowledge the theoretical focus on linear models and RFMs. Extension to nonlinear neural networks is non-trivial and remains an open question except for some extremely special cases [S1, S2].

To validate our theory, we conducted experiments on ResNet18 by reproducing OpenAI's results [S3] on CIFAR-10 with 15% label noise and computing the path norm as defined in [S4] as below.

Based on this, we can plot test error vs. path norm (we cannot put the figure here due to the NeurIPS rebuttal policy). We can find that, in the sufficiently over-parameterized regime, the test risk and norm decrease together, ultimately aligning with the $\varphi$ curve. This suggests that double descent is a transient phenomenon, whereas the phase transition and the $\varphi$-shaped trend reflect more fundamental behavior.

Besides, we also conducted experiments on a 3-layer MLP and a 3-layer CNN. These results demonstrate similar $\varphi$-shaped trends in test error versus path norm and further confirm the generality of our findings. For detailed results, please refer to our answer to Reviewer miNz’s Q1.

Conclusion: These results concide with our theory by demonstrating the existence of phase transitions, while double descent does not always occur—particularly under sufficient over-parameterization. Notably, the $\varphi$ curve exhibits a U-shaped trend, aligning with our theoretical predictions. These findings will be included in the updated version. Besides, we will release the code, model parameters, and visualizations to foster further research in the community, which would be helpful to understand the interplay between data, model complexity, and computational resources in modern ML. We will include these results in our updated version.

---

Q2: The model norm can be controlled via $\lambda$. However, the behavior is different in the under- and over-parameterized regimes, meaning one must know which state the model is in to control the norm, which is a practical challenge.

A2: Identifying whether a model is under- or over-parameterized is straightforward in practice, as it depends on knowing the number of model parameters and the training sample size. Thus, determining the regime is practically feasible.

To guide model selection with the proper regularization parameter $\lambda$ in practice, for under/over-parameterized regimes, one can plot the empirical risk vs. the estimator norm as the regularization parameter (in weight decay) varies, i.e., L-curve [S5], respectively. The “corner” of L-curve reflects an optimal trade-off between data fitting and model complexity. Selecting the regularization parameter at this corner provides an effective strategy for balancing generalization and capacity. We will include more discussion on practical guidelines in our updated version.

---

Q3: Theoretical results rely on specific assumptions (e.g., power-law eigenvalue decay), which may not hold in real-world datasets.

A3: We acknowledge that our theoretical results rely on assumptions such as power-law eigenvalue decay, which may not hold universally. However, these assumptions are standard in the theoretical literature [S6–S8], as they ensure the hypothesis class is well-behaved (e.g., trace-class kernels).

While relaxing these assumptions is technically challenging, our empirical results on shallow and deep neural networks as mentioned before remain consistent with our theory: double descent is a transient phenomenon, whereas the phase transition and the $\varphi$-shaped trend reflect more fundamental behavior.

---

Q4: The work mainly extends prior results from [14] to estimator norm, making it an incremental contribution rather than a fundamentally new framework.

A4: While we build on the analytical tools developed in [14] for deriving deterministic equivalents, the techniques in [14] are insufficient to obtain the deterministic equivalent of the estimator norm. For instance, we generalize their results by deriving expressions of the form $Tr(A X^\top X (X^\top X + \lambda)^{-1})$ for arbitrary PSD matrices $A$, whereas [14] only considered the special case $A = I$.

Moreover, establishing a quantitative relationship between test risk and estimator norm requires nontrivial analysis, including eliminating explicit model size dependence—a key challenge not addressed in [14].

We do not claim to introduce a completely new proof framework, but our contribution aims to address the fudemental question: how test risk evolves with a suitable model capacity? We provide both theoretical and empirical evidence, and offer new insights beyond prior results.

---

Q5: How can the analyses be generalized to other norms? Are other measures like the path norm more suitable for complex models?

A5: Generalizing our analysis to other norms is technically challenging, as it relies heavily on the resolvent structure specific to linear and random feature regression. While recent work has begun to explore deterministic equivalents for general loss functions [S9], the analysis remains considerably more complex and less mature.

However, according to the reviewer's suggestions, we have empirically evaluated the path norm in deep networks, as given in Q1. This trend closely mirrors our theoretical findings for the $\ell_2$ norm, suggesting that the norm–risk relationship may generalize beyond $\ell_2$ and hold for more complex architectures.

---

Q6: How do these insights into norm-based capacity translate to tasks beyond regression, such as classification?

A6: Extending the theory from regression to classification is analytically non-trivial due to differences in loss functions and optimization dynamics. A rigorous treatment would require new techniques beyond our current framework.

Nonetheless, we present empirical results on classification tasks in Table 5, located in Appendix G (page 65), showing that test error (i.e., classification error rate) exhibits the same qualitative behavior as in regression: a U-shaped curve in the under-parameterized regime and a monotonic increase in the over-parameterized regime. These observations suggest that norm-based capacity control remains relevant and informative for understanding generalization in classification.

---

Q7: The study only considers models with closed-form solutions and overlooks the implicit regularization introduced by optimization methods such as SGD.

A7: Thanks for the reviewer pointing out this. Our work requires closed-form solutions and does not address the effects of optimization dynamics. Some recent studies [S10–S13] precisely charecterize the test risk under SGD for linear models. Extending our deterministic equivalent framework to incorporate such optimization effects is an important and challenging direction, which we leave for future work. We will include these references and discuss the possiblity for study.

---

We hope our responses provide a better understanding on this work in terms of problem settings, technical contributions, and practical guidelines. We're happy to discuss more if the reviewer has any concerns.

---

Reference

[S1] Behrad Moniri, Donghwan Lee, et al. "A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks." ArXiv:2310.07891.

[S2] Yatin Dandi, Luca Pesce, et al. "A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilities." ArXiv:2410.18938

[S3] Preetum Nakkiran, Gal Kaplun, et al. "Deep Double Descent: Where Bigger Models and More Data Hurt." ICLR'20

[S4] Yiding Jiang, Behnam Neyshabur, et al. "Fantastic generalization measures and where to find them." ICLR'20.

[S5] Hansen, Per Christian. "Analysis of discrete ill-posed problems by means of the L-curve." SIAM review 34.4 (1992): 561-580.

[S6] Leonardo Defilippis, Bruno Loureiro, and Theodor Misiakiewicz. "Dimension-free deterministic equivalents and scaling laws for random feature regression." NeurIPS'25.

[S7] James B. Simon, Dhruva Karkada, et al. "More is Better in Modern Machine Learning: when Infinite Overparameterization is Optimal and Overfitting is Obligatory." ICLR'24.

[S8] Dominic Richards, Jaouad Mourtada, and Lorenzo Rosasco. "Asymptotics of Ridge (less) Regression under General Source Condition." AISTATS'21

[S9] Kiana Asgari, Andrea Montanari, and Basil Saeed. "Local minima of the empirical risk in high dimension: General theorems and convex examples." ArXiv:2502.01953.

[S10] Courtney Paquette, Kiwon Lee, et al. "SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality." COLT'21.

[S11] Courtney Paquette, Elliot Paquette. "Dynamics of Stochastic Momentum Methods on Large-scale, Quadratic Models." NeurIPS'21.

[S12] Courtney Paquette, Elliot Paquette, et al. "Homogenization of SGD in high-dimensions: Exact dynamics and generalization properties." Mathematical Programming, 2024.

[S13] Elliot Paquette, Courtney Paquette, et al. "4+3 Phases of Compute-Optimal Neural Scaling Laws." NeurIPS'24.

We thank the reviewer for the thoughtful feedback and appreciate the recognition of the theoretical-experimental consistency and the insights into generalization. Below, the concern regarding the universality of the conclusions and their applicability to model selection is addressed.

Q1: The conclusions may not be universally applicable, as the analysis is limited to RFM and linear models, lacking empirical evidence for deep models (more than 2 layers), so supplementary experimental verification in deep cases is necessary.

A1: To validate our theory, according to the reviewer's suggestion, we conducted comprehensive experiments on ResNet18, a three-layer MLP, and a CNN, respectively.

Firstly, we reproduced OpenAI's results [S1] for deep double descent using ResNet18 on CIFAR-10 with 15% label noise, and simultaneously computed the path norm following the definition in [S2]. The table below summarizes the results for various widths in each layer of ResNet:

Based on this, we can plot test error vs. path norm (we cannot put the figure here due to the NeurIPS rebuttal policy). We can find that, in the sufficiently over-parameterized regime, the test risk and norm decrease together, ultimately aligning with the $\varphi$ curve. This suggests that double descent is a transient phenomenon, whereas the phase transition and the $\varphi$-shaped trend reflect more fundamental behavior if a suitable model capacity is used.

Secondly, we also conducted experiments on a 3-layer MLP and a 3-layer CNN trained on the MNIST dataset with 15% label noise, varying the model width (or number of channels) to study the relationship between test error and path norm. For the MLP, when plotting test risk vs. path norm, we can observe $\varphi$ curve, which aligns with our findings on RFMs.

Similarly, in the CNN setting, we have the same observation as below.

Conclusion: These results consistently demonstrate the existence of phase transitions, while double descent does not always occur—particularly under sufficient over-parameterization. Notably, the $\varphi$ curve exhibits a U-shaped trend, aligning with our theoretical predictions. These findings will be included in the updated version. Besides, we will release the code, model parameters, and visualizations to foster further research in the community, which would be helpful to understand the interplay between data, model complexity, and computational resources in modern ML. We will include these results in our updated version.

---

Q2: It is necessary to discuss how to apply the control of norm capacity to actual model selection or training strategies (such as early stopping, weight decay).

A2: Norm capacity can be controlled in practice via standard regularization techniques such as weight decay and early stopping, which implicitly or explicitly constrain model norms. As shown in Figure 2 of our paper, increasing the $\ell_2$ regularization strength leads to a monotonic decrease in the estimator norm.

To guide model selection, one can plot the empirical risk vs. weight decay (i.e., estimator's norm) as the regularization parameter (in weight decay) varies. This typically yields an L-curve [S3], where the “corner” reflects an optimal trade-off between data fitting and model complexity. Selecting the regularization parameter at this corner provides an effective strategy for balancing generalization and capacity. We will include more discussion in our updated version.

---

Reference

[S1] Preetum Nakkiran, Gal Kaplun, et al. "Deep Double Descent: Where Bigger Models and More Data Hurt." ICLR'20

[S2] Yiding Jiang, Behnam Neyshabur, et al. "Fantastic generalization measures and where to find them." ICLR'20.

[S3] Hansen, Per Christian. "Analysis of discrete ill-posed problems by means of the L-curve." SIAM review 34.4 (1992): 561-580.

Dear Reviewer miNz,

Thank you for your support and valuable feedback on our work.

According to your suggestions, we have conducted comprehensive experiments ranging from shallow neural networks to ResNet18 (to reproduce OpenAI's results), and we will release the code publicly to foster further research. Additionally, we have included a discussion on the use of the L-curve for practical model selection of the regularization parameter.

Please feel free to let us know if you have any further questions or suggestions.

Best regards,

The Authors