Thank you for your review. We respond to your questions concerns as follows:

In Section D.1, D.2, the authors should clearly point out where are the experimental results.

Response: Thank you for your suggestion. We will update the text to point this out.

Section D.1 on synthetic tasks pertains to the numerical error points in Figures 3 and S2.A-C.

Section D.2 on the numerical experiments for binarized MNIST and CIFAR-10 with ReLU features pertains to figures 1, 2, 4, S1, S2.D-F, S3, S4, S5, S6, and S7.

The authors missed two important references ([1], [2])...compare the results.

Thank you for bringing these works to our attention. We were aware of ref [1] and many other papers deriving the error... These papers focus on the over-fitting and double descent / multiple descent pheonomena in their analysis. However, these effects vanish entirely when the ridge parameter is optimized at each sample size. By studyiing the behavior of the test risk at optimal ridge, we obtain more practical results that will apply even in settings where the ridge parameter is set to its optimal value using cross-validation.

I have some concerns on the essential contribution ... these two references. Can the authors rely on the results ...$\arg\min_x f_2(x).$

Thank you for bringing these papers to our attention. In short, no, we cannot rely on either of them to derive the test risk of ensembled RFRR.

As for ref. [1], the data model considered here consists of samples drawn uniformly from the unit sphere, whereas the results we use apply to arbitrary (but nicely behaved) data distributions $\mathbf{x} \sim \mu_{\mathbf{x}}$.

As for ref. [2], the loss function considered there encodes joint training of the parameters $\mathbf{a}\_1 \equiv [a\_1, \dots, a\_{N\_1}]^\top$ and $\mathbf{a}\_2 \equiv [a\_{N\_1 + 1}, \dots, a\_{N\_2}]^\top$. This can be written as

$\frac{1}{n} \sum_{i=1}^n \left( y - f_1(\mathbf{a}\_1) \right)^2 + \frac{d}{n} \lambda ||\mathbf{a}\_1||^2 + \frac{1}{n} \sum_{i=1}^n \left( y - f_1( \mathbf{a}\_1) \right)^2 + \frac{d}{n} \lambda ||\mathbf{a}\_2||^2$

The square term here cannot be factored into a sum of contributions depending separately on $\mathbf{a}\_1$ and $\mathbf{a}\_2$, as you have assumed in your comment.

In the notation of [2], the model we consider corresponds to a decoupled loss function which optimizes $\mathbb{a}\_1$ and $\mathbf{a}\_2$ separately:

$\frac{1}{n} \sum_{i=1}^n\left( y - f_1(\mathbf{a}\_1) \right)^2 + \frac{d}{n} \lambda ||\mathbf{a}\_1||^2+ \frac{1}{n} \sum\_{i=1}^n\left( y - f_1(\mathbf{a}\_1) \right)^2 + \frac{d}{n} \lambda ||\mathbf{a}\_2||^2 \,,$

with additional terms if $K>2$.

Additional response to reviewer comments:

Many of your questions and concerns revolve around the derivation of the risk estimate for random feature ensembles, and its presence in prior literature. There are many papers which derive the bias and variance terms for random-features regression using a variety of methods, which we have referenced (Atanasov e tal.,2024; Canatar et al. ,2021; Simon et al., 2023; Adlam & Pennington, 2020; Rocks & Mehta, 2021; Hastie etal., 2022; Zavatone-Vethetal., 2022) and we are happy to add discussions of [1] and [2].

We do not claim to have contributed the error formula for ensembles -- our contribution is a novel and informative analysis of test risk formula.

The error formulas for RFRR ensembles derived in previous works are not easily interpretable, and studying the implications of these error formulas is an important task unto itself. We believe that our paper has addressed an important gap in our understanding of ensembled random feature models. We have used the (known) bias-variance decomposition of the test risk of RFRR to study the tradeoff between model size and ensemble size. Specifically, we have shown that:

• Ensembling is never optimal under a fixed parameter budget at optimal ridge.
• Ensembling can achieve near-optimal performance in both the overparameterized and underparameterized regimes, with precise spectral conditions for near-optimal scaling in the underparameterized regime. If accepted, we will clarify these contributions in the abstract and introduction. We may also change the title of the paper to "No Free Lunch from Random Feature Ensembles: Scaling Laws and Near-Optimality Conditions" to highlight all of our main contributions.

[1] Mei S, Montanari A. The generalization error of random features regression: Precise asymptotics and the double descent curve J. Communications on Pure and Applied Mathematics, 2022, 75(4): 667-766.

[2] Meng, X., Yao, J., & Cao, Y. (2024). Multiple descent in the multiple random feature model. Journal of Machine Learning Research, 25(44), 1-49.

Thank you for your detailed review. Below, we address your questions and concerns:

"The one portion ... would be helpful."

Response: Thank you for this suggestion. We will add more justification. While ensembles are never optimal, they allow parallelization and can be near-optimal, hence useful in practice. Section 5.1 addresses near-optimality in the overparameterized regime. The scaling laws derived in section 6 address near-optimality in the under-parameterized regime -- necessary for a complete characterization.

1. Section D.1 states ... satisfiable?

Response: We will clarify that both the $\eta_t$ and $\bar{w}_t$ are normalized to satisfy these constraints.

2. In Section D.2, why ... clarify this.

Response: Yes, we choose $\mathbf{V}^k_{ij}\sim \mathcal{N}(0, 2/D)$ to converge to the NNGP kernel in Lee et. al. (2018). We will clarify this in the text.

1. "consider the featurization transformation $g$". I ... supposed to be.
2. $\mu_\nu$ was ... for it.

Response: We have revised the text:

"Define the random features $\mathbf{\psi}(\mathbf{x}) \in \mathbb{R}^N$ by $\left[ \mathbf{\psi}(\mathbf{x})\right]\_n = g(\mathbf{v}\_n, \mathbf{x})$, where the $\mathbf{v}\_n$ are random parameter vectors sampled independently from measure $\mu\_{\mathbf{v}}$ on $\mathbb{R}^C$. Here, the function $g: \mathbb{R}^C \times R^{D} \mapsto \mathbb{R}$ is a "featurization transformation," often taking the form $g(\mathbf{v}\_n, \mathbf{x}) = \varphi (\mathbf{v\_n}^\top \mathbf{x})$ for some nonlinear activation function $\varphi(\cdot)$... "

3. **"As $N \to \infty$, this... and $\mu_\nu$?

Response: A careful choice of $g$ and $\mu\_{\mathbf{v}}$ is required to ensure convergence to a particular kernel (see point 2). However, the stochastic kernel in general converges to some deterministic kernel given by $H(x, x') = \mathbb{E}\_{\mathbf{v} \sim \mu\_{\mathbf{v}}} g(\mathbf{v}, \mathbf{x})g(\mathbf{v}, \mathbf{x'})$. We will clarify.

6. Equation (7) seems to ... with one another.

Response: This is a standard bias-variance decomposition of the error (see, [2]). An expectation over $\mathbf{Z}$ is not necessary on the left side of eq. 12 because the test risk concentrates over $\mathbf{Z}$.

7. In Section 5.1, "$\lambda = 1/N = 0$".... $1/N$ and 0.

Response: To clarify, we now say "we expand the risk estimate $E_g^K$ (eq. 9) as power series in $1/N \gtrsim 0$ and $\lambda \approx 0$".

Comments regarding the use of eq. 9, and the slack in this estimate ("where $\mathcal{E}\_g^1$ is ... to be."; "I think... should be stated."; "In Appendix C ... result."; "Also, the proof rigorous conclusions."):

Response: Thank you, we agree that more thorough explanation would be helpful. We call the "true" risk $\mathcal{E}\_g$ the error for a particular realization of the random parameters $\mathbf{v}\_n$. The key result of Defilippis et al. (2024) is to show that the distribution over $\mathcal{E}\_g$ values concentrates around a deterministic‐equivalent expression $E\_g$ which depends on the feature count $N$. The difference between the true random‐features generalization error $\mathcal{E}\_g$ and its deterministic‐equivalent $E\_g$ is controlled by a multiplicative concentration bound:

Thank you for your review!

Rebuttal to Reviewer 2SLL (Complete)

Thank you for your review and questions. It appears that you are convinced of the correctness of our results, but have concerns about the significance of the contribution. We will respond to your concerns and questions individually below:

1. The contribution of this paper is limited; the results are somewhat evident and can be derived from prior works.

Response: The error formulas for RFRR ensembles derived in previous works are not easily interpretable, and studying the implications of these deterministic equivalent errors is an important task unto itself. We believe that our paper has addressed an important gap in our understanding of ensembled random feature models. We have used the (known) bias-variance decomposition of the test risk of RFRR to study the tradeoff between model size and ensemble size. Specifically, we have shown that:

• Ensembling is never optimal under a fixed parameter budget at optimal ridge.
• Ensembling can achieve near-optimal performance in both the overparameterized and underparameterized regimes, with precise spectral conditions for near-optimal scaling in the underparameterized regime. If accepted, we will clarify these contributions in the abstract and introduction. We may also change the title of the paper to "No Free Lunch from Random Feature Ensembles: Scaling Laws and Near-Optimality Conditions" to highlight our contributions beyond Theorem 4.1.

2. The results mainly pertain to the random feature ridge regression model; whether they apply to deep learning models remains unaddressed.

Response: While we are ultimately interested in understanding the utility of ensemble learning using state-of-the-art machine learning models, random-features regression provides an important "ground-floor" to investigate the utility of ensemble learning in a tractable setting. Furthermore, because of the approximate correspondence between RFRR and deep networks trained in the lazy learning regime [3], our results already bear some relevance to deep ensembles. Understanding the properties deep ensembles trained in the rich regime is a critical research direction for our future work, but will be better presented with reference to these baseline results in linear models, which have already saturated the page limit for this venue.

Our results also add to a long history of research on random-feature models in analogy to deep networks. Another example of this fruitful correspondence is work on fine-grained bias-variance decompositions [2]

We have also performed numerical experiments for deep ensembles. In these experiments, we train ensembles of deep convolutional neural networks on a computer vision task (CIFAR10 image classification) using $\mu P$ parameterization, which keeps training dynamics consistent across widths [1]. In addition to the weight decay, there is a "richness" parameter $\gamma$ which controls the amount of feature-learning in the network. Our simulations show that a large network outperforms any ensemble of smaller networks with the same total size when both the weight decay and "richness" parameter are tuned to their optimal values. If it will have a positive impact on your evaluation, we are willing to add these numerical results to the supplemental material. This result is available at this anonymized github repo: https://anonymous.4open.science/r/NoFreeLunchRandomFeatureEnsembles/README.md

3. The layout of the figures is irregular, and the fonts in these figures could be improved.

Response: Thank you for your feedback, we are open to any and all suggestions on how to improve the clarity and appearance of our figures for the final version of this paper if accepted!

"Most of the experiments are based on the random feature (RF) model. Would the same results hold for larger ensemble models?"

Response: See our response to 2. above.

[1] Nikhil Vyas, Alexander Atanasov, Blake Bordelon, Depen Morwani, Sabarish Sainathan, and Cengiz Pehlevan. Feature-learning networks are consistent across widths at realistic scales, 2023. URL https://arxiv.org/abs/2305.18411.

[2] Ben Adlam and Jeffrey Pennington. Understanding double descent requires a fine-grained bias-variance decomposition, 2020. URL https://arxiv.org/abs/2011.03321.

[3] Chizat, L., Oyallon, E., & Bach, F. (2020). On lazy training in differentiable programming (arXiv:1812.07956v5). Retrieved from https://arxiv.org/abs/1812.07956