Theoretical Limitations of Ensembles in the Age of Overparameterization

We thank the reviewer for their detailed review, as well as their suggestions for improving our paper. Below, we address the concerns and questions raised, and outline the changes we will make in response.

Claims And Evidence:

• Clarification of "hockey stick" pattern in Fig. 2: We agree and will adjust our wording in the revised manuscript to reflect that the "hockey stick" pattern is less pronounced for neural networks, likely due to effects that cannot be captured from an RF analysis.
• Interpretation of performance gap and variability in Fig. 3: Thank you for highlighting this point. Indeed, the variability of the curve on the right arises due to variance across single model instantiations. We will update Fig. 3 accordingly, adding shaded regions to represent this variance. Regarding why the gap at the left grows, this is likely due to a) the discrepancy between the finite and infinite cases, and b) numerical instabilities when working with ReLU models (for a more precise explanation, see Appx. A.2). As an example of an experiment where this does not happen, compare Fig. 12 (right), which we deliberately did not include in the paper since this uses the Gaussian Error function.
• Plotting suggestion for Fig. 4: Thank you for this suggestion – we will update Fig. 4 to use shared axes.

Relation to Prior Work and Additional References:

We will add a brief summary of Abe et al. (2022) and cite Neal (1996) and Williams (1997).

Other Comments or Suggestions:

We appreciate your edits and suggestions for clarity, which we will incorporate into our revised version. We address a few specific points below.

• Clarification of Assumption 2.1: We indeed intended the product of the vector and scalar (which produces a vector again). Following standard extensions of subexponentiality to vector valued random variables, our theory relies on any linear combination of the entries of $w\_i w\_{\\bot i}$ being subexponential. We will clarify this point in the revision.
• Phrasing on line 250, right column: We agree and will revise this phrasing accordingly.
• Clarification of Assumption 3.4: We require that the entries of the expected value of the inverse matrix are finite, i.e., that the corresponding expected value exists. This obviously also requires that, for almost all instantiations of the matrix $\\Phi\_{\\mathcal{W}}$, the matrix \\Phi\_{\\mathcal{W}} \\Phi\_{\\mathcal{W}}^\\top is invertible.

Questions for Authors:

• Relation to Ruben et al. (2024): We agree that Ruben et al. (2024) should be discussed in more detail and will include this in the revision; thank you. We would also note that this work is concurrent to ours and should be read as such. Both works conclude that ensembles of overparameterized random feature models do not outperform a single larger model with an equivalent total feature budget. However, we explicitly focus on the overparameterized and zero/small ridge regime, while Ruben et al. primarily analyze generalization under optimal ridge parameters in both the over- and underparameterized regimes, with only brief consideration of our regime. Additionally, our analysis also explicitly considers uncertainty quantification, and we do not rely on Gaussianity assumptions, contrasting their Gaussian universality assumption.
• Ensemble variance and Bayesian uncertainty: As briefly mentioned in Sec. 3.3, only "with Gaussian features, ensemble variance admits a Bayesian interpretation," by which we mean that the ensemble variance matches the posterior variance of a GP with the same limiting kernel. We will ensure this is clearer in the revised version.
• Different ridge parameters in Sec. 3.4: We appreciate this suggestion and agree it would be interesting to explore. Note, however, that when investigating the finite-width regime, we assumed \\lambda \= 0, while in Sec. 3.4, we specifically discuss the infinite-width limit. Additionally, our current focus was explicitly on the small ridge regime, where any implicit regularization parameters (such as $\\tilde{\\lambda}$ from Jacot et al. (2020)) are expected to be very small; therefore, variations in this parameter are likely to have only a minor impact in our setting.

We hope these responses and adjustments clarify our contributions and address your feedback. If you have further questions or suggestions, we would be happy to address them.

We thank the reviewer for their valuable comments and feedback on our paper. Below, we address the concerns and questions raised and outline the changes we plan to make to the manuscript based on your suggestions.

Regarding the Experiments:

We agree that explicitly including the experimental setup in the main text will help clarity. Thus, we will move these details from the appendix into the main body of the paper.

Additional literature:

Thanks for pointing out the additional papers. We will discuss $1$ , $2$ and $3$ in our related works section. Furthermore, we will add relevant references in the field of uncertainty quantification to our related works section and we will address the distinction between epistemic and aleatoric uncertainty.

Specific Comments:

• C2 (Line 297, L2-norm notation): We will correct the notation as suggested.
• C3 (Connection between Figure 5 and Theorem 3.5): We will add a short explanation of and a reference to Figure 5 in the main text.
• C4 (Quantifying the difference in Figure 1): We will include a more detailed plot showing the evolution with increasing ensemble size $M$ in the appendix and reference this in the figure description.
• C5 (Feasibility and intuitiveness of assumptions): We will add a brief intuitive explanation of Assumption 2.1, noting that these assumptions closely hold in most relevant scenarios including e.g., for distributions with bounded support.
• C6 (Stability vs. robustness in Figure 3): Indeed, the observed stability in Figure 3 arises from averaging effects. As suggested by reviewer e9Q9 we will add shaded regions for variance across instantiations of single models/ensembles
• C7 (Counterexamples in Appendix A.2): Due to length constraints, we cannot incorporate Figure 8 directly into the main text but will ensure it is more prominently referenced in the main text.
• C8 (Experiments beyond standard normal distributions): To clarify, even though the weights of feature generating function (e.g. $\\omega$) are normally distributed, the random features themselves (i.e. \\mathrm{ReLU}(\\omega^\\top x) are not, and thus our experiments make use of the generality of our theorem. Regardless, we will supplement Figure 1 and Figure 13 with $\\omega$ drawn from heavy tailed distributions.
• C9 (Invertibility assumption with ReLU activations): We investigated other activations which fulfill the invertibility, including softplus (see e.g. Fig. 2 (left) and Fig. 9; note that using softplus activations, we can approximate ReLU activations arbitrarily precisely, see footnote 2 on page 4) and the Gaussian error function activation function (see e.g., Fig. 10 on the right hand side and Fig. 11 & 12), which satisfy the invertibility assumption.
• C11 (Double descent phenomena): We are not entirely sure what aspect of the double descent phenomena you are referring to. If you clarify further, we would be happy to discuss it.

Responses to Questions:

• Q1 (Training assumption): Yes, in our theoretical analysis, we assume that the training procedure converges to the unique least norm solution, guaranteed for standard SGD initialized at zero. In our empirical analyses, we also always trained all models with the same training algorithms.
• Q2 (Epistemic vs. aleatoric uncertainty): We admit that a single random feature model cannot decouple epistemic and aleatoric uncertainty. This decoupling is a possible advantage of ensembles, though our analysis in Section 3.3 implies that standard ensembles do not cleanly deliniate these sources either.
• Q3 (Feasibility of Assumption 2.1): As discussed in the paper, the first condition is fulfilled whenever $w\_{\\perp i}$ and $w\_i^{\\top}$ are sub-Gaussian, which is true when the features come from activation functions with bounded derivatives and sub-Gaussian weights (which we would argue is a very weak assumption). For the second condition we expect that most nonlinear activation functions, such as sigmoid, tanh, sine, or cosine, satisfy this condition if we assume i.i.d. weights from a distribution with a density function. However, rigorously proving this is beyond the scope of this work.
• Q4 (Impact of regularization parameter $\\lambda$): Intuitively, the training outcome is continuously dependent on the ridge parameter $\\lambda$; thus, small variations in $\\lambda$ typically result in minor deviations in predictions. The proof of our theorem effectively provides a rigorous confirmation for this intuition.

We hope these responses and adjustments clarify our contributions and address your feedback. If you have further questions or suggestions, we would be happy to address them.

We thank the reviewer for their comments and feedback on our paper. Below, we address the concerns and questions raised.

1. Concerns about Theorem 3.3:

"Firstly, there is some problem with Theorem 3.3. The bound term on the right-hand side does not converge to zero as the number of features increases, provided that $\\delta$ is fixed. Thus, Theorem 3.3 is not sufficient to conclude that in the case of finite ensemble budget, the RF ensemble regressor and the RF simple regressor are asymptotically equivalent with high probability as D goes to infinity."

We believe there is a misunderstanding regarding this theorem. Theorem 3.3 does not aim to establish exact asymptotic equivalence. We proved asymptotic equivalence holding almost surely in our main result (Theorem 3.2). Thus, Theorem 3.3 is simply an analogous finite-sample guarantee showing that the single large model and the ensemble behave similarly with high probability at finite scales.

2. Generality and novelty of Theorem 3.2:

"On the other hand, Theorem 3.2 is impressive but not surprising. The linear random feature regression in the case of infinite width simply coincides with kernel interpolation, hence the infinite-width assumption together with the linearity of the model helps simplify the computations”

We believe there is a misunderstanding regarding the key contribution of Theorem 3.2. The main contribution of the theorem is to investigate an infinite ensemble of finite-width random feature regressors rather than the infinite-width model. To our knowledge, no prior work has demonstrated that such an ensemble converges to the kernel interpolator, except in the special case of zero-mean Gaussian random features.

3. Question about the essential aspect preventing ensemble advantages:

"Traditionally, many ensemble methods such as random forest are based on simple but non-linear methods, and with the inspiration of this work, we are curious about what is the essential aspect of random feature model that prevents ensembles from bringing extra advantages? Is it the linearity?"

As suggested by our title, the key attribute preventing ensembles of overparameterized random feature models from providing additional advantages is the overparameterization of the ensemble members. This claim is supported by our theoretical and empirical results contrasting ensembles of underparameterized and overparameterized random feature models (see Figure 2 and Appendix E) as well as other recent work (e.g., Abe et al., 2022).

We hope these responses address your concerns and clarify the contributions of our work. If you have further questions or suggestions, we would be happy to address them.