An Effective Theory of Bias Amplification

Thank you for your thoughtful feedback and questions!

To address the weaknesses/questions that you mentioned:

(W1) We will add the following context to the revised version of our manuscript:

Our theoretical results cannot be derived as a special case of [1] because this work focuses on training and testing a random features model on data from the same, single Gaussian distribution. Furthermore, [2] and [3] focus on training a random features model on data from a single Gaussian distribution and testing the model on a different, single Gaussian. In contrast, we study the random features model in the setting of training on a mixture of Gaussian distributions and testing on each component. Because any Gaussian distribution ${\cal N}_1$ can be expressed as a mixture of Gaussians $p_1 {\cal N}_1 + p_2 {\cal N}_2$ where $p_1 = 1, p_2 = 0$, but a mixture is more expressive than a single Gaussian, our theoretical results cannot be a special case of [1, 2, 3].

(W2) This is a great suggestion. We will add the following content to Section 4.2:

When $\psi \approx 1$ (i.e., the number of parameters and samples are about matched), bias amplification is not monotonic with respect to regularization (or training time); this motivates finding an optimal $\lambda$ or training time to reduce bias amplification. In practice, an optimal $\lambda$ or training time can be selected by searching for values that strike a desired balance between overall validation error and bias amplification. The search space can be further reduced by using insights from our theory such as with overparamaterization, as regularization decreases (or training time increases), bias amplification increases and plateaus, and with underparameterization, as regularization decreases (or training time increases), bias amplification decreases and plateaus. It is further important for ML practitioners the consider the interplay between high vs. low feature-to-sample regimes and overparameterization in inducing bias amplification vs. deamplification, respectively, when selecting optimal hyperparameters (see Figure 2).

References

[1] Adlam, B., & Pennington, J. (2020, November). The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization. In International Conference on Machine Learning (pp. 74-84). PMLR.

[2] Lee, D., Moniri, B., Huang, X., Dobriban, E., & Hassani, H. (2023, July). Demystifying disagreement-on-the-line in high dimensions. In International Conference on Machine Learning (pp. 19053-19093). PMLR.

[3] Tripuraneni, N., Adlam, B., & Pennington, J. (2021). Covariate shift in high-dimensional random feature regression. arXiv preprint arXiv:2111.08234.

Thanks again for your review! Please feel free to let us know if there are any remaining comments not addressed. We appreciate any feedback, and we are happy to answer any further questions.

We have included the additional discussion in the revision (in green). We thank the reviewer again for their helpful feedback and positive reception of this work!

MINOR

(Q1) Yes, Figure 1 is with balanced groups (as stated in line 341).

(Q2) We will revise each of our results sections to regularly remind the reader of the setup and more clearly convey what each regime means in plain terms (e.g., $\psi > 1$ is the overparameterized regime). We described the setup upfront at the start of each subsection (e.g., lines 346-353, lines 408-410, lines 465-483).

(Q3) We do not have any intention of “hiding” the error bars in Figure 3. We made an editorial decision to relegate all the versions of the figures with error bars to the appendix to make the figures more readable. Moreover, the version of Figure 3 with error bars does not convey a different message; our finding still holds, and we are happy to place the error-bar version in the main body. The theoretical predictions associated with Figure 3 are relatively qualitative because CMNIST is a classification problem while our theory is only precise for regression. However, under the assumptions and analogies that we make in lines 437-457, our derivations of the bias and variance terms $B$ and $V$ are correct, with $\approx$ simply capturing the slight discrepancy in $\Sigma_1$ and $\Sigma_2$ due to the colors of the digits.

We hope that the explanations and clarifications provided above address the reviewer’s questions and comments.

References

[1] Mannelli, S. S., Gerace, F., Rostamzadeh, N., & Saglietti, L. (2022). Unfair geometries: exactly solvable data model with fairness implications. arXiv preprint arXiv:2205.15935.

[2] Jain, A., Nobahari, R., Baratin, A., & Mannelli, S. S. (2024). Bias in Motion: Theoretical Insights into the Dynamics of Bias in SGD Training. arXiv preprint arXiv:2405.18296.

[3] Yehudai, G., & Shamir, O. (2019). On the power and limitations of random features for understanding neural networks. Advances in neural information processing systems, 32.

[4] Adlam, B., & Pennington, J. (2020). Understanding double descent requires a fine-grained bias-variance decomposition. Advances in neural information processing systems, 33, 11022-11032.

[5] Bach, F. (2024). High-dimensional analysis of double descent for linear regression with random projections. SIAM Journal on Mathematics of Data Science, 6(1), 26-50.

[6] Goldt, S., Loureiro, B., Reeves, G., Krzakala, F., Mézard, M., & Zdeborová, L. (2022, April). The gaussian equivalence of generative models for learning with shallow neural networks. In Mathematical and Scientific Machine Learning (pp. 426-471). PMLR.

[7] Adlam, B., & Pennington, J. (2020, November). The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization. In International Conference on Machine Learning (pp. 74-84). PMLR.

[8] Ali, A., Kolter, J. Z., & Tibshirani, R. J. (2019, April). A continuous-time view of early stopping for least squares regression. In The 22nd international conference on artificial intelligence and statistics (pp. 1370-1378). PMLR.

Thank you for your detailed review and questions!

We first address each weakness mentioned below:

(W1) Thank you for bringing our attention to [1] and [2]. We were not previously aware of these papers; they were not ignored. We will include the following paragraphs in our revised manuscript, which clearly discusses the connections and differences of our work with [1] and [2]:

The main distinction between our work and [1] and [2] is we provide a precise theoretical characterization of how models amplify bias in different parameterization regimes.

In terms of similarities, [1] and [2] also precisely analyze the bias of models trained on a mixture of data from multiple groups in a high-dimensional setting. Like [1] and [2], we study linear models that are trained with regularization, and measure bias as the difference in test performance of a model between two groups. We further consider some similar factors that give rise to bias amplification, e.g., group imbalance, group variance, inter-group similarity, and dataset size. In addition, we share some theoretical conclusions, e.g., bias can occur even when the groups have the same ground-truth weights (Section 5) and are balanced (Section 4.1). We also discuss positive transfer in our experiment with CMNIST (Section 4.2).

However, our treatment of bias amplification adds nuance to the discussion of positive transfer in [1], which claims that the $EDD$ generally tends to be higher than the $ODD$. Instead, we show that the bias amplification $ADD = ODD / EDD$ can vary greatly (going below 1 and above 1) as a function of the rate of parameters to samples $\psi$, even for a fixed $\phi$, or $\alpha$ as is used in [1] (see Figure 2 in our manuscript).

This is because [1] and [2] only consider the setting where the number of samples $n$ and features $d$ proportionally scale to infinity, while we consider the setting $n, d \to \infty$ and the number of parameters $m \to \infty$. This enables us to expose new, richer insights into the impact of (over/under-)parameterization on bias amplification, such as in Figure 1, lines 368-397, and lines 510-520. For example, we find that “a larger number of features than samples can amplify bias under overparameterization, there may be an optimal regularization penalty or training time to avoid bias amplification, and there can be fundamental differences in test error between groups that do not vanish with increased parameterization” (lines 78-82), among various other findings in Sections 4 and 5.

Additionally, [1] employs the replica method, which is non-rigorous, while we use operator-valued free probability theory (explained in Appendix A), which is entirely rigorous. Furthermore, while [1] discusses the paradigm of training separate models for each group, it theoretically focuses on a single model trained for both groups. In contrast, in our work, we theoretically treat both these paradigms (i.e., to isolate the contribution of the model itself to bias) and validate our theory extensively. Moreover, [1] and [2] study the application of linear classification to Gaussian data and ground-truth weights with isotropic covariance, while we focus on the application of regression with random projections (a simplified model of NNs) to Gaussian data and weights with more general covariance structure. This allows us to analyze additional factors of bias, such as group covariance structure and label noise.

Thanks again for your review! Please feel free to let us know if there are any remaining comments not addressed. We appreciate any feedback, and we are happy to answer any further questions.

To answer your questions:

(Q1) Much research has studied bias amplification empirically (as we detail in Section 1.2), with hypothesized factors such as conditional dependence heterogeneity. This might explain why bias amplification appears “intuitive” in the simple setting that $y = x$ and $y = -x$.

However, our paper goes well beyond prior empirical work by contributing a general yet precise analytical theory of bias amplification that accommodates various model design choices (e.g., number of parameters, regularization penalty) and data distribution properties (e.g., number of features, covariance matrix structures, group imbalance, conditional dependence heterogeneity, label noises). That is, two major goals of the paper are to: (1) derive a precise profile for bias amplification in high dimensions, that goes beyond intuition, and (2) analyze the profile as a function of the aforementioned diverse model design and data distribution factors, to reproduce known bias amplification phenomena and expose new, interesting phenomena in various regimes. For example, we find that “a larger number of features than samples can amplify bias under overparameterization, there may be an optimal regularization penalty or training time to avoid bias amplification, and there can be fundamental differences in test error between groups that do not vanish with increased parameterization” (lines 78-82), among various other findings in Sections 4 and 5.

We would like to clarify that the theoretical results are not based on isotropic covariance. The groups may have different covariance matrices (e.g., as in Section 5 (lines 465-471) and Appendix K), and the only assumption we make about the structure of the covariance matrices is that they are simultaneously diagonalizable (Assumption B.1). Our theory easily accommodates and correctly predicts bias amplification for covariance matrices with non-isotropic structure. For example, we consider fixed covariance in Section 5 (line 468) and covariance that exponentially decays in Appendix K and Figure 10.

(Q2) While we may lack direct access to population parameters like $\Sigma_1$, we can estimate them using the sample covariance $\widehat{\Sigma}_1 = X^\top_1 X_1$. Even if we are unable to robustly estimate the population parameters, our theory still provides valuable insights. Our paper demonstrates that the ratios of parameters for the groups is often what matters, e.g., label noise ratio $\sigma_2^2 / \sigma_1^2$ (lines 361-367), ratio of covariance eigenvalues (Appendix L). Thus, practitioners can use our theory to get intuition about when disparities in the variability of labels and features across groups can severely amplify bias.

(Q3) In practice, an optimal $\lambda$ or training time can be selected by searching for values that strike a desired balance between overall validation error (that is not too high) and bias amplification (that is not too high). The search space can be reduced by using insights from our theory such as with overparamaterization, “as regularization decreases (or training time increases), bias amplification increases and plateaus,” and with underparameterization, “as regularization decreases (or training time increases), bias de-amplification increases and plateaus” (lines 416-419). The heterogeneity definitely can be leveraged, for example, by training a mixture of experts that is regularized to deamplify bias.

We hope that the explanations and clarifications provided above address the reviewer’s questions and comments.

Thank you for your detailed review and questions!

We first address each weakness mentioned below:

(W1) Our analysis does not rely on conditional dependence heterogeneity. To consider the setting where $w^*_1 = w^*_2$, we can simply set $\Delta = 0$ in our theoretical findings. In fact, we empirically verify that our theory still holds and predicts bias amplification occurs even when $w_1 = w_2$ in Section 5 (lines 477-479, Figure 4). In essence, the structure and eigenspectra of the covariance matrices of the two groups still contribute to bias amplification even when the ground-truth weights for the groups are the same. We only allow the possibility of $w^*_1 \neq w^*_2$ to be as general as possible in our theory.

(W2) We draw our definition of bias amplification (Definition 2.1) from [1], which was peer-reviewed and published at FAccT 2023, in order to be consistent with prior literature. At a high level, our definition quantifies how many times worse model bias would be if a machine learning practitioner opted to train a single model on a mixture of data from two groups (i.e., the setting in which bias is observed in practice) vs. separate models for the data from each group (i.e., the setting which corresponds to the bias in the data alone, and thus the a priori amount of bias we would expect in the case of a single model). In sum, we seek to isolate the contribution of the model to bias when learning from data with different groups. This is consistent with the conceptualization of other bias amplification metrics in the literature (e.g., [2]).

(W3) In Line 172, the one-hidden-layer NN model is defined correctly; we do not consider a nonlinear activation. Ridge regression with random features has been posited as a reasonable approximation for NNs in the random features regime [3, 4]. In particular, it has been argued that as the number of parameters $m \to \infty$ (as in our high-dimensional setting), gradient descent effectively learns a linear predictor over $m$ random features [3]. Furthermore, [4, 5, inter alia] are able to reproduce interesting NN phenomena like double descent using the random features model. We choose ridge regression with random projections, as it offers analytical tractability while exposing novel bias amplification phenomena related to model parameterization; such phenomena are not exposed by classical ridge regression. Furthermore, still using free probability theory and the Gaussian equivalence theorem [6], we can easily extend our theory to study bias amplification in NNs with nonlinearities in the NTK regime, as in [7] (see lines 537-539 of the manuscript), albeit with more complex equations.

Thanks again for your review! Please feel free to let us know if there are any remaining comments not addressed. We appreciate any feedback, and we are happy to answer any further questions.

We would like to advise you that a new revision of the manuscript has been uploaded, based on feedback from Reviewer m7Lv, where some line and section numbers have changed. We ask that you reference the first revision that we made in regards to any line or section numbers that we cited in our rebuttal. Thanks for your understanding!

Thank you for your clarification on the theoretical foundation and results presented in this paper. I am happy to increase my score from 5 to 6.

In the revision, I recommend including a discussion on the practical challenge of determining whether a machine learning model trained on data is prone to bias amplification (as your response to (Q2) acknowledges the population parameters cannot be accurately estimated).

Additionally, I am still unclear about your response to (Q3). How can one effectively balance overall validation error and bias amplification in practice when the latter is actually unknown?

(W3) Per your suggestion, we run additional experiments on MNIST and CNN and report the results in the updated version of our paper (Appendix O). We verify our conclusions about label noise and model size on MNIST and CNN. We see in Figure 15 that as we increase the label noise ratio, the $EDD$ generally increases, while the $ODD$ remains relatively low, as claimed in lines 437-457. Furthermore, in Figure 16, as the hidden dimension of the penultimate layer of the CNN increases, the $ODD$ tends towards 0 while the $EDD$ does not, which is in line with what our theory predicts in Figure 2 (in the regime where $\phi < 1$).

Furthermore, ridge regression with random features has been posited as a reasonable approximation for NNs in the random features regime. For example, [1, 3, inter alia] are able to reproduce interesting NN phenomena like double descent using the random features model.

To answer your questions:

(Q1) Yes, the bias amplification phenomenon occurs in low-dimensional regimes. In Sections 4, 5, J, M, N, we show that our theory predicts bias amplification in experiments with models trained on only $n = 400$ samples. The high-dimensional regime is commonly studied in ML theory and statistical physics (as we mention in Section 1.2), as it makes precise analysis more tractable.

(Q2) The phase transition in the lower left corner of the $EDD$ plot occurs at $\phi = \psi$ (i.e., where the rate of parameters to features $\gamma = 1$) (lines 311-312). This is a known interpolation pole in the literature (Appendix S7.2 of [1], [2]). There is not a mismatch with the $ADD$ plot; zooming into the $ADD$ plot reveals the interpolation threshold in light blue. This is due to the granularity of the color thresholding. We are happy to clarify this in the revised version of our manuscript.

(Q3) When the curves are monotone with respect to $\lambda$, the optimal $\lambda$ is either at the left tail of the curve (e.g., $\lambda = 0$) or the right tail (i.e., the largest $\lambda$ among the reasonable options). A practitioner may consider the optimality of $\lambda$ with respect to a tradeoff between overall validation error and bias amplification. In contrast, Figure 11 shows that when $\psi$ is close to the interpolation threshold of 1, the bias amplification curve is often not monotone with respect to $\lambda$.

References

[1] Adlam, B., & Pennington, J. (2020). Understanding double descent requires a fine-grained bias-variance decomposition. Advances in neural information processing systems, 33, 11022-11032.

[2] D’Ascoli, S., Refinetti, M., Biroli, G. & Krzakala, F.. (2020). Double Trouble in Double Descent: Bias and Variance(s) in the Lazy Regime. Proceedings of the 37th International Conference on Machine Learning, in Proceedings of Machine Learning Research 119:2280-2290 Available from https://proceedings.mlr.press/v119/d-ascoli20a.html.

[3] Bach, F. (2024). High-dimensional analysis of double descent for linear regression with random projections. SIAM Journal on Mathematics of Data Science, 6(1), 26-50.

Thank you for your thoughtful feedback!

We first address each weakness mentioned below:

(W1) We will provide clearer intuition for each part of our formulae in the revised version of our manuscript. At a high level, in Definition 3.2, each of the terms $h_1, \ldots, h_4$ capture the limiting values of different sources of covariance between the sample covariance matrices $\overline{M}_1, \overline{M}_2$, the resolvent matrix $\overline{R}$, and the random projections matrix $S$; these sources of covariance are written explicitly in Equations 219-222 (in Appendix E). These terms naturally arise from expanding the solution to the ridge regression model with random projections, as Appendix E demonstrates.

The fundamental constants $e_s, \tau_s, u_s, \rho_s$ (in Definition F.1) can be intuitively interpreted in the setting where a separate model is learned for each group and $\lambda \to 0^+$. In this setting, for ridge regression with random projections, we show in Equations 337-338 (Appendix H) that $e_s, \tau_s$ are linearly related to the first-order degrees of freedom $I_{1, 1}$ of the population covariance matrix $\Sigma_s$. $e_s$ captures the effect of the feature rate $\phi_s$ while $\tau$ captures the effect of parameterization $\gamma$. Similarly, for classical ridge regression, we show in Equations 68-70 (Appendix C) that $e_s$ is linearly related to the degrees of freedom $\overline{\text{df}}_1$ of $\Sigma_s$.

On the other hand, $u_s$ and $\rho_s$ can be understood as pseudo-variances. Indeed, Equations 341, 349, and 52 show that $u_s, \rho_s$ is proportional to $V_s$ for ridge regression with and without random projections. That is, both quantities depend on the second-order degrees of freedom $I_{2, 2}$, or $\overline{\text{df}}_2$. In essence, our theory expands the fundamental constants to the more general setting where a single model is trained on a mixture of data from different groups and $\lambda > 0$ (Definition 3.2).

(W2) We do make some formal statements about bias amplification in the paper. For example, in Appendix H, we directly express the bias and variance of an unregularized model trained on just group $s$ (equivalently, an unregularized model trained on data from both groups when $\Sigma_1 = \Sigma_2$) in terms of the second and first-order degrees of freedom of the population covariance matrix $I_{2, 2}$ and $I_{1, 1}$. Thus, Corollary H.1 captures how the covariance matrix affects the test risk of the model. Corollary H.1 also reveals that in the underparameterized regime ($\psi < 1$), the variance of the model’s test risk strictly increases as a function of $\psi$ (i.e., the rate of model parameters to samples); the test risk of the model skyrockets (i.e., there is catastrophic overfitting) when $\psi$ gets close to 1. Corollary H.1 additionally demonstrates that the bias and variance of the test risk decrease as $\psi$ increases.

Furthermore, in Appendix K, to understand how the structure of the covariance matrix affects bias amplification, we derive the amplification profile of ridge regression with random projections with respect to the ratio $c$ of label noises. We do so in the setting where the eigenspectra of the covariance matrices have power-law decay. We show that bias amplification peaks near $c = 1$, bias reduction peaks when $c \to 0^+$, and amplification does not occur when $c \to \infty$ (lines 2832-2833). Furthermore, in lines 437-457, we mathematically reason about how a large label noise ratio $c$ greatly increases $EDD$ while $ODD$ remains relatively low.

We discuss in Appendix G the difficulties of rigorously isolating the effects of different components on bias amplification. For example, even when the population covariance matrices of the two groups are proportional, the fundamental constants are quartic in the degrees of freedom of the covariance matrix. This is why we empirically investigate how different components (e.g., different covariance structures, group sizes) affect bias amplification and minority-group error in Sections 4 and 5, and extensively validate that our theory predicts these implications.

Thanks again for your review! Please feel free to let us know if there are any remaining comments not addressed. We appreciate any feedback, and we are happy to answer any further questions.

We would like to advise you that a new revision of the manuscript has been uploaded, based on feedback from Reviewer m7Lv, where some line and section numbers have changed. We ask that you reference the first revision that we made in regards to any line or section numbers that we cited in our rebuttal. In particular, the plots for our new experimental results on MNIST and CNN are in Appendix O in the first revision and in Appendix P in the latest revision.

Thanks for your understanding!

Thank you Reviewers HU3q and m7Lv for engaging in the discussion.

Please, Reviewers X48V and KjTp read the authors' replies and feedback from other reviewers. If any concerns remain, request clarifications. This is your last chance to engage.

Thank you for your efforts.

Best regards, Area Chair