For Better or For Worse? Learning Minimum Variance Features With Label Augmentation

We would like to thank Reviewer tHC2 for taking the time to review our paper, and we hope to address the stated weaknesses below.

Weaknesses

1. While we agree that prior works have definitely pointed towards a variance minimization effect for Mixup and label smoothing (e.g. in relation to neural collapse), we are not aware of any work that theoretically establishes this from a feature learning perspective. The goal of our work was to unify and simplify some of the ideas in previous feature learning analyses of Mixup [1, 2].

2. The findings of prior works do not contradict our results -- we now further emphasize in the revisions that low variance features can either be the features that generalize well or the ones that are spuriously correlated with the targets. We hypothesize that due to the repeatedly reported success of label smoothing and Mixup, in practical settings it is likely the case that there are low variance latent features that generalize well as opposed to being spuriously correlated with the targets. We have included multiple different spurious correlation setups in Section 4.2 of our paper (and now also Appendix C.2); we feel that a complete investigation of the existence of low variance/high variance latent features in standard benchmarks is outside the scope of our current work.

Questions

1. Other than the theoretical results we prove, which can be translated to show poor generalization in settings where the spurious correlations are low variance (like in Section 4.2), we are not aware of other works that try to make this point.

We hope the above provides further clarity regarding our paper and we are happy to answer any further questions you may have.

[1] Chidambaram, M., Wang, X., Wu, C., & Ge, R. (2022). Provably Learning Diverse Features in Multi-View Data with Midpoint Mixup. International Conference on Machine Learning.

[2] Zou, D., Cao, Y., Li, Y., & Gu, Q. (2023). The Benefits of Mixup for Feature Learning. International Conference on Machine Learning.

We would like to thank Reviewer LPKu for taking the time to review our paper and for finding our results to be novel, and we appreciate the many detailed comments which we hope to address below.

Weaknesses/Questions

Line 135 on page 3. The identity matrix shouldd be denoted by mathematical expression "$\mathrm{I}_d$" rather than text "Id" right?

While we consider both notations to be standard, we have changed to $\mathrm{I}_d$ to better emphasize the dimension and also pointed this out in the notation.

In the context of binary classification, the label variable should be $Y \in \{-1, +1\}$ right? But in that case, the cross-entropy loss defined in equation 2.1 cannot be applied on binary classification. Or, is the paper implicitly regarding the term "$k$-class classification" to be distinct from the term "binary classification"?

We are considering $k$-class classification to be distinct from binary classification, since taking $y \in \{-1, +1\}$ for binary classification (as opposed to $y \in \{0, 1\}$) makes it much easier to state the binary classification results/experiments. We make this distinction in the discussion right after we introduce our notation in Section 2.

In equation 2.4, I think the second term inside the expectation should be "$\frac{1}{\alpha - 1} \sum_{i \in [k] \setminus Y} \log g^i(X)$".

While there are different stated versions of the label smoothing loss in the literature (depending on the valid range of smoothing hyperparameter), we prefer our version when we consider $\alpha \in (0, 1)$ (and our results use this version as stated). By taking the summation $\frac{\alpha}{k}\sum_{i = 1}^k \log g^i(x)$ over all classes, we ensure that the probability mass assigned to the true target class never becomes smaller than that assigned to any other class, at the cost of interpreting $\alpha$ as the total mass assigned to all other classes.

In the definition of Mixup loss in equation 2.8, are the points pairs $(X_1, Y_1)$ and $(X_2, Y_2)$ sampled independently from $\pi$? In that context, I would denote it as $(X_{1:2}, Y_{1:2}) \stackrel{i.i.d.}{\sim} \pi$ or $(X_{1:2}, Y_{1:2}) \sim \pi \otimes \pi$.

The pairs are indeed sampled independently, and we have fixed the notation as you have suggested to be $\pi \otimes \pi$.

What are the mathematical definitions of low variance features and high variance features? Is it the overall variance? Or in-class variances? Also, how would one justify if a feature is high variance or low? Should there be a predefined threshold? Or maybe even a post-defined threshold, meaning threshold is defined after the groups of high variance feature and low variance features are manually partitioned?

Our notions of low and high variance are codified entirely in Assumption 3.5; i.e. it is a condition on the covariance matrix of the data multiplied by the label and restricted to certain dimensions. We have stated it this way for brevity, but you are definitely correct in that this condition could also be stated in terms of the class-conditional covariance matrices. In particular, "high variance" here just means that those dimensions have nontrivial variance no matter how you combine them linearly. In our experiments, we differentiate high variance and low variance by the individual variances of the dimensions of the input; the low variance "feature" corresponds to those dimensions of the input whose variance falls below some small threshold (which we can vary, so long as it is much smaller than the variance of the high variance dimensions).

Before assumption 3.2, $\mathcal{L}$ and $\mathcal{H}$ are at first defined as subsets of $[d]$, and then regarded as sets of vectors. I understand this is only for the sake of simplicities in notations, but I would suggest that the paper simply uses different notations.

While we agree that it is regrettable that we abuse notation by saying $v \in \mathcal{L}$, we prefer this notation to repeatedly restating $v_i = 0$ for all $i \in \mathcal{L}$.

We would like to thank Reviewer Gh1c for taking the time to review our work and for the useful feedback. We hope to address the stated concerns below.

Weaknesses

1. We apologize for the lack of visual aids for illustrating the theoretical results, this was an oversight on our part and we completely agree that such visualizations provide a lot of practical intuition for the theory. In the revision, we now have an Appendix C.1 that visualizes decision boundaries for weight decay, label smoothing, and Mixup in a simple 2-D case. We have also restructured the main contributions section of the paper (Section 1.1) to better highlight our takeaways.

2. Thank you for pointing out these references that we missed -- we now include all of them, and have also restructured the related work section to include a discussion of the relationship to neural collapse.

3. We agree that our paper does not directly show that minimum variance feature learning leads to better generalization in standard benchmarks, as this would require clearly defining such features with respect to the benchmarks and illustrating that the high variance features are mostly noise in the training data. This type of analysis has been done to an extent in prior work [1, 2], but our goal was to take the observations of the prior work and generalize them at the cost of the setting in which we prove them. Furthermore, as we now emphasize in the main contributions section, our results do not immediately imply generalization results because sometimes the low variance features in the data can generalize poorly, as in the experiments of Section 4.2.

4. You are right, spurious correlations do not need to be low variance -- we merely wanted to illustrate that when they are, the minimum variance learning property can actually be detrimental. We hypothesize that in standard benchmarks the spurious correlations are actually high variance, and the ``good'' features are low variance.

Questions

1. In the CIFAR experiments, there is no pretrained model being used -- we directly train a logistic regression model on the CIFAR datasets. In the setups of Section 4.2, the low variance feature (both the adversarially introduced pixels in Section 4.2.1 and the backgrounds in Section 4.2.2) have smaller magnitudes than the normalized (i.e. $z$-scored) pixels in the rest of the image (the higher variance feature(s) in this case); we intended these setups to combine both the separability and variance assumptions from our theory.

2. The overparameterization definitely affects the learning dynamics of the ResNet models, which definitely interacts with the final layer activation variances we report. However, we do not believe it affects our main takeaways (as our theory either handles linear models or is stated in a level of generality that ignores the model type), and we discuss the impact of scaling overparameterization in Appendix C.3.

We hope the above changes, along with other changes pointed out in the top-level comment, serve to improve upon the weaknesses you pointed out and are happy to answer any further questions you may have.

[1] Chidambaram, M., Wang, X., Wu, C., & Ge, R. (2022). Provably Learning Diverse Features in Multi-View Data with Midpoint Mixup. International Conference on Machine Learning.

[2] Zou, D., Cao, Y., Li, Y., & Gu, Q. (2023). The Benefits of Mixup for Feature Learning. International Conference on Machine Learning.

Thank you for the prompt response. Here we hope to address your additional concerns.

1. The intuition we present still works, i.e. honing in on low variance dimensions/features is still important for minimizing loss since it allows the model to more closely predict the soft labels. However, when we relax the compact support constraint we can't rely on strong convexity of the augmented losses in the way that we have done to separate them from ERM. This is demonstrated by the case of Gaussians where Class -1 is $\mathcal{N}(-\mu, \Sigma)$ and Class +1 is $\mathcal{N}(\mu, \Sigma)$. Here, the Bayes decision boundary is given by $\Sigma^{-1} \mu$ (so we are weighting with respect to the inverse of the variance) and both the ERM and Mixup minimizers were shown to align with this direction by [1]. So while there isn't separation in terms of the learned direction in this case, there is still a preference for low variance effect.

2. We have modified the text so as to clear up the confusion regarding separability, and also made clear that no feature extraction is applied to the CIFAR data before we apply logistic regression.

Feel free to let us know if you have any further questions or suggestions.

[1] Oh, J., & Yun, C. (2023). Provable Benefit of Mixup for Finding Optimal Decision Boundaries. International Conference on Machine Learning.

Thank you for your response. I have reviewed the revised version of the paper and I am happy to see that the authors have effectively addressed my main concern. In particular, the restructured main contributions section is significantly clearer, and I believe it successfully eliminates any potential misunderstandings or confusion for readers. As a result, I am increasing my score to 8.

We would like to thank Reviewer 2SFa for reviewing our paper, and we are grateful that they found our results to be insightful. We hope to address the pointed out weaknesses and questions below.

Weaknesses

• In our work, our goal was only to prove this minimum variance property of Label Smoothing and Mixup -- given that both methods almost always lead to performance improvements in practice, we hypothesize that learning minimum variance features is actually important for the standard datasets/benchmarks. We completely agree that this is only correlation and not causation; this is why we believe that investigating such minimum variance features in standard datasets is a promising direction for future work.

• Thank you for the suggestion -- we have now added Appendix B which covers the settings of the prior works of Chidambaram et al. (2023) and Zou et al. (2023) in more detail and contrasts it to ours.

Minor Comments and Questions

• We have revised the proof of Corollary 3.4 and in the process fixed the typo, thank you for catching.

• Under Assumption 3.2, the max-margin direction must have nontrivial weight in the high variance direction. This should be clearer from the added detail to Corollary 3.4.

We are happy to answer any further questions you may have.

We would like to thank Reviewer Enmj for taking the time to review our paper, and we appreciate that they found our results well-presented. We hope to address the raised questions below.

Weaknesses

1. We agree and as you point out already note in the main paper that Assumption 3.2 is strong; we intended it to set the scene for showing that Mixup/Label Smoothing will still prefer the low variance feature even when it can be worse for classification than the high variance feature. We are not arguing that the high variance features are always, or even typically, the better features in practical settings -- in fact, we expect the opposite since Mixup/Label Smoothing tend to always help performance. And we also agree that, as far as Theorem 3.3 is concerned, $\mathcal{L}$ and $\mathcal{H}$ are just placeholders; we only introduce variance assumptions in Assumption 3.5. Once again, the use of $\mathcal{L}$ and $\mathcal{H}$ here even in Theorem 3.3 is to make it easier to compare the result to Theorems 3.6 and 3.7.

2. The discussion on why Mixup/Label Smoothing learn the low variance features but not the high variance features, whereas ERM can/will learn both is provided in detail in Sections 3 and 4, in particular by contrasting Theorem 3.2 to Theorems 3.6 and 3.7. It is not true that you can just take $\alpha = 0$ and prove the results we prove for Label Smoothing in these sections; that is in fact the purpose of Assumption 3.2 and Theorem 3.3 -- they are agnostic to the variance of the features. We have revised the writing in Section 3 to better emphasize this.

Questions

1. Extending our results to the multi-label case should certainly be possible if we reduce the multi-label problem to several independent binary classification problems (determining whether a certain label should be assigned or not to an input).

We are happy to address any further concerns you may have, and hope that the above clarifies our results further.