Pushing Boundaries: Mixup's Influence on Neural Collapse

The authors would like to extend their gratitude to the reviewer for their constructive response. Below we respond to the weaknesses and questions.

Weaknesses:

• We have updated the paper to include more experimentation and analysis relating to calibration. We believe that this addition provides a connection between our initial findings and the empirical success of mixup. We have updated the formatting of the paper in the way that the reviewer has suggested to improve the clarity.

Questions:

• Train mode was chosen mostly for consistency among plots. Yes, there is a slight difference in the plots for the resnets. We have included a figure for the network in eval mode in the appendix to show this difference. We believe this difference is due to batch normalization but we have not proved such a claim.

We again thank the reviewer for the time and effort spent reading and reviewing our paper. We offer a summary of the overall changes made to the manuscript below.

1. Enhanced Paper Structure and Readability:

• Figures are now strategically placed closer to the relevant text for a coherent reading experience.
• Text from figure captions is integrated into the main body for better context.
• Expanded the experiments section to provide more comprehensive insights.

2. Additional Experiments, Analyses, and Figures:

• Updated the colors in Figure 1 for better visualization and added to it the classifiers.
• Updated Figure 2, showcasing the feature configuration with MSE loss training, and its comparison with CE loss. This is meant to emphasize how the configuration obtained from the MSE loss is expected and uninteresting whereas the one obtained from CE, which is right next to it, is very different and unexpected.
• Introduced Figure 3, illustrating the layer-wise trajectories of the CLS token for a ViT trained with mixup.
• Introduced Table 2 reporting network calibration across various scenarios.
• Introduced Figure 4 which elucidates the geometric configuration's role in improving calibration.
• Introduced Figure 7 which presents changes in the mixup feature configuration once the last-layer classifier is fixed throughout training.

3. Addressing Specific Reviewer Concerns:

• Added detailed explanation of 2D projection methods used in our visualizations.
• Expanded on the theoretical framework, particularly regarding the assumptions about the simplex ETF classifier and its implications.
• Investigated the layer-wise trajectory of representations and its implications in mixup training.
• Discussed the practical significance of our findings in the context of calibration improvements.

Through these revisions, we aim to bridge the gap between the practical success of mixup training and its theoretical understanding, providing a more comprehensive and valuable contribution to the field. In light of these refinements, should you find it fitting, we would be most grateful for any potential reconsideration of our score.

We would like to express our gratitude to the reviewer for the time and effort put into their assessment of our paper. In the following response, we will address each of the weaknesses and questions raised. First the weaknesses:

1, 3, 4. Upon your suggestion, we have made significant changes to the formatting of the paper as well as the coloration of the plots presented. We hope that this new format and visualization have resolved any readability issues you encountered.

2. This is a valid point to raise, and in response to it, we have added an additional section investigating and explaining the relationship that this configuration has with calibration.

3. We have added additional descriptions of the last-layer activations being plotted as well as a detailed explanation of the projection method in the appendix. You are correct about the Neural Collapse projections. The only difference between ours is that in the Neural Collapse paper, to the best of our knowledge, everything is projected onto the span of the class means, whereas we are projecting onto the span of the classifiers. Indeed, we have neglected to display how the shape aligns with the classifier so we have updated the figures to include the classifiers.

4. It would be a strong assumption, so we have incorporated experiments illustrating the convergence of the classifier to a simplex ETF (Figure 5). However, we acknowledge the importance of theoretical proof of the characterization of optimal features without making this assumption, and we have designated this as a direction for future work.

5. If the scenario given by the reviewer were to occur, then let $x_1$ and $x_1’$ be training examples of class 1, $\tilde{\lambda}$ be such that $\tilde{\lambda}x_1 + (1-\lambda)x_1’ = x$, a training example of class 2.

Our objective function involves an expectation over $\lambda$, and since the set {$\{ \tilde{\lambda} \}$} is of Lebesgue measure 0, the case that $x$ is labeled as class 1 is negligible compared to when $x$ is assigned class 2, so $x$ would be aligned with class 2. Thus, we don’t consider this in our theoretical results.

Since in practice lambda is sampled, we can also consider a more discrete approach. This would result in two terms involving the input $x$ contributing to the overall loss, one with label $y_1$ and the other with label $y_2$. That is, there would be a term $\operatorname{CE}(x, y_1)$ and another term $\operatorname{CE}(x, y_2)$. If the first term appears with probability $P$ and the second with probability $(1-P)$, then overall we get:

$P \operatorname{CE}(x, y_1) + (1-P) \operatorname{CE}(x, y_2) = \operatorname{CE}(x, Py_1 + (1-P)y_2)$

Then, the example overall will have an effective label of $Py_1 + (1-P)y_2$, which corresponds to a mixup label with mixup coefficient P, which is covered by our theoretical result.

Interestingly, [1] considers the general setting proposed by the reviewer called manifold intrusion. There the authors claimed that mixup can be detrimental for cases where mixed up examples align with the original training data, exactly the same way the reviewer is proposing. The authors showed that this can occur in datasets like MNIST and that this causes a deterioration in test performance. However, it is unclear to what extent this occurs in more complicated datasets which is why in our paper we did not consider this case.

8. To further investigate the layer-wise trajectory of representations, we have provided an additional experiment doing a layer-wise projection of the CLS token for a ViT trained with mixup. We have also modified the discussion section being referenced to more accurately reflect the results of the paper.

9. We appreciate the different perspectives on the results. Our team holds the view that the result is not superficial. We believe the expected result is a triangular configuration. In order to illustrate this, we conducted an additional experiment incorporating mixup with the MSE loss. The resulting last-layer activations are arranged as convex combinations of the classifier, aligning with what we perceive as a superficial outcome. We hope that this additional experiment emphasizes the unexpected nature of the observed configuration.

We thank the reviewer for the time spent reading our paper and for your helpful and constructive review. Below we address the weaknesses and questions.

• Upon your suggestion, we have revised the formatting of this paper and hope that it has improved its overall readability. Specifically, section 2 has been reworked with figures appearing closer to their relevant text sections.
• We have re-written the problem statement of the paper and additional explanations for the numerical experiments have been incorporated, including a detailed explanation of the projection method.
• We have included a clearer explanation of the interpretation of the theorem after the theorem statement.
• We choose to project the features into 2D mainly to be consistent with the original Neural Collapse visualizations as well as other neural collapse-related papers and visualizations such as those cited in [1] - [4]. To address a more substantial set of classes, we have included a larger subset of classes in the appendix of the paper.
• We have incorporated a more comprehensive exploration of the differences in the theoretical features in comparison to the empirical features. In response to the reviewer's suggestion, we conducted an experiment with the classifier fixed as a simplex ETF, contributing valuable insights to our analysis (as the resulting features align closer to the theoretical features). Additionally, we have included the details of the amplification used to generate Figure 6 in the Appendix.

The authors would like to thank the reviewer for their thoughtful review as the changes implemented in response to the reviewer's comments have improved the paper's quality significantly. Below, we offer a summary of the overall changes to the manuscript.

1. Enhanced Paper Structure and Readability:

• Figures are now strategically placed closer to the relevant text for a coherent reading experience.
• Text from figure captions is integrated into the main body for better context.
• Expanded the experiments section to provide more comprehensive insights.

2. Additional Experiments, Analyses, and Figures:

• Updated the colors in Figure 1 for better visualization and added to it the classifiers.
• Updated Figure 2, showcasing the feature configuration with MSE loss training, and its comparison with CE loss. This is meant to emphasize how the configuration obtained from the MSE loss is expected and uninteresting whereas the one obtained from CE, which is right next to it, is very different and unexpected.
• Introduced Figure 3, illustrating the layer-wise trajectories of the CLS token for a ViT trained with mixup.
• Introduced Table 2 reporting network calibration across various scenarios.
• Introduced Figure 4 which elucidates the geometric configuration's role in improving calibration.
• Introduced Figure 7 which presents changes in the mixup feature configuration once the last-layer classifier is fixed throughout training.

3. Addressing Specific Reviewer Concerns:

• Added detailed explanation of 2D projection methods used in our visualizations.
• Expanded on the theoretical framework, particularly regarding the assumptions about the simplex ETF classifier and its implications.
• Investigated the layer-wise trajectory of representations and its implications in mixup training.
• Discussed the practical significance of our findings in the context of calibration improvements.

Through these revisions, we aim to bridge the gap between the practical success of mixup training and its theoretical understanding, providing a more comprehensive and valuable contribution to the field. In light of these refinements, should you find it fitting, we would be most grateful for any potential reconsideration of our score.

[1] Chenxi Huang, Liang Xie, Yibo Yang, Wenxiao Wang, Binbin Lin, & Deng Cai. (2023). Neural Collapse Inspired Federated Learning with Non-iid Data.

[2] Rangamani, A., Lindegaard, M., Galanti, T. & Poggio, T.A.. (2023). Feature learning in deep classifiers through Intermediate Neural Collapse. Proceedings of the 40th International Conference on Machine Learning, in Proceedings of Machine Learning Research 202:28729-28745 Available from https://proceedings.mlr.press/v202/rangamani23a.html.

[3] Xie, L., Yang, Y., Cai, D., & He, X. (2022). Neural Collapse Inspired Attraction-Repulsion-Balanced Loss for Imbalanced Learning. Neurocomputing, 527, 60-70.

[4] Yang, Y., Chen, S., Li, X., Xie, L., Lin, Z., & Tao, D. (2022). Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?. In Advances in Neural Information Processing Systems (pp. 37991–38002). Curran Associates, Inc..

The authors would like to thank the reviewer for the time spent reading and reviewing our paper. To address the reviewer's point, about the contributions of the paper being limited, the authors have undertaken significant revisions to enhance the clarity, theoretical depth, and empirical evidence of our manuscript, by adding significantly more experimentation and analysis to the paper. Below is a summary of changes made to the manuscript:

1. Enhanced Paper Structure and Readability:

• Figures are now strategically placed closer to the relevant text for a coherent reading experience.
• Text from figure captions is integrated into the main body for better context.
• Expanded the experiments section to provide more comprehensive insights.

2. Additional Experiments, Analyses, and Figures:

• Updated the colors in Figure 1 for better visualization and added to it the classifiers.
• Updated Figure 2, showcasing the feature configuration with MSE loss training, and its comparison with CE loss. This is meant to emphasize how the configuration obtained from the MSE loss is expected and uninteresting whereas the one obtained from CE, which is right next to it, is very different and unexpected.
• Introduced Figure 3, illustrating the layer-wise trajectories of the CLS token for a ViT trained with mixup.
• Introduced Table 2 reporting network calibration across various scenarios.
• Introduced Figure 4 which elucidates the geometric configuration's role in improving calibration.
• Introduced Figure 7 which presents changes in the mixup feature configuration once the last-layer classifier is fixed throughout training.

3. Addressing Specific Reviewer Concerns:

• Added detailed explanation of 2D projection methods used in our visualizations.
• Expanded on the theoretical framework, particularly regarding the assumptions about the simplex ETF classifier and its implications.
• Investigated the layer-wise trajectory of representations and its implications in mixup training.
• Discussed the practical significance of our findings in the context of calibration improvements.

Through these revisions, we aim to bridge the gap between the practical success of mixup training and its theoretical understanding, providing a more comprehensive and valuable contribution to the field. In light of these refinements, should you find it fitting, we would be most grateful for any potential reconsideration of our score.