Neural Collapse To Multiple Centers For Imbalanced Data

We appreciate the reviewer's recognition of our contribution to the field of deep learning and classification. To convey the meaning of our work more fluently, we answer the questions in the following order.

$**Response to weaknesses 1 and 5 **:$

We will enlarge the Fig.1 for a better illustration.

We would like to move Proposition B.1 to the Main Result section (section 3) and introduce the notion of $\tilde{w}_j^{(k)}$ more carefully.

$**Response to weakness 3**:$

We are sorry that our narrative creates a misunderstanding here. In the analysis of the classification rule we are assuming the classifiers and features are well-trained, the optimal symmetric final structure can be seen as the guarantee of the isotropy of the hard-to-predict populations. However, when we use GCR to supervise the model, the training will quickly encounter a gradient vanishing. Indeed, at initialization let $s = argmax_j\lbrace \langle w_j^{(k)},h_{k,i}\rangle\rbrace$, the gradient descent will take $h_{k,i}$ to the direction closer to the specific center $w_s^{(k)}$. Then during training $h_{k,i}$ approaches $w_s^{(k)}$ even faster since $s = argmax_j\lbrace \langle w_j^{(k)},h_{k,i}\rangle\rbrace$ from the beginning. So the model ends up with a structure far from optimal; each class will have several clusters scattered randomly nearly orthogonal to each other. In this case, the model overfits the rule easily. This is the drawback of directly using GCR as a practical objective and the reason we try to use the average of centers instead of the maximum of them.

Both P and Cosine loss are approximations to the Generalized Classification Rule (GCR) while cosine loss is a concise version of P; Both of them use multiple centers, please recall that a center is a vector associated with a certain classifier. The difference between P and Cosine loss is: P uses all centers available while Cosine loss uses centers of one class (the class to which the feature belongs). Unlike P which uses both within-class alignment and between-class separateness (refer to line155 in the main article), Cosine loss uses the within-class alignment only which has faster NC and an advantage in Mixup training (refer to the empirical results in the next Response).

$**Response to weakness 2**:$

Thank you for your question. We did not pay enough attention to the empirical results w.r.t the loss P, since our major concern is the final state of the neural collapse and cosine loss is obviously simpler than P. We now provide a few experimental results w.r.t P, where the classifier is fixed. The table below shows the performance of P on cifar100 with different imbalance ratios.

The performance of P is lower than CAL, and P starts to be incompatible with Mixup training when the ratio decreases (to 0.01). One possible explanation is that the Cosine loss do not have distraction terms from the centers of other classes, while P needs to handle the between-class separateness, which can become a complicated procedure when there are a myriad of centers presented.

$**Response to weakness 4**:$

We add the comparison to the method Rotated Balanced Representation (RBL) of [1] to our paper, and the literature will be discussed in the introduction or related work. It can be observed in the following table (the first four columns are cifar10 test accuracy and the last four are cifar100) that the learnable orthogonal layer is effective when the post-hoc logit adjustment (PLA) is applied (the ablation shows how powerful the post-hoc logit adjustment is). "-" indicates no results recorded (due to time limit).

Our method outperforms RBL in two settings for cifar100. We conjecture that in a setting of a small imbalanced ratio and a large number of classes, the hard-to-predict distribution may dominate the performance. The reason that our method CAL has lower accuracy compared to the method of [1] in other settings, is three-fold: 1. Imbalanced learning with MSE loss is less effective than CE loss in general; 2. Our experiment setting is chosen as close as possible to that where the theoretical analysis (proposition B.1 and C.1) is conducted, for example, to match the isotropy of the Gaussian, we normalize/batch normalize the feature to ensure it is unit norm and centered before the classification, and the weight are unit vectors through training. This setting possibly harms the learnability and flexibility of our model. 3. Under the class-aware MSE loss, we use the original classifier to be the surrogate classification rule of our general classification rule. However, the rule is designed especially for “hard-to-predict” unseen data and thus is not necessarily optimal for the classification of other unseen data. When the hard-to-predict unseen data takes a very small portion of the population, our design may lose its effectiveness. The success of RBL and PLA inspires us to find an optimal classifier for the loss P.

$**Response to weakness 1**$:

The main purpose of this paper is to assign a novel classification rule to the imbalanced classification. The classification rule focuses on the hard-to-predict subpopulation, which is different from the popular margin theory.

In order to justify the usefulness of our theoretical result of the generalized classification rule, we compare the proposed class-aware MSE loss to several baselines. The most useful baseline is SETF, which has nearly the same settings as ours: SETF and CAL both consider fixed classifiers, use only the within-class alignment part of the MSE loss function, and apply the same training schedule. They are comprehensively compared on (four datasets $\times$ three imbalanced ratios $\times$ two backbone networks) (24 settings in total). CAL outperforms SETF in most settings, indicating that assigning more directions for minor classes is a useful strategy aiming more than alleviating minority collapse, and the generalized classification rule is effective. In particular, under either loss, all samples from a class collapse to only one vector, so CAL and SETF should have the same capability of reducing the minority collapse. The significant difference comes primarily from the number of directions: In the supervised training, SETF only has one available direction (the classifier vector) for the feature to align while CAL uses information from multiple directions (the centers of each class).

We further examine the different choices of parameter $f$ and $\theta$; it is worth noting when $\theta = \frac{\pi}{2}$, i.e. the centers are all identical to their corresponding classifier vector, and the performance is no better than SETF. This observation shows reweighting alone does not improve the classification, thus our method is different from reweighting methods.

There are comparisons to other classical methods, the comparable performance (or the marginal improvement) implies fixing classifier has the potential to be developed further and applied in practice.

$**Response to question 1**$: The 3D illustration (Fig.1) of the multi-center frame is presented in the attached pdf. Imagine that all solid-line vectors are orthonormal vectors, among which the red and yellow ones are the $w_1$ and $w_{2}$ in the definition 3.2 which corresponds to class 1 and class 2. The solid green and solid blue are the augmented vectors associated with $w_1$ and $w_2$ resp. Then the green dashed-line vectors are the centers of class 1 and the blue ones are the centers of class 2. Since the classes has more than one center in general, we call them multiple centers.

$**Response to question 2**$: We apologize that $\tau$ is not claimed in the main article before being used. We consider long-tail image data which is generated from imbalance sampling from the data including cifar10, cifar100, SVHN, and STL10. $\tau:= \frac{n_{min}}{n_{max}} \leq 1$ is the imbalance ratio that describes the exponential decay of class sample size $n_k$, more precisely, let $n_1>n_2>\ldots>n_K$, then $n_k = n_{k-1}\tau^{\frac{1}{K-1}}$; $\theta$ is defined in definition 3.2, which is the angle between $\mathbf{w}_{k}$ and its associate centers $\mathbf{w_j^{(k)}}$. Please see more discussions of the experiment settings in Appendix F and the 3D illustration in the attached pdf.

$**Reponse to question 3**$: The logic here is, that the paper finds that a model needs more directions for minor classes (General Classification Rule, GCR), and then proposes loss P and Cosine Loss that lead to a surrogate of the GCR. In particular, we design the MSE-type loss to fit the purpose. It is highly possible to find an analogous conclusion under CE loss or other popular loss [1] if we can figure out the global optimal solution to the corresponding UFM under such losses. Our work demonstrates the effectiveness of the UFM analysis and the methodology can be useful to the model design: given the specific classification rule or other purposes, the joint structure of classifier and feature can be designed.

$**Response to question 4:**$

The Discovery of NC phenomenon [2] and the Classic analysis of UFM [3] tell us that NC is model-agonistic conditional on the high expressivity of the deep neural networks. What we know is NC will not happen if the network does not interpolate the data. We show NCMC occurs through UFM analysis, and it can be verified on loss P and the Cosine Loss. Fig.2 in the attached pdf shows the NC on different backbones. It is observed that ResNet, Densenet, and VGG have more severe collapse than LeNet, a relatively small model that hardly interpolates cifar10 in 200 epochs. However, formulating a complete theoretical justification of the occurrence of either NC or NCMC during SGD training is a hard yet ongoing problem in deep learning theory.

[1] Zhou et al. Are All Losses Created Equal: A Neural Collapse Perspective. NeurIPS 2022.

[2] Papyan et al.. Prevalence of neural collapse during the terminal phase of deep learning training. PNAS, volume 117, 2020.

[3] Zhu et al.. A geometric analysis of neural collapse with unconstrained features. NeurIPS 2021.

Thank you for the recognition of our contribution.

$**Response to the weakness**$:

1. thank you for your advice. We think it is a good idea to move proposition B.1 to the Main Result section and add more explanations of $\tilde{w}_j^{(k)}$. In particular, the set of $\tilde{w}_j^{(k)}$’s form an orthogonal frame (vectors are mutually orthogonal and have unit norm).

2. you are right about the typos, we will check over the formulas and words to guarantee the readability of this paper.

3. Thank you for the recommendations. The citations are recent literature on imbalanced learning and neural collapse. They are closely related to our work, and we will discuss their methods in the introduction and related work.

$**Response to Question 1**$:

We study UFM for the following two reasons:

1. Deep models are generally hard to analyze due to their intricacy and diversity. UFM offers an intuitive explanation of what representation has been learned and how the classification is carried out.

2. Because of the simplicity of UFM, it can be used to help design special representation structures and decision rules of interest, and the conclusions based on UFM are easy to verify.

The underlying assumption of the effectiveness of UFM is the high expressivity of the deep neural networks. The conclusions such as the optimal structure of the classifier and the large margin analysis at the terminal phase of training based on UFM can be generalizable when the model is overparameterized and easily interpolates the data.

Despite the power of UFM, it is no more than an expedient approach. Completely understanding the neural collapse and its connection to other deep learning phenomenons such as benign overfitting and grokking demands deeper theories.

Thank you for your review.

$**Response to weakness 1**$:

As far as we see, the proposed CAL has marginal improvement on cifar10 compared to ARBloss, but clearly outperforms other classical methods in comparison; it also has non-negligible improvements on cifar100. The experiment shows that CAL is effective for the fixed classifiers under MSE loss, which matches our theoretical analysis from the perspective of Neural Collapse. There are quite a few SOTA results in imbalanced learning that outperforms our method, but our primary contributions are more than the performance improvement:

(i) We formalize a novel principle we call “generalized classification rule” that focuses on the hard-to-predict subpopulation of the underlying data distribution, which is a different perspective compared to the popular margin theory and minority collapse. This rule claims that more directions are needed for minor classes in the imbalanced classification. The analysis is inspired by the recently discovered neural collapse phenomenon that for a highly expressive neural network, the classifier and penultimate layer feature has symmetric structures.

(ii) We design the loss that leads the model to a state where the analysis in (i) is effective. In particular, since directly using GCR rule results in gradient vanishing and quick overfitting, we propose to study the surrogate objective P. We perform a theoretical analysis based on unconstrained features model to show that P leads to a neural collapse where the original classifier can be considered a representative of the multiple directions. In practice, we use Cosine Regression loss with a fixed classifier for data training. In contrast to P, Cosine regression loss has similar neural collapse types according to our theory (corollaries for fixed W), but is simpler and easier to train.

(iv) The theoretical analysis is justified. The baseline SETF has nearly the same setting except that we use multiple directions for a class and normalize the features. Our method outperform SETF under most settings, which indicate that the information from multiple directions is useful for the imbalanced classification. This paper demonstrates the power of neural collapse analysis: through Unconstrained feature model we are able to design specific convergence pattern of the classifier and feature.

In short, our work can provide several insights in imbalanced learning and general model design, and we hope the reviewer can re-evaluate our contributions.

$**Response to weakness 2**$:

a) Indeed, the paragraphs in line 43-63 refer to several closely related works to our paper, but they are also the inspiration of our work and serve as the intro to the presentation of our motivation and contributions in the face of the issue that NC does not offer enough information of the underlying data distribution. Since our narrative is not effective according to the reviewer, we will highlight our challenge and motivation more concisely in the revision and offer more discussions about these references in “Related Work” section.

b) We are sorry that there are 4 abbreviated words between line 73 and line 77: UFM (unconstrained feature model), NCMC (Neural Collapse to Multiple Centers), RCR (Regular Classification Rule) and GCR (Generalized Classification Rule). We will reduce the frequency of abbreviate names per sentence in the revision. The whole paper will be carefully polished for high readability.