Thanks to reviewer RW5r for the comments!

Response to “Experimental Designs Or Analyses”: Experiments on Transformer

To further validate the advantages of BCE over CE, we trained ViT [1] and Swin Transformer [2] using BCE and CE on CIFAR10 and CIFAR100. Similar to the experimental setting in Section 4.1 of the paper, we conducted two groups of experiments: 1) with a fixed $\lambda_b=0$, setting the mean of the initialized classifier biases to 0, 1, 2, 3, 4, 5, 6, 8, and 10, respectively; 2) with varying $\lambda_b = 0.5, 0.05, 5\times10^{-3}, 5\times10^{-4}, 5\times10^{-5}$, and $5\times10^{-6}$, respectively, setting the mean of initialized classifier biases to 10. After the training, we visualized the distributions of unbiased positive and negative decision scores and classifier biases using violin plots for the different ViTs and Swin Transformers. One can find the results through the following anonymous link: https://anonymous.4open.science/r/BCE-vs-CE-6F45/

From the results, we obtained the conclusions similar to those from CNNs: for models trained with CE, the final classifier biases have no substantial relationship with the unbiased decision scores of the sample features, which primarily depend on their initial values and regularization coefficients $\lambda_b$. In contrast, for models trained with BCE, regardless of the model or optimizer used, there is a clear correlation between the final classifier biases and the unbiased positive and negative decision scores. These results indicate that the classifier bias of BCE substantially affects the positive and negative decision scores, thereby affecting the compactness and distinctiveness of the sample features, which is consistent with the analysis in the paper.

[1] A. Dosovitskiy, et al, An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR2021

[2] Z. Liu, et al, Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, ICCV2021

Response to “Weaknesses”:

1. theoretical analysis on imbalanced scenarios

On imbalanced datasets, the theoretically analyzing for neural collapse of loss functions is a more challenging task. Currently, the most theoretical analyses of loss function with neural collapse are conducted on balanced datasets, such as [1,2,3,4,5]. Some theoretical works [6] on neural collapse even simplify a class of sample features into a single vector. Currently, we only found few papers [7,8] that rigorously analyze the neural collapse for loss functions on imbalanced datasets. We are also investigating the neural collapse of BCE on imbalanced datasets, and we hope to theoretically address this issue in the near future.

[1] Z. Zhu, et al, A Geometric Analysis of Neural Collapse with Unconstrained Features, NeurIPS 2021.

[2] J. Zhou, et al, Are All Losses Created Equal: A Neural Collapse Perspective, NeurIPS 2022.

[3] M. Munn, et al, The Impact of Geometric Complexity on Neural Collapse in Transfer Learning, NeurIPS2024

[4] P. Li, et al, Neural Collapse in Multi-label Learning with Pick-all-label Loss, ICML 2024

[5] J. Jiang, et al, Generalized Neural Collapse for a Large Number of Classes, ICML 2024

[6] J. Lu and S. Steinerberger, Neural Collapse Under Cross-entropy Loss, Applied and Computational Harmonic Analysis, 2022

[7] C. Fang, et al, Exploring Deep Neural Networks via Layer-peeled Model: Minority Collapse in Imbalanced Training, PNAS 2021

[8] H. Dang, et al, Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Features Mode, ICML 2024

2. decision scores

Although decision scores do not directly measure the intra-class compactness and inter-class differences of features, they can indirectly reflect these two properties by anchoring to the classifier vectors $[w_k]_{k=1}^K$. In practice, the loss functions based on decision scores, such as CE and focal loss, etc., are prevalent in enhancing the feature properties. Therefore, this paper relies on decision scores to analyze CE and BCE. Currently, to fill the gap between "decision score" and “feature property”, we are theoretically analyzing CE and BCE, which directly measure the inner product or cosine similarity among sample features in contrastive learning.

Response to “Other Comments Or Suggestions”: Typos

Thanks to the reviewer for pointing out the typos in the paper. We have conducted a thorough review to avoid such issues in our manuscript.

Thanks to reviewer bxWz for the comments!

Response to “Experimental Designs Or Analyses”: Experiments on Transformers

As required by Reviewers bxWz and RW5r, we train ViT and Swin Transformer using BCE and CE on CIFAR10 and CIFAR100, to further validate the advantages of BCE over CE. The experimental setting and conclusions can be referenced in our response to reviewer RW5r. The results can be found through: https://anonymous.4open.science/r/BCE-vs-CE-6F45/

Response to “Supplementary Material”:

We submitted a 24-page PDF file as supplementary material in the OpenReview, which includes the metrics used in our experiments, additional experimental results, and detailed proofs of the theorems presented in the paper.

Response to “Weaknesses”:

1. t-SNE visualizations for the feature distributions

We thank the reviewer for suggesting the use of t-SNE visualization to compare the feature distributions of CE and BCE. Using t-SNE, we dynamically demonstrate the feature distributions extracted by ResNet18 trained with BCE and CE on CIFAR10, from the first epoch to the 30th epoch. One can find the results through the following anonymous link: https://anonymous.4open.science/r/BCE-vs-CE-6F45/

Although both CE and BCE can lead to neural collapse, ultimately maximizing the intra-class compactness and inter-class differences of the features, BCE leads to neural collapse more quickly during the feature learning. We provide a static display of the features from epochs 12 and 13 for both CE and BCE. One can observe that, compared to CE, the features from BCE at these two epochs are already distributed in distinct regions, within a more compact distribution for each class. In contrast, CE shows unclear boundaries between features of different classes, and the feature distribution within the same class is relatively loose.

2. CE and BCE using cosine similarity and margin

When using cosine similarity as the unbiased decision score, the CE loss still couples the positive and negative decision scores of every sample within one Softmax, constraining their relative values. In this regard, it is fundamentally no different from the original CE, which uses the inner product as the unbiased decision score.

When an angular margin is added, the Softmax-based CE requires that there be at least a margin between the positive and negative decision scores for each sample. Although this margin is consistent across all samples, only when the marginal intervals between the positive and negative decision scores of all samples overlap, a unified threshold that can distinguish the positive and negative decision scores for all samples exists, to ensure that the positive decision scores of all samples are at a uniformly high level (as we have explained in the response to reviewer eGz4 regarding the connection between ``uniformly high level’’ and intra-class compactness), while their negative decision scores are at a uniformly low level. In total, introducing the margin can indirectly enhance the intra-class compactness and inter-class disparity of sample features. In contrast, as explained in Eq. 16 of the paper, BCE uses the unified parameters (i.e., classifier biases) to directly constrain the decision scores of samples within the same class to be at uniformly high level or uniformly low level.

In practice, although the gains brought by an appropriate margin to CE are generally higher than the gains of BCE compared to CE, the margin strategies can also be integrated into BCE for higher performance. This is discussed in more depth with experimental validation in SphereFace2 [1] and UniFace [2], both of which take BCE-based loss functions with cosine similarity and margin, and they achieved better face recognition performance than CE-based ArcFace, A-softmax, and CosFace.

[1] Wen, Y., et al. SphereFace2: Binary classification is all you need for deep face recognition. ICLR 2022.

[2] Zhou, J., et al. UniFace: Unified cross-entropy loss for deep face recognition. ICCV2023.

Response to “Comments Or Suggestions”: experiments with other architectures:

We sincerely appreciate the reviewer for suggesting us to validate the advantages of BCE over CE across more methods and model architectures. We have conducted preliminary explorations of the advantages of BCE over CE using ViT and Swin Transformer. We believe that the benefits of BCE are universal, and in the future, we will consider using BCE in methods including visual prompt tuning and LoRA.

Response to “Questions For Authors”: LTR

LTR is short for long-tailed recognition, which refers to the recognition tasks on imbalanced long-tailed datasets. We apologize for not including its full name in the paper, and we will add it in the revised manuscript.

Thanks to reviewer eGz4 for the comments!

Response to “Claims and evidence”: feature properties on ImageNet

On the validation set of ImageNet, we calculated the feature properties for ResNet50, ResNet101, and DenseNet161 trained by CE and BCE in Table 3 of the paper, and the results are presented in the table below.

In the table, $\mathcal E_{com}$ and $\mathcal E_{dis}$ stand for the intra-class compactness and inter-class distinctiveness, respectively. From the table, one can find that although the more classes and higher complexity result in a less significant gain in feature properties for BCE, it still enhances the feature properties extracted by the model in most cases compared to CE. Only the inter-class distinctiveness of the ResNet101 trained by BCE has decreased, while the compactness and distinctiveness of other models trained by BCE have improved.

In addition to the above results on ImageNet, we also compared BCE and CE using ViT and Swin Transformer. Please refer to our responses to reviewers bxWz and RW5r.

Response to “Essential References Not Discussed”:

We thank reviewer eGz4 for providing three new references. The first paper analyzes the neural collapse (NC) of the MSE loss, while the latter two investigate NC in contrastive learning. These references are indeed relevant to our research and will help advance our theoretical analysis of the BCE in contrastive learning. Once again, we appreciate the reviewer’s suggestions, and we will cite these papers.

Response to “Weaknesses”:

1. Feature properties on ImageNet

In the response to “Claims and evidence”, we have presented the feature properties of ResNet50, ResNet101, and DenseNet161 trained with CE and BCE on ImageNet. Although ImageNet has more classes and higher complexity, the results shows that the features extracted by the models trained with BCE exhibit better intra-class compactness and inter-class distinctiveness.

2. BCE and CE with weight decay

The core argument of our paper is that the classifier bias in BCE plays a significant role in the deep feature learning, while the bias in CE does not, which helps BCE learn features with better properties. In Section 3.2, we theoretically demonstrate this viewpoint through the neural collapse of BCE and CE. In Section 3.3, to provide a simple and intuitive understanding of the above argument, we did not use the weight decay; however, in both the theoretical proof (line 162) in Section 3.2 and the experimental validation (line 258, line 305, and line 322) in Section 4, we did employ weight decay.

Under the same theoretical framework and experimental setting, the theoretical analysis and experimental results in the paper both indicate that when using weight decay, BCE achieves better feature properties and classification results than CE in most cases.

Response to “Other Comments Or Suggestions”: uniformly high level

"Uniformly high level" refers to the situation where the values expressed by the positive (unbiased) decision scores for different samples are consistently at a high level. For the class $k$, it means that the inner products between the different sample features $[h_i^{(k)}]_{i=1}^{n_k}$ and their corresponding classifier vector $w_k$ are all at high level. We believe that, in this case, the sample features within the class exhibit good intra-class compactness.

To understand this, in first, we incorporated the weight decay in the theoretical analysis of the paper, ensuring that the L2 norms of the sample features do not increase indefinitely during the training. The theorems in the paper have also indicated that the L2 norms of different feature vectors will converge to a fixed value. Secondly, as $‖w_k-h^{(k)}‖_2^2 = ‖w_k ‖_2^2 + ‖h^{(k)}‖_2^2 - 2w_k^T h^{(k)},$

for different sample features of class $k$, when they have the same norm, the large unbiased positive decision scores $\big[w_k^T h_i^{(k)}\big]_{i=1}^{n_k}$ implies that they are close to the classifier vector $w_k$. Therefore, if all the unbiased positive decision scores of different sample features of class $k$ are at a uniformly high level, it means all of these features are all close to $w_k$, resulting in small distances between them and thus high intra-class compactness. Conversely, if there is an obviously difference in the distances of the different sample features from $w_k$, it can be reasonably inferred that there will also be considerable differences in the distances between these features, leading to lower intra-class compactness.