Multiplicative Logit Adjustment Approximates Neural-Collapse-Aware Decision Boundary Adjustment

Q2: Why did you create imbalanced versions of balanced datasets instead of using inherently imbalanced datasets? For example, datasets that exhibit natural class imbalances could offer more realistic evaluations.

A6: We utilized CIFAR10/100-LT created from CIFAR10/100 because CIFAR10/100-LT are standard benchmark datasets in LTR. Many LTR studies, including ALA and MLA [1, 2, 3], have employed these datasets for their experiments.

In addition, we followed [3] by incorporating experiments on the Helena dataset, which is inherently imbalanced with $\rho \simeq 40$, as outlined in Appendix F.1.3. The consistent results across both synthetic and naturally imbalanced settings underscore the effectiveness of our method in diverse scenarios.

To highlight this, we added the following sentence to L399 in Section 5.1:
"We can also show the effectiveness of the method on real data using Helena because this is an inherently imbalanced dataset with $\rho \simeq 40$."

Q3: Although the paper focuses on post-hoc LA methods, did you compare the results with loss-function-based approaches? How does MLA's performance compare to other logit adjustment methods?

A7: This paper primarily centers on theoretical analysis, with the theoretical findings validated quantitatively through experiments. Comparing our method with other methods would not yield insights directly relevant to the theoretical contributions of this work. Including such comparisons might introduce experimental results that detract from the central narrative of our paper, potentially diminishing its coherence. It is uncommon in machine learning conferences to present experimental results beyond this type of quantitative validation. For example, several papers that focus on theoretical analysis without comprehensive comparisons [3, 4] have also been accepted at ICLR. Therefore, underrating this work on the grounds of not including such comparisons would compromise the fairness of the review process.

Nonetheless, we conducted supplementary experiments to provide additional context and clarify the relationship with loss-function-based ALA as a highly relevant method. Specifically, we evaluated a loss-function-based ALA with $\gamma_+ = 1$ and compared its performance with post-hoc LA. All other experimental settings followed those described in Section 5.1.

The results for CIFAR10/100-LT datasets are summarized below. As shown, the loss-function-based ALA achieves performance comparable to post-hoc ALA. This demonstrates that MLA achieves accuracy equivalent to, or better than, loss-function-based ALA.

CIFAR100-LT

CIFAR10-LT

In making these revisions, we believe the paper has achieved a higher level of rigor and clarity. We hope these changes address your concerns and align with your expectations for an even more impactful contribution to the field. If you have any further suggestions or specific areas you believe could benefit from additional attention, we welcome your guidance. Your expertise is invaluable to us, and we are committed to delivering a paper that meets the highest standards. If there are no further corrections, we kindly request that you raise the score so that this paper can be accepted.

[1] Long-tail learning via logit adjustment, Menon+, ICLR2021 Spotlight
[2] Adjusting decision boundary for class imbalanced learning, Kim+, 2019
[3] Exploring Weight Balancing on Long-Tailed Recognition Problem, Hasegawa+, ICLR2024
[4] Improved Sample Complexities for Deep Neural Networks and Robust Classification via an All-Layer Margin, Wei+, ICLR 2020

We sincerely appreciate your thorough review of our paper and your positive feedback. Your insights have been invaluable in shaping the direction of our work. We are grateful for the time and effort you dedicated to providing constructive comments. We would like to address some of the points raised in your review and provide additional context or clarification that we believe could contribute to an even more favorable evaluation.

W1: While the theoretical concepts are accurately communicated, the writing could be improved by breaking up overly long sentences to enhance readability (e.g., lines 098-104).

A1: We have split several long sentences in the paper to improve readability. For example, we have revised the sentence in question to describe the four NC phenomena using a bulleted list format as follows.

W2: The results in Tables 6-10 are intriguing, but more explanation is needed to clarify the implications of "Many," "Medium," and "Few" categories. Additionally, the reasons for the Baseline outperforming others in the "Many" category should be discussed in more depth.

A2: The terms "Many," "Medium," and "Few" represent groupings of classes based on the sample numbers, and naively trained models (Baseline) tend to produce outputs biased toward "Many" classes (head classes). Detailed explanations of this grouping are already provided in Appendix F.1. Specifically, classes with a large number of samples are categorized as "Many," those with fewer samples as "Few," and the remainder as "Medium." In Tables 6–10, the "Many," "Medium," and "Few" categories indicate the mean accuracy restricted to each respective class group.

The primary goal of LTR is to increase the overall average accuracy across all classes. Naive training methods result in biased outputs favoring head classes ("Many" classes), which leads to lower overall accuracy ("Average"). Methods such as the 1vs1adjuster and post-hoc LA aim to enhance the accuracy of tail classes ("Few" classes) and improve the overall average accuracy.

To clarify this further, we have added the following explanations to Appendix F.3 regarding the experimental settings and results:
"In addition to the overall average accuracy (Average), we also report the average accuracy for the 'Many,' 'Medium,' and 'Few' categories. Refer to Appendix F.1 for details on these categorizations."

"In all results, the Baseline shows higher accuracy for the 'Many' category, as naive training on long-tailed data results in outputs biased toward head classes. The 1vs1adjuster and post-hoc LA methods successfully adjust for this bias by increasing the accuracy of 'Few' classes, thereby improving the overall average accuracy. As shown in Section 5.3, MLA achieves nearly the same average accuracy as the 1vs1 Adjuster across all datasets. Additionally, MLA outperforms ALA in terms of average accuracy for CIFAR100-LT."

W3: The descriptions for tables and figures are too brief, making it difficult for readers to grasp the full context of the results. Adding more detailed captions would help in understanding the findings.

A3: We have added explanations for the figures and tables. For instance, we have inserted the following sentence in the caption of Table 1:
"For models that achieve 100% training accuracy, $\gamma_\times^\*$ and $\gamma_{\mathrm{1v1}}^\*$ are slightly higher than 0.5."

W4: The paper contains numerous formal definitions and formulas, which can disrupt the reading flow, especially for readers who are not deeply familiar with this specific research area.

A4: To improve the flow of the paper, we have included several supplementary explanations. For example, at L189, we have added the motivation for introducing the Rademacher complexity as follows:
"To evaluate the generalization performance of $\mathcal{H}_l$, we use the Rademacher complexity."
Additionally, at L237, we have included an intuitive explanation of angular bound probability as follows:
"The angular bound probability can also be seen as concentration of the class $k$ features."

Q1: Why did you primarily focus on ResNet models for the image analysis? Were there specific reasons for this choice over other architectures?

A5: We selected ResNet as our primary architecture because it is a widely recognized benchmark in LTR. Numerous LTR studies, including ALA and MLA [1, 2, 3], have utilized ResNet for their experiments.

Additionally, in line with [3], we conducted experiments using an MLP model on the Helena dataset (as detailed in Section 5.1). A similar performance was observed in this experiment. This indicates that our findings are not confined to a specific model architecture.

We sincerely appreciate you for your valuable time and constructive feedback. We would like to address some of the points raised in your review and provide additional context or clarification that we believe could contribute to your acceptance of our paper. Revised sections are indicated in red in the PDF.

Before addressing each specific point, we would like to reiterate that our research centers on the theoretical analysis of post-hoc logit adjustment (LA) within the framework of long-tailed recognition (LTR). This method is particularly advantageous as it requires no specialized training; it only modifies the logits during inference on a conventionally trained model to enhance accuracy. Despite its straightforward implementation and low computational cost, this approach yields significant accuracy improvements, making it highly applicable to real-world scenarios. Notably, additive LA (ALA) [1] was featured as a Spotlight paper at ICLR 2021 and has received over 700 citations. Thus, we believe that theoretical contributions in this area hold substantial potential to advance the field of LTR.

We would like to address the key points:

W1: Motivation: The paper fails to establish a compelling motivation for the study. The claim that "MLA has demonstrated significant empirical success" is not sufficiently substantiated. There is no systematic analysis of the current strengths and challenges of MLA in long-tailed recognition.

A1: Multiplicative LA (MLA) has been confirmed effective in various studies. First, MLA shows performance equivalent to or better than considerably influential ALA. Comparisons between MLA and ALA have been validated by [2] and our experiments on various datasets (see Section 5.3). Second, [6] has shown that the regions occupied by features for each class differ significantly in LTR, which supports the effectiveness of class-dependent temperatures, a method equivalent to MLA. Third, [7] and [8] indicate the efficacy of techniques similar to MLA beyond image classification. For instance, in object detection, MLA based on the frequency of object occurrences, known as inverse image frequency, has been proposed [7]. Similarly, in text retrieval tasks, a widely used method called inverse document frequency [8] adjusts the score function by multiplying it by the inverse of the overall word frequency. These suggest that MLA is not confined to classification in LTR but is a versatile technique applicable across various domains.

However, MLA has lacked a rigorous theoretical analysis until now. Our study addresses that gap by providing a theoretical guarantee of MLA optimality, thereby contributing significantly to the LTR field.

To better convey this, we have included the following explanations of the existing work in Appendix B1.

“MLA has been validated for its effectiveness across various studies. Hasegawa & Sato (2023) demonstrated that MLA performs on par with or better than ALA under realistic conditions. Additionally, Ye et al. (2022) highlighted significant differences in the feature space occupied by each class in LTR, confirming the effectiveness of class-dependent temperatures, which is equivalent to MLA. Beyond image classification, various studies suggest the utility of MLA-like adjustments in other fields. For instance, in object detection, Alexandridis et al. (2023) proposed an MLA variant based on object occurrence frequency, called inverse image frequency. Similarly, in text retrieval tasks, the inverse document frequency (Salton & Buckley, 1988), which adjusts the score function by multiplying the inverse frequency of words in a document, is widely used. Thus, MLA is not only applicable to LTR in classification problems but also serves as a fundamental technique with potential applications across a broad range of domains.”

Thank you for your reply and the revised score. We apologize for any misunderstanding caused by our previous explanations. To address the remaining concerns, we provide detailed responses below:

W1: “For instance, [2], [3] and [9] have been accepted without SOTA comparisons in ICLR. Therefore, a rejection of our work on this basis could compromise the fairness of the review process.” ~ less convincing in addressing the core concerns raised by myself and other reviewers regarding the robustness and applicability of the theoretical contributions in this work.

A1: We emphasize that comparisons with methods outside the scope of post-hoc LA are not directly relevant to this work. Our study is a theoretical analysis specifically within the post-hoc LA framework, and we conducted experiments to validate the theoretical findings. Accuracy comparisons with unrelated baseline or SOTA methods fall outside the theoretical scope of this work and would lack justification.

Moreover:

1. Comparisons with other baseline methods have already been thoroughly covered in the original paper of MLA [1], making further redundancy unnecessary.
2. Techniques such as contrastive learning, which require extended convergence times, are not feasible to evaluate within the time constraints of this review process.

However, we have already conducted additional experiments with loss-function-based baseline methods closely aligned with post-hoc LA. These results include comparisons between the loss-function version of ALA and various post-hoc LA methods. We show the results for CIFAR10/100-LT again in the table below. As shown, the loss-function-based ALA achieves performance comparable to post-hoc ALA. This demonstrates that MLA achieves accuracy equivalent to, or better than, loss-function-based ALA.

CIFAR100-LT

CIFAR10-LT

W2: “The larger number of classes and samples per class align closely with the conditions assumed in our theory, making additional experiments less critical.” less convincing in addressing the core concerns raised by ~  other reviewers regarding the robustness and applicability of the theoretical contributions in this work

A2: Our theoretical analysis assumes a sufficiently large number of classes (addressed in Section 4.3) and a sufficiently large sample size per class (addressed in Section 4.2). These assumptions are inherently satisfied in large-scale datasets, which further enhance the applicability of our theoretical results in such settings.

Additionally, we have already addressed other concerns raised by the other reviewers. For instance, we have clarified that our theoretical framework is model-agnostic and remains valid even for naturally imbalanced datasets.

We deeply appreciate your continued attention to these issues. If the remaining concerns have now been resolved, we kindly request a further score adjustment to reflect this resolution. Should any additional questions remain, please do not hesitate to contact us.

[1] Adjusting decision boundary for class imbalanced learning, Kim+, 2019

We sincerely appreciate your time and thoughtful evaluation of our paper. Your positive feedback is genuinely encouraging, and we are grateful for the constructive comments that have undoubtedly contributed to improving our work. While we understand and value the favorable assessment, we would like to address a few points raised in your review where I believe additional context or clarification could potentially enhance the overall evaluation of our paper. Revised sections are indicated in red in the PDF.

W1: The iNaturalist dataset is a widely used large-scale long-tailed dataset. Conducting experiments on this dataset would be beneficial in examining the behavior of MLA, ALA, and the 1vs1 adjuster in a larger-scale real-world context. Additionally, testing on CIFAR-LT with more extreme imbalance ratios would further evaluate the effectiveness of the proposed method.

A1: We have conducted additional experiments on a more imbalanced CIFAR-LT dataset ($\rho=200$) and included the results in Appendix F.5. All other experimental settings were identical to those described in Section 5.1. Due to space constraints, we present only the average accuracy after hyperparameter tuning on CIFAR100-LT here. Please refer to Appendix F.5 for the comprehensive results. Similar to the case with $\rho=100$, the results demonstrate that MLA effectively approximates the 1vs1adjuster in all scenarios. MLA achieves higher mean accuracy than ALA on CIFAR100-LT in this case as well.

Regarding experiments on iNaturalist, we could not conduct these during the rebuttal period for the following three reasons. First, the computational cost in our environment is prohibitively high. For instance, a typical LTR experiment on iNaturalist requires training with a batch size of 500, which demands substantial VRAM or extensive GPU hours [1]. Second, on such large-scale datasets, the theoretical assumptions underlying our work are more likely to hold, reducing the necessity for these experiments. The larger number of classes and samples per class align closely with the conditions assumed in our theory, making additional experiments less critical. Finally, other LTR papers without experiments on iNaturalist [2], [3] have also been accepted at ICLR. For these reasons, including these experiments is unnecessary to achieve a high evaluation.

W2: The theoretical framework relies on some strict assumptions that may not always hold in practice, such as the assumption of conditions under which NC occurs, which may not be valid in highly imbalanced real-world datasets. Similarly, the assumption that ψ→π/2 does not hold when K is small.

A2: The experimental results suggest that MLA effectively approximates the 1vs1Adjuster even under non-ideal conditions. In the experiments described in Appendix F.2, MLA and the 1vs1Adjuster exhibit similar decision boundaries, even in scenarios where NC does not fully hold (e.g., Helena and ImageNet-LT). Furthermore, Section 5.3 demonstrates that MLA performs nearly identically to the 1vs1Adjuster in terms of accuracy, even in cases with a small number of classes, such as CIFAR10-LT, indicating that these conditions do not pose practical issues.

Additionally, we consider these conditions to be of relatively low importance. By definition, "long"-tailed recognition is characterized by scenarios involving a large number of classes [4]. Moreover, some methods like ETF Classifier [5] promote NC even in imbalanced data settings (as explained in L112–L115). Considering these factors, we believe that the assumptions we made are reasonable.

To better convey this point, we revised the sentence starting from "In LTR" in L36 as follows:
"In LTR, there are many classes, and model predictions are often biased toward head classes."

Additionally, we added the following sentence to L278:
"This is a reasonable assumption in LTR (Yang et al., 2022a)."

W3: There are some typos, such as on line 373, where "the optimal decision boundaries during inference is effected by" should be corrected to "the optimal decision boundaries during inference are affected by".

A3: We have corrected the points you pointed out. Thank you for your comments.