MONICA: Benchmarking on Long-tailed Medical Image Classification

We sincerely appreciate the reviewer’s positive feedback in support of our work. We would also like to thank the reviewer for their valuable efforts and insightful comments, which have greatly contributed to improving our manuscript.

In response to the feedback, we have implemented significant updates, including:

• Providing an anonymous repository for our codebase: https://anonymous.4open.science/r/MONICA-153F/README.md.
• Extending support for more advanced backbones, such as ViT, Swin Transformer, ConvNext, and RetFound, alongside detailed experiments to evaluate their impact.
• Expanding evaluation metrics to include AUROC, AUPRC, and F1 Score, enabling a more comprehensive assessment of model performance.
• Providing detailed responses and analyses regarding the extended tasks, such as OOD detection.

The detailed responses are outlined below:

Writing can be improved throughout.

A: We sincerely appreciate the detailed feedback and will thoroughly revise the manuscript to address all the highlighted issues and ensure clarity, consistency, and professionalism throughout.

It is possible that choosing a fixed set of hyperparameters across methods unintentionally advantages certain methods. Ideally, one could argue that each method should be individually tuned on each task;

A: Thank you for raising this important point. We acknowledge that using a fixed set of hyperparameters across methods may inadvertently favor certain approaches. While ideally, each method should be individually tuned for each task to maximize its potential, this can introduce inconsistencies and make comparisons less fair, as tuning procedures vary widely across methods and datasets. To strike a balance, we chose to use standardized hyperparameters as a baseline for fair and reproducible comparisons, but we also encourage researchers to explore task-specific tuning using our codebase, which provides the flexibility to adjust hyperparameters for individual methods.

We will also include this point as one of the limitations of our work in the discussion section.

Uncertainty estimates should be provided (e.g., bootstrapped confidence intervals or standard deviations over multiple runs);

A: Apologies for the missing details. We ran each methodology on each dataset three times using different random seeds. In the updated version, we will include error bars (e.g., rounded to one decimal place) for greater clarity. Additionally, we plan to create a website to provide more comprehensive results, offering a better overall understanding of this benchmark.

Multiple performance metrics should be provided (e.g., AUROC).

A: In the updated codebase, we expanded the evaluation metrics to include AUROC, AUPRC, and F1 Score, providing a more comprehensive assessment of model performance.

Can the authors provide a summary of practical suggestions for which methods to use in a few sentences near the Conclusion?

A: As discussed in the experimental section, we conducted an in-depth analysis from multiple perspectives, including the use and summarization of re-sampling strategies, employing appropriate data augmentation (e.g., MixUp) to enhance the backbone's representation capability, improving classifiers to mitigate distribution bias, and the necessity of two-stage training methods. Through these analyses, we not only derived many significant conclusions but also identified potential improvement directions for each component, providing a broader design space for future researchers to develop new methods. At the same time, we highlighted the importance of the intrinsic characteristics of medical data, such as incorporating prior knowledge as auxiliary information.

The key conclusions are summarized in the following two paragraphs before the Sec. Conclusion:

• What makes an essential strong baseline?: We synthesize our findings into actionable insights for building robust baselines in long-tailed medical image classification.
• Integration of medical domain prior knowledge: We emphasize the importance of incorporating domain knowledge (e.g., hierarchical relationships) as a key driver for further improvements in medical imaging tasks.

We have also revised structure and hope this will address the concerns.

4.2.1 Overall Performance

• Overall performance evaluation
• Curse of shot-based group evaluation

4.2.2 Decoupled Components Discussion

• Effectiveness of re-sampling across SOTAs
• Use two-stage training as a general paradigm
• ...

4.2.3 Extended Tasks

• LTMIC improves out-of-distribution detection
• Using imbalanced validation dataset for checkpoint selection
• ...

4.2.4 Overall Conclusion

• What makes an essential strong baseline?
• Integration of medical domain prior knowledge

Sec. 5 Conclusion -> Sec. 5 Discussion

More explanations on Fig.4

A: Thank you for highlighting these points. We would like to provide further clarification on the purpose of using an imbalanced validation dataset for checkpoint selection and the reasoning behind our conclusions.

This approach addresses a key difference between long-tailed practices in natural images and medical images. In natural image datasets like CIFAR-LT and ImageNet-LT, the original data distribution is balanced, allowing researchers to construct balanced validation sets. This makes evaluation using metrics like loss value, macro accuracy, or group accuracy straightforward and fair. However, in medical imaging, creating a balanced validation set for checkpoint selection is extremely challenging due to the inherent imbalance in the data. Using an imbalanced validation set, while more challenging, better reflects real-world scenarios and aligns with the nature of long-tailed medical datasets.

Additionally, sampling strategies in long-tailed tasks can lead to instability during training. For instance, some methods, e.g, RW, rely on fixed sampling weights determined at the beginning of training, while others adjust sampling dynamically during training, e.g, focal loss, causing fluctuations in performance. Therefore, beyond measuring a method's absolute performance, its stability during training is also a critical factor to consider.

Given the lack of standardized evaluation methods in the community, this perspective ensures that checkpoint selection and performance evaluation are aligned with real-world data distributions, rather than relying solely on artificial or overly idealized validation settings.

Regarding Figure 4, we apologize for the oversight. The labels for the x and y axes should be AUROC value. We will revise the figure. We also note that the conclusion regarding GCL exhibiting lower fluctuations highlights the importance of stability for practical deployment, which we will make more explicit in the revised manuscript.

Other small questions

Avoid editorializing with value judgments: “benchmark is meticulously designed”; “we… develop a… well-structured codebase”; “our work provides valuable practical guidance”. Simply present your work and let the reader make these judgments!

A: Thank you for pointing this out. We will remove relevant claims.

“a common assumption in long-tailed learning is when the classes are sorted by cardinality in decreasing order” I’m not sure what this means or why this represents an “assumption”. I would just remove this sentence since it does not seem to be used later.

A: Some datasets were originally assigned labels from 0 to 9 (assuming there are ten classes), but these labels were not sorted by the number of samples in descending order. We would like to note that, for the sake of performance calculation, we have re-ordered the categories in the modified datasets. We provided scripts to present how we build this process in MONICA/utils/.

We hope the above discussion will fully address your concerns about our work. We look forward to your insightful and constructive responses to further help us improve the quality of our work. Thank you!

I might suggest including the rank of each method on a given task in all tables.

A: Thank you for your suggestion. Initially, we planned to rank the methods; however, as discussed in the "Curse of Shot-Based Group Evaluation", different scenarios prioritize different shot-based groups, making it challenging to claim that methods with higher average accuracy are universally superior. Therefore, we chose to highlight only the best-performing methods in bold and left room for further exploration of new evaluation metrics in future research. Additionally, we plan to incorporate a ranking function into the benchmark website in future updates to enhance usability.

More details on OOD detection task?

A: We advocate that OOD detection is an extension task that will interest researchers working on long-tailed problems, especially considering the challenges that long-tailed issues bring to OOD detection tasks. Mathematically, OOD detection typically relies on the model's predictive confidence $p(y | x)$ to differentiate ID and OOD samples.

OOD (Out-of-Distribution) detection is a task where a model is trained on a known dataset $D_{in}$ (in-distribution data) and aims to identify whether a given input $x$ belongs to $D_{in}$ or an unknown, unseen distribution $D_{OOD}$ (out-of-distribution data). The core objective can be explained mathematically as follows:

Objective of OOD Detection

Definition of In-Distribution and Out-of-Distribution:

In-distribution (ID) data: $D_{in} = \{ (x_i, y_i) \}_{i=1}^N$,

where $x_i \in X_{in}$ and $y_i \in Y_{in}$ are drawn from a distribution $P_{in}(x, y)$.

Out-of-distribution (OOD) data: $D_{OOD} = \{ x_j \}_{j=1}^M$,

where $x_j \notin X_{in}$ and $y_j \notin Y_{in}$, sampled from $P_{OOD}(x)$.

Predictive Confidence: OOD detection typically relies on the predictive confidence $p(y | x)$ of the model: $p(y | x) = \frac{\exp(f_y(x))}{\sum_{k \in Y_{in}} \exp(f_k(x))}$,

where $f_k(x)$ is the logit (pre-activation score) for class $k$, and $Y_{in}$ represents the set of known classes.

Decision Rule: The task is to define a decision boundary $tau$ such that: g(x) = \begin{cases} ID, & \text{if } \max_{y \in Y_{in}} p(y | x) \geq \tau, \\ OOD, & \text{if } \max_{y \in Y_{in}} p(y | x) < \tau. \end{cases} \

Here, $\tau$ is a threshold, often determined through validation on an auxiliary OOD dataset.

Evaluation Metrics: OOD detection is evaluated using metrics such as AUROC: $AUROC = \int_{0}^{1} TPR(FPR^{-1}(t)) \, dt,$ where TPR and FPR are the true positive and false positive rates, respectively.

In our cases, we have 1000 OOD samples from ImageNet as $D_{OOD}$, where $M$ = 1000, and test set from OrganAMNIST as $D_{in}$. All evaluated OOD methods will give a predictive probability of $p(y | x)$.

In long-tailed settings, this confidence distribution is skewed due to class imbalance.

For head classes (frequent classes), the model is well-calibrated, producing high confidence:

$p(y_{head} | x_{head}) ≈ 1.$

For tail classes (rare classes), the model often produces low and uncertain confidence due to insufficient training data:

$p(y_{tail} | x_{tail}) ≪ 1.$

Consequently, the confidence for tail-class ID samples may overlap significantly with that of OOD samples:

$p(y_{tail} | x_{tail}) ≈ p(y_{OOD} | x_{OOD}),$

leading to poor separation between ID and OOD samples.

In this context, methods designed for long-tailed problems adjust decision boundaries to account for class imbalances. This prevents the model from overly favoring head classes and ensures that tail classes are not confused with OOD samples. A well-calibrated model trained with long-tailed techniques is more likely to differentiate:

$p(y_{tail} | x_{tail}) > p(y_{OOD} | x_{OOD}).$

To conclude, Long-tailed problems in real-world scenarios aim to include as many categories as possible (e.g., rare diseases). However, there is still the potential for unknown categories to exist. The goal of OOD detection is to identify these unknown categories, thereby achieving broader information coverage. This aligns with the motivation of addressing long-tailed problems.

We will include the definition and provide clearer experimental details regarding this in our supplementary documents.

Thank you to the authors for the detailed rebuttal. To clarify one comment, my request for additional evaluation metrics was not meant with respect to the codebase but rather the paper itself: ideally include results with AUROC/balanced accuracy/another appropriate metric in the supplemental materials. I am aware that group-wise accuracy is the primary evaluation metric in the long-tailed learning setting, so I support its use throughout the main text; however, it is entirely possible that a different metric may tell a different story and lead to new interpretations than accuracy alone.

I still do not quite understand the last point regarding sorting labels or what "order" even means in this context. The only impact that the order of labels would have is on how one might index the list of labels to assess the subgroup of interest -- it does not impact the results whatsoever. I think this only causes confusion and represents a trivial implementation detail of performance metric calculation (whether to subset for consecutive elements of a list or not), unless I am misunderstanding something.

Overall, this is a strong submission and I will maintain my original score.

We thank the reviewer for the positive opinions and helpful suggestions.

We have conducted extensive updates and additional experiments based on the reviewers' valuable feedback. Specifically,

• We added an anonymous repository for our codebase: https://anonymous.4open.science/r/MONICA-153F/README.md
• We released the download link, train/val/test split and preprocessing codes for all the datasets.
• We added support for more advanced backbones, including ViT, Swin Transformer, ConvNext, and RetFound, and performed detailed experiments to evaluate their impact.
• We added the results for SSL methods.

The detailed responses are outlined below:

Main Questions:

This work doesn't introduce any new datasets or methods. It is a collection of datasets (multi class or multi label) that are already publicly available without justifications as they are many other such kind of long tail datasets available. Also, they have changed some of the original datasets, it would not be useful if they don't share the modified datasets.

A: We have released the download link, train/val/test split and preprocessing codes for all the datasets. We also provided a guide for the users to build their own customized datasets for training. We hope this can address your questions.

They only tried ResNet for the tasks, would be nicer to make comparisons with other models.

A: Unless explicitly specified in the original paper that a particular backbone must be used (e.g., RSG uses ResNext), we choose ResNet as our backbone for the following reasons: the methods supported by our benchmark predominantly use ResNet as the backbone in their original papers. This is especially important for methods that require modifications to the network structure (e.g., the ResNet in SADE). Attempting to modify ViT in the same way as ResNet to achieve a similar motivation could introduce discrepancies, potentially leading to unfair evaluations.

However, we appreciate the suggestions from the reviewer to add more backbones with different architectures. Our updated codebase now supports more advanced backbones such as ViT, Swin Transformer, ConvNext, as well as relevant foundation models like RetFound for ophthalmology. We have provided corresponding alerts in the codebase for those fixed backbones proposed by the original paper. We completed training for ViT, Swin Transformer, and ConvNext on the ISIC dataset. Here, we present partial results, and the complete results will be included in the appendix.

The key observations:

• Overall, increasing the number of parameters does improve performance for most methods, particularly for MixUp. This is likely because MixUp, as a data augmentation and regularization technique, benefits from greater model capacity, allowing it to learn more robust feature representations.

• In terms of efficiency, as the level of imbalance increases, simply increasing model capacity shows diminishing returns, indicating the need to explore other strategies to enhance performance. Additionally, under the Pareto distribution sampling setting, the increase in imbalance significantly reduces the number of training samples, which may lead to overfitting in larger models. Therefore, the number of model parameters should be appropriately aligned with the amount of training data to achieve optimal performance.

Also the discussions on SSL models seem not supported by any data.

A: We apologize for the missing experimental results on SSL methods.

The following are the SSL results. Our codebase does not support all SSL methods directly; instead, the methods mentioned in other papers are pre-trained using their official code and then loaded into MONICA as initialized weights for further training.

Table 7. Experimental results on ISIC2019-LT (r=100) trained from different self-supervised methods.

As we discussed in our manuscript, we conclude this for several reasons: (1) The amount of medical imaging data in this study is limited, especially for a LTMIC setting, while SSL often relies on large quantities of unlabeled data. (2) SSL typically depends on specific hyper-parameter designs/tuning such as data augmentations, which are crucial to the second-stage fine-tuning. However, the lack of guidance in these aspects diminishes the possibility of achieving optimal practice.

In the future work, we will explore some SSL methods designed for the specific medical domain, e.g., Li et al. for ophthalmology [1].

[1] Li, Xiaomeng, et al. "Self-supervised feature learning via exploiting multi-modal data for retinal disease diagnosis." IEEE Transactions on Medical Imaging 39.12 (2020): 4023-4033.

There are some typos such as quotation markers etc, please correct.

A: Thank you for pointing this out. We will thoroughly review the manuscript to address any typos and ensure clarity.

We hope the above discussion will fully address your concerns about our work, and we would really appreciate it if you could be generous in raising your score. We look forward to your insightful and constructive responses to further help us improve the quality of our work. Thank you!

Dear reviewer gxGu,

We would like to know if our response has addressed your concerns and questions. If you have any further concerns or suggestions for the paper or our rebuttal, please let us know. We would be happy to engage in further discussion and manuscript improvement.

Thank you again for the time and effort you dedicated to reviewing this work.

Sincerely,

The Authors

The section on using an imbalanced validation dataset for checkpoint selection is unclear about its purpose. Figure 4 lacks labels for the x and y axes, making interpretation difficult.

A: Thank you for highlighting these points. We would like to provide further clarification on the purpose of using an imbalanced validation dataset for checkpoint selection and the reasoning behind our conclusions.

This approach addresses a key difference between long-tailed practices in natural images and medical images. In natural image datasets like CIFAR-LT and ImageNet-LT, the original data distribution is balanced, allowing researchers to construct balanced validation sets. This makes evaluation using metrics like loss value, macro accuracy, or group accuracy straightforward and fair. However, in medical imaging, creating a balanced validation set for checkpoint selection is extremely challenging due to the inherent imbalance in the data. Using an imbalanced validation set, while more challenging, better reflects real-world scenarios and aligns with the nature of long-tailed medical datasets.

Additionally, sampling strategies in long-tailed tasks can lead to instability during training. For instance, some methods, e.g, RW, rely on fixed sampling weights determined at the beginning of training, while others adjust sampling dynamically during training, e.g, focal loss, causing fluctuations in performance. Therefore, beyond measuring a method's absolute performance, its stability during training is also a critical factor to consider.

Given the lack of standardized evaluation methods in the community, this perspective ensures that checkpoint selection and performance evaluation are aligned with real-world data distributions, rather than relying solely on artificial or overly idealized validation settings.

Regarding Figure 4, we apologize for the oversight. The labels for the x and y axes should be AUROC value. We will revise the figure. We also note that the conclusion regarding GCL exhibiting lower fluctuations highlights the importance of stability for practical deployment, which we will make more explicit in the revised manuscript.

References

[1] Ju, Lie, et al. "Hierarchical knowledge guided learning for real-world retinal disease recognition." IEEE Transactions on Medical Imaging (2023).

[2] Wu, Tong, et al. "Distribution-balanced loss for multi-label classification in long-tailed datasets." ECCV 2020.

[3] https://github.com/Westlake-AI/openmixup

[4] Kang, Bingyi, et al. "Decoupling Representation and Classifier for Long-Tailed Recognition." International Conference on Learning Representations.

[5] Zhang, Yifan, et al. "Deep long-tailed learning: A survey." IEEE Transactions on Pattern Analysis and Machine Intelligence 45.9 (2023): 10795-10816.

[6] Li, Xiaomeng, et al. "Self-supervised feature learning via exploiting multi-modal data for retinal disease diagnosis." IEEE Transactions on Medical Imaging 39.12 (2020): 4023-4033.

We hope the above discussion will fully address your concerns about our work, and we would really appreciate it if you could be generous in raising your score. We look forward to your insightful and constructive responses to further help us improve the quality of our work. Thank you!

The analysis in Section 4.2 is poorly organized. Line 504 claims that "the most advanced long-tailed learning methods no longer focus on improving a single strategy," but this claim is not well-supported by the preceding analysis.

A: We sincerely apologize for the lack of clarity and structure in Section 4.2 of the initial submission. We have carefully revised this section to improve its organization, readability, and focus, ensuring that the discussions are logically connected and serve to provide actionable insights for designing better models. Below, we outline the revised structure and rationale behind the organization:

4.2.1 Overall Performance

• Overall performance evaluation: We begin with a discussion of the overall performance across methods and datasets to provide readers with a high-level understanding of the results.
• Curse of shot-based group evaluation: We delve into the challenges posed by shot-based group evaluation, highlighting the trade-offs between head and tail class performance and their implications for model design and constraints for real-world applications (e.g., sensitivity to rare diseases).

4.2.2 Decoupled Components Discussion

• Effectiveness of re-sampling across SOTAs: We discuss and summarize how different re-sampling strategies affect state-of-the-art methods.
• Use two-stage training as a general paradigm: Two-stage training is presented as an essential traning pipelines for the best performed methods in our benchmark, with observations on how it helps decouple backbone training from classifier adjustments. In this context, in the subsequent paragraphs, we focus on discussing solutions for improving feature representation training (e.g., MixUp and SSL) and enhancing classifier performance.
• MixUp can improve the feature representation: We discuss how MixUp enhances feature learning and its consistent benefits across various models.
• Dilemmas of self-supervised learning for LTMIC: We highlight the challenges and limitations of applying self-supervised learning (SSL) to long-tailed medical image classification (LTMIC).
• Modify classifier to reduce prediction bias: We discuss the role of classifier modifications in mitigating prediction bias, particularly for tail classes, and connect this to the broader two-stage training paradigm.

4.2.3 Extended Tasks

• LTMIC improves out-of-distribution detection: We explore the potential of LTMIC approaches in out-of-distribution detection tasks and their practical significance in medical imaging, e.g., low-quality images, non-medical images and images with unknown rare diseases.
• Using imbalanced validation dataset for checkpoint selection: We provide insights on why using an imbalanced validation set is more aligned with real-world scenarios and how it affects model selection.
• Multi-label classification is more challenging: We discuss the unique challenges of multi-label classification, as detailed in the responses above.

4.2.4 Overall Conclusion

• What makes an essential strong baseline?: We conclude our findings with actionable insights for building robust baselines in long-tailed medical image classification.
• Integration of medical domain prior knowledge: We emphasize the importance of incorporating domain knowledge (e.g., hierarchical relationships) as a key driver for further improvements in medical imaging tasks.

We hope that this revised structure and enhanced discussion will address the concerns.

Methods like ERM, cRT, and LWS represent, are not referenced properly. Section 3.2.3 does not fully align with the table.

A: ERM refers to Empirical Risk Minimization, which involves directly training on the original distribution using cross-entropy loss. cRT represents classifier re-training, and LWS refers to the learnable weight-scaling classifier; both methods are derived from Kang et al., 2020. [4] We will ensure the inclusion of the relevant references.

For consistency between Table 2 and Sec. 3.2.3, we have clarified this in Line 314 - 315: We introduced and implemented over 30 methods but only present results for the most relevant ones to avoid redundancy and maintain clarity. Methods with similar or suboptimal performance were excluded to highlight those that best support our primary findings in line with the study's main motivation. The complete results are available in the supplementary documents.

Lack of insights on different datasets. Why did the authors not include a domain-specific analysis?

A: In addition to the earlier discussion on overall performance, we also tested the impact of different data augmentation methods on the performance of various datasets. The current version of codebase supports RandAug and we plan to support some other augmentation codebase such as OpenMixup[3] in the future update.

Table 2. Results with Different Augmentation Techniques

The key observations:

• Using more complex data augmentation methods like RandAug and AugMix can further improve performance. For datasets with diverse image patterns, such as dermoscopy images (e.g., with color variations), the performance gains from applying complex augmentations are often greater compared to others such as radiology datasets.
• Cropping is a common method in data augmentation, but it may not be suitable for all medical datasets, as medical data often relies on specific regions of lesions for classification or diagnosis, such as in cases of diabetic retinopathy grading.
• Skin lesions are often characterized by certain specific local features, such as the shape of the lesion, irregular edges, and color distribution. By applying cropping, the model can focus more on the detailed features of the lesion area, reducing background interference and thereby improving classification or detection performance.

Finally, we would like to emphasize that incorporating prior knowledge in medical imaging can be very helpful in addressing long-tail problems. We acknowledge that one limitation of this paper is the inability to incorporate extensive medical knowledge for different datasets. We will address this point in the revised version of the paper.

We appreciate the reviewer’s constructive comments and valuable suggestions.

In response to the feedback, we have implemented significant updates and conducted additional experiments, including:

• Providing an anonymous repository for our codebase: https://anonymous.4open.science/r/MONICA-153F/README.md.
• Extending support for more advanced backbones, such as ViT, Swin Transformer, ConvNext, and RetFound, alongside detailed experiments to evaluate their impact.
• Expanding evaluation metrics to include AUROC, AUPRC, and F1 Score, enabling a more comprehensive assessment of model performance.
• Conducting additional ablation studies to analyze variations in input size, backbone comparisons, and augmentation techniques like RandAug and AugMix.
• Providing detailed responses and analyses regarding the extended tasks, including OOD detection and self-supervised learning.

We believe these updates address the concerns raised and significantly enhance the clarity and depth of our study.

The detailed responses are outlined below:

Motivation. The paper lacks sufficient justification for evaluating long-tailed problems specifically in medical imaging tasks...

A: Thank you for highlighting this concern. We appreciate the opportunity to clarify the motivation and address the fundamental differences between long-tailed problems in medical imaging and classical long-tailed learning tasks for natural images.

Medical Background

• Intrinsic Nature of Medical Imaging Data: Unlike natural image datasets, long-tailed distributions in medical imaging are a direct result of real-world prevalence rates (In this work, we consider an extreme challenging scenario with higher imbalance ratio under Pareto distribution). Rare diseases naturally appear less frequently, but their accurate diagnosis is critical in clinical settings due to their high-risk nature. This makes the long-tailed challenge in medical imaging inherently different, as it directly impacts patient outcomes and decision-making in healthcare.

• Importance of Rare Classes: In medical imaging tasks, rare classes often correspond to severe or life-threatening conditions. Misclassifying these rare cases can have dire consequences, unlike conventional tasks where tail classes may not carry the same level of significance. This necessitates methods that can effectively handle rare but critical samples, beyond merely improving overall performance. For example, in clinical scenarios, many use cases may prioritize sensitivity, whereas screening scenarios often focus on a balance between sensitivity and specificity. To address this, our updated codebase includes additional metrics such as AUROC, AUPRC, and F1 Score.

• Integration of Domain Knowledge: Medical imaging tasks often rely on incorporating hierarchical relationships among diseases (e.g., different stages of a disease) or prior clinical knowledge, which is rarely considered in conventional tasks. This additional layer of complexity requires adapted methodologies that leverage domain-specific insights, further distinguishing medical imaging from other domains.

Benchmark and Codebase

-Lack of Standardization in Dataset Settings: There is no unified standard for defining the degree of imbalance between head and tail classes in long-tailed datasets (e.g., imbalance ratio). Some studies use artificially generated long-tailed distributions (e.g., Pareto distribution), while others rely on real-world data distributions, making it difficult to directly compare results across studies. Even with the same dataset, differences in preprocessing methods (e.g., data augmentation, cropping, resolution adjustments) can lead to significant performance differences. The lack of standardized preprocessing protocols impacts the reproducibility of results.

-Evaluation Discrepancy: Conventional long-tailed tasks often use balanced test sets to evaluate models, focusing on overall accuracy or head-to-tail trade-offs. In contrast, medical imaging typically evaluates models on naturally imbalanced test sets to simulate real-world distributions. This difference shifts the emphasis to metrics like sensitivity, AUROC, and AUPRC, making certain methodologies unsuitable or less effective in the medical context.

-Empirical Evidence: Our comprehensive analysis across multiple datasets and methods highlights that existing long-tailed learning approaches do not always generalize well to medical imaging tasks. For example, methods like GCL or MixUp may show inconsistent results across datasets with varying levels of imbalance or complexity. These observations underline the need for tailored approaches specific to the medical domain.

We will incorporate these clarifications into the revised manuscript to strengthen the motivation and provide additional insights into the distinct challenges and requirements of long-tailed medical imaging tasks.

We sincerely appreciate the reviewers' valuable time and effort, especially considering the detailed, point-by-point constructive feedback provided. We deeply respect these insights and have made every effort to address each comment in our response. We have not avoided any questions raised by the reviewers and have instead tried to answer them as comprehensively as possible. Once again, we thank the reviewers for their patience in reading our detailed response.

Legality of data distribution and factual inaccuracies in dataset details

A: Thank you for pointing this out. We sincerely apologize for any misleading information regarding the data descriptions in the code repository.

We will take this concern very seriously. Providing high-quality public resources is a responsibility we fully acknowledge, and we are committed to addressing the identified issues in legality, accuracy, and implementation. We will thoroughly audit and revise the dataset information in the repository, ensuring all licenses and links are accurate. For example:

• Correcting CheXpert’s license details.
• Properly linking WILDS-Camelyon17 to its original source.
• Correctly identifying PathMNIST’s license.

Other Details: Except for CheXpert, the distribution of all other datasets for non-commercial purposes is legal. CheXpert, however, prohibits distribution without written permission. It should be noted that the initial code repository we provided did not include any redistributed CheXpert data; we only provided processing scripts for reference. Additionally, all NumPy files only contain sequences of filenames and did not modify any of the original data, including images and labels.

Additionally, we will include a clear disclaimer in the repository to guide users on license compliance and give proper credits for each dataset.

Code quality and design

A: We appreciate the detailed feedback on the code quality and design. Our goal is to make the codebase as accessible and extensible as possible, and we take these suggestions seriously. We have revised most of the redundant components and will include proper citations to clarify whether specific implementations are original or adapted. Additionally, based on the suggestion from @Reviewer FbkS:

Avoid editorializing with value judgments: “benchmark is meticulously designed”; “we… develop a… well-structured codebase”; “our work provides valuable practical guidance”. Simply present your work and let the reader make these judgments!,

we have removed statements claiming contributions regarding the codebase to avoid causing any potential confusion.

We also would like to emphasize that we do not see the publication of this work as the conclusion of the work on this code repository. Instead, we view it as a starting point. Our aim is to continuously maintain and improve the repository over time—for example, by incorporating new long-tailed learning methods, introducing emerging foundation models, supporting additional backbones, and exploring more diverse data augmentation techniques. While we acknowledge that the current code design has redundancies and limitations, it has been structured to allow for these future extensions. This is also reflected in the additional experimental results we provided during the rebuttal phase.

Table 4. ISIC2019-LT (r=100)

Table 5. ISIC2019-LT (r=200)

Table 6. ISIC2019-LT (r=500)

The key observations:

• Overall, increasing the number of parameters does improve performance for most methods, particularly for MixUp. This is likely because MixUp, as a data augmentation and regularization technique, benefits from greater model capacity, allowing it to learn more robust feature representations.

• In terms of efficiency, as the level of imbalance increases, simply increasing model capacity shows diminishing returns, indicating the need to explore other strategies to enhance performance. Additionally, under the Pareto distribution sampling setting, the increase in imbalance significantly reduces the number of training samples, which may lead to overfitting in larger models. Therefore, the number of model parameters should be appropriately aligned with the amount of training data to achieve optimal performance.

Add augmentation techniques such as AugMix?

A: The current version of codebase supports RandAug and we plan to support some other augmentation codebase such as OpenMixup[3] in the future update.

We have also tested different augmentation techniques:

Table 7. Results with different augmentation techniques

The key observations:

• Using more complex data augmentation methods like RandAug and AugMix can further improve performance. For datasets with diverse image patterns, such as dermoscopy images (e.g., with color variations), the performance gains from applying complex augmentations are often greater compared to others such as radiology datasets.
• Cropping is a common method in data augmentation, but it may not be suitable for all medical datasets, as medical data often relies on specific regions of lesions for classification or diagnosis, such as in cases of diabetic retinopathy grading.
• Skin lesions are often characterized by certain specific local features, such as the shape of the lesion, irregular edges, and color distribution. By applying cropping, the model can focus more on the detailed features of the lesion area, reducing background interference and thereby improving classification or detection performance.

References

[1] Deep Long-Tailed Learning: A Survey.

[2] Hierarchical knowledge guided learning for real-world retinal disease recognition.

[3] https://github.com/Westlake-AI/openmixup

We hope the above discussion will fully address your concerns about our work, and we would really appreciate it if you could be generous in raising your score. We look forward to your insightful and constructive responses to further help us improve the quality of our work. Thank you!

We thank the reviewer for the encouraging feedback and helpful suggestions.

We have conducted extensive updates and additional experiments based on the reviewers' valuable feedback. Specifically,

• We added an anonymous repository for our codebase: https://anonymous.4open.science/r/MONICA-153F/README.md
• We introduced new datasets and transformed existing ones (e.g., WILDS) to better align with long-tailed settings (in progress).
• We added support for more advanced backbones, including ViT, Swin Transformer, ConvNext, and RetFound, and performed detailed experiments to evaluate their impact.
• We expanded the evaluation metrics to include AUROC, AUPRC, and F1 Score, providing a more comprehensive assessment of model performance.
• Additional ablation studies were conducted to analyze input size variations, backbone comparisons, and augmentation techniques such as RandAug and AugMix.

The detailed responses are outlined below:

Weakness

More datasets Pathology such as WILDS (medical subset).

A: Thank you for your valuable insights on pathology datasets. We have chosen PathMNIST as a representative dataset for this study. Our codebase supports customized datasets, enabling researchers to conduct their own experiments.

Regarding WILDS, we observed that it processes the original whole-slide images into patches to identify the presence of tumors, resulting in a binary classification task. However, this setup may not align well with our long-tailed setting. To address this, we used the "stage label" as the new patch annotation, transforming the task into a 9-class classification problem.

The experiments are currently in progress as we prioritized supplementing some other important experimental results within the limited time. The complete results will be updated in the final revision. Thank you for your patience and understanding.

More complex backbones such as ConvNext / Swin or foundation model backbones for different datasets.

A: Unless explicitly specified in the original paper that a particular backbone must be used (e.g., RSG uses ResNext), we choose ResNet as our backbone for the following reasons: the methods supported by our benchmark predominantly use ResNet as the backbone in their original papers. This is especially important for methods that require modifications to the network structure (e.g., the ResNet in SADE). Attempting to modify ViT in the same way as ResNet to achieve a similar motivation could introduce discrepancies, potentially leading to unfair evaluations.

However, we appreciate the suggestions from the reviewer to add more backbones with different architectures. Our updated codebase now supports more advanced backbones such as ViT, Swin Transformer, ConvNext, as well as relevant foundation models like RetFound for ophthalmology. We have provided corresponding alerts in the codebase for those fixed backbones proposed by the original paper. We completed training for ViT, Swin Transformer, and ConvNext on the ISIC dataset. Here, we present partial results, and the complete results will be included in the appendix.

Table 1. ISIC2019-LT (r=100)

Generally the community uses pretrained backbones rather than training from the scratch.

A: Unless otherwise specified (e.g., for SADE or other foundation models), our methods use backbones pre-trained on ImageNet. In contrast, the community often employs training from scratch, as datasets like CIFAR-10/100 and ImageNet-LT are widely regarded as important benchmark datasets.

The same backbone is used for every task for fairness but generally a sweep over backbones would help since different modalities and tasks require different approaches.

A: As our response to More Backbones, our updated codebase supports additional backbones, and we will include discussions based on the results. Furthermore, we have added a set of experiments with varying input sizes. Partial results are shown below, and we believe these experiments provide valuable insights into how different input sizes impact performance across various datasets and methods.

Top-1 accuracy is an in appropriate metric for model selection in imbalances settings and AUROC, AUPRC, F1 should be used.

A: In long-tailed learning, the community typically uses shot-based group Top-1 Accuracy as a key metric to evaluate models. We hope the reviewers understand that this choice is based on the findings of supported methods from relevant references and some important surveys, e.g, Zhang et al. [1], rather than being a subjective decision. However, we believe adding more metrics can help users better understand the differences between methods in practical applications. For example, in clinical scenarios, many use cases may prioritize sensitivity, whereas screening scenarios often focus on a balance between sensitivity and specificity. To address this, our updated codebase includes additional metrics such as AUROC, AUPRC, and F1 Score.

Error bars are missing in experiments.

A: Apologies for the missing details. We ran each methodology on each dataset three times using different random seeds. In the updated version, we will include error bars (e.g., rounded to one decimal place) for greater clarity. Additionally, we plan to establish a website to provide more comprehensive results, offering a better overall understanding of this benchmark.

More thorough error analysis, clearer articulation of novel insights, better connection to clinical relevance, more detailed ablation studies.

A: We sincerely appreciate the reviewers’ constructive feedback regarding the need for more thorough analysis and discussion.

We will include a deeper investigation into error cases, particularly focusing on challenging classes in long-tailed datasets, to provide better insights into model limitations and potential improvements. We will contextualize our results with respect to real-world medical applications, discussing how our findings could be utilized in practice and their potential impact on clinical workflows (e.g., with experiments on augmentations).

These suggestions are invaluable for improving the quality and impact of our work.

Questions:

Key trends observed across the board?

A: As discussed in the experimental section, we conducted an in-depth analysis from multiple perspectives, including the use and summarization of re-sampling strategies, employing appropriate data augmentation (e.g., MixUp) to enhance the backbone's representation capability, improving classifiers to mitigate distribution bias, and the necessity of two-stage training methods. Through these analyses, we not only derived many significant conclusions but also identified potential improvement directions for each component, providing a broader design space for future researchers to develop new methods.

Furthermore, we would like to emphasize that the codebase we provide enables the design, improvements and potential combination of modules, offering greater flexibility. The additional experiments we conducted with different backbones and data augmentation strategies also release new insights.

Additionally, we explored the extension tasks on key issues relevant to this field, such as OOD detection. At the same time, we highlighted the importance of the intrinsic characteristics of medical data, such as incorporating prior knowledge as auxiliary information—for example, hierarchical information[2], which serves as an excellent case in point.

Do stronger backbones can help learn better features?

A: Based on the experimental results with the additional backbones, it is evident that stronger backbones can indeed help learn better features. However, due to differences in network architecture (CNN-based, transformer-based) and the number of parameters, it is challenging to directly compare their differences horizontally. To address this, we added a new set of results to specifically compare the performance differences between ResNet50 and ResNet101 across methods.