Pursuing Better Decision Boundaries for Long-Tailed Object Detection via Category Information Amount

Response to Weaknesses 1, 2, and 3:

We greatly appreciate your recognition of this work. Your careful attention to the scaling issue in Figure 1 demonstrates your rigorous academic pursuit. In line 170, $m$ represents the number of instances in category $i$, and in line 176, $p$ denotes the dimensionality of the instance embeddings. We have added explanations for these two symbols in the revised manuscript.

Regarding your suggestion to present the CIA plot in the ablation study section, we hope you can understand our decision to place it in the introduction. Our goal was to tell a cohesive story, with each step supported by results, which is why we positioned it in the introduction.

Once again, we sincerely wish you a wonderful day.

I will keep my rating.

Response to Question 1 and 2:

In the introduction section, we wrote: "Recent studies have shown that the response of deep neural networks to images is similar to human vision, following the manifold distribution hypothesis, where the embeddings of images lie near a low-dimensional perceptual manifold embedded in high-dimensional space. Continuous sampling along a dimension of this manifold corresponds to continuous changes in physical features." This viewpoint is supported by theory and is based on recent research, which I find fascinating for its connection to neural mechanisms. You may refer to the following papers for further reading: Separability and Geometry of Object Manifolds in Deep Neural Networks and Representations and Generalization in Artificial and Brain Neural Networks. I believe the theory of disentangling manifolds can be used to further explain some phenomena in deep neural networks, and if you are interested, we can connect after the conference for further discussion and potential collaboration.

Regarding Question 2, maintaining a queue for all categories is sufficient. In our code implementation, we appended the category label to each instance embedding, so we only need to retrieve the corresponding category based on the label to calculate the category information amount. Thank you once again for your recognition of our work.

Response to Question 1:

The purpose of Figure 2 is to demonstrate the correlation between category information and mean average precision (mAP) for each class. All models were trained using the standard Faster R-CNN model from MMDetection, without employing IGAM. The results in this section are solely intended for observing the phenomenon, and no modifications were made to the standard Faster R-CNN model. Therefore, we omitted the mAP for each class to reduce the space occupied in the paper.

Response to Question 2:

We are glad to help clarify your doubts. Let us explain how IGAM affects the decision boundary using a binary classification example. Please refer to the cross-entropy loss functions shown in Equations 3 and 4. For a sample of class 1, suppose the angle between $W_1$ and $W_2$ is $t$. The decision boundary of the IGAM loss is given by $\cos(\theta_1) = \cos(\theta_2 + m_{12})$. Since $\theta_1 + \theta_2 = t$, the decision boundary is $\theta_1 = (t + m_{12}) / 2$. When the information content of class 1 is less than that of class 2, $m_{12} < 0$, so $\theta_1$ is smaller than the angle bisector $t/2$, compressing the space for class 1.

Response to Question 3:

Your concern is important, and we respond to it below. Suppose that the information content of class $i$ is small, but the cross-entropy loss does not adjust the decision boundary based on the information content. Therefore, we have modeled this constraint. Please note that a smaller information content does not imply a smaller decision space, and our experimental results also demonstrate that the modeling we have shown is necessary and effective.

**Additional Response: **

We have also added the performance of IGAM under the DETR framework on the LVIS v1.0 dataset in Section 4.3 of the revised manuscript. The experimental results further demonstrate the versatility of our method across various object detection frameworks.

Notes:

• The $mAP^b$, $AP_r$, $AP_c$, and $AP_f$ (%) for each method are reported.
• 🟢↑ indicates performance improvements.

Response to Weakness 3:

Your concern directly addresses the core of our research, demonstrating your professional and rigorous academic expertise. We would like to clarify that, on the relatively balanced Pascal VOC dataset, our primary goal was to demonstrate the effectiveness of IGAM. Therefore, we believed it unnecessary to include additional comparison methods. However, your feedback made us realize that we should include the experimental results of the baseline model (trained with cross-entropy loss) in Table 5 (Table 6 in the revised manuscript) to show the significant improvement IGAM brings to the baseline model. As shown, using Faster R-CNN as the target detection framework with ResNet-50 and ResNet-101 as backbone networks, IGAM improves the overall performance of the baseline model by 4.9% and 5.1%, respectively, which is consistent with the excellent performance on LVIS v1.0. This work has demonstrated through extensive experiments that category information amount significantly enhances the baseline methods, validating the necessity and effectiveness of the core contribution proposed in this work.

Additionally, in response to your concern, we have also included the improvement brought by IGAM to the baseline methods when using Cascade Mask R-CNN and DETR as target detection frameworks. The experimental results are shown in the table below, and it can be observed that IGAM significantly improves the performance of the baseline methods in all four cases. We have also added these results in Appendix B of the revised manuscript.

Your feedback prompted us to further validate IGAM’s applicability to multiple target detection frameworks on Pascal VOC, which has elevated the quality of our work. We sincerely thank you for this valuable input.

Response to Weakness 2:

We are more than happy to address your concerns. In classification tasks, two papers (Ref 1 and Ref 2) officially reported the more widespread issue of model bias in 2023. Therefore, in the introduction, we stated, “However, recent research in image classification suggests that category bias is not only caused by the imbalance in sample numbers but may also be closely related to the complexity of intra-category features.” Inspired by these two papers, we became curious about target detection tasks, and our initial starting point was to identify factors affecting model bias in target detection. With this motivation in mind, we proposed the concept of category information amount and studied the relationship between category information amount and model bias.

Given the scope of the work, we did not apply category information amount to classification tasks in this study, but we plan to explore this aspect in future work. If you have any further questions, please don’t hesitate to contact us. We are available at any time to answer your inquiries. We sincerely wish you a wonderful day.

Ref 1: Curvature-Balanced Feature Manifold Learning for Long-Tailed Classification
Ref 2: Balanced Data, Imbalanced Spectra: Unveiling Class Disparities with Spectral Imbalance

Dear Reviewer 7hzz,

Thank you very much for your valuable and insightful comments.

Thank you for your meticulous guidance in improving the logical flow of our paper. We trained Faster R-CNN on the LVIS v1.0 and COCO-LT datasets using CE, SeeSaw, and Focal Loss, respectively, and then calculated the correlation between category information amount and average precision (AP) per category. The experimental results are as follows (also included in the revised version's introduction). Combined with the existing experimental results on Pascal VOC, it can be observed that category information amount more accurately reflects model bias in both long-tailed and non-long-tailed scenarios.

Table: Pearson correlation coefficient between category information amount and class average precision on long-tailed datasets. The model is Faster R-CNN with R-50-FPN backbone.

We may not have fully understood your suggestion. If you have further recommendations for improving our writing, we would be more than happy to receive your guidance. Once again, we sincerely thank you for your invaluable help.

Thank you for your time and your detailed response. As for the Question 1, I notice that the caption of Figure 2 mentions that three values ​​are plotted in the image. If the author only plot two, it is recommended to delete mAP and explain it to facilitate understanding. For Weakness 3, I seem to have asked two questions, but I only understood the response to the second one. The first one is about the methods in Table 5 are all from 2021 to 2022. Are there any other updated methods in the past two years, such as BACL [Qietal.(2023)] in Table 4. If so, can the authors make a comparison?

Thank you for your further response. I have increased my score from 5 to 6. Wishing you a wonderful day!

Response to Weakness 5:

Thank you for raising this important question. We fully agree that object scale and occlusion are critical factors influencing model performance and contributing to category bias, as extensively discussed in existing literature. For instance, research in small object detection and occluded object recognition has shown that when objects in a category are generally smaller or more heavily occluded, the model tends to perform worse on that category. This is because these factors directly limit feature extraction and the acquisition of semantic information. MMDetection even supports multiple algorithms designed to improve recognition of occluded objects.

We would like to clarify that previous studies have indeed recognized object scale and occlusion as factors affecting category performance and causing model bias. However, our research focuses on uncovering additional potential sources of category bias. While object scale and occlusion are known influencing factors, they are not the sole causes of category bias.

In this study, we define and introduce the concept of category information amount as a new perspective, revealing how the diversity and richness of category representation in the feature space impact model performance. As an important metric for assessing the representational capacity of a category, category information amount reflects the internal complexity of a category and its significance in model learning.

Our research not only identifies the significant impact of category information amount on model performance but also demonstrates that dynamically adjusting decision boundaries to optimize category information amount can partially mitigate model bias. This finding provides researchers with a new perspective, encouraging them to analyze and address category bias from broader angles beyond object scale or occlusion factors.

Response to Weaknesses 3 and 4:

Thank you for providing additional references and constructive suggestions. Your feedback reflects an exceptional level of expertise in the field of long-tail object detection. We have thoroughly reviewed [Ref_1] and [Ref_2] and analyzed their methods in comparison to our proposed IGAM approach. Below is a detailed explanation and comparison:

(1) Core Differences in Method Design:

• [Ref_1] (RichSem):
  This method relies on external classification datasets (e.g., ImageNet-21k) to provide additional training samples and uses visual-language models (e.g., CLIP) to extract "soft semantics" for enhancing tail-class feature representation.
  In contrast, IGAM is entirely based on the target detection dataset and does not depend on external data, pre-trained models, or external annotations.
  The performance improvement of [Ref_1] partially stems from the additional resources used, such as external classification datasets and pre-trained models. By comparison, IGAM addresses class imbalance solely using the detection dataset, without introducing external data. Therefore, a direct comparison with [Ref_1] may not be entirely fair, as the two methods fundamentally differ in resource utilization and design goals.

• [Ref_2] (Step-wise Learning):
  This method introduces several steps and modules, such as data splitting (e.g., separating "head-dominant" and "tail-dominant" data), exemplar replay, and classifier head distillation. It addresses the long-tail distribution problem through multi-stage step-wise training, including pretraining, fine-tuning, and knowledge distillation.
  The step-wise learning framework requires multiple training phases and intricate data operations, resulting in a more complex overall process. In contrast, IGAM achieves the same goal using a single optimization objective, significantly simplifying the training pipeline.

(2) Model Complexity and Resource Dependency:

• Complexity of [Ref_1]:
  RichSem involves an additional semantic branch, external pre-trained models (CLIP), and classification datasets, which substantially increase implementation complexity and resource dependency. Additionally, it combines detection data with external classification data to extract and process soft semantics, further complicating the training process.

• Complexity of [Ref_2]:
  Step-wise Learning requires data grouping, saving intermediate models (e.g., head-class expert models), and conducting knowledge distillation on tail-class data. These multi-stage processes lead to higher training resource requirements.

• Simplicity of IGAM:
  IGAM employs a dynamic loss function based on class information content, directly optimizing model performance on detection datasets without external data, pre-trained models, or multi-stage training processes. This simplicity makes IGAM more practical for integration and real-world applications.

(3) Differences in Applicability:
Both [Ref_1] and [Ref_2] primarily focus on long-tail datasets. IGAM, however, not only performs well on long-tail datasets (e.g., LVIS and COCO-LT) but also demonstrates significant performance improvements on non-long-tail datasets (e.g., Pascal VOC), highlighting its broader applicability.

In summary, the core strengths of IGAM lie in its simplicity and efficiency. Unlike [Ref_1] and [Ref_2], our method achieves significant performance improvements without relying on external data, introducing additional modules, or employing multi-stage training. This demonstrates its practical value, particularly in resource-constrained scenarios. Based on the above analysis, we believe that [Ref_1] and [Ref_2] differ significantly from IGAM in terms of design goals and implementation pathways. A direct comparison may not fully reflect IGAM’s core advantages.

In the revised manuscript, we have clarified the differences between these methods in the related work section and emphasized IGAM's unique strengths in simplicity and broad applicability.

Dear Reviewer NFCd,

Your suggestion is highly valuable, and we are happy to follow it to demonstrate the generality of our method.

We would like to clarify that, the mainstream benchmarks in the field of long-tailed object detection primarily adopt Faster R-CNN and Cascade Mask R-CNN as detection frameworks, with ResNet-50-FPN and ResNet-101-FPN as the typical backbone networks. Therefore, our work follows the conventions of prior research in this area.

To address your suggestion, we have added evaluations on the DETR detection framework and conducted additional experiments using Swin-T as the backbone network in Faster R-CNN, Cascade Mask R-CNN, and DETR frameworks. The new experimental results are summarized in the table below (also included in Table 3 of the revised manuscript). It can be observed that IGAM improves the baseline models by 6.3%, 6.6%, and 7.1% overall performance under Faster R-CNN, Cascade Mask R-CNN, and DETR frameworks with Swin-T as the backbone network, respectively. This demonstrates the generality and effectiveness of our method across different frameworks and backbone types.

We would also like to emphasize that the core idea of category information amount is independent of the feature representation network architecture. Therefore, it is expected that IGAM can generalize to other detection frameworks.

In addition, compared to [Ref_1] and [Ref_2], our experimental validation is more comprehensive. Specifically, [Ref_1] did not include evaluations on the Cascade Mask R-CNN framework, while [Ref_2] did not evaluate Faster R-CNN or Cascade Mask R-CNN frameworks and lacked experiments using ViT-based models as backbone networks.

Notes:

• The $mAP^b$, $AP_r$, $AP_c$, and $AP_f$ (%) for each method are reported.
• 🟢↑ indicates performance improvements.

Dear Reviewer NFCd,

Thank you very much for your constructive comments, which enabled us to polish the paper.

First, our response to Weakness 1 is as follows.

Your suggestion has greatly improved the completeness of our work, and we sincerely thank you for that. We have supplemented our observations on the LVIS v1.0 and COCO-LT datasets. It can be observed that there is a significant negative correlation between category information amount and category detection accuracy on these two datasets, with numerical values exceeding -0.65. This further demonstrates that the proposed category information amount metric more accurately reflects category difficulty in both long-tailed and non-long-tailed scenarios. Additionally, we have incorporated the newly added experimental results into the second paragraph and Table 1 of the revised manuscript, highlighting them in blue for clarity.

Table: Pearson correlation coefficient between category information amount and class average precision on long-tailed datasets. The model is Faster R-CNN with R-50-FPN backbone.