Transferring Labels to Solve Annotation Mismatches Across Object Detection Datasets

$**Dear Reviewer 2kAn**$

Thank you for your positive comments regarding the good results presented in the paper.

In the following, we answer the reviewer's concern with the scope of the paper:

1. Confusion with the introduction and Figure 1: We introduce the label translation problem as a general and prevalent type of mismatches among object detection datasets. While resolving arbitrary label mismatches is extremely non-trivial, we consider our streamlined version of addressing specifically bounding box mismatches as the necessary first step in this novel line of research. Fortunately, our metric of concern (target performance) provides strong evidence that solving the simpler problem remains effective. We would appreciate if the reviewer could expand on any points that they feel we haven't addressed.
2. Narrowing down the scope of the mismatch problem:
   • We argue that optimizing only the target dataset performance is not narrowing down from multi-dataset object detection. Instead, they have distinct values depending on the application. Optimizing target dataset performance is particularly important when there is a specific target domain of interest, such as in Sim2Real. The performance on simulated data is relatively unimportant and only serves the purpose of improving target performance on real-world data.
   • While we narrowed down the mismatch problem to a shared class label space, LGPL can handle the setting when two datasets have only partially overlapped categories, by translating the overlapped categories and leaving the other as is.
3. Assumptions in Section 4.1: The assumptions are made based on the observation from seven datasets surveyed in Section 3.
   • To further justify the second point “the source labeling function either detects or over-detects all the objects specified in the target labeling function”, we manually collect the gold translated labels and perform TIDE analysis to understand the types of translation required (See the details in Section 5.1 and Section A.3). The results in Table 9 show the low value of “Missed error” and high value of “Background error”, which supports the need to fix background errors.
4. A multi-dataset version: As pointed out in the second point (Narrowing down the scope of the mismatch problem), performing multi-dataset object detection has distinct application values that differ from our goals in this work.
5. The term "label translation": We thank the reviewer for the thoughtful concerns. The confusion with language translation is unfortunate, and we are keen to hear suggestions on how this might be mitigated in future.
6. The role of Section 3: Section 3 introduces the concept of annotation mismatches and categorizes them into four common types. While it is challenging to quantitatively measure how severe annotation mismatches are between each pair of datasets, we manually collect gold translation labels and quantify the TIDE errors in Section 5.1 and Appendix A.3.

Empirical evaluation

The reviewer asked why we chose YOLOv3 instead of newer variants or one-stage detectors. In fact, we specifically choose three different types of object detectors: two-stage object detector (Faster-RCNN), one-stage object detector (YOLOv3), and transformer-based object detector (Def-DETR).

We choose YOLOv3 out of three YOLO variants in MMdetection due to its stable training. But in any case, we feel that we have adequately demonstrated the robustness of our technique to different object detectors. If the reviewer would like to see the performance on another detector, please kindly provide us with a reference and we will aim to incorporate these results into a later version.

We would also want to emphasize that we have tested on four different scenarios with three different architectures as well as ablations with model-centric alternatives such as domain adaptation and foundation model alternatives such as SAM.

Suggestions of LRP metric

We appreciate the reviewer's great suggestion of the LRP metric and we will include the LRP metric in our camera-ready version.

Minor comments

We appreciate the feedback and have revised the paper submission accordingly.

$**Dear Reviewer MRBF**$

Thank you for your positive comments regarding the interesting research problem, the proposed usage of RoI head, and the good performance on various detectors and datasets.

Handling unique categories that are in one dataset but not in the others

This is a great point. Indeed, LGPL can handle the setting when two datasets have only partially overlapped categories, by translating the overlapped categories and leaving the others as is.

To specifically translate unique classes, we note that LGPL is model/training-agnostic and can be applied with any other data augmentation or training algorithm. For example, if the target dataset has unique categories, we can use any off-the-shelf semi-supervised learning to learn the unique categories. If the source dataset has some unique categories, we can just drop those labels.

Necessity of training a RPN that generates source-style boxes

We interpret the reviewer's suggestion as: "Using the RPN trained on the target dataset so that the training/inference is not tied with any source dataset."

We highlight that the pseudolabeling baseline is exactly training the RPN together with RoI on the target dataset. However, as shown in Table 1, pseudolabeling under-performs in the label translation problems, often with worse performance than "No translation".

Handle target datasets with more than one category

We apologize for the confusion. In fact, most of our experiments handled multiple categories, i.e., nuScenes $\rightarrow$ nuImages (10 classes), Synscapes $\rightarrow$ Cityscapes (7 classes), and Internal-Dataset $\rightarrow$ nuImages$^\dagger$ (3 classes). We provide the dataset statistics in Table 6 in the Appendix and the full class names in Section A.2. We have clarified in Section 5.1

Evaluation on other detection datasets

We expect our strategy to work on other common detection data sets like COCO and Object365 since our method and problem are not restricted to driving data. We focus on driving data sets because this is the real-world motivation of our work (e.g., see our Internal-Dataset) where the practice involves combining multiple data sets from different vendors [1] or mixing synthetic data [2]. We also emphasize that we considered four scenarios from seven different data sets and arrived at consistent results.

The role of translation-mAP

We absolutely agree with your assessment on the effectiveness of translation-mAP, which reflects the false intuition that visual inspection of translated labels might be sufficient to assess label quality. We believe it is important to disprove the false notion and emphasize the importance of downstream training when evaluating label quality.

Label translator is source-target specific

We agree that a universal label translator that would not require retraining on the source and the target datasets would be impactful and impressive. However, we emphasize that we are the first one that formally introduce the label translation problem. We feel it is necessary for this work to streamline the problem setup. Furthermore, implementing a universal label translator is non-trivial (e.g., off-the-shelf foundation models are not always effective as we show in Table 5). We hope to extend this framework to a universal translator in future work.

[1] Yan Wang et al., Train in Germany, Test in The USA: Making 3D Object Detectors Generalize, Conference on Computer Vision and Pattern Recognition , 2020.

[2] Viraj Uday Prabhu et al., Bridging the sim2real gap with CARE: Supervised detection adaptation with conditional alignment and reweighting. Transactions on Machine Learning Research,2023.

$**Dear Reviewer XHr3**$

Thank you for the positive comments regarding the practical importance of our problem, the presentation of our work, and the promising results achieved compared to the baselines.

Novelty and technical contributions

We revised the paper to clarify our novelty and the contributions in the Introduction and the Conclusion.

To clarify our novelty and technical contributions:

1. Novelty: We are the first to introduce and formalize the label translation problem, to the best of our knowledge. We demonstrate that this problem is prevalent with an extensive study of several object detection datasets, by categorizing a set of four common types of annotation mismatches.
2. Technical/Methodology: We propose a simple but effective label translation algorithm, which, although motivated by pseudolabeling, expands upon this naive baseline. Unlike pseudolabeling, LGPL effectively leverages source dataset bounding boxes and class labels during label translation.
3. Empirical: We extensively validate our approach on seven datasets and four source-target combinations. Our approach consistently outperforms all the baselines in every setting. Importantly, we compare against strategies like supervised domain adaptation [1], demonstrating that our data-centric approach is a new alternative to the standard method for addressing image/label mismatch.

Comparison with supervised domain adaptive object detection method [1]

We apologize for the confusion on this experiment. In Figure 4, we indeed compare the proposed approach with supervised domain adaptation (SDA) [1]. Prior art [1] adopts popular unsupervised domain adaptation (UDA) approaches to the supervised settings, where both the source and the target labels are accessible. We adopt two variants S-DANN and S-CycConf from [1]. Note that we prefix the supervised version with "S".

SDA approaches proposed in [1] directly align instance features given the source and the target labels, while cross-domain weakly supervised object detection (CD-WSOD) either aligns instance-level features via pseudolabels [2,3] or aligns image-level features [4]. We choose SDA as our baselines since SDA avoids noisy pseudolabels and is able to align instance-level features between the source and the target domain effectively.

We appreciate the reviewer's careful review. We have fixed the submission to clarify the comparison in Figure 4 and added a brief background to the supervised domain adaptation in Section C.1.

Details of choosing class-conditional thresholds $\sigma_c$

Our revision clarifies this point in Appendix D.1.

We choose $\sigma_c$ by quantizing the confidence score [5] for each class and applying it to all the label translators. Annotations falling into the last bin or with a confidence score lower than $0.001$ are dropped. In this way, the classes that are more challenging get the lower thresholds and vice versa. We leave further investigation of the class-conditional threshold to future research.

Let $v_i$ be the value of the $i^{th}$ bin. We performs sensitivity test w.r.t. $\sigma_c$ on nuScenes $\rightarrow$ nuImages:

From the experiments above, the choices of the class-conditional threshold do have impacts on the downstream-mAP but all of them still outperform the baselines.

[1] Viraj Uday Prabhu et al., Bridging the sim2real gap with CARE: Supervised detection adaptation with conditional alignment and reweighting. Transactions on Machine Learning Research,2023.

[2] Naoto Inoue et al., Cross-Domain Weakly-Supervised Object Detection through Progressive Domain Adaptation, CVPR2018

[3] Shenxiong Ouyang et al., Pseudo-Label Generation-Evaluation Framework For Cross Domain Weakly Supervised Object Detection, ICIP 2021

[4] Yunqiu Xu et al., H2FA R-CNN: Holistic and Hierarchical Feature Alignment for Cross-domain Weakly Supervised Object Detection, CVPR2022

[5] Herbert A. Sturges, The choice of a class interval, Journal of the American Statistical Association 1926