Open-Vocabulary Object Detection via Language Hierarchy

[Response 1] The affect of the reliability score:

Thank you for your suggestion. As suggested, we conduct the new ablation study to examine the effect of the reliability score in the proposed DetLH. As the table below shows (on Object365), DetLH effectively mitigate the label noises with the design of reliability score.

[Response 2] More comparisons:

We would clarify that we did not provide much comparisons with those methods as mentioned in Lines 661-671 because they focus on different tasks with different objectives (also use different training data, backbones and benchmarks). We largely focus on benchmarking DetLH with Detic as both methods work on the same task WSOD and use the same training data (ImageNet-21K and LVIS) and backbone. As suggested, we will provide more comparisons between DetLH and those detectors and include the comparisons in the updated manuscript or appendix.

In addition, we would clarify that we focus on cross-dataset zero-shot evaluations [48, 68] (i.e., evaluation on unseen datasets like CLIP) as discussed in Lines 117-119. We did not benchmark on LVIS/COCO (both use the same data but different annotations) since we used LVIS/COCO in network training.

As suggested, we benchmark DetLH on LVIS and COCO as shown below and will include these results in the updated manuscript or appendix.

[Response 1] The necessity of the investigated problem:

Thank you for your suggestion. We would clarify that, compared with recent MLLM-based detection/grounding methods, our approach of scaling the vocabulary of detectors using image labels offers significant advantages in training efficiency. As shown in Table 14 (as copied and posted below), DetLH is up to ten times more efficient than current visual grounding/MLLM-based methods such as GLIP and DetCLIP. The superior efficiency is critical to the scalability of detectors in many practical tasks.

[Response 2] The novelty of the proposed method:

Thank you for your suggestion! We would share that, different from existing WSOD and semi-supervised WSOD methods, DetLH tackles the image-to-box label mismatch by introducing language hierarchy into self-training: 1) it introduces WordNet's language hierarchy to expand the image labels and accordingly enables co-regularization between the expanded labels and self-training. As shown in Table 10 (in the submitted manuscript), just introducing language hierarchy or simply combing self-training and language hierarchy does not work without our co-regularization design; 2) it introduces language hierarchy into prompt generation to bridge the vocabulary gaps between detector training and testing, leading to better prompt generation and detection performance. Extensive experiments validate the effectiveness of our designs in DetLH. We will highlight the above points in the updated manuscript.

[Response 3] Comparisons with other semi-supervised WSOD methods:

Thanks for sharing [a,b,c,d]. We will review them and highlight how DetLH differs in the revised paper. As the table below shows, DetLH clearly outperforms [a,b,c,d] on Object365.

[Response 4] Visualizations of label expanding:

Thank you for your suggestion! Figure 1 in the attached PDF shows that the labels expanded by WordNet are not all reliable. We will include the suggested visualization in the updated manuscript or appendix.

[Response 5] Missing oracle values on Object365 in Table 1:

We reported AP value only for Object365 as we referenced to the Detic paper which only reports AP value. For Pascal VOC, we trained the model to get the oracles of all AP metrics. We will train models on Object365 to obtain the complete AP values and update them in the updated manuscript.

[Response 6] The efficiency of WordNet in training and inference:

Dear Reviewer PjbE,

Thank you for your insightful feedback. We have carefully considered your questions and suggestions and have addressed them accordingly. We sincerely appreciate your constructive comments, which have helped strengthen our paper. As the discussion phase is nearing its conclusion, we would appreciate it if you could let us know if there are any additional questions or suggestions.

Best regards,

Authors

[Response 1] Analysis of discrepancies of the taxonomies of image vs. box categories:

Thank you for pointing out this issue! As suggested, we analysed the mismatch between image-level and box-level categories and how much it could affect the detection performance. As the table below shows, the mismatch between image-level and box-level categories varies across datasets, and the proposed DetLH improves more with the increase of mismatch levels. We will include a subsection to discuss this issue in the updated paper.

[Response 2] More ablation studies:

Thank you for your comment! As suggested, we conduct new ablation studies to analyse how effective DetLH deals with noisy labels. Specifically, we compare DetLH with and without using reliability scores (the latter means uniform category weights) over Object365. As the table below shows, including the adaptive weighting mechanism (i.e., reliability scores) helps mitigate the label noises effectively.

Regarding the impact of using proxy vocabulary, we conduct new experiments on Object365 dataset to compare LHPG with CLIP embeddings only and LHPG with both CLIP embeddings and proxy vocabulary. As the table below shows, using a proxy vocabulary performs clearly better, demonstrating its effectiveness in narrowing the distribution gap between training and test labels.

[Response 3] Misleading Claims:

Thank you for your suggestion! To avoid confusion, we will rephrase the text "These detectors usually yield constrained detection... [68]" to "As reported in [68], detectors trained solely on detection datasets often exhibits constrained performance.", and the claim "DetLH introduces large-scale image-level datasets [...]." to "Similar to [68], DetLH follows semi-supervised WSOD and introduces large-scale image-level datasets for detector training.".

[Response 4] More insights and analysis should be provided:

Thank you for pointing out this issue. We would clarify that we provided extensive discussion and related insights in the appendix (e.g., Section B Additional Discussion, and Section C Additional Comparison), including more ablation studies, strategy studies, parameter studies, comparisons, etc. In addition, we benchmark DetLH across multiple datasets and discuss its generalization ability and parameter sensitivity In Section 4.1 Comparison with the SOTA and Section 4.3 Discussion. We will revise the two Sections to remove repetitive text and include some insight analysis from the appendix.

Regarding Wildlife datasets, our statement is largely based on the intuition that these datasets with many fine-grained categories should suffer from clear vocabulary mismatch, and the language hierarchy introduced by DetLH should help mitigate such mismatch and improve the detection. The new experiments in Response 1 show that wildlife datasets do suffer from more severe mismatch and DetLH helps mitigate the mismatch effectively.

Dear Reviewer TMzm,

Thank you for your encouraging feedback and positive evaluation for our work. We will include the updated ablation study, as well as further discussions regarding experiments (and taxonomy analysis) in the main paper in the revised version, and we agree that these are important points that should not be left in the supplementary material. We sincerely appreciate your constructive comments, which have strengthened our paper.

Best regards,

Authors