Domain Adaptation for Large-Vocabulary Object Detectors

Response to Weaknesses-1:

The Proposal Network of Faster R-CNN generates a large number of region proposals on the input image (i.e., thousands to tens of thousands of region proposals), which make VILD-like methods very slow. On the other hand, our KGD is performed only on the selected box predictions (i.e., the box predictions after the confidence thresholding), where the number of involved predictions is much smaller (i.e., a few to several dozen), which only introduces a few additional computation overhead. In another word, Eq. (6) in our manuscript works by cropping the selected box predictions (i.e., the pseudo labels after the prediction filtering and thresholding), instead of cropping all region proposals as in VILD [1], which significantly reduces the number of regions to be cropped and is much more efficient.

We validate above statements by examining the training and inference time of all the compared methods, as shown in Table 11 in our manuscript. It shows that the operation of cropping object regions and using CLIP introduces a few additional computation overhead in training time and almost does not affect the inference time. The reason lies in that we only crop a limited number of object regions (i.e., selected ones) and process them with CLIP model in a parallel manner during training, while the inference pipeline does not involve these procedures.

Response to Weaknesses-2:

Thank you for your suggestions! We would clarify that we have introduced the WordNet and CLIP into the compared methods for fair comparison, which are signified by $\star$ as shown in Table 1 and 2 in the manuscript. Specifically, the methods employ WordNet to retrieve category descriptions given category names, and CLIP to predict classification pseudo labels (CLIP scores) for objects. The experiments results show that KGD still outperforms the state-of-the-art clearly when CLIP and WordNet are incorporated into the compared methods, validating that the performance gain largely comes from our designed KGD method (including the GCN) instead of merely using CLIP scores. Moreover, we would like to clarify that we compared our KGD with existing CLIP knowledge distillation methods designed for detection tasks (including VILD, RegionKD, and OADP) in Table 4 in the manuscript.

Table 4 in the manuscript reports the experimental results over the Cityscapes dataset, which shows existing CLIP knowledge distillation methods do not perform well in adapting LVDs to downstream tasks, while our KGD works for LVDs adaption effectively, which further validate that the performance gain largely comes from our designed KGD method (including the GCN) instead of merely using CLIP scores.

Response to Weakness-1: Thanks for your comments. As suggested, we compare the memory usage and computational overhead with other methods in the table below, where $\star$ signifies that the methods employ WordNet to retrieve category descriptions given category names, and CLIP to predict classification pseudo labels for objects. The experiments are conducted on one RTX 2080Ti. It can be seen that while the involvement of CLIP during training increases memory usage and computational overhead due to the processing of cropped object regions, the memory usage and computational overhead during inference remain comparable to baseline methods. This is because the inference pipeline does not involve CLIP, thus maintaining efficiency and ensuring that its practicability for deployment.

Response to Weakness-2:

Many thanks for your suggestions! We would like to clarify that we introduced the WordNet and CLIP into the compared methods for fair comparison, which are signified by $\star$ as shown in Table 1 and 2 in the manuscript.
Specifically, for the methods signified by \dag, we employ WordNet to retrieve category descriptions given category names, and CLIP to predict classification pseudo labels for objects. The experiments results show that KGD still outperforms the state-of-the-art clearly when CLIP and WordNet are incorporated into the compared methods, validating that the performance gain largely comes from our novel KGD (including the GCN) instead of merely using CLIP and WordNet.

Response to Question-1: As suggested, we conducted the experiment of applying traditional methods (Mean Teacher(MT) [53], SHOT [29], SFOD [27], HCL [18], and IRG-SFDA [55]) on RegionCLIP. As the table below shows, KGD performs consistently well on this scenario. We will include the new experiments into the updated paper later. Thank you for your suggestion!

Response to Weaknesses-1: Thanks for your comments. We would clarify that the motivation for using knowledge graphs is to explicitly and comprehensively extract CLIP knowledge for effectively de-noising pseudo labels generated by LVDs when adapting LVDs. On the other hand, directly utilizing CLIP to obtain pseudo-labels could also benefit unsupervised domain adaptation of LVDs, but it may be less effective. The reason lies in that knowledge graphs carry not only the information of each category but also inter-class relations, while pseudo-labels only carry the former information. As the table bleow, we conduct new experiments that adapt Detic with semi-supervised learning [Ref 1] using CLIP-generated pseudo-labels. The experimental results show that our proposed KGD outperforms the semi-supervised learning using CLIP-generated pseudo-labels, validating the performance gain largely comes from our novel KGD designs instead of merely using pseudo-labels from CLIP.

[Ref 1] Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, ICML, volume 3, page 2, 2013

Response to Weaknesses-4: Thanks for your comment. We would clarify that all methods (including MT [53] and Ours) in all tables use Detic [81] as the baseline in the manuscript. In another word, the MT [53] in all tables actually denote the mentioned “Detic [81]+MT [53]”.

Response to Weaknesses-5: Thanks for your comment. We would clarify that we studied the training and inference time of all the compared methods in the manuscript, and Table 11 in the manuscript shows the results on Cityscapes. It shows that incorporating CLIP into unsupervised domain adaptation introduces a few additional computation overhead in training time and almost does not affect the inference time. The reason lies in that we only crop a limited number of object regions (i.e., selected ones) and process them with CLIP model in a parallel manner during training, while the inference pipeline does not involve these procedures.

Response to Weaknesses-2:

Thanks for your comments. As suggested, we introduce the Periodically Exchange Teacher-Student (PETS) [Ref 1] and Target Prediction Distribution Searching (TPDS) [Ref 2] as the comparison methods. As the table below shows, KGD consistently outperforms PETS [Ref 1] and TPDS [Ref 2].

In the tables below, $\star$ signifies that the methods employ WordNet to retrieve category definitions given category names, and CLIP to predict classification pseudo labels for objects. We adopt AP50 in evaluations. The results of all methods are acquired with the same baseline (Detic[81]) as shown in the first row. We will include the new experiments into the updated paper later. Thank you for your suggestion!

[Ref 1] Liu, Qipeng, et al. "Periodically exchange teacher-student for source-free object detection." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[Ref 2] Tang, Song, et al. "Source-free domain adaptation via target prediction distribution searching." International journal of computer vision 132.3 (2024): 654-672.

Response to Weaknesses-3: Thanks for your comment. As suggested, we conduct new experiments to compare our KGD with prior knowledge graph distillation methods[Ref 3, Ref 4] . The results in the tables below show that our KGD outperform [Ref 3, Ref 4] clearly, largely becuase the knowledge graphs in [Ref 3, Ref 4] are hand-crafted by domain experts while ours is built and learnt from CLIP. We will include the new experiments into the updated paper later.

[Ref 3] Christopher Lang, Alexander Braun, and Abhinav Valada. Contrastive object detection using knowledge graph embeddings. arXiv preprint arXiv:2112.11366, 2021

[Ref 4] Aijia Yang, Sihao Lin, Chung-Hsing Yeh, Minglei Shu, Yi Yang, and Xiaojun Chang. Context matters: Distilling knowledge graph for enhanced object detection. IEEE Transactions on Multimedia, 2023, doi: 10.1109/TMM.2023.3266897.

Response to Question-5:

Thanks for your detailed comments! We will check through our manuscript carefully and improve the table layout in the revised manuscript. As suggested, we conduct experiments and report the results of using ''LKG Extraction with category name'' and ''Static VKG Extraction'' jointly in the following table. As a comparison, our proposed KGD shows clear improvements as the language and vision information extracted along the training process dynamically stabilizes and improves the model adaptation, validating the performance gain largely comes from our novel KGD designs instead of jointly using ''LKG Extraction with category name'' and ``Static VKG Extraction''.

As suggested, we conduct additional ablation study on 3 object detection datasets that span different downstream domains including the object detection for intelligent surveillance (BAAI), common objects (VOC), and artistic illustration (Clipart1k). As the table below shows, the behavior consistent across datasets that span different downstream domains. Later, We will conduct ablation study on all the 11 object detection datasets and include the results into the revised manuscript. Thank you for your suggestion!

Response to Weakness-1: Thanks for your detailed comments. In our context, the edges in LKG and VKG represent affinity edges based on the distance between node representations. On the other hand, we model these distances as the semantic similarities measured by CLIP model, where the CLIP model largely captures rich semantic information among various categories and the resulted distances thus reflect meaningful semantic relationships among different nodes. We will clarify this distinction in the revised manuscript. The term “encapsulation” was to convey the process of integrating extracted knowledge graphs into the object detector. We acknowledge that this term may obscure the visual adaptation steps described in Eq. (11) and Eq. (12). We will provide detailed explanations to accurately reflect these adaptation in the revised manuscript.

Response to Weakness-2: Thanks for your comments! We will check through our manuscript text carefully and improve the paper presentation later. Please refer to the responses to Question 1-2.

Response to Weakness-3: Thank you for your comments! As suggested, we provide detailed explanations and conducted experiments for your concerns. Please refer to the responses to Question 3-5.

Response to Question-1: The dashed reddish lines between LKG and VKG in Figure 2 represent the cross-modal edges that connect the nodes between vision and language modalities. The purpose of these edges is to enable the integration of both language and visual information, allowing the model to leverage complementary information from both modality. We will revise the caption of Figure 2 and include a more detailed explanation in the main text to ensure this is clear to the readers. Thanks for your comments!

Response to Question-2: In this paper, the hyponym set contains both the hyponym class names and their descriptions. Specifically, each hyponym in the hyponym set is the concatenation of the class name and its descriptions, which contains both the class names and their description.
In the same way, the initial class name is used by concatenating the initial class name and its description to formulate the definition of a certain category as the following:

[class name]+[':']+[category description]

Response to Question-3: In the Eq. (12), the nodes of VKG are preliminarily updated with a pre-defined $\lambda$. $\lambda$ is set as 0.9999. We study $\lambda$ by changing it from 0.99 to 0.999999. The table below reports the experiments over the Cityscapes dataset. It shows that both an excessivel small $\lambda$ or excessively large $\lambda$ lead to performance degradation, largely because an excessively small $\lambda$ (i.e., 0.99) introduces more noise and fluctuation, while an excessively large $\lambda$ (i.e., 0.999999) results in a sluggish response to the latest data changes, failing to update VKG nodes promptly. However, an appropriate value (0.9999) of $\lambda$ can suppress noise and data fluctuation while promptly updating VKG nodes to timely respond to the latest data distribution shift.

Response to Question-4: Eq. (13), incorporate the downstream visual graph knowledge into VKG with a pre-defined $\alpha$. $\alpha$ is set as 0.001. We study $\alpha$ by changing it from 0.001 to 1.0. The table below reports the experiments over the Cityscapes dataset. It shows that both an too small $\alpha$ or too large $\alpha$ lead to performance degradation, largely because a too small $\alpha$ may cause the model to fail to effectively utilize the information from neighboring nodes, thus not fully capturing the structure of the graph and the relationships between nodes, while a too large $\alpha$ may cause noise to propagate through the graph, making the node updates more susceptible to outliers or noisy data.

Response to Question-6: Thank you for your detailed comments! We will carefully check and improve the paper presentation in the revised manuscript.

Response to Question-7:

Thank you for pointing these issues and we will revise the paper accordingly such that all typos are corrected. We will make the following revisions:

''Dynamic VKG Extraction without Smooth''-->''Dynamic VKG Extraction without Smoothing''.

''but without smooth''-->''but without smoothing''.

I have read all the reviewers' comments and the authors' answers, I believe my initial rating of 6: Weak Accept is a fair assessment for this contribution and thus maintain it.