Cyclic Contrastive Knowledge Transfer for Open-Vocabulary Object Detection

Q2. The relationship between regional-contrastive distillation loss and semantic guidance.

Reply: Thank you for pointing out that

“As indicated in the third row of Table 4, the absence of this loss results in a performance decline of 5.4% AP50 on novel classes.”

It is indeed a clerical error. We are sorry for the mistake and misunderstanding inccurred. The correct version should be the following "As indicated by the difference between the first row and last row of Table 4, the absence of this loss results in a performance decline of 5.4% AP50 on novel classes."

Effectiveness of regional contrastive training: It's worth noting that the results of row 2 and row 4 comes from the same model weights. The only difference between these two is how the model gives class prediction: whether using the similarity-based classification or not. In other words, the abaltion experiment comes from three different parameters that learned with three different strategies.

These three models corresponds to the version whether trained using the components (Semantic Guiding, Regional-Contrastive Distillation) introduced in Sec. 3.

In our preliminary experiment, we observed the performance of version 1 stagnates even with longer training schedule. After incorporating the regional contrastive loss, the performace boosts from 32.6 to 38.0. This phenomenon also indicates regional contrastive distillation also helps model get out of local optima to some exent.

Regional Contrastive loss should be used in conjunction with Similarity Classification to form cyclic knowledge transfer loop. Since the same model presents disparate results with different post-process, we reckon the reason behind is: after training, the embedding space of our detecor model is pulled close to the hidden space of the teacher. Spcifically, added with textual features, the regional-level query embeddings are refined by decoder, then guided to be aligned with visual features encoded by the teacher. The tight integration of textual and visual features within the detection loop set the optimization direction towards the student detector imitating its teachers' in a regional level.

Using vanilla post-process to give classification score fails to fully unleash the model's ability. It should be used in conjuction with its teacher's classification style--CLIP also give zero-shot class predictions by similarity score. the holistic operations complete the loop of our cyclic knowledge transfer for OVD.

Thank you for your insightful feedback and for highlighting the strengths of our work, including the structured methodology, comprehensive experiments, and the significance of the CCKT-Det framework in advancing OVD. We appreciate your recognition of our approach to addressing key limitations without requiring additional supervision. The following is our response with necessary explnations.

Q1. Similarities and differences with GroundingDINO.

Reply: Thanks for your suggetion, we add more descriptions in refined paper to make the comparison more comprehensive following your advice.

Concretely, though Grounding DINO [R10] bears certain similarity with our proposed CCKT-Det in spirit at first glance, the following aspects differentiates ours from theirs:

• The formulation of semantic priors: For each incoming image, Grounding DINO forwards the same texts into text encoder usually extracted from dataset labels. Then top N pixel features from image backbone are selected and treated as object queries based on their max similarity scores with text features. However, our proposed mechanism for selecting semantic priors are based on MLLMs, which is indeed a binary classification for concept existence problem. This eliminates the potential interventions among similar concepts since similar concepts may compete with each other thus the similarity score might fall out of the top N operation. More importantly, we forward different texts tailored to each input image rather than statically fixed input texts. The object queries in Grounding DINO may still integrate text features of non-existent concepts in current image from its text cross-attention in cross-modality decoder as in Fig. 3 of their paper. In contrast, our object queries only learn from text features of existent concepts.
• Different architectures: Grounding DINO injects additional cross-modality fusion after text encoder and image backbone, which is proved to improve performance [R11], while we follow the standard Deformable DETR architecture with no post cross-modality fusion.
• Different training data: Grounding DINO uses various source pre-training data (object 365, GoldG, COCO, RefCOCO, etc.) while we keep a modest data usage to demonstrate the effectiveness of CCKT-Det.

References:

[R10] Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. In ECCV 2024.

[R11] TransVG++: End-to-End Visual Grounding with Language Conditioned Vision Transformer. In TPAMI 2022.

We appreciate your acknowledgment of our method as a novel framework for the OVD task, recognizing it as a significant advancement, particularly in scenarios where data may not be readily available. Here's our response with some necessary explanations.

Question: Report performance on base categories on COCO and LVIS.

It is better to add a more detailed result comparison in Tab.1 and Tab.2. For example, show the AP_Base result for COCO and AP_c and AP_f results for LVIS.

This method has shown promising results in detecting novel objects, but it seems not very good for detecting base objects.

Reply: Thanks for your advice, here we report results including performance of base classes on COCO and LVIS dataset. Results on the COCO dataset:

We note that on base classes, our method is not as good as others. However, previous approaches might suffer from issue of overfitting on base classes. Here we define a new metric termed $\triangle \text{AP}_ {50}=|\text{AP}_ {50}^{B}-\text{AP}_ {50}^{N}|$ to quantitatively measure this phenomenon. The $\triangle \text{AP}_ {50}$ reflects the gap between performances on base and novel classes, open-vocabulary detection should achieve both high $\text{AP}_ {50}^{B}$ and $\text{AP}_ {50}^{N}$, but more importantly, a low $\triangle \text{AP}_ {50}$ to avoid skewed performance. As this new metric shows in the above table, the overfitting phenomenon is rampant especially on methods relying on extra image-caption pairs (CoDet) and self-training techniques (OV-DETR) since the generated pseudo boxes of base classes are more accurate than pseudo boxes of novel classes. Though our method is relatively low on base classes, we maintain a good balance between base and novel classes, achieving the lowest 0.3 $\triangle \text{AP}_ {50}$.

We also conduct a new experiment that trains CCKT-Det using data provided by OV-DETR (it includes generated pseudo boxes) to support our guess: we can observe performance of base objects surpassing novel objects by integrating more generated pseudo boxes as supervisions

If an open-vocabulary detector can only achieve superior performance on base categories but poor performance on novel classes, its effectiveness will be diminished, given that the evaluation of OVD essentially reflects not only AP but also the model's generalization ability to novel categories.

Since the hidden space of VLMs exhibits minimal bias regardless of the division of base and novel classes on COCO dataset under OVD setting [R4], there is no distinct 'base' or 'novel' for VLMs, this balanced structural knowledge is effectively distilled to our detector. A similar trend can also be observed in other knowledge transfer or knowledge distillation-based methods, such as ViLD-img and CORA, achiving a relatively low $\triangle \text{AP}_ {50}$ compared to methods that utilize additional training data.

In other words, the student performance is affected by teacher. If teacher has limitations, they could propagate to student. We can further observe that when we employ a more robust MLLM (GVT) to guide our model, the performance on both base and novel classes is boosted (c.f. to CCKT-Det and CCKT-Det++ in Tab. 1).

To summarize, there is an inherent trade-off between base and novel categories, knowledge distillation based method could inherit both balanced semantics and limitations from the teacher. With the major evaluation metric for open-vocabulary detection being the novel AP, our method achieves superior performance while striking a good balance between base and novel classes.

Results on LVIS dataset with $\text{AP}_ {c}$ and $\text{AP}_ {f}$:

[R4] Open-Vocabulary Object Detection Using Captions. CVPR 2021.

Dear reviewer,

Today is the last day for reviewers to ask questions to authors. Did the authors' rebuttal address your concern? Do you have any additional questions?

I decide to maintain my original rating after reading the other reviewers' questions and your reply.

Thanks for your response and for taking the time to consider our rebuttal. Based on your suggestions, we have conducted the relevant experiments and provided an analysis of the results. Besides, we have addressed the second concern mentioned in the "Weaknesses" section. If you have any further comments, suggestions, or points that require clarification, please let us know. Thank you very much!

We also conduct a new experiment that trains CCKT-Det using data provided by OV-DETR (it includes generated pseudo boxes) to support our guess: we can observe performance of base objects surpassing novel objects by integrating more generated pseudo boxes as supervisions

If an open-vocabulary detector can only achieve superior performance on base categories but poor performance on novel classes, its effectiveness will be diminished, given that the evaluation of OVD essentially reflects not only AP but also the model's generalization ability to novel categories.

Since the hidden space of VLMs exhibits minimal bias regardless of the division of base and novel classes on COCO dataset under OVD setting [R4], there is no distinct 'base' or 'novel' for VLMs, this balanced structural knowledge is effectively distilled to our detector. A similar trend can also be observed in other knowledge transfer or knowledge distillation-based methods, such as ViLD-img and CORA, achiving a relatively low $\triangle \text{AP}_ {50}$ compared to methods that utilize additional training data.

In other words, the student performance is affected by teacher. If teacher has limitations, they could propagate to student. We can further observe that when we employ a more robust MLLM (GVT) to guide our model, the performance on both base and novel classes is boosted (c.f. to CCKT-Det and CCKT-Det++ in Tab. 1).

To summarize, there is an inherent trade-off between base and novel categories, knowledge distillation based method could inherit both balanced semantics and limitations from the teacher. With the major evaluation metric for open-vocabulary detection being the novel AP, our method achieves superior performance while striking a good balance between base and novel classes.

Results on LVIS dataset with $\text{AP}_ {c}$ and $\text{AP}_ {f}$:

References:

[R4] Open-Vocabulary Object Detection Using Captions. In CVPR 2021.

[R5] Open-Vocabulary DETR with Conditional Matching. In ECCV 2022.

[R6] fvcore https://github.com/facebookresearch/fvcore

[R7] CORA: Adapting CLIP for Open-Vocabulary Detection With Region Prompting and Anchor Pre-Matching. In CVPR 2023.

[R8] Deformable DETR: Deformable Transformers for End-to-End Object Detection. In ICLR 2021.

Q4: Report performance on base categories on COCO and LVIS.

Reply: Thanks for your suggestions, we report them and provide relevant analysis.

Results on the COCO dataset:

We note that on base classes, our method is not as good as others. However, previous approaches might suffer from issue of overfitting on base classes. Here we define a new metric termed $\triangle \text{AP}_ {50}=|\text{AP}_ {50}^{B}-\text{AP}_ {50}^{N}|$ to quantitatively measure this phenomenon. The $\triangle \text{AP}_ {50}$ reflects the gap between performances on base and novel classes, open-vocabulary detection should achieve both high $\text{AP}_ {50}^{B}$ and $\text{AP}_ {50}^{N}$, but more importantly, a low $\triangle \text{AP}_ {50}$ to avoid skewed performance. As this new metric shows in the above table, the overfitting phenomenon is rampant especially on methods relying on extra image-caption pairs (CoDet) and self-training techniques (OV-DETR) since the generated pseudo boxes of base classes are more accurate than pseudo boxes of novel classes. Though our method is relatively low on base classes, we maintain a good balance between base and novel classes, achieving the lowest 0.3 $\triangle \text{AP}_ {50}$.

We extend our sincere gratitude for recognizing our method as reasonable and effective, addressing OVD with comprehensive experiments. Here's our response with some necessary explanations. If you have any further questions, please feel free to let us know.

Q1: Performance drop of regional contrastive distillation loss.

Reply: From line 2 to line 3, we can observe that adding regional contrastive loss alone decreases the effect of semantic guiding, but from line 2 to line 4 we can see that regional contrastive loss together with similarity classification improves the performance from 32.6 to 33.7, adding semantic guiding to line 4 further boosts to 38.0.

Line 3 and line 5 are models with identical weights (the model trained with distillation loss), the only difference lies in how they give classification predictions in inference, line 5 is a similarity-score classification while line 3 derives the prediction that incorporates logits score from classification branch.

Since the same model presents disparate results with different post-process, we reckon the reason behind is: after training, the embedding space of CCKT-Det is pulled close to the hidden space of the teacher. Spcifically, added with textual features, the language queries are refined by decoder, then guided to be aligned with visual features encoded by the teacher. The tight integration of textual and visual features within the detection loop sets the optimization direction towards mimicking teacher on region level.

Using vanilla post-process to give classification score fails to fully unleash the full potential of the model. Instead the classfication prediction should be follow the classification style of the teacher--CLIP performs zero-shot class predictions by measuring cosine similarities between images and texts. The similarity classfication postprocess completes the loop of our cyclic knowledge transfer, i.e., without introducing a gap between how student and teacher perform classification.

Q2: Suggestions on supplementing introduction to the task setup, workflow details.

Reply: Thanks for your suggestions, we add more descriptions mentioned above and refine them in the paper.

Q3: Comparison or analysis of time complexity.

Reply: Our CCKT-Det follows Deformable-DETR [R8], here we report training and inference time complexity of original Deformable-DETR, OV-DETR [R5] (another comparable Deformable-DETR style OVD detector) and our CCKT-Det.

For training epoches and total GPU hours, our knowledge distillation based method requires less computation and training resources to converge.

For computational complexity, we use the FLOPs and fvcore [R6] to get the statistics. OV-DETR [R5] suffers from repetitive per-class inference: both computation and memory consumption scales linearly with the vocabulary size [R7]. Since there is $C_{all} \times N_{queries}$ instead of only $N_{queries}$ needs to be fed into decoder each forward pass, we fail to get the FLOPs for OV-DETR with limited GPU memory at hand.

For inference time, we follow OV-DETR to report the second per iteration. Our model is trained and tested on NVIDIA RTX 3090, all statistics related with NVIDIA Tesla V100 comes from [R5, R8].

Dear reviewer,

Today is the last day for reviewers to ask questions to authors. Did the authors' rebuttal address your concern? Do you have any additional questions?

Thank you for acknowledging our framework as explicit and effective. Here's our response with some necessary explanations.

Q1: The motivation of this paper.

Reply: Thanks for your feedback. We clearly stated the motivation of our work as bellowing. Open Vocabulary Object Detection (OVD) aims to identify objects from novel categories that were not encountered during the training process. According to the training data, OVD methods can be mainly grouped into [R1]: 1) methods that utilize additional image-caption pairs beyond the detection training data; 2) methods that employ self-training strategy and generate additional pseudo labels for both base and novel classes on detection training data; 3) methods that rely on knoledge distillation from teacher VLMs. Our approach falls into the third group.

While methods that leverage 1) additional image-caption pairs or 2) pseudo labels can enhance the performance of open-vocabulary detectors, they often exhibit low data efficiency and incur high computational costs. Our proposed cyclic contrastive knowledge transfer seeks to optimize the knowledge transfer efficacy of the third knowledge distillation group without the need for supplementary data. Existing knowledge distillation approaches primarily focus on learning from the visual knowledge embedded in the teacher model. In contrast, we advocate for a cyclic learning process that incorporates both text and visual knowledge from the teacher by enveloping the detector in this cycle, while excluding the object queries, as they serve as inputs to the detector. This methodology significantly enhances cross-modal learning, as evidenced by our experimental results.

It is also worth noting that OV-DQUO adopts an iterative training strategy that gradually generates and adds unknown object proposals into the training set, hence it aligns more closely with pseudo-labeling group.

References:

[R1] A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Future. TPAMI 2024.

Q2: The definition of semantic priors and its difference between captions and prompts.

Reply: Sorry for the confusion. The semantic priors denotes the presence of a specific concept in current image. In existing OVD, the input texts to text encoder are typically category names from datasets, and they are fixed for each image. In contrast, we filter out non-existent concepts based on either VLMs, i.e., CLIP, or MLLMs, i.e., GVT. Hence the input texts are dynamic for each image. The similarity logits w.r.t. categories that are below threshold ρ are filtered out to ensure that only highly confident classes are identified as present ones. This avoids novel classes being overwhelmed by many non-existent base classes as our ablation in Sec. 4.1 confirms.

The semantic prior embeddings are text features of those remaining classes (above the threshold ρ) encoded by CLIP text encoder via filling their category names into template prompts. These embeddings are added to the learnable object queries to form the language queries and forwarded to decoder for box regression and classification.

Difference between semantic priors and captions or prompts: Prompts can be categorized into template prompts (e.g., "a photo of [CLASS].") and soft continuous prompts by prepending several learnable vectors before the [CLASS] token. We use semantic priors to resolve the concept existence problem stated in Sec. 3.2 and apply the template prompts to filtered classes.

Q3: The performance on base classes

Reply: Current setting of open-vocabulary detection is to train the detector on base categories $C_B$ and test on both base and novel categories $C_N$, where $C^B \bigcap C^N = \emptyset$. OVD mainly uses the AP for novel classes as the evaluatation metric, and we follow standard reporting practices [R2-R4] in literature that includes AP on novel and all categories.

Here we include the AP for both base and novel classes for a complete reference, and provide relevant analysis:

Results on COCO dataset:

We note that on base classes, our method is not as good as others. However, previous approaches might suffer from issue of overfitting on base classes. Here we define a new metric termed $\triangle\text{AP}_ {50}=|\text{AP}_ {50}^{B}-\text{AP}_ {50}^{N}|$ to quantitatively measure this phenomenon. The $\triangle \text{AP}_ {50}$ reflects the gap between performances on base and novel classes, open-vocabulary detection should achieve both high $\text{AP}_ {50}^{B}$ and $\text{AP}_ {50}^{N}$, but more importantly, a low $\triangle \text{AP}_ {50}$ to avoid skewed performance. As this new metric shows in the above table, the overfitting phenomenon is rampant especially on methods relying on extra image-caption pairs (CoDet) and self-training techniques (OV-DETR) since the generated pseudo boxes of base classes are more accurate than pseudo boxes of novel classes. Though our method is relatively low on base classes, we maintain a good balance between base and novel classes, achieving the lowest 0.3 $\triangle \text{AP}_ {50}$.

We also conduct a new experiment that trains CCKT-Det using data provided by OV-DETR (it includes generated pseudo boxes) to support our guess: we can observe performance of base objects surpassing novel objects by integrating more generated pseudo boxes as supervisions

If an open-vocabulary detector can only achieve superior performance on base categories but poor performance on novel classes, its effectiveness will be diminished, given that the evaluation of OVD essentially reflects not only AP but also the model's generalization ability to novel categories.

Since the hidden space of VLMs exhibits minimal bias regardless of the division of base and novel classes on COCO dataset under OVD setting [R4], there is no distinct 'base' or 'novel' for VLMs, this balanced structural knowledge is effectively distilled to our detector. A similar trend can also be observed in other knowledge transfer or knowledge distillation-based methods, such as ViLD-img and CORA, achiving a relatively low $\triangle \text{AP}_{50}$ compared to methods that utilize additional training data.

In other words, the student performance is affected by teacher. If teacher has limitations, they could propagate to student. We can further observe that when we employ a more robust MLLM (GVT) to guide our model, the performance on both base and novel classes is boosted (c.f. to CCKT-Det and CCKT-Det++ in Tab. 1).

To summarize, there is an inherent trade-off between base and novel categories, knowledge distillation based method could inherit both balanced semantics and limitations from the teacher. With the major evaluation metric for open-vocabulary detection being the novel AP, our method achieves superior performance while striking a good balance between base and novel classes.

Results on LVIS dataset with $\text{AP}_ {c}$ and $\text{AP}_ {f}$:

References:

[R2] F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models. In ICLR 2023.

[R3] Region-centric Image-Language Pretraining for Open-Vocabulary Detection. In ECCV 2024.

[R4] Open-Vocabulary Object Detection Using Captions. In CVPR 2021.

Dear reviewer,

Today is the last day for reviewers to ask questions to authors. Did the authors' rebuttal address your concern? Do you have any additional questions?

Thanks for your response and for taking the time to consider our rebuttal. We appreciate your acknowledgment that your concerns have been addressed. However, we would like to understand your reasoning behind maintaining a negative rating. If there are specific aspects of our work that you believe still require improvement or clarification, we would be grateful for your feedback. Our goal is to enhance the quality of our paper, and your insights would be invaluable in this process. Thank you once again for your time and consideration.

Thanks for your valuable comments and constructive suggestions. The following is our response with some necessary explanations. If you have any further questions, please feel free to let us know.

Q1: How would hungarian matching based on CLIP embeddings expected to significantly improve performance.

Reply: In our proposed contrastive knowledge transfer cycle, query embeddings are designed to align with region-level visual features provided by the teacher model. The contrastive loss establishes the optimization target to improve this alignment. To provide a more precise response, let’s consider this from the perspective of positive pairs in the contrastive loss framework.

Teacher Embeddings: In various region-level contrastive learning frameworks for object detection tasks, positive instances are generated in a self-supervised manner. These instances are derived from augmented views of the same object, characterized by variations in scale and location. In our method, positive instances are sampled from a teacher model using an object-specific feature cache. We utilize a visual encoder to extract features from cropped images, which are obtained from the bounding boxes of base categories. These region-level features are subsequently stored in a cache and sampled during each training iteration. For further details, please refer to Figure 3 in our main paper.

Student Embeddings: We enhance query embeddings through the integration of semantic priors. In contrast to the original queries in standard DETR, which are initialized to zero embeddings with learnable weights, this approach enables the queries to demonstrate increased differentiation during the Hungarian matching phase. Consequently, queries that are more likely to correspond to valid objects are selected by the Hungarian matcher and regarded as positive instances from the student model. In essence, the incorporation of semantic guidance enriches the semantic priors for the queries during the subsequent Hungarian matching, thereby facilitating the optimization process.

Q2: Limitations or biases in the teacher model may propagate to CCKT-Det.

Reply: Yes, we agree with your opinion. We reckon that CCKT-Det excels by leveraging advanced teacher models for semantic priors and feature extraction, it depends on their quality. Consequently, any limitation or biase inherent in these sources can affect our accuracy and introduce biases. However most open-vocabulary detection methods rely on teacher VLMs to achieve a good visual-semantic alignment, hence we leave this for future work.

Q3: Computational costs for researchers lacking high-performance infrastructure.

Reply: Leveraging VLMs/MLLMs indeed requires high-performance infrastructure. Scaling the teacher models provides an alternative to directly scaling the backbone of the student model for improved performance. The latter approach incurs increasing FLOPs with each forward pass, however, for researchers without high-performance devices, features from the teacher can be extracted once, saved as a .pkl file, and then loaded as needed.

Q4: Pseudo labels incur noise.

Reply: CCKT-Det utilize pseudo labels to determine whether a concept exist within one image. To mitigate noise, ground-truth annotations of base classes are employed during the training process. Pseudo labels are only utilized during the inference phase.

Q5: Semantic priors not generalizable to real-world applications.

Reply: Semantic priors enhance the model to detect novel objects within the learned semantic scope of teacher. Current evaluation settings and performance already demonstrate the generalization ability of CCKT-Det to novel concepts. While promising, the open-vocabulary detection community faces challenges in more fine-grained and unexpected scenarios, as the pretraining corpus of teacher VLMs may lack some "out-of-distribution" domain-specific data.

Dear reviewer,

Today is the last day for reviewers to ask questions to authors. Did the authors' rebuttal address your concern? Do you have any additional questions?