LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors

We are heartened by the reviewer's recognition of our method's motivation and utility. We value your insightful summary and constructive feedback and address each point as follows:

Q1: Comparison with region-level prompting in CORA (CVPR 2023)

In contrast to CORA's approach, our visual prompts integrate background information from images. Table 9 of our paper shows that incorporating background information leads to performance improvements, illustrating the effectiveness of our method. We believe that this design offers a more comprehensive and contextual understanding of the image, aligning more closely with the visual prompt concept. Here are the experimental results:

This distinction in prompt design underlines the originality of our approach and its effectiveness in enhancing detection performance.

Q2: Lack of Parameters Experiment on Update Frequency for Dictionary Update

We would clarify that the update frequency of the descriptors dictionary is not equal to "the number of high-frequency descriptors". We conduct new experiments to assess its impact on performance. As the table below shows, DVDet maintains stable performance when the dictionary update frequency remains within a certain range

Q3: Effect of LLM on DVDet's Running Speed

• We appreciate the reviewer pointing out the need for experimental analysis on efficiency. In our study, the primary factor impacting training efficiency is the update of category descriptors. Notably, the update time for the descriptor dictionary accounts for only 20.16% of the total training time when the update frequency is set at every 100 iterations in our implementation. This ratio demonstrates that our approach achieves an efficient balance, being comprehensive while remaining resource-effective.

• We further conducted experiments to evaluate this, as shown in the table below. The results indicate that within a certain range, reducing the update frequency of the dictionary allows DVDet to maintain stable performance.

Q4: Handling Multiple Hyperparameters

Actually, our method focuses on three key parameters: the length 'm' of the enlarged proposals, the update frequency 'N' of descriptors, and the number 'TopN' of high-frequency descriptors. However, our experimental data indicate that the algorithm's performance is not highly sensitive to these parameters. This is supported by the results shown in Table 6 of our paper regarding the 'TopN' parameter. In response to Q2, we have analyzed the impact of the Descriptor Update Frequency 'N'. Additionally, to further demonstrate the importance of the parameter 'm', which determines the width and height of the proposal, we present the following experimental results:

Moving forward, we plan to implement learnable parameters and structures, aiming to reduce the necessity for manual hyperparameter tuning.

We are grateful that the reviewer recognizes the value, straightforwardness, and effectiveness of our method. We appreciate your concerns about the algorithm's efficiency increase and stability and will address each of your points in turn.

Q1: A comparison of computation costs with the baseline.

• We appreciate the reviewer pointing out the need for experimental analysis on efficiency. In our study, the primary factor impacting training efficiency is the update of category descriptors. Notably, the update time for the descriptor dictionary accounts for only 20.16% of the total training time when the update frequency is set at every 100 iterations in our implementation. This ratio demonstrates that our approach achieves an efficient balance, being comprehensive while remaining resource-effective.

• We further conducted experiments to evaluate this, as shown in the table below. The results indicate that within a certain range, reducing the update frequency of the dictionary allows DVDet to maintain stable performance.

Q2: Prompt Generation Randomness from LLMs

We are pleased that the reviewer has pointed out the impact of descriptor randomness on experimental performance. In fact, DVDet's interactive mechanism was designed with the generative model's randomness in mind. We have implemented two relevant mechanisms:

• Descriptor Update Mechanism: By retaining high-frequency descriptors and discarding low-frequency ones, we ensure the visual relevance of the descriptors.
• Descriptor Selection Mechanism: During the inference phase, we discard irrelevant descriptors, thereby enhancing stability.
• We further present the performance variability across five experimental runs to demonstrate the robustness of our algorithm.

Q3: Lack of Ablation Experiment on Hyper-Parameter m of Conditional Context Prompt:

• Firstly, in Table 9 of our paper, we provided ablation studies regarding background information, showing the experimental outcomes when parameter m is set to 0.

• We conducted further experiments about the parameter m. As the table below shows, DVDet's performance remains stable when m lies within a certain range:

We are grateful for the reviewer's interest in understanding how descriptors and regional prompts influence detection results. We address each point of interest in turn.

W1: when LLM interacts with VLM, Could you give more examples of prompts to use and how the prompt may influence the final results:

In this paper, we utilize three prompts to interact with LLMs to gather descriptors, 'Template N' only includes the category name, ‘Template H’ [1] includes high-frequency descriptors and the category name, and ‘Template C’ includes confusing categories and the category name. The performance using different prompts is as follows:

Our experimental findings show that our designed templates, namely templates C and H, are more effective than the template N used in [1] for prompting LLMs. This results in the generation of effective descriptors and an overall improvement in performance, highlighting the efficacy of our method in descriptor generation.

[1] Menon S, Vondrick C. Visual classification via description from large language models[J]. ICLR2023.

W2: What will happen if we just let LLM do some general description of objects without seeing the objects?:

• Firstly, in our method, the Large Language Models (LLMs) understand objects through their interaction with Vision Language Models (VLMs). So, during training, the LLMs don't see novel classes. The improvement in detecting these novel classes, which are unknown to LLMs, shows how effective our generated descriptors are.

• Secondly, we present experimental results where all classes were unknown to the LLMs. The results from Table 5 of our paper are as follows:

These results indicate that even without interacting with detection models, the descriptors from LLMs still improve the performance of these models.

W3: if the regional prompt is not accurate, how will the performance look like when interacting with LLM?

As the reviewer correctly points out, the accuracy of prompts can indeed affect the algorithm's performance. As shown in Table 3 of our paper, we present the performance of DVDet with Prompt Learning. To further illustrate the impact of prompt accuracy, we added a new experiment using a Random Prompt. This additional experiment clearly demonstrates that inaccurate, randomly selected prompts lead to decreased performance. In contrast, our original approach with prompt learning significantly improves detection accuracy.

We are really encouraged that the reviewer recognizes our method to be intuitive and effective.

We thank the valuable comments and insightful suggestions, and we hope our detailed responses below can address your concerns.

W1: Discussion with Concurrent Work [1] and [2].

As the reviewer noted, our work mainly involves the generation of category descriptors and the regional visual prompt. We appreciate the reviewer pointing out the need to compare our approach with the latest methods CORA [1] and MCVOD [2], and we provide a comparison with each below.

Firstly, regarding the generation of descriptors, our approach differs significantly from MCVOD [2] in the way of interacting with LLMs. Specially, we leverage the interactive capabilities of LLMs to dynamically generate visually relevant descriptors for each category, rather than treating the process as a one-time interaction. As shown in Table 5 of our paper, introducing this interaction mechanism generally leads to a performance gain of around 1.5%. The experimental results are as follows:

Secondly, regarding visual prompts, our design differs from the prompts in CORA [1] by integrating background information from images. As Table 9 of our paper shows, we observe further performance improvements when the background information is incorporated. As the reviewer pointed out, prompts without background information can be considered tailored for detection tasks (prompting). Incorporating background information further merges contextual information (context), making it more consistent with the image. The experimental results are as follows:

We will add the discussion with the concurrent methods[1, 2] in our updated manuscript.

[1] Wu X, Zhu F, Zhao R, et al. CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching[C]. CVPR 2023.

[2] Kaul P, Xie W, Zisserman A. Multi-Modal Classifiers for Open-Vocabulary Object Detection [C]. ICML 2023

W2: Gains on Base vs. Novel Categories.

Thank you for highlighting this important point. We acknowledge that our method does not show much more improvements in novel categories compared to base categories. Nonetheless, it's notable that our algorithm achieves consistent enhancements in both.

As illustrated in Figure 1 of our paper, unlike classification models using VLMs, current open-vocabulary detectors struggle to align visual features with descriptions of categories. Our ablation studies in Table3 of our paper, visualized in the results, demonstrate that category descriptors do not boost the pretrained model in detection tasks as effectively as they do in classification tasks [3], occasionally leading to reduced performance.

In our training process, we focus on aligning both base categories and descriptors, endowing them with better cross-modal alignment capability. This approach further yields improvements in novel categories, thereby confirming our method's effectiveness in enhancing open-set detection performance. The visualizations in Figure 5 of our paper further demonstrate how descriptors progressively refine detection results in new categories.

[3] Menon S, Vondrick C. Visual classification via description from large language models[J]. ICLR2023.

W3: Decreased Performance with Descriptors without Prompt Learning

Thank you very much for the constructive suggestion. The reason for this phenomenon, where adding fine-grained descriptors by itself without prompting hurts the performance of the model, lies in the disparity between the capabilities of VLMs and current open vocabulary detectors in cross-modal alignment. As shown in Figure 1 of our paper, the visual features extracted by VLMs maintain high consistency with the textual embeddings of descriptors. Therefore, in classification tasks [3], they can directly enhance performance in the inference stage without additional training. However, this is challenging to achieve in detection tasks. To make this conclusion more convincing, we have also compiled results on Detic and provided the following ablation study:

We will update the experiments and discussions.

We thank the valuable comments and insightful suggestions, and we hope our detailed responses below can address your concerns.

W1: Analysis of Descriptors from LLMs, essential for evaluating their quality and impact on improving object detection.

Thank you for highlighting the importance of descriptor analysis, which is indeed a central aspect of our work. Effectively generating descriptors through LLMs forms the core of our approach.

• Firstly, we designed an interactive mechanism for refining descriptors through ongoing interactions with LLMs. In contrast to static descriptors, which are produced through a one-time interaction, our interactive approach updates existing descriptors based on their contributions in enhancing detection performance. This point is well demonstrated in the experiment table below:

• Secondly, our semantic selection mechanism chooses visual-related descriptors, thereby avoiding the influence of descriptors that are not visible in the current sample. This design highlights the importance of the relevance of the descriptors as validated in the table below.

• Lastly, we provide extensive visualization in Figures 7 and 8 of our paper, which illustrate how the descriptors improve detection performance. These visualizations clearly demonstrate the effectiveness of our descriptor generation in enhancing object detection performance.

W2: The performance enhancements (~2% on average) in object detection.

We would like to share that a 2% improvement in performance metrics is quite acceptable in the current setting. For instance, methods like VLDet [1] which incorporates caption data also achieve only around 2% improvement compared to those without caption data. However, we would like to reiterate two key points: first, our method employs lightweight prompt learning, and second, it achieves consistent improvements across multiple detectors.

[1] Lin C, Sun P, Jiang Y, et al. Learning object-language alignments for open-vocabulary object detection[J]. ICLR, 2023.

Q1: The training efficiency affected by the descriptors generation process.

We appreciate the reviewer pointing out the need for experimental analysis on efficiency. In our study, the primary factor impacting training efficiency is the update of category descriptors. Notably, the update time for the descriptor dictionary accounts for only 20.16% of the total training time when the update frequency is set at every 100 iterations in our implementation. This ratio demonstrates that our approach achieves an efficient balance, being comprehensive while remaining resource-effective.

Q2: Complexity and Performance Trade-offs:

Building on our previous discussion, we identified that the frequency of dictionary updates (i.e., the frequency of interactions with LLMs per number of iterations) is a key factor influencing training efficiency. We conducted experiments to evaluate this, as shown in the table below. The results indicate that within a certain range, reducing the update frequency of the dictionary allows DVDet to maintain stable performance.

Q3: Performance Under Varying Complex Conditions

In Figure 7 of our paper, we demonstrate how fine-grained descriptors correct the model's detection results. Further visualizations of this process are provided in Figure 8 of our paper. These visualizations clearly show that our approach significantly improves detection, particularly in challenging scenarios. These scenarios include (1) detecting distant or occluded objects, and (2) distinguishing objects with small inter-class variations.