DiPEx: Dispersing Prompt Expansion for Class-Agnostic Object Detection

We thank the reviewer for the positive comment and constructive feedback! Please find our detailed response below:

R4.1 Clarification on "semantic overlap".

We appreciate the reviewer’s detailed feedback and understand that the section on “semantic overlaps” may seem disjointed. We conducted a pilot study using UNIVERSAL-query and CLASS-WIDE query sourced from ChatGPT and WordNet, respectively, to validate the efficacy of GroundingDINO in recognizing semantically similar and dissimilar words.

These results, shown in Table 1, support our hypothesis that semantic overlap negatively impacts detection performance. Additionally, a case study provided in Appendix A.2 reveals diminished confidence in GroundingDINO when presented with semantically overlapping queries, as exemplified by the contrast between "plates . cups ." (semantically different) and "plates . dishes ." (semantically similar), as illustrated in Figure 7.

Due to page limits, we regret any confusion caused by the placement of this crucial information in the supplementary materials, which might have been overlooked. To improve the clarity, we will incorporate this case study into the main body of the revised version.

R4.2 Inference cost comparison

Great suggestion! We have provided an additional inference cost comparison between our proposed method and two handcrafted baselines. Our results, detailed in Table 2, demonstrate that while the naive query-merging method reduces the computational cost, it significantly compromises detection accuracy due to overlapping semantics. In contrast, our proposed method strikes a balance, achieving superior detection performance. Specifically, our Dipex approach is 6.69% slower compared to the naive query-merging method, but it shows an impressive 89.34% reduction in inference time compared to the prediction-merging method.

R4.3 Technical details clarification

Thank you for your feedback! We apologize for the confusion regarding the notations. To clarify:

• In Section 3.1, the contextual embeddings $\mathbf{v}$ are denoted as $\mathbf{v} =\lbrace\mathbf{v}_i\rbrace ^{M} _{i=1} \in\mathbb{R}^{M\times d}$, where $M$ represents the number of $d$-dimensional learnable tokens appended to a text query $\mathbf{c}$. This follows the conventional way of introducing learnable context vectors [A].
• In Section 3.2, we introduce a novel prompt expansion method. Here, the prompts hierarchy is denoted as $\mathbf{v}_{l,k}$. The subscript $l$ indicates the layer in the tree or training round, while $k$ represents the number of learnable tokens in the $l$-th layer, similar to the notation in Section 3.1.

We hope this clarification clears up any confusion regarding the notations.

[A] Kaiyang Zhou et al. Learning to Prompt for Vision-Language Models, in IJCV.

R4.4 Concerns on Potential Random Noise

Thanks! The COCOEval metric is particularly strict in evaluating detection performance, as it only matches detections with the highest Intersection over Union (IoU), which can penalize minor localization errors. Additionally, the MS-COCO dataset is not comprehensively annotated, which can even lower the average precision (AP) as correct predictions for unannotated instances are mistakenly counted as false positives. At the very initial training stage (l=1), the learnable tokens can be highly uncertain. Optimizing a small number of prompts can lead to instability, given the insufficient parameters to capture the diverse pseudo-labels. Selecting only one parent prompt with high uncertainty for expansion in the first iteration may not provide enough stability. However, as the number of prompts progressively increases, the model becomes more mature, and we observe a steady improvement in performance. This trend illustrates how increasing the number of prompts helps manage uncertainties and improves overall performance, thereby validating our approach.

R4.5 In line 269, the paper claims that a higher MAC correlates with a broader spectrum of vocabularies. Aside from intuitions, is the range of vocabularies measured in some way? How did paper come to this conclusion?

We appreciate the reviewer's thoughtful question! The concept of MAC (Mean Angular Coverage) is inspired by the WordNet hierarchy, where a larger MAC indicates the discovery of more high-level semantics. While MAC serves as an approximation rather than a direct measure of vocabulary range, it provides a practical criterion for determining when to stop the training process. The main motivation behind MAC is to capture the breadth of concept coverage without the need for direct interpretation of the learned tokens, which is inherently complex. Although MAC does not explicitly measure the vocabulary range, its correlation with high-level semantic discovery offers a useful approximation for our purposes. We hope this clarifies our rationale.

R4.6 How exactly is the prompt logit activation computed?

Thanks for raising this question. As stated in line 285, prompt logit activation refers to the number of activated prompts based on the confidence threshold. Figure 5 illustrates the activation frequency (in log scale) of each expanded child prompt, providing a clear visualization of how different prompts contribute to the detection process.

Thanks again for reviewing our paper! We are more than willing to have a follow-up discussion with you if you still have any further concerns!

We thank the reviewer for the constructive comments and suggestions, which we address below:

R3.1 Discussion on more related works

Thanks for bringing the literature to our attention!

[1] proposed POMNet, which leverages a transformer-based Keypoint Interaction Module (KIM) to capture interactions between keypoints and the relationship between support and query images. [2] introduced CapeFormer, a two-stage framework where keypoints are matched and treated as similarity-aware position proposals in the first stage, addressing noisy matching results due to the open-set nature of the problem. [3] introduces a meta-learning approach combined with iterative point refinement techniques for class-agnostic pose estimation.

Please note these methods are specifically designed for pose estimation, which involves predicting key-points on objects. Object detection, on the other hand, focuses on identifying and localizing entire objects within an image. The granularity and nature of the tasks are different, making direct application challenging. Though we find these methods relatively out of scope to our current work, we acknowledge the importance of this area. We will add a detailed discussion accordingly in our revised version to broaden its relevance.

R3.2 The Abstract is not abstract enough.

Thanks for your constructive comment! Our intent was to provide a comprehensive summary of our work, outlining the goals and motivation clearly. While we believe it accurately reflects our research, we will endeavor to make it more concise.

R3.3 Clarification on evaluation.

Thanks for your suggestion! We confirm that our evaluation approach involves casting all existing categories from MS-COCO and LVIS into a single class, and we will provide a detailed explanation in our revised manuscript. As highlighted in our introduction, the primary goal of our paper is to enhance Average Recall (AR), which measures how comprehensive objects are captured. Through visual inspections of ground truth (GT) against our results, we observed that COCO and LVIS are not densely annotated (see Figure 9-10 in the manuscript for more visualization). This is the primary reason our AP appears low; correct predictions for unannotated instances are mistakenly counted as FP in COCOEval. To address this, we plan to develop a fairer and more comprehensive benchmark in the future.

R3.4 How about the efficiency of the proposed model in training/inference?

Thanks for raising this important question! Regarding the efficiency of our model, we have noted this as a limitation in the conclusion section of our paper. We have provided an additional inference cost comparison between our proposed method and two handcrafted baselines in the attached PDF. Our results, detailed in Table 2, demonstrate that while the naive query-merging method reduces the computational cost, it significantly compromises detection accuracy due to overlapping semantics. In contrast, our proposed method strikes a balance, achieving superior detection performance. Specifically, our Dipex approach, is 6.69% slower compared to the naive query-merging method, but it shows an impressive 89.34% reduction in inference time compared to the prediction-merging method. We will incorporate the efficiency analysis into the main body of the revised version!

Thanks again for reviewing our paper! We are more than willing to have a follow-up discussion with you if you still have any further concerns!

We thank the reviewer for the positive comments and constructive feedback.

R2.1 I prefer to owe the detection degradation to the disturbed attention, rather than the so-called semantic overlap.

We appreciate the insightful feedback. To clarify our "semantic overlap" hypothesis, we refer to the example in Appendix A.2. Our empirical study on "semantic overlap" is illustrated in Figure 7, where detection confidences are maintained with less similar words ("plates . cup ."), but performance drops with similar words ("plates . dishes ."). We conjecture that this is highly attributable to the dataset that was used during pre-training - there is a low chance of an image containing similar words [C], which may explain the reason why Grounding DINO was structured to favor specific categories over generic ones. Owing to this observation, we gained inspiration to hierarchically discover more fine-grained concepts from higher-level semantics.

[C] Shuai Shao et al. Objects365: A Large-Scale, High-Quality Dataset for Object Detection, in ICCV.

R2.2 Clarification & Additional Experiments

Great suggestion! We have run an additional experiment based on your suggestion, where we add additional prompts at the initialization stage. Please refer to Table 1 in the attached PDF for your reference. By increasing the number of initial prompts from ($k=1$ to $k=5$), we observe an improvement in the detection performance.

R2.3 Why MAC metric for expansion determination? How about using the mean angular or KNN metrics?

We appreciate the reviewer's suggestion regarding the MAC metric for expansion determination. Our choice of maximum angular coverage (MAC) is inspired by the observation of semantically dissimilar words have larger angular discrepancies as hypothesized in our case study in Appendix A.2. We aim to capture semantics as comprehensively as possible to improve recall. The angular discrepancy between prompts is enforced by dispersion loss, where we intend to push child pairs further, and maximum angular deviation represents the comprehensiveness of the vocabs. While using mean angular metrics is plausible, it tends to find fine-grained concepts such as "apple" & "oranges", which can be easily dominated by the abundance of fine-grained words. This contrasts with our goal of promoting diversity among learned prompts from a global perspective. KNN on the other hand may signify how detailed for a particular pair and the results can be very restricted to a particular concept. Due to limited time during rebuttal, we will add comparisons in the revised version.

R2.4 I wonder what will happen when adding the constraints to the encoded text features.

Thank you for your feedback！ As we are unsure about your concern, we interpret your questions as "how dispersion loss can affect the encoded text features". Firstly, we would like to clarify that MAC is the criterion for early stopping and it is not directly used in our optimization process. Dispersion loss, on the other hand, is our training objective which ensures that child prompts are sufficiently distinct from each other while not straying too far from their parent prompts. This combination ensures that our approach effectively captures diverse and unique concepts while maintaining coherence with the broader parent prompts. The learnable prompts are directly appended to the encoded text. The encoded textual query (e.g., "generic") is only used as a guide to acquiring pseudo-labels and it is not optimized as part of the learning objective.

R2.5 How to ensure the quality of pseudo labels for class-agnostic object detection?

Thanks! To re-iterate, we iteratively refine our pseudo-labels to ensure their quality during the training process. We start with predictions from an off-the-shelf GroundingDINO model, removing low-confidence detections and excessively large bounding boxes. At each self-training stage, we update the pseudo-labels by performing inference using the learned prompts. To avoid duplication, we eliminate boxes with an IoU greater than 0.5 compared to the previous "ground-truth" boxes and apply SoftNMS.

R2.6 Comparability with UniDetector

Thanks for the great suggestion! It is important to clarify that UniDetector is an Open-Vocabulary detector (similar to what GroundingDINO was originally designed for), which differs from our class-agnostic settings. UniDetector requires comprehensive class labels for training, whereas our approach only relies on pseudo-box supervision. Notably, UniDetector also proposes a class-agnostic detector (CLN), which combines both the RPN and RoI head to generate proposals for universal object detection. This highlights the significance of class-agnostic object detection in Open-World scenarios, aligning with our focus on detecting every object in the scene without relying on predefined class labels.

Q2.1 Detailed operations in Table 1

Thank you for your suggestion! As described in lines 52-57, query-merging involves concatenating all queries into a single string (e.g., "objects . generic . entities ."), whereas prediction-merging entails combining the results from separate inferences using individual text prompts (e.g., "objects", "generic", "entities"). We apologize for any confusion and will clarify this distinction in the revised manuscript.

Q2.2 Misuse of Notations

We appreciate your feedback! In this paper, we have used the terms "prompt" and "word" interchangeably, which may have caused some confusion. The notation in lines 134 and 137 is accurate; however, we will revise the manuscript to ensure that $P$ consistently refers to "prompt embeddings" for clarity.

Thanks again for reviewing our paper! We are more than willing to have a follow-up discussion with you if you still have any further concerns!

Dear Reviewer 86g8,

Thank you for your positive response and for raising your score! We are glad that our clarifications addressed your concerns.

Best,

Authors

We thank the reviewer for the constructive comments and suggestions, which we address below:

R1.1 Novelty and comparisons with [1,2]

Contributions. We appreciate your feedback and would like to clarify the major contributions of our work:

(1) General Impact: Our work presents an early analysis of the bottleneck of current class-generalized detection tasks such as OOD detection. The challenge of locating every possible object remains a fundamental challenge before being able to improve AP for classes of interest (line 28-32) has been rarely studied. Further, prior OOD works are incomparable due to inconsistent evaluation metrics (e.g., open-set benchmark uses FPR95, and unsupervised class discovery uses CorLoc). DiPEx is the first work to comprehensively benchmark existing OOD detection (open-set/open-world detection) and unsupervised object discovery against class-agnostic OD setting, providing valuable guidance for future research.

(2) Technical Contributions: We would like to point out that instead of using off-the-shelf contrastive loss ([1]&[2]), all our design is motivated by angular perspective to discover unrevealed semantics by pushing children prompts away while maintaining coherence with the parent prompts. The rationale behind this was based on our observation in pilot study (Appendix A.2 & Figure 7), where two text tokens that are semantically dissimilar exhibit large angular distances lead to more confident box predictions. This motivates us to hierarchically decompose high-level semantics into finer-grained ones, aiming to uncover more semantics.

Compared to SOTA. We appreciate the reviewer for highlighting [1, 2]. Key differences include:

(1) Task. Both [1] and [2] focus on domain adaptation for style diversification with classes available, or in other words, diversifying the suffixes prior to the class token. Additionally, both methods are constrained to a fixed number of style prompts. On the contrary, our design aims to disentangle arbitrary numbers of fine-grained classes, followed by a MAC early stopping strategy to prevent the excessive growth of class prompts. Notably, we are the first one to introduce progressive prompt expansion, proving its effectiveness. Our task, without class vocabulary available, is much more challenging.

(2) Objectives. Both [1] & [2] use off-the-shelf contrastive loss, which significantly differs from our dispersion loss. Contrastive loss separates inter-class samples; this contradicts our motivation of revealing classes in a tree hierarchy fashion. Additionally, orthogonal constraint in [1] is unsuitable; recall our ablation study (see Appendix A.2) - semantically similar words can have small angular discrepancy (e.g., "plates" & "dishes" have angular distance $\theta$ of 53.73$^\circ$). This means that the orthogonal ($\theta=90^\circ$) constraint is too much and will push prompts too far and distort underlying representation. Therefore, we have opted to use dispersion loss as a softer constraint to enforce separability between child prompts while maintaining child-parent coherence.

R1.2 The proposed method may also be suitable to other vision tasks.

We appreciate the reviewer's feedback. Adapting existing prompting methods (e.g., CoOp), designed primarily for classification tasks, to detection yields suboptimal performance (Table 2, 3), especially with a large number of classes (Figure 3). This is because Grounding DINO uses class and query confidence to find top-k proposals, and if class prompts are not accurately learned, many correct boxes will be missed due to low confidence. Our hierarchical expansion allows us to find accurate fine-grained semantics, specifically designed to help detection-based foundation models identify more objects and improve recall. Furthermore, detection serves as a basis for downstream applications, such as multi-object tracking, which we plan to explore in future work.

R1.3 Does the pseudo labels get updated during prompt-based optimization? If updated, how to do this?

Yes, our pseudo-labels are iteratively refined with each training process during the training process. (1) We start with inference using an off-the-shelf GroundingDINO model with "generic" and preprocess the predictions by removing low-confidence detections and excessively large bounding boxes. (2) At each self-training stage, we perform inference again with the expanded prompts to obtain updated pseudo labels. (3) For quality control, we remove boxes with an IoU greater than 0.5 with the previous "ground-truth" boxes and apply SoftNMS. This ensures the continual improvement of box accuracy and promotes the model to discover emerging concepts.

R1.4 What does the learned prompt resemble?

Great question! The interpretability of context tokens remains as an open question [A, B]. Prior arts [A] have attempted to use NN words from pre-trained embeddings to interpret the learned vectors. However, since text features operate within a continuous embedding space, they likely carry more abstract meanings that are not readily interpretable. Regardless, we have provided a visualization of the attention features from each distinct prompt in the attached PDF for your reference.

[A] Kaiyang Zhou et al. Learning to Prompt for Vision-Language Models, in IJCV.

[B] Brian Lester et al. The Power of Scale for Parameter-Efficient Prompt Tuning, in EMNLP.

R1.5 However, I fail to understand how the 52.46% and 23.64% reductions are computed by the results in Table 1.

Thanks! The percentage reduction was calculated using (query_merged-prediction_merged)/query_merged. We will provide more clarification in the revised version.

Thanks again for reviewing our paper! We are more than willing to have a follow-up discussion with you if you still have any further concerns!

Dear Reviewer Jt2M,

Thank you for your positive response and for raising your score! We are glad that our clarifications addressed your concerns. In our revised manuscript, we will make sure to include a discussion on the contrasts between our work and the literature you referenced [1, 2]. We will also further elaborate on the additional concerns you have raised in your review.

[1] Promptstyler: Prompt-driven style generation for source-free domain generalization. ICCV, 2023.

[2] Distilling Vision-Language Foundation Models: A Data-Free Approach via Prompt Diversification. ACM MM, 2023.

Best,

Authors.