Open-Det: An Efficient Learning Framework for Open-Ended Detection

Dear reviewer JNJc,

We sincerely appreciate your valuable feedback and have incorporated all suggestions into the manuscript for the next version.

Q1: OED is close to OWD. It could ... section.

We have carefully checked the OWD [1], which enables the detector to label unknown objects as ''unknown'' and incrementally learn these identified unknown categories without forgetting previously learned classes. In contrast, our method could directly detect novel/unseen categories without requiring any prior human knowledge. In other words, both OWD and OED are capable of detecting and recognizing new categories, demonstrating their relevance. Following your suggestion, we will include this work in the related work section in the new version.

Q2: The authors ... insights ... improve it.

Similarly to the existing OED framework of GenerateU, Open-Det's performance is primarily constrained by cross-modal semantic discrepancies between visual regions and image-like textual embeddings. These discrepancies arise from the interactions among the detector, the VLM, and the LLM. To mitigate this limitation, employing stronger foundation models, such as ALIGN and CogVLM-17B for VLM models and DeepSeek for generative language model, is an efficient method for further performance improving. Additionally, training on supplementary datasets can serve as an effective approach to enhance performance.

Furthermore, integrating a segmentation module into Open-Det framework will offer mask priors for more precise regional and semantic feature extraction. This enhancement can further address semantic gap in cross-modal representations, transforming Open-Det into a unified detection-segmentation framework and boosting performance in both tasks. We will incorporate this discussion in the new version and propose it as a direction for future research.

Q3: Why the ... Swin-L model ... GenerateU?

Compared to Swin-T, the Swin-L model has a significantly larger number of parameters (197M vs. 29M, a difference of 6.8$\times$) and supports a higher input image size (384 $\times$ 384 vs. 224 $\times$ 224, with GenerateU using a pretrained Swin-L at a resolution of 384 $\times$ 384). This results in a substantial increase in GPU memory usage during model training (e.g., Swin-L requires approximately 4 $\sim$ 5 times more memory than Swin-T), making it infeasible for Open-Det to train on 4 V100 GPUs.

It is fair and effective to evaluate the performance of GenerateU and Open-Det models with the same backbone (such as Swin-T). Therefore, we did not use Swin-L for performance comparisons in the first instance.

However, as suggested, we managed to conduct additional experiments with two backbones: Swin-S (using 4 V100 GPUs) and Swin-L (using 4 A800 GPUs). The main experimental results are as follows:

Results:

• Open-Det-Swin-S achieves an improvement of +5.0% in AP$_r$ (26.0% vs. 21.0%) and +3.4% in AP (30.4% vs. 27.0%) than Open-Det-Swin-T, demonstrating its efficiency.

• Using only 1.5% training data, Open-Det-Swin-S outperforms GenerateU-Swin-L by +3.7% in AP$_r$ and +2.5% in AP.

• When utilizing the larger backbone of Swin-L, Open-Det-Swin-L significantly outperforms GenerateU-Swin-L by +8.9% in AP$_r$（31.2% vs. 22.3%）and +5.2% in AP（33.1% vs. 27.9%）, further confirming its superior effectiveness and efficiency.

We will include these results in the new version.

Q4: Did you try to ... is improved?

Intuitively, more training data would further improve the performance of Open-Det. Unfortunately, due to current limitations in available time (within the rebuttal days) and GPU resources, conducting experiments with the large-scale dataset (such as GRIT5M) presents significant challenges.

As an implementable scheme, we further extended Open-Det's training on the VG dataset from 31 to 50 epochs to investigate potential performance improvements under practical resource limitations. The results show that extended training further improves the performance with +0.9% (21.9% vs. 21.0%) in AP$_r$ and +0.4% (27.4% vs. 27.0%) in AP, confirming its effectiveness. However, we believe it remains valuable to investigate the impact of training data scale on performance for furture studies when more GPU resources become available.

[1] Towards open world object detection. CVPR 2021.

Dear reviewer AMke,

We thank your time and feedback. However, we respectfully disagree with comments (Q1 to Q7) and the assertions（the omitted citations and misleading novelty) for the following reasons:

Q1: The ..., yet ... cite ... novelty.

(1) Reviewer's Factual Error: The reviewer claimed 3 times: we failed to cite any of the 5 specified papers. In fact, we have explicitly cited 3 out of 5 in multiple places in main paper: Deformable DETR [1] (Line 338), DINO [2] (Lines 87, 132, 152, 295, 796), and DAC-DETR [4] (Lines 79, 155, 161, 295, 797). The assertion that we deliberately omitted citations - and thus misleading novelty - has no ground and constitutes a factual inaccuracy requiring correction.

(2) Exclusion of [3,5]: DAB-DETR [3] differs from Open-Det in task and method (e.g., Q2). Similarly, ScaleDet [5] tackles label unification for multiple-dataset training, differing from our focus in Q7. We initially excluded them but may be included in Related Works if space permits.

(3) Open-Det enhances detection across object granularities (coarse-to-fine) and scales (large-to-small) (not solely in training acceleration). Notably, it demonstrates additional vocabulary priors are not necessary in inference, with 11.7% training data achieving +6.8% AP$_r$ and +8.5% AP over GLIP(A) of OVD. We believe our work establishes a foundational framework for future OED research.

Q2: Object is ... found in [1][2][3].

Reviewer's Factual Error:

• [1]: Obtains Content Query (CQ)/Positional Query (PQ) via Top-K selected sinusoidal-encoded anchors (no threshold);

• [2]: Initializes CQ statically, derives PQ from Top-K boxes (no threshold);

• [3]: Uses static initialization for both CQ/PQ (no threshold)

Innovation of Open-Det: Different from [1-3], Open-Det introduces a NOVEL single adaptive query type (rather than CQ and PQ), transforming fixed number of objects prediction into flexible.

Q3: Object ... DAC-DETR.

• Reviewer's Factual Error: DAC-DETR [4] employs a dual-decoder architecture (standard O-Decoder and auxiliary C-Decoder) to enhance training efficacy, rather than dividing the first and last two decoder layers.

• Advantages of Open-Det: In contrast, our Open-Det directly optimizes the architecture of standard single-decoder (Sec.3.2), enabling lower training costs and superior efficiency.

Q4: Vision ... deformable attention (DA).

This claim misinterprets our contribution. We did NOT claim DA is our contribution. It is not suprise that DA is widely used. We also follow this practice. However, one of our novelty is the design of VLD-M (Fig.3), leveraging it to construct image-like queries for visual-text alignment. This represents both a novel architectural design and distinct application.

Q5: Vision ... supervise ...

• This claim without gives any references. Text-embedding to supervise query feature is NOT a common trick but a priciple paradiam for vision-text alignment. Exisiting mehtods differ in how the supervison is done, rather than whether or not simply using it. The reviewer's claim is NOT professional.

• Innovation of Proposed VLD-M: Our VLD-M aims to adaptively construct image-like queries to reduce region-text annotations (Sec.3.4), improving AP$_c$ by +3.9% and AP by +2.8% (Table 2).

Q6: ONG: Noisy ... [2].

DINO adds biased coordinates noise into box to accelerate box convergence. However, Open-Det adds Gaussian noise into text embeddings of VLM to speed up LLM's convergence. They differ fundamentally in both objectives (box vs. LLM) and noise types (biased coordinates vs. Gaussian).

Q7: MAL: ... ScaleDet [5].

• Reviewer's Factual Error: MAL is NOT used for text-classifier but for query-text alignment.

• Text Label Unification ≠ Multimodal Alignment: They are fundamentally different in motivation and method. ScaleDet unifies text labels via semantic similarity (as soft label in MSE) to combine multi-dataset training, while MAL resolves query-text matching conflicts through similarity-binarized BCE updates (Sec.3.6).

Q8: Missing ... .

• We conducted test on COCO. Open-Det still outperforms GenerateU by +2.8% AP (35.8% vs. 33.0%).

• Furthermore, additional experiment shows that Open-Det-SwinL surpasses GenerateU-SwinL by +8.9% in AP$_r$ on LVIS (see Q3 responses to JNJc), demonstrating significant superiorities.

---

According to our responses to Q1~Q8, our method is NOT an adaptation of existing techniques, but rather respresents a fundamentally different approach, innovating in framework, modules, loss functions, performance and efficiency.

Overall, we respectfully request the reviewer to cross-check the citations, methodologies, as well as all our responses, and we appreciate a fair judgement.

Dear reviewer E2VT,

We deeply appreciate your suggestions and will include them into our revisions.

Q1: Training free VL-SAM .. better.

A direct performance comparison requires careful consideration of model scales:

• VL-SAM: VLM: 17B, LLM: 7B, SAM: 0.632B, totaling 24.632B
• Open-Det: VLM: 0.307B, LLM: 0.250B, Detector: 0.049B, totaling 0.606B (ONLY 2.46%)

With significantly less parameters (40.6$\times$ ↓), Open-Det is still on par with VL-SAM in AP$_c$ (24.8% vs. 25.3%) and even higher in AP$_f$ (30.1% vs. 30.0%), indicating better trade-off between resouce and performance.

Q2: (1) How ... ambiguous? (2) What's ... accuracy? (3) How ... consistent results?

Language (text) inherently exhibits ambiguity, a characteristic independent of the design of OED models.

(1) Similar to GenerateU, Open-Det also adopts ''beam search'' strategy to generate multi-level object names , effectively addressing the ambiguity issue and enhancing accuracy.

(2) For fair evaluation, we adopt GenerateU's standardized protocol: comparing latent space similarities score between generated and annotated texts, eliminating evaluation biases from textual variability.

(3) The adopted beam search strategy ensures consistent results, and Open-Det achieves higher text similarity scores (Appendix Sec. C.1) than GenerateU.

Q3: (1) How ... same category? (2) Does ... conflicting gradients, or (3) could ... detection tasks?

(1) MAL transfers one-to-one matching (query to category, Fig. 6) to one-to-many matching (single query to multiple same-category instances) via:

• Computes text similarities between all object-name pairs, offering a quantitative basis for dynamic label matching correction.

• Corrects dynamic label matching (from 0 to 1) by transforming unmatched same-category pairs into matched ones (one-to-many) based on text similarities, resolving contradictory loss terms during optimization.

(2) Yes, MAL efficiently addresses contradictory loss generation at its source.

(3) MAL depends solely on textual ground truth, making it invariant to scene variations, but may be related to the annotations accuracy and text encoder.

Q4: (1) The use of AP ... category names? (2) how ... descriptions (e.g., ...)?

(1) The evaluation assesses both box accuracy (IoU) and object naming performance (via cosine similarity (CS) score between generated and annotated texts). If object name is wrongly predicted, the CS score will be lower, thereby reducing the performance of AP.

(2) Generated name granularity varies by training data annotations. Similar to GenerateU, Open-Det focuses on OED task, using detection annotations (i.e., object names and boxes) without utilizing complex object descriptions. Thus, Open-Det mainly predicts object categories (Fig.4, 10, 11), rather than fine-grained descriptions (i.e., obtaining relations between objects). We view the task of finer-grained name generation as an important area for future research.

Q5: The ... Grounding DINO or ... CoDET.

The comparison between Grounding DINO and Open-Det are as follows:

Using only 5.3% training data, Open-Det-SwinT still significantly outperforms Grounding-DINO-SwinT by +6.6% in AP$_r$, +5.2% in AP$_c$, and +1.4% in AP, further confirming its effectiveness and efficiency.

CoDET [1] is designed for programming problems and has NOT been evaluated on object detection tasks. Due to fundamentally different objectives (Code Generation vs. Object Detection), a direct detection performance comparison is invalid. We also cross-check Co-DETR [2], which is designed for closed-set detection and can't be directly compared to our OED model.

Q6: (1) While ..., ... low-data regimes, such as ...? (2) Does ... training set?

(1) Like most OVD and OED models, Open-Det also faces common challenges related to low-data regimes and few-instance learning.

(2) However, Open-Det demonstrates superiorities in both efficiency and generalization via VLM knowledge distillation into proposed VL-prompts. Notably, using significant less training data, it obtains superior performance on rare categories (namely only few instances of new category): +6.8% AP$_r$ over GLIP(A), +6.6% AP$_r$ over Grounding-DINO-SwinT, and +3.6% AP$_r$ over GenerateU. Using 1.5% training data, Open-Det-SwinL outperforms GenerateU-SwinL by +8.9% in AP$_r$ (Please see our responses to JNJc in Q3)

[1] Codet: Code generation with generated tests. ICLR 2023.

[2] Detrs with collaborative hybrid assignments training. ICCV 2023.

Dear reviewer EV8Z,

We sincerely appreciate your time and constructive suggestions. We have carefully integrated your suggestions into the manuscript and will include these updates in the next version.

Q1: (1) Can the proposed ... more training data? (2) Comparing to ... improvement. It would be interesting to see the performance with VG and GRIT5M.

Thank you for your valuable suggestion.

(1) Intuitively and empirically, it is commonly observed that larger datasets can enhance model accuracy in deep learning, as demonstrated in studies such as [1,2,3]. This principle is applicable to the Open-Det model as well.

(2) There are two key reasons why Open-Det was not trained using the combined VG and GRIT5M datasets. Firstly, Open-Det's design prioritizes model efficiency, minimizing reliance on large datasets and training resources. Remarkably, when trained on the single VG dataset for only 20.8% of GenerateU's training epochs, our model outperforms GenerateU which used both datasets. This result effectively demonstrates the effectiveness of our approach. Secondly, the GRIT5M dataset is 64.6$\times$ larger than VG dataset, which would require either:

• 64.6$\times$ more GPU resources for the same training time (for instance, GenerateU was trained using 16 A100 GPUs), or
• 64.6$\times$ longer training time on our 4 V100 GPUs

These demands would result in substantial resource requirements and costs for model training. As a result, we excluded GRIT5M training from our initial submitted manuscript.

Unfortunately, due to current limitations in available time (within the rebuttal days) and GPU resources, conducting experiments with the large-scale GRIT5M dataset presents significant challenges. However, following the reviewer's suggestion, we implemented a more computationally efficient alternative by extending Open-Det's training on the VG dataset from 31 to 50 epochs. This modified experimental design serves to:

• Investigate potential performance improvements under practical resource limitations;
• Validate the model's effectiveness without requiring additional combined dataset training;

The experimental results are as follows:

Our extended training experiments yielded consistent improvements across all evaluation metrics. Compared to the 31-epoch baseline, additional training achieved: +0.9% improvement in AP$_r$ and +0.4% improvement in AP. Notably, Open-Det trained solely on VG dataset outperforms GenerateU (trained on both VG and GRIT5M), demonstrating +1.9% higher AP$_r$ and +0.6% higher AP. These comparative results provide further evidence of Open-Det's superior efficiency and effectiveness in the OED task.

Experiments on Larger Backbones: In addition, when using larger backbones, such as Swin-S and Swin-L, Open-Det achieves significant improvements across all metrics. For instance, Open-Det-Swin-S obtains an improvement of +5.0% in AP$_r$ and +3.4% in AP than Open-Det-Swin-T. Using only 1.5% training data, Open-Det-Swin-L significantly outperforms GenerateU-Swin-L by +8.9% in AP$_r$（31.2% vs. 22.3%）and +5.2% in AP（33.1% vs. 27.9%）, further confirming Open-Det's significantly superior effectiveness and efficiency. For more information, please refer to our responses to the Reviewer JNJc in Q3.

Q2: Figure 2 is complex and it is not easy to understand the entire pipeline.

Thank you for your valuable suggestion. Figure 2 may appear complex due to the extensive detailed information it contains. In Appendix A.1 (page 12) of the submitted manuscript, we have presented a simplified version of the Open-Det pipeline (Figure 5), which highlights the core workflow and provides a clearer illustration of the entire process for better clarity. Additionally, the caption of Figure 2 has explicitly indicated the relationships between the two figures, contributing to a better understanding of the main pipeline. We deeply acknowledge the reviewer's suggestion and will further modify Figure 2 by reorganizing the figure layout to make it more understandable.

[1] Grounded language-image pre-training. CVPR 20222.

[2] Grounding dino: Marrying dino with grounded pre-training for open-set object detection. ECCV 2024.

[3] Detclipv3: Towards versatile generative open-vocabulary object detection. CVPR 2024.