QFree-Det: Query-Free Detector with Transformer and Sequential Matching

Q5: Dynamic query selection and the dual-step decoding process may require more computational resources, potentially impacting processing speed in real-time detection applications.

In QFree-Det, the dynamic query selection and the dual-step decoding process do not require substantial additional computational resources.

For the dynamic query selection, the AFQS algorithm introduces a simple yet effective threshold-based method, which maintains consistent complexity. Specifically, the AFQS algorithm serves two main purposes: converting fixed queries into free queries and addressing the high training GPU memory demand issue associated with excessive query numbers during training. It consists of two steps:

(1) generating classification scores for all encoder tokens, which has a fixed computation complexity;

(2) selecting queries using a threshold-based method that compares the scores of all encoder tokens to get a global solution, also with fixed computation complexity.

Additionally, regarding inference time, most of the duration is spent on step one (approximately 7ms for a 2080 $\times$ 800 image), while step two takes less than 1 ms (For more details, please refer to our responses in the General Responses section).

For the dual-step decoding process, as mentioned in the responses to Q3 and Q6, our LDD decoder consists of 4 LBP layers and 2 DP layers (totaling 6 CA and 4 SA layers), reducing both the number of parameters and computational load compared to existing methods, which use 6 decoding layers (with 6 CA and 6 SA layers). We further tested the model's inference speed, and our LDD decoder is faster than the decoder of DINO, with an improvement of 34.8% under the same number of queries (more details in General Response), indicating the effectiveness of our AFQS method and LDD framework.

Transformer-based detectors commonly face challenges in real-time applications due to their core multiple components, which involve numerous time-consuming attention operations. Our newly designed efficient LDD decoder framework helps mitigate this issue to some extent, making it possible to integrate with lightweight models like YOLO and further enhance the model's speed. Additionally, our free-query model holds significant potential for open-ended object detection task. This task directly generates object names through a generative large language model, where the query count greatly influences the model's complexity. The adaption characteristic of our model enables reduced queries to feed into generative large language model, thereby decreasing the workload for subsequent category generation processing, which highlights its potential applications and importance for the future research in vision community.

---

Q6: According to Table 5, free-query with AFQS only is worse than fixed-query.

Yes, the performance of the free-query approach using only AFQS is slightly below (by 0.3% AP) than fixed-query method. Naturally, within an appropriate range of query numbers, more queries can lead to more detections, thereby enhancing model accuracy. A fixed number of predefined queries (such as 900) maintains a constant query count across both sparse and dense scenarios. In contrast, when applying the AFQS algorithm alone, the number of queries adapts to the input image, which presents a greater challenge for SA, as it must accommodate various patterns of query numbers. This may lead to a slight decrease in accuracy, as shown in Table 5 (line 477 in the main paper). However, compared to the fixed-query detector, the free-query detector is crucial for enhancing model's flexibility, applicability, and scalability in real-world scenarios, making it a promising trend for future research. Furthermore, the issue of accuracy decrease can be effectively mitigated through other methods.

The decrease in accuracy resulting from the reduction in the number of queries is a fundamental limitation of the query-based detection framework, where each object prediction is from a single query. This issue is hard to be effectively addressed by relying solely on query selection algorithms like AFQS. As a result, we introduce a more efficient decoder of LDD architecture, which incorporates 2 SA units in each DP module, enhancing the ability of decoder to handle different patterns of query numbers. After using the LDD decoder, the accuracy of our model improved by +1.2% AP, as shown in Table 5 (line 478) of the main paper.

Our LDD architecture effectively leverages the roles of CA and SA, enabling the DP module to adapt to these pattern changes of query numbers. In addition, the overall number of SA used in our decoder is two fewer than in existing methods (4 vs. 6), and there are NO additional decoder branches in LDD architecture compared to other methods, further demonstrating the effectiveness of our AFQS and LDD approaches.

Continue the answer of part 1 to Q2:

At the same time, we explore another dimension of one-to-one and one-to-many method (beyond the label assignment level), specifically the opposing effects of CA and SA (with CA used to aggregate predictions of boxes to a single object (one object to many queries); SA used to disperse boxes for a single query to one object (one query to one object) and reduce the confidence scores of similar queries, as detailed in Sec.3.1.3 of the main paper). By effectively leveraging the CA and SA in decoder structure, we have alleviated the recurrent shifting issue associated with both CA and SA. This is illustrated in Appendix Table 12 (the ablation on BLP and DP layers) and Table 14 (the ablation on the order of CA and SA) of the Appendix in main paper, which highlight our ''sequential matching'' impact on accuracy.

---

Q3: Deduplication Part (One-to-one matching) can be replaced by the parameter-free NMS. The Deduplication Part (One-to-one matching) looks more complex than NMS.

Thanks very much for the significant comment. DP is an integral part of the decoder and not a post-processing algorithm. Although we designed DP to eliminate duplicates, it is distinct from NMS and cannot be directly replaced by it. Our model's decoder, similar to those in existing DETR-like models, comprises six layers: 4 LBP layers and 2 DP layers, connected in series. As analyzed in original DETR [1] paper, the decoder primarily reasons about the relations of queries and image context to generate detections, which is necessary for the detector. In Table 12 of the Appendix of main paper, we conducted ablations on the BLP and DP layers. The results (line 941 of main paper) indicate that reducing the number of DP layers (for example, using only one DP layer) is harmful for the performance, underscoring the importance of the DP layer.

In Table 2 (line 249 and 250) of the main paper, we present a free-query model that removes all SA (approximating the removal of the DP module), while employing the NMS to eliminate duplications. This model achieved only 47.5% AP on COCO, which is significantly lower than the 50.5% AP achieved with our model with DP, clearly highlighting the importance and irreplaceability of the DP.

On the other hand, NMS can only perform deduplication on the detection results that have already been obtained, and it is sensitive to parameters; it cannot generate detection results directly, making it unsuitable as a replacement for the decoder. The DETR model was originally designed to simplify the detection process in an end-to-end pipeline, removing post-processing steps like NMS and improving model robustness. Our method aligns well with this goal, enabling the model to function effectively without relying on NMS, but not to pull NMS back again.

The original DETR paper also notes that the FFN (Feed Forward Network, FFN) in the decoder has a significant impact on model accuracy. We have made a test on FFN of DINO model to reduce computational complexity by decreasing the hidden layer dimension in the FFN from 2048 to 768. However, this change resulted in a 1.6% decrease in AP, underscoring the critical role of the decoder network layers in maintaining performance. This further underscores the importance of the decoder.

[1] Carion N, Massa F, Synnaeve G, et al. End-to-end object detection with transformers[C]// ECCV2020.

---

Q4: PoCoo loss is useful for small object detection. However, its connection to the motivation of designing a query-free detector is very weak.

Thanks, we agree with it, and PoCoo loss is a supplementary contribution of our method. As presented in Sec.3.5 of main paper, PoCoo loss is designed based on IA-BCE loss, and IA-BCE loss efficiently addressed the correlation between classification score and localization precision. Our PoCoo loss further effectively leverages the prior information of box annotation, playing a crucial role in the model's training process.

Dear reviewer,

We sincerely appreciate your thoughtful and constructive feedback. We are eager to respond the concerns and hope our responses will align with your expectations.

Q1: The contribution of AFQS algorithm.

We appreciate for the valuable feedback and suggestions. The decoder query plays a crucial role in reasoning about the relations of the object and the global image context to output the detections, connecting the components of the decoder and greatly impacting the performance. Transforming a fixed-query detector into a free-query detector is not merely a matter of switching from a predefined to a dynamic number of queries in DETR models; it is a complex process. As discussed in Sec.3.1.2 and Sec.3.1.3 of the main paper, this transformation is influenced by the decoder architecture, the statically initialized queries, and the using of self-attention module.

Specifically, the proposed AFQS algorithm is novel in three key aspects:

(1) New Query Type with Object Information: AFQS introduces a new query type that alters the existing query composition in DETR models (from content and positional queries to SADQ). This approach effectively leverages encoder token information and reduces the need of random embeddings for query initialization. Most existing methods direct follow the query design of DETR, where the positional query is original adopted to increase the difference between quries to produce different results (as outlined in Sec.3.2 of DETR [1] paper). However, our experiment and analysis indicate that the positional query is unnecessary (please see details in our response to the Reviewer TZb2 of Q1.1), which is an important insight for future researches on both the fixed-query and free-query detectors.

(2) Adaptive Queries: AFQS switches from a fixed, predefined number of queries to an adaptive number of queries, using a simple yet effective threshold-based method to filter appropriate encoder tokens. As the model training progresses, this method gradually enables the classification head to distinguish between positive and negative samples among all encoder tokens, obtaining a global solution.

(3) Addressing Training GPU Memory Limitation for Using SA module: AFQS effectively addresses training GPU memory limitations, allowing for training the model with the essential self-attention (SA) module within the decoder architecture. In contrast, directly switching from fixed queries to free queries would significantly degrade performance (e.g., a 4.9% AP drop as shown in Table 2 of the main paper).

[1] Carion N, Massa F, Synnaeve G, et al. End-to-end object detection with transformers[C]// ECCV2020.

---

Q2: Sequential matching is incremental: One-to-many matching and One-to-one matching are the existing methods.

One-to-one matching and one-to-many matching label assignments have been demonstrated in several studies (such as H-DETR, Align-DETR, and DAC-DETR, etc) to enhance the training convergence by increasing the number of positive query samples. In Appendix Sec.A.4 of the main paper, we have discussed the differences between our method and existing one-to-many methods in terms of motivation, implementation, and performance. Unlike simply employing one-to-one and one-to-many matching label assignments, our method features a sequential matching process that constructs a new decoder. This design is based on experiments and an analysis of the roles of SA and CA (as outlined in Sec.3.1.3 of main paper) and addresses the need to transition from a fixed-query to a free-query detector. For more discussion about ''Sequential Matching'', please see our response to Reviewer rMNn of Q1.

Furthermore, our goal is NOT to accelerate model convergence as these methods do. Instead, we address the matching ambiguity that arises from the mixed use of one-to-one and one-to-many matching—a problem that these methods have overlooked and not effectively resolved. We tackle this issue at the label assignment level by designing the LDD decoding structure with one-to-many label assignment in BLP and one-to-one label assignment in DP, which decouples the two types of matching while enabling each to fulfill its role effectively.

Dear reviewer,

We greatly appreciate your valuable feedback and suggestions, and we have carefully analyzed and compared the differences between these methods, in hopes of effectively addressing the issues.

Q1: The sequential matching process is similar to the hybrid layers in H-DETR and MS-DETR

In the Sec.A.4 (line 800 to 816) of Appendix in the main paper, we have discussed the difference between our approach with existing one-to-many methods, such as H-DETR [1] and DAC-DETR [2]. And, we previously overlooked the discussion of our work with MS-DETR [3] approach. For convenience, here, we discuss the differences between our method and both two approaches. Following the suggestion, we will include these citation and discussion in the next version of the paper.

Specifically, the sequential matching process in our work differs from H-DETR and MS-DETR in four main aspects: (1) Problem Addressed: The primary goal of both H-DETR and MS-DETR is to enhance the model's training efficiency to speed up the training convergence. In contrast, our sequential matching model focuses on resolving detection ambiguity caused by mixed label assignments (i.e., combining one-to-one and one-to-many) and addressing the recurrent shifting issue (line 273 in the main paper) that arises from the interaction between CA and SA. (2) Design of Decoder Structure: H-DETR uses additional branches to learn one-to-many assignments, which significantly increases the training cost. The MS-DETR shares a similar decoder structure as DINO, and introduces additional heads (box and class predictors) for one-to-many supervised to further enhance the training efficiency. Different from these two methods, our approach employs a single branch and implements an efficient end-to-end structure by dividing the decoder into decoupled locating and recognition stages: BLP for localization and DP for refining and de-duplicating detection boxes. Our approach effectively alleviates the ''detection ambiguity'' from mixing label assignments to sequential label assignments, incorporating with the optimized decoder structure by leveraging the unique characteristics of CA and SA to stop the ''recurrent shifting'' problem, not only ensuring faster convergence, reducing the complexity, but also further mitigating the ''detection ambiguity'' at the same time. (3) Detection Capability: H-DETR and MS-DETR are both limited to detecting a fixed number of objects, whereas our decoder is designed to detect an adaptive number of objects, which is beneficial for many applications, such as sparse/dense/open-ended detection tasks. (4) Detection Accuracy: Compared to H-DETR, as shown in Table 3 (line 399) of the main paper, with the same backbone (Swin-T), our model achieves higher performance, increasing by +2.1% AP (52.7% vs. 50.6%) and +2.9% AP$_S$ (36.3% vs. 33.4%). For your convenience, we list it as below:

Compared to MS-DETR, as shown in the above table, with the same backbone (ResNet50), our model also achieves higher performance, increasing by +0.2% AP (50.5% vs. 50.3%) and +1.6% AP$_S$ (34.3% vs. 32.7%) than MS-DETR, further confirming our model's effectiveness. As mentioned in MS-DETR, we also believe that the one-to-many supervision using additional head modules in MS-DETR is a complementary approach to our model, which could potentially further enhance the training efficiency and accuracy of our method.

[1] Jia D, Yuan Y, He H, et al. Detrs with hybrid matching[C]//CVPR 2023.

[2] Hu, Z, Sun, Y, Wang, J, et al. DAC-DETR: Divide the attention layers and conquer[C]//NIPS 2023.

[3] Zhao C, Sun Y, Wang W, et al. MS-DETR: Efficient DETR Training with Mixed Supervision[C]//CVPR 2024.

---

Q2: The dynamic query selection mechanism is similar to the distinct query selection process in DQS.

Thank you very much for your comment. We apologize, but due to the lack of reference, we cannot clearly identify which specific paper needs to be compared. We are unsure if the mentioned DQS method refers to the DDQ [1] approach. If that is the case, please refer to our response to Reviewer Pjik regarding Q1. Thank you very much.

[1] Zhang S, Wang X, Wang J, et al. Dense distinct query for end-to-end object detection[C]//CVPR 2023.

Q6: Detailed analysis of the trade-offs between performance and computational cost in QFree-Det.

It's a significant issue for the model design. For the model with an adaptive free number of queries, computational complexity and performance are related to the number of objects in the test images. In the Appendix Sec.A.5 of the main paper, we have conducted ablation studies on the classification score $S$ using the COCO dataset, exploring how the number of queries and FLOPs change with variations in the classification score S. The results are shown in Table 11 of Appendix in the main paper. For your convenience, we have copied it below:

From this table and Fig.4 of main paper, we can observe that a lower threshold S leads to a higher number of queries, which improves the model's accuracy but also increases its computational complexity. When S ranges from 0.01 to 0.05, the model's performance varies in a small range of 50.2% $\sim$ 50.6% AP, indicating good robustness. To balance the trade-offs between performance and computational cost, we set $S=0.02$ for the configuration used in other experiments in the main paper. With this configuration, our model achieves higher performance (50.5% vs. 49.0% in AP) while maintaining smaller computational complexity (273G vs. 279G) and faster decoder inference speed (9.2ms vs. 14.1ms) than DINO.

---

Q7: How does the AFQS algorithm scale with increasing image complexity and object density?

Thanks for your valuable comment. In the Sec.B.2 (line 849 to 885) of Appendix in main paper, we have tested the proportional relation between the number of objects and the number of queries. The results show that as the number of objects in an image increases, the rate at which the number of queries increases slows down. To better illustrate this, we calculated the ratio of the model's queries to the number of objects in the image, presented in the table (row 5) below:

From the table, it can be observed that as the number of objects increases, the growth rate of queries slows down and gradually declines, falling from 25.08 to 16.37. Concurrently, our model demonstrates a growing advantage over DINO as the number of objects increases across the subsets (31–35, 36–40, 41–45, 46+), as illustrated in Fig.5 of Appendix in main paper. Notably, the subset of 1-5 objects accounts for 55.9% of the images in the val2017 set of COCO. For this subset, our model utilizes only 7.25% of the queries (65.22 vs. 900) while achieving a higher performance with a +0.9% AP, suggesting that the queries selected via AFQS are more effective.

In terms of complexity, we have discussed the changes in FLOPs with varying queries in Appendix A.5 of the main paper. As shown in Fig.4, the reduction in the number of queries (cyan dashed line) effectively lowers the model's FLOPs (purple solid line). These experiments underscore the effectiveness of our method's adaptive characteristics in object detection.

Q4: In-depth analysis: why PoCoo loss function can be particularly effective in improving small object detection performance.

Thanks for your valuable comment. PoCoo loss works by integrating additional prior information regarding box sizes in training process. During training , our PoCoo loss automatically categorizes smaller objects as hard samples based on their annotation information of box size within the image. Explicitly assigning higher weights to small objects would up-weight hard samples and thus focus training on them, which has a similar effect as the data-based method. Specifically, the $[* ]$ term in Eq.(6) of our PoCoo loss, which explicitly introduces the prior of box size information into the loss function, shares a similar motivation with the modulating factor of $(1 - p_t)^\gamma$ in focal loss [1].

[1] Lin T Y, Goyal P, Girshick R, et al. Focal Loss for Dense Object Detection[C]//ICCV2017.

---

Q5: Experiments on other datasets? Are there specific scenarios where QFree-Det might underperform?

Q5.1 Experiments on other datasets. For the experiments in the main paper, we have built various variants of QFree-Det on common benchmark datasets of COCO and WiderPerson, with 5 different backbone models across CNN-based, MLP-based, Transformer-based, and Mamba-based backbones, and the large models of Swin-L is also included. The comprehensive experimental results presented in Table 3,4 and other extensive ablation studies in the main paper have demonstrated the effectiveness and superiority of our QFree-Det approach.

However, as suggested, we still conducted a new experiment on CrowdHuman [1] dataset, which is also a challenging datasets for dense pedestrian detection in the wild. The results listed below show that QFree-Det obtains overall higher performance compared to DINO variants, further confirming the effectiveness of our approach. We will include the experiment results on the CrowdHuman dataset in the new version of the paper.

Q5.2 Specific scenarios. The variation in detection scenarios reflects differing levels of detection difficulty, which significantly impacts model performance. The primary significance of QFree-Det lies in its ability to transform a fixed-query detector into a free-query detector while maintaining the model accuracy and not increasing computational requirements. Similar to many advanced models (such as Co-DETR and DAC-DETR), QFree-Det is designed to enhance model accuracy and improve detection robustness, but it may still faces challenges on blurriness and occlusion object detection. These detection scenarios remain ongoing challenges to be addressed.

On the other hand, the accuracy of detection models is significantly influenced by the backbone used to extract initial feature representations, as this backbone forms the foundation for the model's subsequent decoder during detection. The variations in accuracy across different backbones can be quite substantial. For instance, QFree-Det-VMamba-T achieved improvements of +3.9% AP and +4.5% AP$_S$ over QFree-Det-ResNet50 (54.4% vs. 50.5% AP and 34.3% vs. 38.8% AP$_S$) under same 12 epochs training. We hope our method could spark new insights for developing high-accuracy, robust, query-free detection models in further research within the vision community.

[1] Shao S, Zhao Z, Li B, et al. Crowdhuman: A benchmark for detecting human in a crowd[J]. arXiv 2018.

Q2: Comparision with SOTA: DAC-DETR, Co-DETR

Thanks for the comment. The comparisons of Co-DETR (in line 389) and DAC-DETR (in line 393) are listed in Table 3 of the main paper. Under the same backbone and training epochs, our model obtains higher performances than Co-Detr-4scale and DAC-DETR, with improvements of +1.0% (50.5% vs. 49.5%) and +0.5% (50.5% vs. 50.0%) in AP, +1.9% (34.3% vs. 32.4%) and +1.4% (34.3% vs. 32.9%) in AP$_S$, respectively. This confirms the effectiveness of our model. For your convenience, we list the results below:

---

Q3: Computational complexity analysis or efficiency comparison with other methods: inference time and memory usage data.

Thanks for the comment.

(1) For the computational complexity, we have included analysis in Appendix A.5 of the main paper, indicating that our model can effectively reduce the number of queries, and the corresponding FLOPs are also reduced (in Fig.4 of Appendix in the main paper).

(2) For the efficiency comparison, we have conducted experiments on both COCO and WiderPerson datasets to show the relations between the number of objects, adaptive query numbers, and the corresponding performance (see Appendix B.2 in the main paper). The results (in Table 9, Fig.5) indicate that our model obtained overall higher performance across all subsets of COCO than DINO. Especially for the subset of 1-5, our model only uses 7.25% of the queries (65.22 vs 900) while achieving higher performance by +0.9% AP. For the more challenging dataset of WiderPerson, our model obtains overall higher performance under both sparse and dense scenes (in Appendix Table 10, Fig.6 of the main paper). These results clearly demonstrate the effectiveness of our method.

(3) For the inference time, we conducted additional comprehensive tests on inference speed, varying the number of queries from 10 to 1800. The results indicate that, with the same number of queries as DINO (900 queries), our LDD decoder framework has improved the inference speed of the model's decoder by +34.8% over DINO. For more details, please refer to our responses in the General Responses section. On the other hand, our query-free model, which adapts the number of queries based on the image itself, holds significant potential for open-ended object detection task. This task directly generates object names through a generative large language model, where the query count greatly influences the model's complexity. The adaption characteristic of our model enables reduced queries to feed into generative large language model, thereby decreasing the workload for subsequent category generation processing, which highlights its potential applications and importance for the future research in vision community.

(4) In terms of inference memory usage, our model is similar to DINO. During inference, modules like CA and SA generate intermediate variables, causing dynamic changes in GPU memory. We test and record the maximum memory allocation (batch size = 1): 912.3 MB for DINO (900 queries) and 914.9 MB for QFree-Det (900 queries), indicating only a small difference between the two models. This slight variation may be due to differences in the implementation of decoder structures, such as the intermediate variables within the code. Since the adaptive number of queries learned by the model is limited to a small range, the differences in memory usage are minimal. It is noteworthy that the significance of the model's adaptive free number of queries, derived from the image itself, lies in its ability to detect a flexible number of objects while reducing computational load, as illustrated in Appendix A.5 of the main paper.

In addition, the AFQS algorithm serves two main purposes: converting fixed queries into free queries and addressing the issue of high training GPU memory demand associated with excessive query numbers during training, which inhibits the usage of SA (see Sec.3.1.3 of the main paper). Its primary goal is not to directly reduce memory usage. Furthermore, since the model does not require additional gradient information during inference, the overall memory usage has little impact on the model's inference process, even with a larger number of queries.

Dear reviewer,

We really appreciate your thorough and constructive feedback and suggestions. We have carefully considered your suggestions and made every effort to address the issues well by adding relevant experiments and in-depth analyses.

Q1: In-depth analysis about removing PQ dose not significantly affect model performance and how the LDD framework solve the detection ambiguity problem.

Thanks for the feedback. These two questions are very worthwhile points to be discussed.

Q1.1 Removing PQ ... Our experiments (Table 1 and Table 13 of the main paper) show that removing PQ has a slight effect on the model's accuracy. We believe this is primarily due to the interaction mechanisms of CA and SA in the decoder, as well as the specific role of PQ for the decoder.

In original DETR [1], PQ was randomly initialized and learned to increase the differences between query embeddings (as outlined in Sec.3.2 of the DETR paper). The follow-up works adopted a similar query structure to DETR, referring to them as content queries (CQ) and positional queries (PQ) (as mentioned in the Introduction of DAB-DETR [2]). We have discussed the role of CQ and PQ in the Sec.3.1.2 of the main paper and Sec.A.1 of the Appendix. With the development of PQ reformulating the box coordinates into PQ embeddings, we can observe that PQ provides the essential object location information for CQ (via plus operation).

However, instead of the static random initialization in DETR, obtaining adaptive query directly from encoder tokens would inherently contain these object location information. Specifically, these queries integrate the token information of the regions where the object itself is located, which implicitly contain bounding box positions or offset information at each layer of the decoder, supervised by the ground truth boxes. This is one of the reasons why adding additional PQ positional information has a minimal impact on model accuracy.

Additionally, the interaction mechanisms of CA and SA eliminate the need to explicitly include PQ in queries. CA employs deformable attention [3] to interact information between the query and encoder features. By inputting the bounding box position of the current query, CA samples points around this bounding box to interact with the query, effectively updating the query's information, so that reducing the need for additional positional priors to indicate the object's location. For SA, the queries are derived from encoder tokens, which contain inherent differences in information between different objects. This enables the SA module to learn these differences without needing additional positional priors.

Q1.2 LDD framework ... In the Introduction (lines 74 to 81), Related Work (lines 135 to 145), and Sec.3.1.3 (lines 264 to 277) of the main paper, we have discussed that detection ambiguity arises from two main reasons: one-to-one and one-to-many label assignments with shared decoder weights, and the opposing roles of CA and SA with recurrent shifting (line 273 in the main paper) operation. To address this, we designed a unified LDD framework that effectively decouples mixing label assignments and explicitly removes the ''recurrent shifting" operation.

Specifically, the BLP module contains only CA module and employs only the one-to-many supervision mechanism, while the DP module incorporates multiple SA and employs the one-to-one matching label assignment mechanism. These designs effectively eliminate the sources of detection ambiguity from the outset.

When connecting the BLP and DP modules, the conflict between the two label assignments still exists, due to the query is sequentially updated by the BLP and DP modules (see details in Sec.3.4, lines 367 to 372 of the main paper). To address this, we designed a simple yet effective method for Stopping Gradient back-propagation of Query (SGQ) from DP to BLP, which helps mitigate conflicts between the two modules. As shown in Table 16 of Appendix in the main paper, the absence of the SGQ approach resulted in a 2.0% AP drop in the model's performance, highlighting the significance of our approach in alleviating detection ambiguity and further validating LDD's effectiveness.

[1] Carion N, Massa F, Synnaeve G, et al. End-to-end object detection with transformers[C]// ECCV2020.

[2] Liu S, Li F, Zhang H, et al. DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR[C]//ICLR2022.

[3] Zhu X, Su W, Lu L, et al. Deformable DETR: Deformable Transformers for End-to-End Object Detection[C]//ICLR2021.

Q5: Speed of inference: How fast can the QFree-Det be for some sparse scenario, i.e., only a instance in a image?

Thank you for the comment. We conducted additional comprehensive tests on inference speed, varying the number of queries from 10 to 1800. The results indicate that, with the same number of queries as DINO (900 queries), our LDD decoder framework has improved the inference speed of the model's decoder by 34.8% over DINO. When the number of queries is further reduced, the decoder’s inference speed remains stable, primarily due to the parallel computation performed by CA and SA within the decoder. For more details, please refer to our responses in the General Responses section.

On the other hand, our query-free model, which adapts the number of queries based on the image itself, holds significant potential for open-ended object detection task. This task directly generates object names through a generative large language model, where the query count greatly influences the model's complexity. The adaption characteristic of our model enables reduced queries to feed into generative large language model, thereby decreasing the workload for subsequent category generation processing, which highlights its potential applications and importance for the future research in vision community.

---

Q6: Is the same experiments (44.1) as that of line 248?

Yes, it is the same experiment in line 476 and line 248. This experiment is conducted to test the impact on model accuracy after removing SA. By training the model without SA, we can perform multiple tests with any number of queries without needing to retrain the model. However, this model still lacks dynamic adaptability to queries and can only perform testing with manually defined varied number of queries.

In line 248, we labeled it as ''fixed query number'' because the test is still conducted with a fixed query count (900). In line 476, we labeled it as ''free'' because this model can accommodate any number of query tests, without the need to retrain the model. It may not be appropriate to directly label it as ''free'' directly, so we will update it to ''free-conditioned query'', indicating that it is free but subject to the constraints of the manually defined testing parameters.

Continue the answer to Q1:

The experimental results are presented in the table below:

(Note: DETA with * uses 9 encoder layers. Detectron2 and MMDetection are both improved codebase for DETR-like detector.)

It is important to note that our model was developed using the public official DINO [3] code, rather than improved frameworks such as DETA with Detectron2 [4] and DDQ-DETR with MMDetection V3.0 [5]. This may lead to an unfair comparison of their accuracy, as the variations in codebases have not been eliminated. Despite this, our model still achieves comparable results and even outperforms others in small object detection, demonstrating its effectiveness. The suboptimal implementation of our model may not fully showcase its advantages, as the reviewer has mentioned in Q3 regarding the improved MMDetection repository. Due to time constraints, we will leave this issue in future work.

[1] Ouyang-Zhang J, Cho J H, Zhou X, et al. Nms strikes back[J]. arXiv preprint, 2022.

[2] Zhang S, Wang X, Wang J, et al. Dense distinct query for end-to-end object detection[C]//CVPR 2023.

[3] Zhang H, Li F, Liu S, et al. DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection[C]//ICLR 2022.

[4] Wu Y, Kirillov A, et al. Detectron2. 2022.

[5] Chen K, Wang J, Pang J, et al. MMDetection: Open MMLab Detection Toolbox and Benchmark. arXiv preprint 2019.

---

Q2: Comparative Experiments in Table 3: comparison with Stable-DINO and CO-DINO.

Thanks for the valuable comment. The comparison of Co-DETR [1] with our method was listed in Table 3 (line 389) of the main paper. Following the suggestions, we further list the results about Stable-DINO [2] method as below:

(Note: detrex and MMDetection are both improved codebase for DETR-like detector.)

As we can see, compared to these two methods, our approach achieves the overall best performance, especially for the small objects detection, increasing by +1.4% and +1.9% AP$_S$ than Stable-DINO and Co-DETR, respectively. We will include this comparison with the Stable-DINO model in Table 3 of the new version of the main paper.

[1] Zong Z, Song G, Liu Y. Detrs with collaborative hybrid assignments training[C]//ICCV2023.

[2] Liu S, Ren T, Chen J, et al. Detection transformer with stable matching[C]//ICCV2023.

[3] Ren T, Liu S, Li F, et al. detrex: Benchmarking detection transformers[J]. arXiv preprint 2023.

Dear reviewer,

We sincerely thank you for putting in the time to provide us with a thorough review. We have carefully considered your suggestions and have done our best to address these concerns.

Q1: Comparison with DETA and DDQ-DETR: the disscussions about NMS and Self-attention (SA).

Thanks very much for the valuable comments and suggestions.

DETA [1] and DDQ-DETR [2] both use NMS to eliminate duplicates in their models. Removing duplicate detection boxes is a crucial step for reducing false positive samples in detection systems. We categorize existing methods of deduplication into two types based on their principles: box-based and class-based. Box-based methods, such as NMS, work by comparing the overlapping (IoU) between predicted boxes. While straightforward, they are sensitive to the IoU threshold and struggle with overlapping targets. In contrast, class-based methods utilize self-attention to compare features between queries, influencing classification scores to achieve deduplication, which results in lower scores for redundant boxes. This approach can mitigate the issue of overlapping objects and is commonly employed in DETR-like models. Intuitively, combining both box-based and class-based methods may further enhance the model's overall performance.

Compared to DETA and DDQ-DETR, our approach addresses different problems (in motivation and objective), introduces different solutions to issues (in query selection, decoder architecture, and loss function), and achieves comparable results in performance. In addition, these differences do not diminish the contributions of each method to solving the respective issues.

DETA is a box-based method for deduplication. Specifically, DETA is designed to investigate the impact of one-to-many assignment-based training on enhancing the training efficiency for DETR models, which differs to our motivation and objective. It achieves this by employing one-to-many IoU assignments in conjunction with NMS method, which is applied during both query selection and final prediction post-processing. This study shows that the one-to-many IoU assignments, combined with NMS, effectively improve training efficiency.

DDQ-DETR combines both box-based and class-based deduplication methods. It specifically explores how query distinctness affects the model's optimization process and accuracy, which also differs to our motivation and objective. DDQ-DETR introduces the Distinct Query Selection (DQS) module, which uses training-unaware NMS to filter dense queries into distinct queries (box-based method). Then, DDQ-DETR further applies Hungarian matching (class-based method), considering both bounding box scores and class scores, to generate final one-to-one detection results. Additionally, DDQ-DETR also uses one-to-many label assignments and incorporates an auxiliary head along with Auxiliary Loss for Dense Queries to maintain training efficiency (but this kind of mixing label assignments on same decoder weights still introduces the issue of ''detection ambiguity''). Overall, this approach highlights that both sparse and dense queries in end-to-end detection are problematic. By explicitly combining box-based and class-based methods, DDQ-DETR ultimately enhances model accuracy.

Unlike these two methods, our approach focuses on transforming a fixed-query detector into a free-query detector, thereby addressing the fixed capacity of DETR-like models. We deeply explore the role of SA (which has NOT been examined in DDQ-DETR) to address challenges related to the existing decoder structure during this transition, particularly the training GPU memory demands issue associated with SA. In contrast to DETA and DDQ-DETR, our object is not to investigate how to enhance model accuracy through NMS. Furthermore, we observe that the existing connection between CA and SA can result in the recurrent shifting problem (as noted in line 273 of the main paper). To tackle these challenges, we developed a more effective decoder structure (LDD) and implemented decoupled one-to-one and one-to-many label assignments. This design significantly alleviates the detection ambiguity issue (Table 12 and Table 16 in Appendix of the main paper). Notably, DETA and DDQ-DETR share a similar decoder structure with existing methods, which highlights the contribution of our approach of novel LDD decoder. Compared to previous class-based deduplication method, QFree-Det further optimizes the decoder structure, enabling query-free detection while minimizing the ''detection ambiguity" associated with one-to-many assignments.