DECO: Unleashing the Potential of ConvNets for Query-based Detection and Segmentation

Q1: My main concern is the use of cross-scale feature-fusion module which is proposed in RT-DETR, which weakens the contribution. It would make the paper more strong if the authors could further discuss the difference.

A1: Thanks for the concern. We would like to highlight that our DECO also outperform DETR with the same settings, i.e., training receipt, architecture etc. The comparisons are shown in Table 2 (as also shown in Figure 1 in supplementary material) and we can see that our DECO obtains better performance than DETR, which justifies the effectiveness and also the main contribution of our proposed method.

Due to the powerful capacity of DETR framework, lots of follow-up methods have improved DETR from different perspectives to greatly boost the performance and RT-DETR is one of the excellent work. To make more proper comparisons with state-of-the-art methods, we incorporate the multi-scale structure of RT-DETR into our DECO framework and obtains stronger performance than most DETR variants. The core DECO encoder module for extracting global feature is different from RT-DETR. Besides, as stated in the paper Line 335-338, the combination of features between different scales is also different from RT-DETR.

We would like to argue that it actually showcases our DECO is a general framework to be compatible with other modules and more explorations are left for future work.

Q2: There are many recent studies speeding up transformers, it would be nice to discuss more on using convolutions. Claims in L073-L074 are not convincing to me.

A2: Thanks for the question.

• There are quite a few studies for speeding up detection transformer (DETR) like Focus-DETR, Lite-DETR etc. As shown in Figure 1, our proposed method obtains similar accuracy/latency tradeoff with Focus-DETR and outperforms Lite-DETR, demonstrating the strong performance of our model.
• There are also some popular attempts for speeding up attention module like FlashAttention. The FPS for DETR reported in the manuscript is measured in Pytorch 1.10 without FlashAttention. We utilize XFormers to adopt flash attention to DETR and measure the latency on V100 GPU. It should be noted that flash attention is only supported in Pytorch 2.0+ and our DECO also obtains higher FPS in this setting. Though DETR obtains more speedup by using Pytorch 2.1 and FlashAttention, our DECO still achieves better tradeoff with respect to FPS and AP, as shown in Figure 2 in supplementary material.

Thanks for the rebuttal. My concerns are well addressed and I'm increasing my score.

Q1: Although this paper presents the convolution-version DETR which replaces attentions with convolutions, the overall performance is inferior to recent transformer-based methods, such as DINO, Co-DETR, and Stable-DINO[1], showing the limitations of using convolutional architectures.

A1: Thanks for the comments. We have already included the comparisons with DINO and Co-DETR in our original manuscript (Figure 1 and Table 1). As suggested, we also include the results of Stable-DINO in the updated manuscript and also highlight the comparisons of these models below.

As shown in the above table, our model actually obtains better AP than Co-DETR and comparable AP with DINO and Stable-DINO, but with much faster inference speed. A more clear visualization in Figure 1 of the manuscript also demonstrates that our model achieve better accuracy and latency trade-off than these models.

Q2: In Tab. 2, it shows that DETR achieves better performance than DECO on both R50 and ConvNeXt-T while the improvements in inference speeds are not significant. In addition, DETR can be further optimized for acceleration therefore obtaining better inference speed and accuracy. Hence, the effectiveness of the proposed method in this paper is rather limited.

A2: Thanks for the concern. We would like to highlight that our DECO also outperforms DETR with the same settings, i.e., training receipt, architecture etc. The comparisons are shown in Table 2 (as also shown in Figure 1 in supplementary material) and we can see that our DECO obtains better performance than DETR, which justifies the effectiveness and also the main contribution of our proposed method.

There are also some popular attempts for speeding up attention module like FlashAttention. The FPS for DETR reported in the manuscript is measured in Pytorch 1.10 without FlashAttention. We utilize XFormers to adopt flash attention to DETR and measure the latency on V100 GPU. It should be noted that flash attention is only supported in Pytorch 2.0+ and our DECO also obtains higher FPS in this setting. Though DETR obtains more speedup by using Pytorch 2.1 and FlashAttention, our DECO still achieves better tradeoff with respect to FPS and AP, as shown in Figure 2 in supplementary material.

Q3: This paper lacks experimental comparisons with many recent works, such as Stable DINO[1], RT-DETRv2[2], DDQ[3], and SpeedDETR[4], in terms of both accuracy and speed.

A3: As suggested, we include the results of Stable-DINO, DDQ, SpeedDETR in the updated manuscript and also highlight the comparisons of these models below.

As shown in the above table, our model obtains similar FPS with SpeedDETR and has 3.8% higher AP. Though our DECO has slightly lower AP than Stable-DINO and DDQ, it obtains much faster inference speed. A more clear visualization in Figure 1 of the manuscript also demonstrates that our model achieve better accuracy and latency trade-off than these models.

RT-DETRv2 is a strong real-time detector which mainly introduce modifications within the deformable attention module and improved training strategy to achieve enhanced performance. According to the results of DCNv3 presented in the response of Q4, DECO can gain from using deformable convolution. Thus, this bag of freebies for training strategy in RT-DETRv2 can also be generalized to our DECO framework, including those related to deformable attention. We will explore more in future work.

We also would like to highlight that this paper aims to explore an alternative solution using CNNs and prove that it can also achieve quite competitive performance with Transformers, in terms of both object detection and segment anything tasks. We hope that this fresh idea of convolutional query-image fusion architecture can shed some light on the future design. Our DECO is a general framework to be compatible with other modules and more explorations are left for future work.

Q1: On Lines 074-075, within the scope of object detection, the statement ``whose weights and inputs are both generated during runtime'' is unclear, as the only ones dynamically generated are intermediate variables (including attention weights).

A1: Thanks for the question. To be more clear, we aim to emphasize that the attention mechanism introduces dynamic memory access where both attention weights and inputs are generated during runtime. In contrast, the weights in convolution is static and only the inputs are dynamic. The above discussions are related to the attention and convolution operations, and are actually decoupled with the tasks. Therefore, this statement is equally valid within the scope of object detection.

Q2: The performance can be further improved using more convolutional-based techniques like deformable convolutions.

A2: Thanks for the good suggestion. We replace the dwconv in Self-Interaction Module (SIM) with deformable convolution, using the implementation of DCNv3. As shown in the below table, utilizing deformable convolution brings +1.0 AP gain and even obtains slightly faster FPS (66 vs. 67). We are trying to also use deformable convolution in Cross-Interaction Module (CIM) and hoping to obtain further improvement.

Q3: Can the denoising training technique in DN-DETR and DINO be introduced into convolutional-based DECO?

A3: Thanks for the question. The denoising training in DN-DETR and DINO aims at speeding up the convergence of bipartite matching. It introduces the noised ground-truth bounding boxes with their labels into the training process to make the matching loss more stable. To avoid leakage of ground truth, attention mask is used to separate the real object queries and the denoising ones.

As this bipartite matching is also used in DECO, we also face the need to speed up the convergence. Denoising training is possible to be applied to our convolutional design in a similar way. For example, the denoising groups of queries can be inserted into the initial object queries. The number of denoising queries can be controlled to ensure the entire queries are able to be reshaped to a 2D arrangement. To avoid knowledge leakage, similar ideas of masked convolutions can be adopted in the training process.

Q1: Clarity: Fusion methods and query map upsampling strategies should be clearly stated and explained in the method section.

A1: Thanks for your suggestions. We provide more details in the method section for the fusion and upsampling strategies in Line 237-246.

There are also other design choices for the upsampling function, \ie, resizing both the object queries and encoder embeddings to a fixed size before fusion, or directly upsampling the object queries to dynamic size of encoder embeddings, which is related to the resolution of input image. We will provide more analysis in experimental section. Then the upsampled object queries and the image feature embeddings are fused together using $\mathrm {Fusion}(\cdot)$ function, followed by a large kernel depthwise convolution to allow object queries to capture the spatial information from the image feature. An intuitive implementation for $\mathrm {Fusion}(\cdot)$ is element-wise add operation. There are also some alternatives like element-wise multiply operation, or first concatenating two features then using convolution to reduce the number of channels. More discussions will be presented in the ablation studies.

Q2: The paper is missing some important references, such as the large kernel CNN paper and the SparseInst paper, which is also a pure-conv query-based object detection/segmentation method.

A2: Thanks for pointing out. We have included related papers regarding large kernel convolutions[1,2,3]. We utilize a large kernel depth-wise convolution in the proposed InterConv module, to enhance the interaction of image embeddings and query features.

We also include detailed discussions with SparseInst in related work section.

Cheng et. al proposed SparseInst, an efficient and fully convolutional framework for real-time instance segmentation. SparseInst utilizes a sparse set of instance activation maps to predict objects in end-to-end style, which could be viewed as an alternative for the query-based mechanism of DETR. In this paper we would like to design a DETR-like detection pipeline but built with standard convolutions, which could inherit both the advantages of ConvNets and the favorable properties {of the} DETR framework. Compared with SparseInst, our method follows the similar query-base mechanism of DETR, and also demonstrates good generalization capability like segment anything models.

[1] Ding et. al, Scaling up your kernels to 31x31: Revisiting large kernel design in cnns, CVPR 2022
[2] Liu et. al, More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity, ICLR 2023
[3] Ding et. al, UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio Video Point Cloud Time-Series and Image Recognition, CVPR 2024
[4] Cheng et. al, Sparse Instance Activation for Real-Time Instance Segmentation, CVPR 2022