To Reviewer C9T7

Q1: It would be beneficial to provide a detailed analysis of the computational cost.

RE: We provide analyses of the computational cost in Table 3, Table 4, and Table 5 in the main text. Besides, more discussion can be found in the rebuttals for all reviewers above.

---

Q2: The performance improvement appears to diminish when using more advanced models like Co-DETR, especially during extended training periods such as 24 or 36 epochs.

RE: A significant advantage of most SOTA query-based detectors is that they can converge extremely fast within 12 epochs. As a result, the performance gain may not be that obvious within a long training schedule. This phenomenon is also reported by Hybrid Proposal Refine [a], where DDQ with HPR merely gains +0.1% AP improvement (from 52.4% to 52.5%) as the training epochs increase from 12 to 24. Some training strategies are excepted to further improve the performance with longer training schedules, e.g., large-scale jitter used in [a] and other data augmentation. We will add relative discussions in the revision.

[a] Hybrid Proposal Refiner: Revisiting DETR Series from the Faster R-CNN Perspective, In CVPR 2024.

---

Q3: Apply Frozen-DETR to other SOTA models, such as MS-DETR, Hybrid Proposal Refiner and KD-DETR.

RE: Thanks for your advice. However, we would like to highlight that Hybrid Proposal Refiner and KD-DETR are not publicly available when submitting our work. These two papers are only available on CVPR2024 openaccess after 13th June. And KD-DETR is not a new detector but a distillation method. As a result, we provide the experiments on MS-DETR and Hybrid Proposal Refiner below. As shown in the table, our method can be easily applied to recent SOTA models.

---

Q4: Why is the performance of the Co DINO baseline in Tables 12 and 13 inconsistent with the main paper?

RE: The baseline in Tables 12 and 13 is the same baseline as in Section 4.2, which uses four-scale feature maps and does not use co-heads. Due to space limitations of the main text, these two tables are presented in the appendix. We will add more experimental details.

---

Q5: Previous work noted that features from CLIP's vision encoder can have artifact problems. Would applying the methods from it help mitigate these artifacts and improve performance?

RE: Thanks for your advice. Since this work only releases the checkpoint for DINOv2-FM. Thus the following experiments are conducted on DINOv2-FM and DINOv2-FM with registers (DINOv2-reg). In the table, we find that using DINOv2 can even get better results than using CLIP, which is different from Table 2. We hypothesize there are two reasons: First, DINOv2-FM has both global-wise and token-wise pre-training pre-tasks. Thus the patch tokens are more informative. Further, DINOv2-FM ViT-L is distilled from ViT-giant, which equals to a larger foundation model. We find that DINOv2-reg can mitigate artifacts in DINOv2-FM and further improve the performance. We will add these results to the manuscript.

---

Q6: The performance gains heavily depend on the quality and characteristics of the pre-trained foundation models.

RE: Since different vision foundation models are pre-trained with different pre-tasks and Frozen-DETR does not train the vision foundation models, it is quite normal and foreseeable that the choice of vision foundation models will affect the performance, and not all of them are suitable for Frozen-DETR. We have tested and compared many representative foundation models with various pre-training methods in Table 2 with some intuitive explanations. These empirical practices can be used as the selection guidelines.

---

Q7: Solutions such as knowledge distillation might help improve efficiency.

RE: Thanks for your instructive advice. In recent distillation methods, the teacher model and student model should always be trained with the same task and use similar architectures. Since foundation models are not pre-trained for detection, how to transfer the knowledge from them to detectors is still an open question. We also notice that some open-vocabulary methods distill the knowledge from CLIP. However, they can not improve the base class performance from distillation or even pursue high novel class performance at the price of low base class performance [1,2,3]. While our method can easily apply to many query-based detectors, boosting all classes' performance.

[1] Open-vocabulary object detection via vision and language knowledge distillation. In ICLR, 2022.

[2] Distilling detr with visual-linguistic knowledge for open-vocabulary object detection. In ICCV, 2023.

[3] Aligning bag of regions for open-vocabulary object detection. In CVPR, 2023.

To Reviewer QbuY

Q1: The idea of feature fusion seems incremental from the perspective of technical contributions. By looking at the design, the feature fusion of the frozen foundation model is the only principle claim of the paper.

RE: We respectfully disagree with this point. We would like to highlight the core contribution of this work, which is a novel paradigm to integrate frozen vision foundation models with query-based detectors. We are the first to show that frozen foundation models can be a versatile feature enhancer to boost the performance of detectors without any fine-tuning, even though they are not pre-trained for object detection. Such a paradigm enjoys many advantages over using vision foundation models as the backbone. Please see the rebuttals for all reviewers above for more details.

---

Q2: Frozen-DETR is only tested with DINO, which has huge computational costs due to dense multiscale feature utilization in the encoder. To demonstrate the utility of the presented idea, there should be results with affordable DETRs, e.g. DN-DETR, Conditional-DETR, DAB-DETR, Anchor-DETR, IMFA -DETR, Deformable-DETR etc.

RE: We have applied the method to AdaMixer, DINO-det, and Co-DETR in the paper, which can demonstrate the generalization ability. We have also applied our Frozen-DETR to MS-DETR and HPR (See Reviewer C9T7 Q3 below). Due to limited time and computation resources, we select DN-DETR and DAB-DETR as the representatives of single-scale query-based detectors. Both models are trained with 12 epochs. Experimental results show that Frozen-DETR can also significantly enhance the performance of single-scale detectors (+ 4.0% AP on DAB-DETR and + 2.7% AP on DN-DETR) with acceptable additional computation cost.

We also notice that the models can be further accelerated without using DC5 but with lower performance (lags behind around 2% AP). Without DC5, DN-DETR has 104 GFLOPs and runs at 21.0 FPS, while Frozen-DETR has 298 GFLOPs and runs at 14.1 FPS. Unfortunately, these single-scale models lag behind SOTA models by more than 10% AP and need 50 epochs to converge. Thus, these models can not meet practical needs.

---

Q3: I am more concerned about the applicability of this method in the real world, considering its weaker accuracy improvement tradeoffs vs runtime. By looking at Tables 12 and 13, changing the image resolution does not aggressively improve the accuracy. Similar effects are seen in Table 13.

RE: We believe the accuracy improvement of Frozen-DETR is significant and all other reviewers agree with us on the promising performance:

• On the COCO dataset, we increase 2.9% AP for DINO-det (Table 6).
• On the challenging large vocabulary LVIS dataset, we increase 6.6% AP for DINO-det (Table 7).
• On the challenging long-tail scenario, we increase 8.7% APr and 7.7% APc for DINO-det (Table 7), showing the potential to alleviate the class imbalance problem.
• On the challenging open-vocabulary scenario, we increase 8.8% novel AP for DINO-det (Table 8), showing strong open-vocabulary ability.
• In the real world, input images always suffer from natural distribution shifts. We also find that Frozen-DETR inherits great domain generalization ability from frozen foundation models. We directly transfer the model trained on the COCO dataset to the COCO-ood dataset [1] without fine-tuning, which is a dataset having the same classes as COCO but different domains, such as sketch, weather, cartoon, painting, tattoo, and handmake. As shown in the table below, Frozen-DETR achieves almost the same performance on both datasets, while other detectors degrade a lot on the COCO-O. The performance of Frozen-DETR on COCO-O is two times higher than the baselines and even higher than detectors with strong backbones, showing its strong robustness. We will add these results to the appendix.

Further, in Table 12, we aim to validate the property that Frozen-DETR enjoys the asymmetric input size. Using a small image resolution is already feasible to achieve promising results thus achieving a good performance-speed trade-off. In Table 13, we aim to demonstrate that a stronger foundation model can obtain larger improvements. Please see the rebuttals for all reviewers above for more discussions on the performance-speed trade-off.

[1] COCO-O: A Benchmark for Object Detectors under Natural Distribution Shifts. In ICCV 2023.

Thanks for your feedback and agreement on two key contributions of our work: 1) the first attempt to apply frozen foundation models in the downstream object detection task and 2) the improved detection performance.

For the technological contribution, we design a decoupled feature fusion method by viewing each component of the foundation models as a special part of detectors, such as class tokens as image queries and patch tokens as another level of feature pyramid. Such a feature fusion technique can effectively transfer the image understanding ability from foundation models to detectors with minimal modifications to detectors so that the method can be easily applied to various query-based detectors. Considering the various benefits of our Frozen-DETR compared to other ways to use foundation models in detection discussed before, we hope that the Frozen-DETR paradigm can arouse researchers' interest in introducing foundation models into detection.

For the computation cost concern, we provide the detailed FLOPs and FPS for all the models in the above table. We would like to highlight some key points:

• Our Frozen-DETR can be applied to various query-based detectors, including both single-scale and multi-scale detectors. The inference time is within 1.5x the time of baselines, which is acceptable considering the performance improvement. Reviewer RksL agrees with us.
• The detectors without multi-scale feature maps or encoders run faster but lag behind SOTA models by a clear margin (for example, DAB-DETR-DC5 only gets 38.0% AP with training 12 epochs). To get better performance, they should be equipped with a stronger backbone or other modules, which will also slow down the speed. While our Frozen-DETR can achieve high performance, especially in many challenging scenarios mentioned in the rebuttals above.
• As discovered by many papers, FPS does not strictly increase in proportion to FLOPs, as many factors also influence the FPS, such as the complexity of operators, memory access cost (MAC), and GPU parallel computing utilization [1]. For example, Deformable DETR and Sparse RCNN have similar FLOPs but with different fps (13.4 vs 17.8), as shown in the above table. Since foundation models enjoy a simple architecture and Attention is highly optimized for modern GPUs, most operators can be computed in parallel. The fps gap is smaller than that of FLOPs. We recommend to use FPS as the main metric for computation cost.
• Further, many inference-time acceleration methods [2-5] for foundation models have been proposed, which can further accelerate Frozen-DETR. Besides, Frozen-DETR can be equipped with different foundation models. A faster foundation model can also accelerate Frozen-DETR.
• According to your comments, we also change the V100 GPU to a slower 1080 Ti GPU. The trend in FPS is the same for different types of GPU.

[1] An Energy and GPU-Computation Efficient Backbone Network for Real-Time Object Detection. In CVPRW 2019.

[2] FasterTransformer. By NVIDIA.

[3] TensorRT-LLM. By NVIDIA.

[4] Flashattention: Fast and memory-efficient exact attention with io-awareness. In NeurIPS 2022.

[5] Efficient Memory Management for Large Language Model Serving with PagedAttention. In SOSP 2023.

Thanks for your quick reply!

We have checked the code (the code is modified from benchmark.py from Deformable-DETR and we will release our code in the future) and tested the fps for DAB-DETR-DC5/Frozen-DETR (DAB-DETR-DC5) on 1080ti three more times and got the same results:

DAB-DETR-DC5: 4.9, 4.9, 5.0

Frozen-DETR (DAB-DETR-DC5): 4.7, 4.7, 4.7

We also provide the fps for DAB-DETR-DC5/Frozen-DETR (DAB-DETR-DC5) on various types of GPUs: 1080Ti (5.0/4.7), P100(6.0/5.4), 2080Ti (8.2/7.6), V100 (10.2/8.5), 3090 (12.6/11.8), A100 (21.4/19.9). The fps gaps are similar across various GPUs. We find that the GPU utilization in 1080Ti is around 90% and around 70% in V100 and Frozen-DETR has a higher GPU utilization rate than baselines. We hypothesize that the GPU utilization rate is one of the reasons for the similar fps gaps across different GPUs and small fps gaps compared with FLOPs gaps. As mentioned above, many other factors will also influence the FPS.

Once again, we highly recommend you select fps (the real runtime) as the main metric instead of flops. The runtime can be further improved by many off-the-shelf acceleration techniques.

Thank you for the additional results.

However, I am still unconvinced with GPU FPS performance across GPUs. They seem erratic. The primary reason is that the Frozen foundation model is run sequentially (i.e. before the main pipeline backbone or right after the main pipeline backbone). Hence 1080Ti like GPUs should technically have more FPS gap which is contrary to the evaluations provided by the authors. The current evaluations are showing 4.9FPS without the frozen model and 4.7FPS with the frozen model.

Understanding, in terms of latency, 4.9FPS is 204ms while 4.7FPS is 212ms i.e. a gap of 8ms. In fact, with a stronger GPU this latency should decrease and hence on the stronger GPU, the FPS gap should be smaller. Given the size of the frozen model (DINO (transformer-based or any other)), the latency introduced by the frozen model of 8ms on 1080Ti is not justifiable in any case. Moreover, with these values, 204ms was introduced by the main detection pipeline while 8ms by the frozen foundation model which is only 3%. This directly counters your response of "Frozen-DETR has a higher GPU utilization rate".

Considering the above argument, I'll keep my original rating.

To Reviewer QbuY:

Thanks for your patience and quick reply. We understand that the newly added DAB-DETR-DC5 experiment during the rebuttal, with its slightly different setup, may cause some confusion. We appreciate the opportunity to clarify any misunderstandings, and all clarifications will be incorporated into the main text.

As mentioned in our last reply, all single-scale detectors use standard dense self-attention in the encoder, which has the quadratic computation w.r.t. token numbers. Taking DAB-DETR-DC5 as an example, for an image with the input size 800*1200, the token numbers in the encoder will be (800 / 16) * (1200 / 16) = 3750.

In multi-scale detectors, it is unreasonable to concatenate all the multi-scale feature maps along the token number dimension because there are too many tokens for multi-scale features. Thus, all multi-scale query-based detectors use deformable attention in the encoder, which is sparse attention with linear complexity to approximate the standard self-attention. Our frozen-DETR (DAB-DETR-DC5) has multi-scale feature maps and uses deformable attention in the encoder, following common practices in multi-scale query-based detectors. In summary, the foundation model in Frozen-DETR introduces extra latency, while deformable attention reduces latency. This results in an insignificant FPS gap between DAB-DETR-DC5 and our Frozen-DETR, which is the reason for your confusion.

Regarding parallel computing, we have demonstrated that it does accelerate our Frozen-DETR even on a 1080Ti GPU. Besides, many researchers are now focusing on the efficient inference for large foundation models and LLMs. We believe these techniques could also be effectively applied to our Frozen-DETR.

Considering the various advantages of Frozen-DETR, especially under many challenging scenarios (which you also acknowledged regarding the improved performance), we note that all other reviewers have reached a consensus with us on the good performance-speed trade-off of Frozen-DETR and have given us a positive rating. We are looking forward to your feedback.

To Reviewer RksL

Q1: When switching from R50 to Swin-Large or even ViT-Large, does this approach of inserting a frozen CLIP still lead to noticeable improvement?

RE: Yes. We conduct experiments with Swin-L based on the Co-DETR detector. In this experiment, we equip the Co-DETR with DFN5B-CLIP-ViT-H-14-378 as the feature enhancer and fine-tune their Objects356 pre-trained checkpoint. Our Frozen-DETR is not used during the Objects365 pre-training. The results in the table below show that frozen-DETR can still gain noticeable improvement even with a strong backbone and strong pre-training.

---

Q2: I am still curious whether the R50-CLIP would boost the performance of an R50 backbone detector. Does this increase necessitate that the model size of CLIP backbone surpasses that of the detector backbone?

RE: Here we use the same baseline as in Table 13. Using the R50-CLIP feature enhancer can still increase by 0.5% AP. Nevertheless, Frozen-DETR aims to equip detectors with a strong foundation model. If we use the R50 version of foundation models, we think using it as a backbone is a better choice.

To Reviewer V83J

Q1: A few suggestions on writing

RE: Thanks for your helpful and detailed advice. We will carefully revise our manuscript. We rename the detector DINO as DINO-det and the self-supervised foundation model DINOv2 as DINOv2-FM.

---

Q2: EVA-CLIP-18B is equipped with a ViT rather than R50. The version of CLIP using R50 as the backbone lagged behind. This evidence cannot support the claim that the improvement is from large-scale training.

RE: This example aims to emphasize that a large-scale pre-trained foundation model has strong image understanding abilities such as zero-shot ability, even without fine-tuning on a specific downstream dataset. This property demonstrates that the foundation model has the potential to serve as a plug-and-play feature enhancer for downstream tasks. In the revision, we will replace the compared R50 with an ImageNet-22k pre-trained and ImageNet-1k fine-tuned ViT-H-14, which achieves 85.1% top1 accuracy and merely surpasses the zero-shot CLIP model by +1.3%, showing the foundation CLIP model's generalizable image understanding abilities.

---

Q3: Is DINO-det the SOTA query-based detector?

RE: DINO-det is one of the most well-known query-based detectors due to high performance and fast convergence speed and most recent SOTA detectors are based on it. Here we list some top detectors in the paperwithcode:

• Co-DETR (the rank-1 detector) adds additional co-heads to DINO-det for fast and better convergence. We have also applied our method to Co-DETR in Table 6.
• InternImage-H (the rank-2 detector) and M3I Pre-training (the rank-3 detector) equip DINO-det with a strong backbone InternImage-H which is pre-trained by M3I Pre-training.

---

Q4: The setup of Table 1 is a little unclear, is the backbone tunable? I assume they are not. Then, the comparison is slightly unfair -- the backbones are frozen for the top lines, and there are additional trainable parameters in the encoder backbone. In other words, if we fine-tune the encoder backbone of Table 1, will the gap between the two variants be even smaller, which questions the current design?

RE: All backbones in Table 1 are tunable. Thus, the comparison is fair and the current design is reasonable. We will add more details to clarify the setup in Table 1.

---

Q5: In Table 2, DEiT-III has comparable results with CLIP, how will this backbone perform for the main experiments?

RE: We find DEiT-III can also boost the performance in the main experimental setup, but slightly lower than CLIP, as shown in the table below.

---

Q6: One limitation of this approach is the running time. in Table 4, the authors reported the FPS, running time time has been 1.5x compared with the baseline model. However, if using the foundation model in a conventional way (i.e. directly using them as the backbone), the running time will not be affected in the inference time. Moreover, if using a ViT-based foundational model, how will the inference time and GPU memory change, since most of the recent powerful foundation models are equipped with ViT?

RE: We believe the additional computation cost is acceptable, which is also pointed out by Reviewer RksL. If we directly use ViT-L as the backbone, it will take 10G inference memory (3x larger than the baseline) and 2.1 FPS (4x slower than the baseline), which is much larger and slower than our Frozen-DETR. Please see the rebuttals for all reviewers above for more details.