DON’T NEED RETRAINING: A Mixture of DETR and Vision Foundation Models for Cross-Domain Few-Shot Object Detection

We sincerely thank Reviewer KWMR for the constructive feedback, insightful questions, and valuable suggestions!

Below we address your concerns.

Q1: About some unclear parts in the paper.

R1: Thank you for your valuable questions.

Below is our detailed response to the three issues you raised in Question 1:

• R1.1: In DINOv2, the spatial dimensions of the feature maps differ from those of the ResNet backbone’s feature maps. In the revised main paper., we will use (H, W) to denote the spatial dimensions of backbone features and (Hd, Wd) to denote the spatial dimensions of DINOv2 features. We will revise this phrasing in the revised main paper.

• R1.2: Our feature fusion strategy follows the illustration: We first process the output of the previous stage with the backbone layer. The processed features are then fused with the VFM features. Finally, the fused features are used as the input for the next layer.

• R1.3: In Tables 4 and 5, “DINO v2” refers to replacing the ResNet50 backbone of original DINO DETR with DINO v2, while keeping the detection head and training settings unchanged. This ensures a fair experimental comparison.

Q2: Comparison with Frozen DETR on cross‑domain datasets.

R2: Thank you for bringing Frozen DETR [1] to our attention.

We compare our method with Frozen DETR. We adopt the same baseline structure as Frozen DETR. Frozen DETR follows a two‑stage training paradigm of base training and finetuning. To align the experimental and dataset settings, we skip the base training step, finetuning only on the cross domain datasets. As shown in Table 1, our method consistently outperforms Frozen DETR across six crossdomain datasets. We will include these comparison experiments in the revised main paper.

Table 1: Comparison Results Against Frozen DETR under 10-shot setting.

Discussion between our method and Frozen DETR.

• Frozen DETR modifies the original DINO DETR architecture and introduces many additional parameters, thereby requiring base training followed by fine-tuning on novel classes to achieve strong performance. In contrast, our approach achieves best performance by fine-tuning only on novel classes, without any base training.
• Frozen DETR enhances detector features using only the final‑layer output of the VFM. Because it relies on self‑attention for feature enhancement, it incurs significant computational overhead. In contrast, our method employs the SEP and PEP modules, avoiding the $O(n^2)$ complexity, which allows us to fuse multi‑layer VFM features efficiently.

References:

[1] Fu S, Yan J, Yang Q, et al. Frozen-DETR: Enhancing DETR with Image Understanding from Frozen Foundation Models[J]. arXiv preprint arXiv:2410.19635, 2024.

Q3: Comparison with FrozenDETR on the COCO dataset.

R3: Thank you for your constructive suggestion.

We train and evaluate our method on the COCO dataset and compared it to the performance of Frozen DETR reported in the original paper. As shown in Table 2, our method outperforms Frozen DETR on COCO, further validating its general applicability. We will include these comparison experiments in the revised supplementary material.

Table 2: Comparison Results Against Frozen DETR on the COCO dataset.

Q4: Analysis of method limitations.

R4: Thanks for your insightful comment.

One limitation of our approach is that it is currently confined to visual tasks and does not yet support multimodal applications such as text‑guided visual grounding or video synthesis. However, our enhancement strategy can be easily applied to the visual backbones of multimodal models. As shown in Table 3, the method achieves strong performance when combined with Grounding DINO, even without explicit optimization for textual features. Extending the approach to multimodal scenarios will be explored in future work.

Table 3: Comparison results of original Grounding DINO and our method under 10-shot setting.

Q5: About the symbol used in Table 1.

R5: Thank you for your constructive suggestion.

Distill‑CDFSOD, CD‑ViTO, and our method all need to finetune on novel classes. We will revise this phrasing in the revised main paper.

Thanks for the authors' response. And my question has been resolved.

Q1: About some unclear parts in the paper.

Regarding the unclear parts in the paper, I strongly recommend that the authors release their complete code after the paper is accepted.

Q2 & Q3: Comparison with Frozen DETR.

First, I would like to emphasize that Frozen DETR, just like the method proposed in this paper, requires both base training (e.g., on COCO) and novel fine-tuning (on cross-domain datasets). Therefore, the authors should not highlight that their method skips base training.

Secondly, the result presented in Table 2 of R3, on the COCO dataset, are surprising to me. It indicates that incorporating features from vision foundation models can significantly improve object detection performance. I suggest the authors include this part in the appendix of the paper.

Thank you for your positive feedback!

Below we address your concerns.

Q1: About code.

R1: Thank you for your valuable suggestions.

We will release our complete code if the paper is accepted.

Q2: Comparison of our method and Frozen DETR.

R2: Thank you for your insightful comment.

As shown in Table 1, a key difference is that our method can directly use DETR checkpoint pretrained on COCO and finetune the added modules on novel classes, making it simple and efficient to apply. In contrast, Frozen DETR modifies the original detector architecture and cannot directly make use of pretrained DETR weights, so it requires training a new model from scratch. We will include relevant discussions and comparison results with Frozen DETR on six cross-domain datasets in the revised supplementary material.

Table 1: Method comparison of our method and Frozen DETR.

Q3: Experiment results on COCO dataset.

R3: Thank you for your recognition of our work.

We will include comparison results with Frozen DETR on COCO dataset in the revised supplementary material.

We sincerely thank Reviewer Xh4w for the constructive feedback, insightful questions, and valuable suggestions!

Below we address your concerns.

Q1: Clarification of VFM feature aggregation without ER and RR.

R1: Thank you for your insightful comment.

When the ER and RR modules are removed, we assign a unit weight to each VFM feature map. We then directly apply the SEP and PEP modules to all VFM features maps without backbone‑conditioned guidance or additional gating network.

Q2: Additional ablation studies.

R2: Thank you for your careful attention to our ablation studies.

We conduct comprehensive ablation studies, as shown in Table 1. We will include comprehensive ablation studies in the revised main paper.

Table 1: Results of ablation studies. 'SEP' denotes shared expert projection，'PEP' denotes the private expert projection，'ER' denotes the expert-wise router，'RR' denotes the region-wise router.

Q3: Evaluation of simpler feature aggregation strategy.

R3: Thank you for your valuable suggestions.

We compare our method to three simpler feature aggregation strategies. Specifically, we designed and evaluated three variants:

• All Stages: Retain all stage PEP modules and replace them with simple MLPs, removing all other modules.

• Final Stage: Retain only the final-stage PEP module and replace it with simple MLP, removing all other modules.

• Linearly Adding: Remove all modules and aggregate features at the backbone stage via linear addition,

As shown in Table 2, simple feature aggregation strategies lead to a substantial performance decline, thereby validating the efficacy of our approach. Moreover, aggregating features only at the final backbone stage performs worse than aggregating at every stage, highlighting the importance of our iterative design. We will include these comparison experiments in the revised supplementary material.

Table 2: Comparison Results of our method and simple feature aggregation strategies.

Q4: About recent works.

R4: Thank you for pointing us to [1] ,[2] and [3].

We acknowledge their relevance to our work and will cite these papers in the revised main paper.

ETS[1] proposes a two-step augmentation and grid-search strategy to adapt foundation models for cross-domain few-shot object detection. CDFormer[2] introduces transformer-based OBD and OOD modules to mitigate object-background and inter-class feature confusion in cross-domain few-shot detection. IFC[3] enhances cross-domain few-shot detection by using learnable instance feature caches with an instance reweighting module for robust prototypes.

References:

[1] Pan, Jiancheng, et al. "Enhance then search: An augmentation-search strategy with foundation models for cross-domain few-shot object detection." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

[2]Meng, Boyuan, et al. "CDFormer: Cross-Domain Few-Shot Object Detection Transformer Against Feature Confusion." arXiv preprint arXiv:2505.00938 (2025).

[3] Huang, Yali, et al. "Instance Feature Caching for Cross-Domain Few-Shot Object Detection." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.

Q5: About notations and typos.

R5: Thank you for your careful attention to our paper.

We have carefully revised all the writing issues you pointed out and re‑checked the entire paper to ensure that similar mistakes will not occur again.

Q6: Analysis of routing network based on the OD feature.

R6: Thank you for your insightful question.

We empirically observe that detectors possess strong localization and regression capabilities, accurately identifying object regions. However, due to domain shift, object features within these regions are often incomplete. This aligns with our paper’s conclusion that detectors have strong localization but weak classification. Therefore, we leverage the detector’s localization ability to guide VFM feature maps in supplementing the missing key features in salient regions so that the more generalizable VFM features can compensate for the information loss caused by domain shift. We will include the corresponding visualizations and discussions in the revised supplementary material.

Q7: Analysis of abnormal variations in DINOv2’s classification capability.

R7: Thank you for your constructive feedback.

We argue that the increased sample size leads the model to prioritize localization regression at the expense of its classification accuracy under the 10‑shot task setting. As shown in table 3, We observe that although DINOv2 exhibits strong classification performance(dAP: 27.7) in the 5‑shot setting, its average performance(mAP: 23.3) is low. In contrast, despite weaker classification performance(dAP: 30.3), the average performance(mAP: 27.4) increases substantially in the 10‑shot setting, supporting our conclusion.

Table 3: Comparison Results of DINOv2 under 5/10 shot task settings.

Q8: Analysis of method limitations.

R8: Thank you for your constructive suggestion.

One major limitation of our approach is that it currently focuses solely on visual tasks and has not been extended to multimodal applications such as text‑guided visual grounding or video synthesis. Nevertheless, the proposed enhancement strategy can be easily applied to the visual backbones of multimodal models. As demonstrated in Table 4, our method still performs strongly when combined with Grounding‑DINO, even without explicit optimization for textual features. We intend to further explore this direction in future work.

Table 4: Comparison results of original Grounding DINO and our method under 10-shot setting.

We sincerely thank Reviewer cjXu for the constructive feedback, insightful questions, and valuable suggestions!

Below we address your concerns.

Q1: Comparison to ETS.

R1: Thank you for bringing ETS method [1] to our attention.

We compare our method with ETS on six cross domain datasets. For fairness, we reproduce the experiments using the official ETS code and also report the results from the original paper. As shown in Table 1, our method can be directly applied to Grounding DINO without any modification and achieves similar performance to ETS, even though it is not specifically designed for Grounding DINO. In future work, we will explore designs better tailored to Grounding DINO. We will include these experimental results in the revised main paper.

Table 1: Comparison Results of ETS method and our method under 10-shot setting.

References:

[1] Enhance Then Search: An Augmentation-Search Strategy with Foundation Models for Cross-Domain Few-Shot Object Detection, CVPRw 2025.

Q2: Analysis of SEP and PEP modules.

R2: Thanks for your valuable question.

The SEP module captures general saliency features that are common across all expert features. However, these feature maps lack the precise boundary details present in individual expert features. In contrast, our PEP module is tailored to extract those boundary cues unique to each expert, effectively filling in the missing boundary information. We validate our conclusion through PCA‑based feature visualizations. We will add visualization results in the revised main paper.

Q3: Analysis of DETR’s classification capabilities.

R3: Thank you for your constructive question.

We have demonstrated DETR’s relatively weak classification performance in Table 4 of main paper. We use the AP drop(dAP) caused by false‑positive samples to reflect the model’s classification capability. We show the experimental results in Table 2 below for easier checking. We identify notable weaknesses in the classification performance of DINO DETR.Specifically,the dAP values reach 44.45/40.12/35.68 under the 1/5/10-shot settings. This is consistent with our paper’s conclusion that DETR has relatively weak classification capability.

Table 2: Comparison results of classification capability.

Q4: Comparison results on different detector backbones.

R4: Thank you for your valuable suggestions.

We conduct additional comparison experiments using differenet backbone architectures. A shown in Table 3, Applying our method to stronger backbones achieves better results than result we report in the main paper. For example, integrating our method into DINO with a Swin‑B backbone improves the average performance by 2.4 AP, while using a ViT‑L backbone achieves a gain of over 4.6 AP. We will include these comparison experiments in the revised main paper.

Table 3: Comparison results of different backbone.

Q5: Analysis of different foundation models.

R5: Thank you for your valuable question.

CLIP [4] uses large‑scale image text contrastive learning to capture high‑level semantics, excelling at zero‑shot classification and retrieval. However, it lacks the spatial granularity needed for region‑level tasks [1], making it less suitable for dense predictions such as detection and segmentation. In contrast, DINOv2 [5] uses self‑distillation with patch‑level reconstruction to learn fine‑grained, spatially coherent features. This leads to superior dense‑vision performance, as confirmed by studies [2,3] on visual grounding and segmentation.

References:

[1] Refining CLIP’s Spatial Awareness: A Visual‑Centric Perspective

[2] From CLIP to DINO: Visual Encoders Shout in Multi‑modal LLMs

[3] Learning Vision from Models Rivals Learning Vision from Data

[4] Learning Transferable Visual Models From Natural Language Supervision (CLIP)

[5] DINO v2: Learning Robust Visual Features Without Supervision

Q6: Analysis of method limitations.

R6: Thank you for your valuable comment about our method limitation.

The increased computational cost caused by the larger number of detector layers and expert features is indeed a limitation of our method. This limitation can be easily mitigated. As the number of backbone layers increases, we can sparsely select a subset of layers for processing, which has little impact on performance. Moreover, the routing mechanism introduces only minor computational cost and has minimal effect on inference speed. Another limitation of our approach is that it is currently designed only for visual tasks and does not yet handle multimodal scenarios, such as text‑guided visual grounding or video synthesis. However, our enhancement strategy can be easily applied to the visual backbones of multimodal models. As shown in Table 4, our method achieves strong performance when combined with Grounding DINO, even without explicit optimization for textual features. We will further explore these directions in future work.

Table 4: Comparison results of original Grounding DINO and our method under 10-shot setting.

Thank you for your positive feedback!

We are delighted that we addressed your concerns.

We also sincerely appreciate your efforts to improve our work.

We sincerely thank Reviewer ZRE6 for the constructive feedback, insightful questions, and valuable suggestions!

Below we address your concerns.

Q1: Inference Speed Comparisons.

R1: Thanks for your insightful question.

We compare inference speed with YOLO‑World and Qwen Model using Frames Per Second(FPS). Our method achieved similar FPS like YOLO‑World while attaining the highest performance, as shown in Table 1.

Table 1: Comparison results of inference speed and performance. 'mAP' denotes the mean performance across the six cross‑domain datasets.

Q2&Q3: Comparison to MLLMs and OVMs.

R2&R3: Thanks for your constructive suggestion. We compare our method with multimodal large language models(MLLMs) and open‑vocabulary methods(OVMs). Using their open‑source code, we conducted fair comparisons on the same dataset. Qwen model and Ferret model obtain results through text‑guided visual grounding. Grounding‑DINO and YOLO‑World derive detection results via image‑text matching. All codes are publicly available on their official websites.

As shown in Table 2, our method achieved highest performance compared to OVMs and MLLMs across six cross domain datasets. Compared to Grounding‑DINO and YOLO‑World model, our method achieves the improvements of 22.1mAP/26.3mAP. For Qwen model and Ferret model, our method achieves the improvements of 27.5mAP/36.2mAP. We argue that models such as Qwen, despite being trained on large‑scale image‑text datasets, have never seen the novel classes in the cross domain dataset, resulting in weak zero‑shot performance. In contrast, our method adopts a cross‑domain learning strategy that integrates with visual foundation models, achieving superior performance on cross‑domain tasks. This also points to an important future direction, where we plan to extend our cross‑domain learning approach to MLLMs to further improve their performance on cross‑domain tasks.

Table 2: Comparison results of our method, MLLMs and OVMs under 10-shot setting.

Q4: Analysis of method limitations.

R4: Thanks for your insightful question.

A key limitation of our method is that our current approach is limited to purely visual tasks and does not yet support multimodal scenarios such as text-guided visual grounding or video synthesis. Nonetheless, the proposed enhancement strategy is readily adaptable to the visual backbones of multimodal models. Even without optimizing textual features, our method still achieves strong performance when combined with Grounding DINO, as shown in Table 3. We plan to explore in future work.

Table 3: Comparison results of original Grounding DINO and our method under 10-shot setting.