Test-Time Adaptive Object Detection with Foundation Model

W1: About fair comparison.

Previous methods rely on source domain data for fine-tuning or statistical computation, whereas our method has no access to source domain data. And considering the differences in the adopted detectors, achieving a completely fair comparison is challenging. However, we have made efforts to ensure a fair comparison with previous methods. For example, in Table 2, we compare with CTTAOD based on Faster R-CNN (Swin-T). Its results are even worse than using ResNet-50/101 as backbones, because ResNet-50/101 use SoftTeacher weights pretrained on the COCO training set with extensive data augmentation.

We also conduct thorough comparisons with the strong baseline Mean-Teacher using Grounding DINO, which demonstrate the performance improvements achieved by our method.

W2: Ablation on the threshold $th_{pl}$.

Thank you for pointing out this issue. We select thethreshold $th_{pl}$ under Gaussian noise corruption on Pascal-C. When $th_{pl}$ is too high, only a small number of pseudo-labels are available for test-time adaptation, which hinders the model from effectively adapting to the target domain. Conversely, if $th_{pl}$ is set too low, noisy pseudo-labels may be introduced, which can also degrade the model's performance. Therefore, we set it to 0.3.

W3: About the applicability in real-time and resource-constrained settings.

The latency of our method compared to directly testing with a vision-language detector is 693.8 ms/img vs. 213.1 ms/img. However, it brings significant performance improvements, with gains of +11.4% AP50 on Pascal-C and +5.4% mAP on COCO-C. For more analysis of inference speed and storage costs, please refer to our response to Reviewer 4Xpg W1.

Typos: Thank you, we will revise the typos.

Q1: The performance under significant distribution shifts or unseen categories.

For Pascal-C and COCO-C, we follow previous works and use the most challenging corruption level for evaluation. Additionally, ODinW-13 contains 13 datasets with significant domain shifts (e.g. remote sensing, underwater, etc.). These benchmarks ensure significant distribution shifts from the pre-training data. On the other hand, since the pre-training data of Grounding DINO includes grounding data, it is difficult to determine whether the categories in OdinW-13 are entirely novel. Our key contribution, by contrast, lies in introducing a vision-language detector to overcome the closed-set limitation. The experimental results in Table 1, Table 2, and Fig. 4 demonstrate that our method performs well under significant distribution shifts and in open-set scenarios.

Q2: About the IDM module for distribution drift in continual test-time settings and the error accumulation from noisy pseudo-labels in continual scenario.

Our method focuses on TTAOD, and continual test-time setting is beyond the scope of our paper. Theoretically, the IDM module can be extended to handle distribution drift in a continual test-time scenario. Like CTTAOD [1], we can detect sudden distribution shifts to identify the arrival of a new target domain. Then, we clear the triplets in the IDM module and re-maintains them on the new target domain to handle distribution drift. As for error accumulation, upon detecting the arrival of a new target domain, we can reinitialize new text-visual prompts in our MPMT to alleviate this issue.

[1]Yoo J, Lee D, Chung I, et al. What how and when should object detectors update in continually changing test domains?[C]. In CVPR, 2024.

Q3: About the runtime latency compared to direct testing.

Please refer to our detailed response to W3.

Q4: About the hyperparameters $th_{pl}$ and memory capacity.

The ablation study of the threshold $th_{pl}$ is provided in W2, and the analysis of IDM’s memory capacity is presented in Fig. 5 and Section 4.4 of the paper. Regarding hyperparameter tuning, we strictly followed prior work by selecting hyperparameters under a single type of corruption (Gaussian noise in our paper).

Limitations: Please refer to Q1 and W3. Thank you for the reminder. We will include a discussion on the detection performance and inference speed of our method in the limitations section.

W1: About inference speed and storage costs.

Thank you for pointing out the potential limitation. As shown in the table below, compared to the Mean-Teacher baseline using Grounding DINO, our approach achieves nearly a 2% performance improvement with only minimal additional inference time. And our method overcomes the closed-set limitation and enables us to completely eliminate reliance on source domain data. Although previous approaches have faster inference speeds, they require pre-training weights for each source domain data and extract statistical characteristics from the source data—both of which also incur significant time costs. For example, pre-training takes about 2 hours for VOC and 15 hours for COCO on 8 V100 GPUs.

In the following table, CTTAOD-Skip* demonstrates excellent latency performance, primarily because it skips adaptation for some test samples based on the distribution gap. Recent works, such as Efficient TTAOD [1], also specifically focus on the efficiency issue. These techniques can also be integrated into our method to further improve inference speed.

Employing IDM does introduce an increase in storage cost, primarily due to loading DINOv2-L, which requires approximately 1 GB. The triplets maintained by IDM itself require only about 50 MB. In real-world deployments, techniques like quantization and pruning can be considered to reduce storage costs.

Thank you again. We will provide a detailed discussion of these limitations in the paper.

[1]Wang K, Fu X, Lu X, et al. Efficient Test-time Adaptive Object Detection via Sensitivity-Guided Pruning[C]. In CVPR, 2025.

W2: Preventing representation collapse.

This is a key issue that we have also noticed, and we have already discussed it in Section 3.3. Since the text prompt is added to the language branch in the form of a residual (Eq. 1), representation collapse can be avoided by initializing it with all zeros. For the visual prompts, the proposed TTWS is specifically designed to address this issue, as shown in Fig. 3.

We include results for directly testing with added learnable vectors in the following table. Using randomly initialized learnable vectors indeed leads to representation collapse, but this issue can be addressed by incorporating the proposed TTWS. Furthermore, representation collapse can cause catastrophic collapse of the detector during test-time adaptation. Our proposed TTWS effectively addresses this issue, as demonstrated in Tables 3 and 6 of the paper.

Q1: The change in mAP after each epoch.

First, there may be a misunderstanding. In TTA, we perform adaptation with only one pass over the test data, and the model learns while making predictions. The evaluation is conducted using the accumulated prediction results. We will emphasize this point in the paper. To more clearly demonstrate the evolution of mAP, we evaluated our method at different iterations, and the results show that as the number of iterations increases, the model's performance gradually improves, indicating a more thorough adaptation to the target domain.

Q2: About the ablation results.

We have already investigate the impact of using TPT, VPT, TTWS, and MR in Table 3, and provide detailed results in Table 6. This might have been overlooked.

Q3: About the latency.

Please refer to our response to W1.

Q4: Comparisons with recently proposed SOTA methods.

We have compared our method with recent SOTA approaches, including STFAR, CTTAOD, and the strong baseline Mean-Teacher. We also noticed a recent work, Efficient TTAOD*[1], which primarily focuses on the efficiency issue in Continual Test-time Object Detection. Since its code has not been released, we are only able to compare with it on its first target domain of the first run. Additionally, we compare our method with several TTA approaches designed for image classification [2,3,4]. As illustrated in the following table, our method achieves SOTA performance.

[2] Shu M, Nie W, Huang D A, et al. Test-time prompt tuning for zero-shot generalization in vision-language models[J]. In NeurIPS, 2022.

[3] Feng C M, Yu K, Liu Y, et al. Diverse data augmentation with diffusions for effective test-time prompt tuning[C]. In ICCV, 2023.

[4] Zhang J, Huang J, Zhang X, et al. Historical test-time prompt tuning for vision foundation models[J]. In NeurIPS, 2024.

Typos: Thank you. We have carefully proofread the paper and corrected the typos.

W1: Compare with other PEFT methods.

Thank you for your suggestion! Prompt tuning has been widely used for fine-tuning VLMs and has achieved impressive results. Following your advice, we compared our method with two PEFT approaches, LoRA and Adapter, based on the implementations provided in [1]. Both LoRA and Adapter show improvements over direct testing, but our method consistently achieves superior performance.

[1] Yin D, Hu L, Li B, et al. 5% $>$ 100%: Breaking performance shackles of full fine-tuning on visual recognition tasks[C]. In CVPR, 2025.

W2: Reasons for adopting DINOv2 for IDM.

DINOv2 is pretrained on large curated data from diverse sources, enabling highly generalizable feature representations. To determine the optimal feature extractor for the IDM module, we conducted a comprehensive comparison between DINOv2 and CLIP across various model sizes. Experiments demonstrate DINOv2's superior performance over CLIP, with model scaling further enhancing MR. Considering computational efficiency, we use DINOv2-L to maintain the IDM module.

W3: Performance with large backbones.

Considering computational cost, Swin-T is commonly used in tasks based on vision-language detectors [2]. Accordingly, we also adopt Swin-T for our experiments. In general, using larger models yields further performance improvements, which is also confirmed by the results shown in the table below. Moreover, our method consistently brings performance gains across Swin-T, Swin-B, and Swin-L.

[2] Deng J, Zhang H, Ding K, et al. Zero-shot generalizable incremental learning for vision-language object detection[J]. In NeurIPS, 2024.

W4: Rename Memory Reinforcement and Memory Hallucination.

Thanks for your suggestion. We will revise "Memory Reinforcement" to "Memory Enhancement". As for "Memory Hallucination", in the context of few-shot learning [3], "hallucination" is commonly used to refer to the generation of new samples based on existing data. Therefore, we adopt this term. If you have a better suggestion, we would be glad to consider it.

[3] Zhang W, Wang Y X. Hallucination improves few-shot object detection[C]. In CVPR, 2021.