Thank you so much for the great comments. Our response to your concerns is presented as follows.

$\textcolor{orange}{\text{Q1:}}$ It would be interesting to see additional results in a similar scenario, such as test-time domain adaptation.

$\textcolor{green}{\text{Response:}}$ Great suggestion. We have evaluated the proposed method on the Office-Home dataset in the Test-Time Adaptation (TTA) setting. As shown in the table below, our method demonstrates advantages over previous state-of-the-art methods (results cited from [1], where all methods maintain a fixed batch size of 64, similar to ours). We will include these results in the revised manuscript to enhance our analysis.

Table: Comparison results (%) in the TTA setting.

[1] Benchmarking test-time adaptation against distribution shifts in image classification. arXiv preprint arXiv:2307.03133, 2023.

[2] Tent: Fully test-time adaptation by entropy minimization. In ICLR20.

[3] Test-time classifier adjustment module for model-agnostic domain generalization. In NeurIPS21.

[4] Continual test-time domain adaptation. In CVPR22.

[5] Efficient test-time model adaptation without forgetting. In ICML22.

[6] Towards stable test-time adaptation in dynamic wild world. In ICLR23.

$\textcolor{orange}{\text{Q2:}}$ Could the authors provide more detailed descriptions for Fig. 1? Additional explanations would help readers better understand the key ideas of the paper.

$\textcolor{green}{\text{Response:}}$ To address this issue, we will revise the caption of Fig. 1 as follows:

“Conceptual illustration of ProDe. We align the adapting direction with the desired trajectory by leveraging a proxy space that approximates the latent domain-invariant space. This process incorporates direction adjustments based on proxy error correction, effectively implementing proxy denoising, and finally achieves enhanced model adaptation.”

We believe this revision will clarify the key ideas presented in the figure and improve understanding for our readers.

Thank you so much for the great comments. Our response to your concerns is presented as follows.

$\textcolor{orange}{\text{Q1:}}$ Ambiguity and Misleading Terminology in Domain Invariance Claims.

$\textcolor{green}{\text{Response:}}$ Insightful discussion. In the context of domain adaptation, "domain invariant space" refers to an ideal latent embedding space where the mapped features from different domains align with the same probability distribution. A general goal of domain adaptation is to approach this ideal space, although achieving it perfectly in practice is often not feasible. This terminology is widely accepted within the domain adaptation community.

While we understand the suggestion to redefine the term, introducing a new term is often conservative unless a fundamentally new concept is presented, which is not the case here. Therefore, we prefer to retain the term "domain invariance" while further elaborating on its meaning to enhance understanding. However, we are still open to more suggestions.

$\textcolor{orange}{\text{Q2:}}$ Unsupported Assumption Regarding and Invariant Space Error.

$\textcolor{green}{\text{Response:}}$ We appreciate your concern regarding the assumption. As noted in our analysis presented in Figure 4 (d) (Lines 510-519), our findings indicate that the impact of denoising $e_{VI}$ is negligible during the early phases of domain adaptation. That means, in this initial stage the divergence between the Vision-Language model’s space and the domain-invariant space does not significantly influence adaptation outcomes.

We will ensure to clarify this in the revised manuscript.

$\textcolor{orange}{\text{Q3:}}$ Oracle Configuration.

$\textcolor{green}{\text{Response:}}$ Apologies for this typo, and we will correct it.

$\textcolor{red}{\text{Q4:}}$ Over-Reliance on a Single Vision-Language Model (CLIP).

$\textcolor{green}{\text{Response:}}$ Our selection of CLIP is based on its widespread use in existing UDA and SFDA research, ensuring fair comparison. We recognize the importance of evaluating our method with other vision-language models and appreciate the suggestion to expand our analysis. We have conducted this test with OpenCLIP [1], selecting the previous best ViL method DIFO [2] as a comparison. The results in the table below indicate that the proposed method is generic with the ViL model, and can readily benefit from the advancement in ViL models.

We will add the detailed results in the revised manuscript.

Table: Reliance analysis on ViL models.

[1] Reproducible scaling laws for contrastive language-image learning, In CVPR23.

[2] Source-free domain adaptation with frozen multimodal foundation model, In CVPR24.

$\textcolor{red}{\text{Q5:}}$ Generalization Claims to SFDA Settings.

$\textcolor{green}{\text{Response:}}$ Thank you for your suggestion. We have evaluated ProDe in broader SFDA settings, specifically for source-free multi-target domain adaptation (SF-MTDA) and source-free multi-source domain adaptation (SF-MSDA). For SF-MTDA, we treat multiple target domains as a single integrated domain and adapt the source model accordingly. For SF-MSDA, we follow the ensembling approach from [1], passing the target data through each adapted source model and averaging the soft predictions to derive the test labels. The results, as shown in the table below, demonstrate that our method substantially outperforms state-of-the-art alternatives in both settings.

We will include these findings in the revised manuscript.

Table: SF-MTDA results (%) on Office-Home

Table: SF-MSDA results (%) on Office-Home

[1] Unsupervised multi-source domain adaptation without access to source data, In CVPR21.

[2] Conmix for source-free single and multi-target domain adaptation, In WACV23.

[3] Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation, In ICML20.

Thank you so much for the great comments. Our response to your concerns is presented as follows.

$\textcolor{orange}{\text{Q1:}}$ I would like to know if using ViL for pseudo-supervision is suboptimal when the distance between the source domain space $D_S$ (or training dataset $D_{Tt}$) and $D_I$ is closer than the distance between $D_V$ and $D_I$. Additionally, the formulas (5) and (6) in the paper contain many hyperparameters; is tuning these hyperparameters a challenge?

$\textcolor{green}{\text{Response:}}$ Thank you for this insightful question. We believe that using ViL for pseudo-supervision, despite potential distance discrepancies between the domain spaces, still provides valuable generic knowledge throughout the adaptation process. In such cases as mentioned, our proposed Mutual Knowledge Distillation method plays a crucial role. It effectively mitigates potential negative effects by integrating task-specific knowledge from the in-training target model with the generic knowledge from the ViL model. This integration ensures that our approach is not overly reliant on the ViL model alone, allowing it to adapt more effectively to the nuances of the target domain.

Regarding the hyperparameters in Equations (5) and (6), we have provided their values used in our experiments in the "Hyperparameter Setting" section of Appendix-D. Specifically, we set the values of $(\alpha, \beta, \omega)$ to (1, 1, 0.4) consistently across all datasets, which did not require extensive fine-tuning. We have also discussed their insensitivity in Appendix E.2 (see Fig. 5). The parameter $\gamma$ is the only one that necessitates fine-tuning, as it is sensitive to the dataset scale, also noted in the TPDS method [1].

[1] Source-free domain adaptation via target prediction distribution searching. International Journal of Computer Vision (IJCV), 132(3):654–672, 2024.

Thank you very much for the great comments. Our response to the your questions are elaborated below.

$\textcolor{orange}{\text{Q1:}}$ The intuition behind how a Gaussian distribution is considered for the VLM’s predictions is not entirely clear. Moreover, the conversion in Eq. 2 also seems unclear, in terms of how the conversion is possible.

$\textcolor{green}{\text{Response:}}$ Thank you for your insightful question regarding Theorem 1 and the treatment of the VLM's predictions as a Gaussian distribution. This assumption stems from the Central Limit Theorem, which suggests that, under certain conditions, the sum of a large number of independent random variables will tend to be distributed normally, regardless of the original distributions of the variables. In our context, we consider the VLM’s predictions to be influenced by various sources of noise and uncertainty, which justifies the Gaussian approximation.

Regarding the conversion in Eq. 2 (also see Lines 195-204), we express this relationship in terms of probability distributions to facilitate the understanding of how the confidence of the VLM's predictions relates to the current training model and the source model. By framing the prediction as a probabilistic event, we can leverage the concept of proxy confidence, $P(GP(V)=True,t)$, to quantify how reliable we consider the VLM’s predictions to be at any point in the adaptation process. In essence, this conversion allows us to connect the notion of prediction reliability with the underlying distributions, making it easier to reason about the impact of proxy errors and their effect on the adaptation process.

We will further clarify these points.

$\textcolor{orange}{\text{Q2:}}$ How does the proposed method ProDe perform in a multi-source or multi-target domain adaptation setting?

$\textcolor{green}{\text{Response:}}$ Please take the responses to $\textcolor{red}{\text{Q5}}$ of reviewer SdQs.

$\textcolor{orange}{\text{Q3:}}$ Why have the authors used DomainNet-126 rather than the full DomainNet dataset?

$\textcolor{green}{\text{Response:}}$ This is to ensure fair comparison with existing works as this version DomainNet-126 has been extensively used with cleaned labels as compared to the original version (see Appdendix-C).

We will clarify.

$\textcolor{orange}{\text{Q4 and Q5:}}$ Following the discussion on fair comparisons in the previous section, can the authors present results in a fair setting? Additionally, if the above is not possible, could the authors present results with ViTs rather than ResNet for a more fair comparison?

$\textcolor{green}{\text{Response:}}$ Per suggestion, we present comparisons with typical SFDA methods using ViT backbones (cited from DPC [1]), employing ViT-B/16. The results in the table below show that ProDe-V16 consistently outperforms DPC in most cases. An exception is that ProDe-V16 is only 0.8% behind on Office-31, which may be attributed to potential overfitting on this relatively small dataset. Notably, even with a ResNet backbone for the target model, ProDe-V16 still surpasses DPC, which utilizes a ViT. Generally, using a ViT for such a small training dataset is unnecessary due to the tendency for overfitting.

We will add this test.

Table: Comparison results (%) on Office-31, Office-Home, VisDA and DomainNet-126.

[1] Towards Dynamic-Prompting Collaboration for Source-Free Domain Adaptation, In IJCAI24.

[2] Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML20.

[3] Exploring domain-invariant parameters for source free domain adaptation, In CVPR22.

[4] Domain-specificity inducing transformers for source-free domain adaptation. In ICCV23.

[5] Attracting and dispersing: A simple approach for source-free domain adaptation, In NeurIPS22.

Thanks again for the valuable comments and suggestions. Our response to these new comments are elaborated below.

$\textcolor{orange}{\text{Q6}}$: Insightful comment. The proposed method ProDe utilizes learnable prompts (Fig. 2 right) that are passed along with the template text prompts “a photo of a <cls>”. However, there is no discussion on this aspect of the method. Why are these prompts being used? How many prompts are being used? How essential are the learnable prompts to ProDe? Can the authors present an ablation study on these learnable prompts in addition to addressing the above queries?

$\textcolor{green}{\text{Response}}$: In the proposed approach, we employ the initialization template of “a photo of a <cls>” for each class because it is the most used template to initiate the learnable prompt. In addition, we have evaluated the effect of prompt learning with this initiation (see Table 20 in the revised manuscript).

For further analysis, we conduct an ablation study on nine typical templates. As shown in the table below, there are no evident performance variations, indicating our method is insensitive to the selection of templates.

We have added the results in the revised manuscript.

Table: Ablation study results on initialization template selection.

$\textcolor{orange}{\text{Q7}}$: Regarding the question from Reviewer SdQs about the proposed method's overreliance on CLIP, the reviewer believes that ProDe can work with any discriminative Vision-Language Model (VLM), i.e., a VLM that generates embeddings from the vision and text encoders as outputs, such as CLIP or ALIGN. The authors can present results with ALIGN, FILIP, or BLIP to support this claim.

$\textcolor{green}{\text{Response}}$: Great suggestion. In response to reviewer SdQs's concern about the overreliance on CLIP, we have further tested the generality of our method with OpenCLIP [1] as the ViL model. Prefere refer to the responses to $\textcolor{red}{\text{Q4}}$ of reviewer SdQs for more details. Additionally, the revised paper includes a thorough analysis in Tables 15–19 and the section "Reliance Analysis on ViL Models" in the supplementary document. We can test a third ViL model for the final version if needed.

[1] Reproducible scaling laws for contrastive language-image learning, In CVPR23.

Thank you for your insightful comment and for drawing our attention to this issue.

To begin with, Source-Free Domain Adaptation (SFDA) inherently involves uncertainty arising from unknown noise. Accordingly, Equation (1) can be interpreted through the lens of Distributionally Robust Optimization (DRO) theory, where the upper bound is understood as a worst-case estimate. Under this interpretation, the conclusion that $\eta_t$ gradually increases remains valid within the DRO framework, which takes uncertainty into consideration.

We will revise the ArXiv version to clarify this point more explicitly.