COSDA: Counterfactual-based Susceptibility Risk Framework for Open-Set Domain Adaptation

Dear Reviewer rce5,

Thank you for your decision and constructive feedback. These detailed and professional comments have highly enlightened and encouraged us to make every effort to improve our work. We hope our responses could resolve the concerns.

Claims And Evidence. What is the meaning of “c' is the specific implementation of C” in Definition 1.

A1. We sincerely appreciate the opportunity to clarify this important conceptual point. We provide a simplified binary example to clarify the meaning. As shown in Fig. 7 (Appendix), for maple leaf classification, let $C$ denote the presence of palm-shaped lobes. There the specific implementation $c\in \{0,1\}$:

• $c=1$: Sample exhibits palm-shaped lobes
• $c=0$: Absence of this trait

Then the different implementations of $C$ in the sample will influnence the probablity of the label. We hope this illustrative example has clarified the causal feature representation.

Methods And Evaluation Criteria. I think the authors could conduct experiments on more challenging datasets (such as DomainNet)

A2. We sincerely appreciate the suggestion. We have expanded our experiments on both DomainNet and VisDA.

Table 1. Comparison results on DomainNet and VisDA with CLIP backbone*( Baseline results from [R1]).***

[R1] Cross-domain Open-world Discovery, ICML 2024

Key Findings from Table 1:

1. Substantial HOS Improvements:
   • On DomainNet, COSDA-CLIP achieves 73.9% HOS, outperforming CROW by 14.9%
   • On VisDA, COSDA-CLIP reaches 78.4% HOS, outperforming CROW by 9.2%
2. Dual Strengths in OS* and UNK: With CLIP, COSDA particularly further enhances known-class classification and unknown-class detection (UNK).

Table 2. Performance comparison with [1][3] on Office-Home and Office-31

Table 2 shows leading performance of COSDA on small-scale datasets compared with [1][3]. These experiments could further confirm the effectiveness of COSDA. We will well cite [1][2][3] in the updated version.

Experimental Designs: It is more appropriate to visualize the feature space of a more advanced method, especially ANNA (2023).

A3. Thanks for your suggestions. We have added the requested comparison with ANNA in our feature space analysis. The visualizations can be found at https://anonymous.4open.science/r/tsne-5F2D/

Essential References Not Discussed: A close relative idea has been proposed in prior work DCC[1] published in CVPR 2021, which is not discussed in the related work. Several classical researches are missing, such as OVAnet [2] and UADAL [3].

A4. We appreciate your insightful comments. We will incorporate [1][2][3] into the updated related work. DCC proposes Domain Consensus Clustering for universal domain adaptation, using semantical and sample-level consensus to effectively separate and distinguish common classes from private ones. OVANet proposes a universal domain adaptation method that learns an open-set threshold from source data via one-vs-all classifiers and adapts it to the target domain by minimizing class entropy. UADAL addresses open-set domain adaptation with unknown-aware adversarial learning, aligning known classes while segregating unknowns in feature space. We carefully read [1][2][3], and the difference is we exclude all known-class positive samples and cluster only the remaining negative samples (i.e., those not belonging to any known class). Besides, unlike one-vs-all strategies requiring C clustering operations per epoch (C = the number of known classes), COSDA achieves comparable performance with only one clustering per epoch.

Other Comments Or Suggestions

A5. We sincerely appreciate your thorough review of our paper. We will carefully correct the errors in the updated version and thoroughly review the text to prevent similar mistakes.

LLine 133-135: The parameter $\epsilon$ represents the degree of intervention on the causal features. It should be large to ensure that the semantic information can be disentangled (i.e., so that it influences the probability of label $Y$).

LLine 163-164: The command \LB (which represents the label $Y$) was missing its backslash due to a typesetting error.

The anonymous code repository is unavailable: We have updated the code in the previously provided anonymous link while also including necessary details for running it.

We sincerely appreciate your insightful comments. Please let us know if you need any further information or if there are additional points you would like to discuss with us.

Best regards,

Authors of #10234

Dear Reviewer zv4D,

Thank you for your decision and constructive feedback. We have studied the comments carefully and made thorough revisions. We also greatly appreciate your insightful questions and hope that our responses have helped to clarify them.

Weakness 1: As reported in the main tables, in most cases the proposed methods are not good and fail to be SOTA (State of the Art). While the methods may show promise or have certain advantages in specific scenarios, they do not consistently outperform existing techniques across all metrics and benchmarks.

A1. We appreciate the reviewer's attention to our experimental evaluation. Our comprehensive assessment includes three key metrics: OS* (known-class accuracy), UNK (unknown-class detection), and HOS (balancing both aspects). As highlighted in our contributions, COSDA achieves consistent improvements over SOTA methods:

• Absolute HOS gains: +2.9% (Office-Home), +2.2% (Office-31), +1.0% (Image-CLEF)
• Dominant rankings:
  • Image-CLEF: Achieved the highest HOS in 11 out of 12 subtasks, and the best UNK score in 9 out of 12 subtasks.
  • Office-31: Achieved the highest HOS in 3 out of 6 subtasks, and the best UNK score in 3 out of 6 subtasks.

The method demonstrates both generalizability (highest average performance across all benchmarks) and robustness (most of the subtasks show statistically significant improvements). These results validate our design's effectiveness in handling the known-unknown class trade-off in many cases.

Weakness 2: I think this task could potentially be completed by leveraging Video-Language Models (VLMs). However, the authors did not demonstrate scenarios or cases where VLMs might fail to work effectively. It's important for a comprehensive evaluation to include both the capabilities and limitations of such models.

A2. We sincerely appreciate this valuable suggestion. We have conducted additional experiments with CLIP ViT-L on challenging datasets DomainNet and VisDA.

Table 1. Performance comparison (%) on DomainNet and VisDA datasets using CLIP backbone. (All baseline results are obtained from [R1]).

[R1]Cross-domain Open-world Discovery, ICML 2024

Implementation Details. Considering both time constraints and GPU memory demands (particularly for the larger models), we utilized six 40GB NVIDIA A100 GPUs to execute the new experiments. DomainNet and VisDA use the same hyperparameter settings as smaller-scale datasets, specifically $\lambda_s=0.2$, $\lambda_{exo}=1$. But the learning rate has been reduced, specifically $lr = 5e-4$.

Key Findings from CLIP Backbone Experiments:

1. Substantial HOS Improvements:
   • On DomainNet, COSDA-CLIP achieves 73.9% HOS, outperforming CROW by 14.9%.
   • On VisDA, COSDA-CLIP reaches 78.4% HOS, outperforming CROW by 9.2% .
2. Dual Strengths in OS* and UNK: With CLIP, COSDA particularly further enhances known-class classification and unknown-class detection (UNK).
   • For DomainNet OS*, a 1.7% improvement;
   • For DomainNet UNK, a 25.3% improvement;
   • For VisDA OS*, an 8.2% improvement;
   • For VisDA UNK, a 9.8% improvement.

Table 2. Sub-Task Performance of COSDA on DomainNet with CLIP bakcbone

Table 2 exhibited detailed results of COSDA for 6 subtasks on DomainNet with CLIP. These additional experiments could further confirm the effectiveness of COSDA. We fully agree with the reviewer's insightful suggestion about VLMs. This represents a promising new direction worth exploring in OSDA.

Weakness 3 & Other Comments: No codes are provided.

A3. We sincerely appreciate the reviewer's emphasis on reproducibility. As noted in our submission (Page 6, Line 321), we open-sourced the complete implementation. For greater visibility, we will relocate the code announcement to the abstract/introduction in the updated version.

Current Implementation Overview:

• Benchmark Support: Office-Home, Office-31, Image-CLEF, DomainNet, VisDA
• Architecture Flexibility:
  • CNN backbones (ResNet, VGG)
  • VLMs (CLIP)
• Training Frameworks:
  • Multi-GPU distributed training
  • Single-GPU training

We sincerely appreciate your insightful comments once again. Please let us know if you need any further information or if there are additional points you would like to discuss with us.

Best regards,

Authors of #10234

Dear Reviewer pQYc,

We sincerely appreciate your constructive comments on our work. We have carefully addressed each point raised and incorporated corresponding improvements. We hope our responses have adequately addressed the concerns.

Weaknesses 1 & Weaknesses 3. More ablations on ViT backbone should be added; Lack of experiments on large-scale benchmarks such as DomainNet and VisDA.

A1. We fully agree with the reviewer's valuable suggestion regarding the importance of evaluating our method on large-scale datasets with diverse backbones, as this would further validate the generalization of COSDA. In response to this suggestion, we implemented additional experiments on VisDA and DomainNet [1][2][R1][R2] and used CLIP ViT-L14-336px as the backbone.

[R1]Cross-domain Open-world Discovery, ICML 2024
[R2] Domain consensus clustering for universal domain adaptation, CVPR 2021

Implementation Details. Considering both time constraints and GPU memory demands, we utilized six 40G A100 GPUs to execute the new experiments. DomainNet and VisDA use the same hyperparameter settings as smaller-scale datasets, specifically $\lambda_s=0.2$ and $\lambda_{exo}=1$. The learning rate is $5e-4$.

Table 1. Performance comparison (%) on DomainNet and VisDA using CLIP backbone. (Baseline results from [R1]).

Key Findings from CLIP Backbone Experiments:

1. Substantial HOS Improvements:
   • On DomainNet, COSDA-CLIP achieves 73.9% HOS, outperforming CROW by 14.9%.
   • On VisDA, COSDA-CLIP reaches 78.4% HOS, outperforming CROW by 9.2% .
2. Dual Strengths in OS* and UNK: With CLIP, COSDA particularly further enhances known-class classification and unknown-class detection (UNK).
   • For DomainNet OS*, a 1.7% improvement;
   • For DomainNet UNK, a 25.3% improvement;
   • For VisDA OS*, an 8.2% improvement;
   • For VisDA UNK, a 9.8% improvement.

Table 2. Performance comparison on VisDA (VGG19). (Baseline results from [R2]).

Table 3. Sub-Task Performance of COSDA Across Backbones on DomainNet

Table 4. Performance comparison with [1][2] on Office-Home and Office-31

Table 2 shows leading performance on VisDA (VGG-19). Table 3 exhibited detailed results of COSDA for 6 subtasks on DomainNet with ResNet-50 and CLIP. Table 4 supplements the performance comparison between COSDA and references [1][2] on small-scale datasets. These additional experiments could further confirm the effectiveness of COSDA. Additionally, we will well cite [1][2] you provided in the updated revisions.

Weaknesses 2. The OS* of the proposed framework is worse than baselines on all datasets. For example, 0.3 on Image-CLEF, 3.2 on Office-31, and 6.7 on Office-Home.

A2. We sincerely appreciate the reviewer's insightful observation regarding the OS* performance. We acknowledge that our method demonstrates a modest compromise in OS* scores. However, this trade-off enables significant improvements in UNK (33.2%, 43.4%, and 49.4% gains, respectively). HOS is the most crucial metric, which achieves a balance between known-class and unknown-class recognition. For HOS, COSDA outperforms the highest OS* methods by 19.3%, 28.1%, and 36.5% on three small-scale datasets. Notably, COSDA achieves OS*, UNK, and HOS improvement on challenging DomainNet (Table 1). These results could substantiate the advantages of our approach.

Other Comments. There are too many loss terms in Eq (25) which is complex and may make optimization difficult. How do you balance the contribution of different terms?

A3. Thank you for raising this important point. To balance the contributions of different loss terms in the overall objective, we introduced weight hyperparameters for loss items. These hyperparameters were optimized via a grid search strategy to ensure an appropriate trade-off for the dataset. As shown in Fig. 6, we analyzed the impact of parameters.

We sincerely appreciate your insightful comments again. Please let us know if you need any further information or if there are additional points you would like to discuss with us.

Best regards,

Authors of #10234

Dear Reviewer 6vAQ,

Thank you for your decision and constructive feedback. We have stuied the comments carefully and made through revisions. We hope that our responses have helped to clarify the concerns.

Experimental Designs Or Analyses & Q1. whether the method can perform well on harder data (like DomainNet [1]) remains unknown ; All the conclusions and insights can change when using different backbones. Why ResNet-50 remains the preferred choice?

A1. We sincerely appreciate your insightful suggestions and questions. Initially, we adopted ResNet50 as the backbone to maintain consistency with prior work in this domain. However, as you rightly pointed out, evaluating the model with more advanced backbones and larger-scale datasets would better demonstrate its robustness. Following this suggestion, we implemented addtional experiments on VisDA[3] and DomainNet[1][2] and discovered that our method demonstrates good performance on both CNN-based and CLIP-based architectures.

Table 1. Comparison results on DomainNet and VisDA with CLIP backbone (All baseline results are obtained from Reference [2]).

Implementation Details. Considering both time constraints and GPU memory demands (particularly for the larger models), we utilized six 40GB NVIDIA A100 GPUs to execute the new experiments. DomainNet and VisDA use the same hyperparameter settings as smaller-scale datasets, specifically $\lambda_s=0.2$, $\lambda_{exo}=1$. But the learning rate has been reduced, specifically $lr = 5e-4$.

Key Findings from CLIP Backbone Experiments:

1. Substantial HOS Improvements:
   • On DomainNet, COSDA-CLIP achieves 73.9% HOS, outperforming CROW by 14.9%.
   • On VisDA, COSDA-CLIP reaches 78.4% HOS, outperforming CROW by 9.2% .
2. Dual Strengths in OS* and UNK: With CLIP, COSDA particularly further enhances known-class classification and unknown-class detection (UNK).

Table 2. Performance of COSDA on DomainNet with ResNet50

We also note that with ResNet50, COSDA surpasses DCC [3] and UNIOT [1], confirming its robustness across backbones. The suggested references will be well cited in our updated version.

Q2. The step of building the centroids of all known and unknown classes shares the same idea as in [2] and [3]. Please discuss the main difference between method level and motivation level.

A2. We sincerely appreciate your insightful questions. In our work, the VMP module serves as a functional component to ensure CFA, rather than acting as a standalone clustering solution. I carefully read the papers you provided, and the difference lies in the fact that we exclude all known-class positive samples and cluster only the remaining negative samples (i.e., those not belonging to any known class), unlike [2][3], which cluster all target samples. Due to the reduction in the number of samples and clusters, VMP has saved on the computational cost. Besides, unlike one-vs-all strategies requiring C clustering operations per epoch (C = the number of known classes), COSDA achieves comparable performance with only one clustering per epoch. This design reduces complexity from O(CN) to O(N). Additionally, we acknowledge that advanced clustering could further optimize this step and will explore this in future work.

Q3. Could you clarify the motivation behind designing SRE, CFA, and VMP, their specific roles in causal inference, and how they interconnect within the overall methodology?

A3. We appreciate your questions regarding the methodological framework. SRE quantifies causal feature representation ability via susceptibility analysis. Direct SRE estimation in the target domain is infeasible due to label scarcity. To resolve this problem, we first decompose target-domain expected risk into open-set and closed-set, then bridge source/target risks using domain-invariant representations, and finally derive generalization bounds to narrow the gap between empirical and expected susceptibility risks (Theorems 1-2 & Corollary 1). To satisfy the exogeneity assumption for causal identifiability, we propose CFA, which encourages independence across causal features belonging to different categories via information bottleneck (Proposition 2) and then introduces VMP pseudo-label strategy (Eq. 17-19). Additionally, our code is open-sourced to show how SRE, CFA, and VMP are integrated end-to-end.

We sincerely appreciate your insightful comments once again. Please let us know if you need any further information or if there are additional points you would like to discuss with us.

Best regards,

Authors of #10234