UMFC: Unsupervised Multi-Domain Feature Calibration for Vision-Language Models

W1: The proposed methods rely on multi-domain data for calibration, which slightly weakens its application scenarios.

(1) In fact, our method is not limited to multi-domain scenarios; it is also applicable to single-domain scenarios. In single-domain case, we employ the same calibration process used in UMFC. This involves gathering statistical information from the clustered training set and utilizing this information to calibrate the features.

(2) Following the unsupervised calibration setting in Section 5.2, we conduct experiments on DomainNet, where both the unlabeled training data and test data originate from the same domain. As shown in the Table below, UMFC outperforms vanilla CLIP even in single-domain scenarios, and its performance is only marginally affected by the number of clusters (M).

W2: The comparison in TTA is not enough. As the paper use prototype for calibration, the proposed method be compared with other state-of-the-art prototype-based methods such as T3A [1].

Thanks for pointing out this. In the following, we compare T3A and our UMFC under test-time adaptation setting. To ensure a fair comparison, we integrated T3A into CLIP framework, using CLIP model instead of a pre-trained source domain model. Additionally, we set the hyperparameter N in T3A to 1 (the optimal value in our experiment), which determines the number of supports to restore. We set the batch size to 100 the same as UMFC. The results are shown in the below table.

As shown in the below table, T3A's performance improved only in the Infograph (I) domain but declined in other domains, particularly in the Quickdraw (Q) domain. We attribute this to domain bias inherent in CLIP, which is exacerbated by T3A. In contrast, our UMFC leverages the characteristics of CLIP across different domains for calibration, achieving better performance across all domains.

W3: The ablation of the number of clusters M is insufficient. Since we do not know the number of domains, what would the results be if we set M higher than the actual number of domains?

Thanks for pointing out this. We conduct ablation studies on cluster number M on both DomainNet and ImageNet-Variants (see the table in Global Response 2). The results indicate that our method is not sensitive to the value of M, as it still delivers significant improvements even when M does not match the actual number of domains (8/10 for DomainNet and 6 for ImageNet-Variants).

Q1: The author set the batch size of TTA to 100. Is the proposed method sensitive to the batch size?

In the table below, we present the results of UMFC with different batch sizes in test-time adaptation scenario. The results validate that our proposed method is not sensitive to the batch size.

W1: I think the improvements seem minor. In table 1, compared to the really easy approach CLIP-D, UMFC fails to improve it and UMFC + CLIP-E outperforms it by only 0.65% points. What about CLIP-D + CLIP-E? I think you should use this one to agast UMFC + CLIP-E.

There are some misunderstandings here, and I would like to clarify them in the following.

• Minor improvement compared to CLIP-D. In CLIP-D, we design customized prompts that incorporates corresponding domain label and name for each test sample. CLIP-D shows better performance than vallina CLIP, demonstrating the benefit of integrating domain information in multi-domain scenarios. Notably, our comparison between UMFC and CLIP-D is not meant to imply that UMFC outperforms CLIP-D, as CLIP-D utilizes domain labels and names for all test samples, whereas UMFC operates without such explicit supervision. Rather, the purpose of this comparison is to demonstrate that UMFC can extract domain information from data without supervision, highlighting it greater versatility in various situations.

• Disadvantages of CLIP-D. The CLIP-D model, while simple and straightforward, has several disadvantages: 1. it requires domain labels and names for all test samples, which is impractical in many real-world scenarios. 2. it is sensitive to the choice of domain names; Our findings show that CLIP-D's performance degrades when synonyms are used instead of the original domain names. Detailed results can be seen the below table.

  	C	I	P	Q	R	S	Avg
  CLIP-D	73.90	55.84	67.75	17.84	83.26	67.56	61.03
  CLIP-D (Synonyms)	72.11	54.83	66.09	17.73	83.34	65.74	59.97

• More comparison baseline. We build CLIP-DE by replacing the single domain-specific prompt in CLIP-D with an ensemble of prompt template incorporating domain names. The results are shown in below table. As shown, CLIP-DE performs only marginally better than CLIP-E, and slightly worse than CLIP-D on DomainNet. This suggests that CLIP-D may require a more refined prompt design and ensemble strategy to improve performance in multi-domain scenarios.

  	C	I	P	Q	R	S	Avg
  CLIP-E	73.16	54.17	67.02	15.86	84.30	67.49	60.33
  CLIP-D	73.90	55.84	67.75	17.84	83.26	67.56	61.03
  CLIP-DE	73.62	54.34	67.64	16.34	84.49	67.45	60.65

W2: In table 2, why did CoOp perform worse than the original CLIP? I believe it's the receipt problem, and I think you should at least perform some hyper-parameter choice to get a reasonable baseline, since finetuning on those in-domain data can easily get performance improvements.

There may be a misunderstanding, and I would like to clarify. In Table 2, we train CoOp using single-domain few-shot labeled data. For example, CoOp(Q) represents CoOp model trained using data from Quickdraw domain. As shown in Table 2, CoOp(Q) performs better than vallina CLIP and our UMFC in the Quickdraw domain, but significantly worse in the other 5 domains. This is because single-domain supervised fine-tuning tends to overfit to the specific training domain and cannot generalize well to other new domains.

W3: What's the extra computation cost involved in this approach and how it compares to other methods? Since the proposed approach uses test-time adaptation, I believe the authors should tell us the computation cost.

Here we report the computational cost of UMFC and other comparison methods under different scenarios. For analyzing computational cost, we report the training time, inference time and memory.

• Unsupervised Calibration: In this scenario, the entire unlabeled training set is provided for training. Corresponding computation cost comparisons are shown in below Table. Firstly, UMFC incurs minimal training and inference overhead compared to CLIP. This is because UMFC only requires a single forward pass to extract features and then calculate statistics for feature calibration. Secondly, when compared to few-shot fine-tuning methods like CoOp, UMFC also demonstrates lower consumption of computational resources and time.

  Method	Training Time	Inference Time	Epoch	Memory
  CLIP	-	86 seconds	-	1797MiB
  MUST	10 hours (2 GPUs)	92 seconds	30	25944MiB
  UMFC	2.3 seconds + 55 seconds	86 seconds	-	1887MiB
  CoOp (6*1 shot)	32 minutes	83 seconds	50	7007MiB
  CoOp (6*4 shots)	160 minutes	83 seconds	100	7007MiB

• Test-time Adaptation: In this scenario, no training data is provided and the test data arrives in batches. Corresponding computation cost comparisons are shown in below Table. As seen, UMFC requires less memory than TPT and shows greater computational efficiency. Specifically, UMFC takes only 296 seconds, whereas TPT requires nearly 197 minutes. This is because TPT requires fine-tuning the text prompt for each test sample and augmenting each test sample 64 times to ensure the reliability of the fine-tuning results, which significantly slows down TPT's inference speed.

  TTA	Inference Time	Memory
  UMFC	296 seconds	1790MiB
  TPT	197 minutes	6872MiB

W1: CLIP's ability to encode domain information is limited to specific domains.

This is a valuable question. We would like to clarify that the feature bias of CLIP is not limited to some specific domains, but is a general issue.

• Explanation of Figure 1(b): We use t-SNE for visualization, which maps high-dimensional data into lower-dimensional space while preserving relative distances. Large distances between I/P/Q and other domains cause the relative distance among C, R, and S domains to shrink. To mitigate this effect, we visualize only C, R and S domains, As shown in Figure 1(a) of global response PDF, image features from R, C, and S domains are clustered rather than dispersed.
• Quantitative results of model bias: To further illustrate model bias, we calculate the Euclidean distances of image features across 6 domains in DomainNet. See the detailed results and analysis in global response R2.
• Model bias in other datasets: We utilize the image features from ImageNet-Variants, then visualize clustered features with t-SNE (Figure 1(b) of global response PDF) and report the Euclidean distances. These results validate the model bias in CLIP is a general issue.

W2: For figure 1(c), it is suggested to show the percentage of predictions rather than the absolute numbers. It would be interesting to evaluate such bias in other domains.

(1) We have redrawn the figure, shown in Figure 2 of global response PDF. (2) The Table 1 of global response PDF lists the top-5 predicted classes for each domain, shown as "class name / percentage". Note that a uniform prediction probability in DomainNet is 1/345 = 0.29%. The table reveals that CLIP favors different classes across domains, which is consistent with conclusion of our paper.

W3: Does it mean that the proposed method need prior knowledge on the number of domains in the target dataset?

Our method does not need prior knowledge on the number of domains in the target dataset. See the ablation studies in global response R3.

W4: Whether the observations persist when the pre-training dataset distribution changes.

To answer this question, we use pre-trained models from OpenCLIP[1], which explores scaling laws with LAION dataset. We visualize the image features on DomainNet with different pre-training data distribution. See the details and analysis in global response R2.

Moreother, in Table 2 of global response PDF, we futher verify the effectiveness of our method on OpenCLIP series models. The results demonstrate the universal applicability of our method to various VL models with different architectures and pre-training data.

[1] Reproducible scaling laws for contrastive language-image learning. CVPR23

W5: Compared with more recent methods, such as MaPLe[B] and PromptSRC[C].

We compare UMFC with MaPLe and PromptSRC under different scenarios. We will add these results in final version.

• Domain-balanced Labeled data: We first consider the domain-balanced data for FSL methods. Specifically, we train MaPLe and PromptSRC with k (k=1 or 4) labeled data per class for each domain on DomainNet. The Table 3(a) of global response PDF shows that UMFC achieves comparable performance to MaPLe with limited labeled data, but underperforms them when more labeled data is available.

• Domain-imbalanced Labeled data: We next consider domain-imbalanced labeled data, as balanced annotations in multi-domain scenarios are expensive. Specifically, we train FSL methods with 1 labeled data per class from a single domain and test them on all domains. The Table 3(b) of global response PDF shows that the choice of training domain significantly impacts the performance of two FSL methods, suggesting that supervised FSL methods may overfit to the training domain.

In summary, FSL methods require domain-balanced labeled data for fine-tuning. When such high-quality labeled data is expensive to acquire in multi-domain scenarios, UMFC offers a cost-effective alternative, as collecting unlabeled data is more economical.

W6.1: Collecting unlabelled data for clustering is even more expensive than labeled data.

We need to highlight that collecting unlabeled data is more cost-efficient than labeled data, especially in multi-domain scenarios.

• No Need for Expert Annotation: Unlabeled data collection avoids the need for expert annotation across domains and categories, unlike labeled data, which requires expert involvement.
• Challenges with Labeled Data Quality: Although FSL methods like CoOp and CLIP-Adapter require only few labeled data, they demand high-quality data. Issues such as category imbalance, domain imbalance (shown in Table 2), and data noise can easily cause models overfitting to fine-tuned data, making labeled data collection costly in multi-domain scenarios.

W6.2: In Table 2, it is unfair to show the superiority of UMFC by comparing it with CoOp trained on one single domain.

Table1 compares UMFC with FSL methods with multi-domain training data, and Table2 use single-domain data. These experiments show UMFC as a viable alternative when collecting extensive multi-domain (balanced) labeled data for FSL methods is impractical. UMFC also generalizes well to unseen domains rather than overfitting to training domains. See our response to Weakness 2 for Reviewer Xgmy for detailed results.

W6.3: It can be beneficial to compare the clustering time of UMFC with CoOp training time.

We report the computational cost of UMFC and CoOp on DomainNet. UMFC's clustering time is negligible (2.3 seconds). The entire training process is significantly faster than CoOp. UMFC does not require gradient back-propagation, leading to lower memory usage. See more details in our response to Weakness 3 for Reviewer jj75.

W7: In Line 228, it is union or intersection of class sets in ImageNet-A and ImageNet-R?

The class space of ImageNet-Variants is union of class sets in ImageNet-A and ImageNet-R.

Dear Reviewer cRCJ,

As the rebuttal period is ending soon, please let us know whether your concerns have been addressed or not, and if there are any further questions.

Thanks, Authors.

def kmeans(data, n_clusters=3):
    from sklearn.cluster import KMeans
    model = KMeans(n_clusters=n_clusters, max_iter=500)  
    y_pred = model.fit_predict(data)  
    return y_pred

W1: While effective, the calibration process might introduce additional complexity in terms of understanding and implementing the recalibration mechanisms, particularly the calculation and subtraction of domain-specific biases.

In fact, our calibration method is straightforward and computationally efficient.

• Easy to implement: The calibration process involves three main steps:

  1. Assigning cluster label: (stage1) Unlabeled samples are clustered, and cluster labels are assigned.
  2. Collecting statistical information: (stage2) Statistics for each cluster are gathered based on the assigned labels.
  3. Feature calibration: (stage3) The original image and text features are calibrated using the collected statistics.

• Computation efficient: We utilize the k-means algorithm to cluster image features and calculate the mean feature of each cluster. Therefore, steps 1 and 2 are efficient and can be completed within one minute in our experiments. In step 3, the feature calibration mechanism does not require backpropagation or modification of model parameters, leading to minimal additional computational overhead. For further details, please refer to our response to Weakness 3 for Reviewer jj75.

W2: There's a risk that overly aggressive calibration could lead to overfitting on the specific domains included in the training set, potentially reducing the model’s ability to generalize to entirely new or unseen domains.

Thanks for this valuable suggestion. To evaluate UMFC's generalization capability on unseen domains, we follow the unsupervised calibration setting in Section 5.2. We randomly select disjoint domains as unlabeled training or test data. Specifically, we randomly select 3 domains in DomainNet as unlabeled training data (source domains) and use the remaining 3 domains as test data (target domains). The results are shown in below table.

From this table, we can observe that, UMFC consistently outperforms vanilla CLIP model across different source domains, demonstrating its robust generalization ability. Additionally, our method can be easily deployed in Test-Time Adaptation (TTA) scenarios. This allows for continuous updates of statistical information as new test samples from emerging domains arrive. This capability ensures that the model maintains its generalization effectiveness over new domains.

My major concern is whether the proposed methodology is a real improvement over CLIP-D (e.g., 61.68 vs. 61.03 in unsupervised calibration), considering the requirement for extra computational overhead and unlabeled dataset.

Thanks for this question, and I would like to clarify this point in the following.

• Improvement over CLIP-D. In CLIP-D, we design customized prompts that incorporates the corresponding domain label and name for each test sample. Compared to vallina CLIP, CLIP-D shows better performance, which demonstrates the benefit of considering domain information under multi-domain scenarios. However, our comparison between UMFC and CLIP-D is not intended to show that UMFC performs better than CLIP-D, as CLIP-D utilizes domain labels and names for all test samples, while UMFC does not. Instead, this comparison aims to illustrate that UMFC can extract domain information from data without supervision, making it more versatile in various situations.

• Disadvantages of CLIP-D. Although CLIP-D a is simple and effective method in multi-domain scenarios, it has many disadvantages: 1. it requires domain labels and names for all test samples. 2. it is sensitive to the choice of domain names. See more details in our global response R1.

W0: How does UMFC compare to CLIP-DE (Domain-Specific Ensemble Prompting)?

We build CLIP-DE by replacing the single domain-specific prompt in CLIP-D with an ensemble of prompt template with domain names. As shown in the table below, CLIP-DE performs only marginally better than CLIP-E, and slightly worse than CLIP-D on DomainNet. We analyze this is because the ensemble strategy weakens the domain information utilized in CLIP-D. Furthermore, our UMFC outperforms CLIP-DE, validating its superiority.

W1: Gaussian assumption is a bit too strong for each domain.

Thanks for your suggestions. In domain generalization, it is commonly assumed that domain features follow a multivariate Gaussian distribution [1]. Moreover, since our method relies only on the mean values for feature calibration (Equation 4 of main paper), it can generalize to other distributions where the mean effectively captures the overall information.

[1] Domain Generalization With Adversarial Feature Learning. CVPR 2018

W2: I would assume the dataset to be unlabelled, as described in L179-L180. However, for text feature calibration, the description in L208 is 'By clustering image features, we can compute the average feature µ of unlabelled images in each domain . representing the domain-specific features of that domain' . Does the domain here refer to the clustering in the image feature calibration step?

Yes, we apologize for this confusion. Here, 'domain' refers to the cluster labels assigned during the clustering stage. We will revise it in the final version.

W3: Equation (3) should be $\triangleq$.

Thank you for pointing this! We will revise it in final version.

W4: More details are needed for each experimental section.

Thanks for your suggestions. Our work can be deployed across various scenarios, including Unsupervised Calibration (UC), Test-Time Adaptation (TTA), and Transductive Learning (TL). We will add the following details for the experimental section in final version.

• In the UC scenario, we provide an unsupervised training set and use K-Means to assign cluster labels to the training samples, while also saving the corresponding cluster prototypes. Then, we create an unlabeled training set from a mixed domain by sampling 16 instances from each class across 6 domains in DomainNet. UMFC derives image bias and text bias for different clusters based on the cluster labels. During the testing phase, we first predict the cluster labels of the test samples using the cluster prototypes obtained during training, and then calibrate the predictions using the bias information derived from UMFC.

• In the TTA scenario, no training data is provided. Test data from mixed domains arrive in batches, with a batch size set to 100. For the first batch, we perform initial clustering using K-Means. For subsequent batches, we assign cluster labels to the samples based on cluster prototypes and continuously update these prototypes. Once the cluster labels are obtained, UMFC calculates the bias information for the current batch, updates the bias information for each cluster based on the new labels, and then calibrates the data for the current batch.

• In the TL scenario, we provide the entire test set. Similar to UMFC, we gather statistical information and calibrate the predictions for the test data. TL can be viewed as an extreme case of TTA, where the entire test set is treated as a single batch.

prompts=[
  "a {domain} image of a {class}. itap of a {class}.",
  "a {domain} image of a {class}. a bad photo of the {class}.",
  "a {domain} image of a {class}. a origami {class}.",
  "a {domain} image of a {class}. a photo of the large {class}.",
  "a {domain} image of a {class}. a {class} in a video game.",
  "a {domain} image of a {class}. art of the {class}.",
  "a {domain} image of a {class}. a photo of the small {class}."
]