CLIPCEIL: Domain Generalization through CLIP via Channel rEfinement and Image-text aLignment

We appreciate the reviewer's detailed comments and offer our responses below.

Weakness

[W1]: Present the comparisons on ImageNet

Thank you for your insightful suggestion. We conducted experiments on ImageNet (w/ 1000 classes) using the same setting as CoOp. Since the setting is the single source domain. We modify our loss and remove the domain variance term in the $\mathcal{L}_{\rm{ref}}$. Similar to CoOp, we train our model on few-shot ImageNet (16 samples per class) and test the model on different variants of ImageNet datasets. Table 5 in the rebuttal PDF indicates that our CLIPCEIL outperforms both the "CLIP Zero-Shot" and "CoOp".

[W2]: It appears that the improvements diminish once the model is fine-tuned

We appreciate the reviewer's insightful question and are eager to engage in further discussion. One possible explanation for the observed phenomenon is that when the backbone is frozen, we can only adjust the adapters and loss functions, meaning that all the improvements come from these adjustments. In contrast, fine-tuning the entire backbone allows us to leverage a large number of model parameters to enhance performance, which might limit the potential benefits of the adapters and loss functions.

[W3]: It would have been helpful to see the model's performance on other ViT backbones.

Thank you for your insightful suggestion. We conducted experiments using different ViT backbones, i.e., ViT-B/32 and ViT-L/14, on the OfficeHome dataset. The performance results are presented in Table 6 of the rebuttal PDF. It shows that CLIPCEIL consistently outperforms the zero-shot prediction on these two ViT backbones, demonstrating its generalization ability on different architectures.

Question

[Q1]: Did the authors consider ablating this component with other contrastive loss objectives or alignment methods?

Thank you for your insightful suggestion. We try another contrastive loss i.e., SLIP, which combines CLIP loss and SimCLR. We conducted experiments by replacing $\mathcal{L}_{\rm{CE}}$ with $\mathcal{L}_{\rm{SLIP}}$. As shown in Table 7 of the rebuttal PDF, combining our proposed loss with $\mathcal{L}_{\rm{SLIP}}$ achieves the best performance on the OfficeHome dataset, indicating the effectiveness of our proposed loss.

[SLIP] SLIP: Self-supervision meets Language-Image Pre-training, ECCV 2022.

[Q2]: What do the authors mean by "multi-scale information?

Thank you for your clarifying question. "Multi-scale information" refers to the latent representations obtained from different transformer blocks at various levels within the CLIP visual encoder. While the term "multi-scale information" is typically associated with CNNs and might not be entirely accurate for ViTs, we use it to convey the concept that features are derived from multiple levels.

Limitations

[L1]: Could the authors provide a comparison of their method against previous state-of-the-art methods, such as CoOP, in terms of computational efficiency?

Thank you for your insightful question. We measured the GPU memory usage and average training time per step for various methods on the OfficeHome dataset using a single A100 GPU. The table below indicates that CoOp is the most memory-efficient and fastest model, while CLIPCEIL offers comparable computational efficiency to CoCoOp and ERM.

We appreciate the reviewer's detailed comments and offer our responses below.

Weakness

[w1]: The adapter is designed for ViT, and is hard to use for ResNet backbone.

Thank you for your insightful comments. Our proposed method can also be extended to the ResNet backbone. The primary difference lies in the way we extract latent representations, while other components, such as the architecture of the adapter $g$, remain unchanged. For ResNet backbone, we use the latent feature map as the latent feature and apply Attention Pooling to convert the 2D feature map into a 1D vector. The vectors of different layers are then fed into the Transformer layer of adapter $g$. We conducted experiments using the ResNet-50 backbone, and our method outperformed other ResNet based models, as shown in Table 2 in the rebuttal PDF.

[w2]: How the Transformer layer is necessary.

Thank you for your insightful comments. We conducted an ablation study to investigate the Transformer layer. We replaced the Transformer layer with Average Pooling or an MLP. As shown in the orange block in Table 4 of the rebuttal PDF , the Transformer layer outperformed the other fusion strategies, indicating its necessity.

[w3]: Training from scratch or fine-tuning a pre-trained model.

Thank you for your clarifying question. Our proposed method builds upon a pre-trained CLIP model, which we then fine-tune on the source domains.

[w4]: CLIPCEIL currently only applies to multi-source domain generalization.

Thank you for your insightful comments. We also put this in the limitation section of the main text. However, CLIPCEIL model can be adapted to single source domain generalization by simply removing the domain variance term in the loss function. We conducted experiments on ImageNet (w/ 1000 classes) using the same few-shot single source domain generalization setting as CoOp. Table 5 in the rebuttal PDF indicates that CLIPCEIL outperforms "CoOp".

Question

[Q1]: How many data points were used in the experiment in Table 1?

Thank you for your clarifying question. Table 1 in the main text presents the results of a simple proof-of-concept experiment based on our observations. This experiment is training-free, with no data points used for training, and is tested on the OfficeHome test set. We first calculate the domain variance in text embedding channels across 80 prompt templates and the class variance across 65 classes. We then select the channels with larger class variance and smaller domain variance. Assuming effective alignment of visual-language features in CLIP, we use the same selected channels for visual embeddings. During inference, we use the inner product of the visual and text feature vectors, similar to the approach used in CLIP zero-shot.

[Q2]: How the model trained on OfficeHome performs on other datasets?

Thank you for your insightful perspective. The domain generalization task aims to enhance the model generalizability, and as such, the domains in the benchmarks designed for domain generalization are already very diverse. Moreover, domain generalization assumes that the source and target domains share the same label space (i.e., the categories are consistent across different domains), which is not the case with different benchmark datasets. Therefore, a model trained on one benchmark dataset (e.g., OfficeHome) typically is very difficult to test on another one (say PACS).

Nevertheless, we are intrigued by the reviewer's suggestion about evaluating the performance of a model trained on one benchmark dataset on another dataset. To investigate this, we identified a common category, "person", within the PACS and VLCS datasets. We then conducted an experiment by testing a model trained on the PACS dataset in the "Labelme" domain of the VLCS dataset. The results below shows the performance drop but we would be happy to discuss this interesting result with the reviewer.

[Q3]: Why do Table 4 and Table 5 demonstrate different performances?

Thank you for your clarifying question. Since the content in Table 5 is irrelevant, we assume that the reviewer referred to Table 3 and 4. Table 3 in the main text presents the ablation studies for different loss terms, while Table 4 focuses on the ablation studies for various adapter architectures. This distinction may be somewhat confusing, and we will address this in the revised paper. The term "Multi-scale" in Table 3 indicates training CLIPCEIL using only $\mathcal{L}_{\rm{CE}}$. In contrast, "Multi-scale" without "bypass" in Table 4 signifies training CLIPCEIL with all loss terms except for the bypass connection. Thus, they have different performances.

[Q4]: Can you experiment with as a relatively simple architecture?

Thank you for your insightful suggestion. We conduct the experiment with a simple adapter $g$, without considering the multi-scale information. $*i.e.,*$ one-layer MLP projector, and evaluate the impact of our proposed loss. The pink block in Table 4 of the rebuttal PDF indicates that incorporating $\mathcal{L}_{\rm{ref}}$ and $\mathcal{L}_{\rm{dir}}$ with a simple adapter $g$ still improves the performance.

Q4: These experiments do not conclusively demonstrate that their generalization performance is superior to CLIP's.

Thank you for your insightful comments. The CLIP model is indeed powerful and has found widespread application across various fields. Its alignment of visual and textual information enables impressive zero-shot capabilities for unseen categories. However, in this paper, our goal is to enhance the generalizability of vision-language models (e.g.CLIP) within the context of standard domain generalization settings, specifically on established domain generalization benchmark tests.

We greatly appreciate the time you've taken to review our rebuttal and for recognizing our contributions and the effectiveness of our proposed method on multi-domain datasets. We are pleased to address the reviewer's follow-up questions and discuss these intriguing points.

Q1: The experiment in Table 5 to be somewhat unfair because Coop was not designed for domain generalization. Subsequent works, like CoCoop, were developed specifically to address this issue.

Thank you for your insightful comment. We have also compared our CLIPCEIL model with CoCoOp on the ImageNet under exactly the same few-shot single domain generalization setting, using the performance results for CoOp and CoCoOp as reported in the original CoCoOp paper. As shown in the table below, our proposed method outperforms both CoOp and CoCoOp, achieving the highest average target domain accuracy among the three models.

It's important to clarify that these experiments on ImageNet are not part of the standard domain generalization benchmark, which typically involves a multi-source domain generalization setting. This is why we did not include them in the main text. Instead, the ImageNet experiments referenced in Table 5 in the rebuttal PDF focus on a few-shot single-source domain setting, which is not the design focus of our proposed model either.

Q2: It's unclear whether a given dataset is truly out-of-domain.

Thank you for this interesting point. As we mentioned in our response to Reviewer cn9a, the datasets used to train the CLIP model are indeed quite diverse, covering a wide range of image types such as digits images, human faces, traffic signs, remote sensing, self-driving, pathology, human actions, natural photographs/pictures, etc., but these images are typically photographs of objects and scenes taken from the real world. Some domains that are commonly featured in standard domain generalization benchmarks are not adequately represented in CLIP's pre-trained datasets. For example, domains like quickdraw, infograph, clipart, sketch in DomainNet, clipart, art in OfficeHome, Cartoon, Sketch in PACS are examples of underrepresented categories. Also, the unique image styles from the Terra Incognita dataset, which features camera trap images for monitoring animal populations, appear to be underrepresented. As a result, we believe that the data in the domain generalization benchmark datasets are not fully represented in CLIP’s training data, which may explain why the domain generalization performance, even with the CLIP model, remains relatively low (around 50%-60%) on the DomainNet and Terra Incognita datasets.

Q3: Additionally, the original CLIP paper emphasizes that its superior performance compared to ImageNet models might be due to CLIP’s ability to avoid easily learning dataset-specific biases. For these reasons, I’m not particularly in favor of imposing the domain concept on CLIP.

Thank you for this interesting point and your insightful comments. We agree that CLIP does have better generalizability in many widely used datasets than ImageNet trained models, largely due to its extensive and diverse training datasets. However, as mentioned earlier, after carefully checking the datasets used to train the CLIP model, we found these images are typically photographs/video frames of objects and scenes taken from the real world. although CLIP was evaluated on 27 datasets as shown in Table 10 of the original paper, a close examination of these datasets reveals that they also consist of photographs/video frames of objects and scenes taken from the real world. From this perspective, the CLIP model also exhibits bias to images taken from the real world, and has not yet demonstrated sufficient generalizability to other domains featured in domain generalization benchmarking datasets, such as quickdraw, clipart, cartoons, and sketches, as indicated by the "CLIP zero-shot" results in Table 2 of the main text. Our model aims to narrow this gap.

We appreciate the reviewer's detailed comments and offer our responses below.

Weakness

[Proposed Method]

[W1]: The idea and motivation are the same as DomainDrop [14]. Some of the ideas in CLIPCEIL are similar to the following works, [R8-R11].

Thanks for your insightful comments. The technique used in the DomainDrop [14] paper differs from CLIPCEIL. DomainDrop explicitly drops the domain-specific feature channels identified by a domain discriminator. However, each feature channel may include both domain-specific and domain-invariant information. Thus, directly dropping entire feature channels can lead to the loss of class-relevant information for downstream tasks and may not be optimal. In contrast, CLIPCEIL implicitly drops the domain-sensitive information by minimizing the inter-domain variance and maintaining class-relevant information as much as possible by maximizing the inter-class variance. Moreover, CLIPCEIL also proposed the Image-text alignment that specifically designs to the vision-language model, compared to DomainDrop only utilizes visual features.

The histograms in Table 1 are the visualization tool adapted from DomainDrop [14]. Our observations align with those in the DomainDrop paper; features extracted from pretrained models without specific adaptation for the domain generalization task exhibit large variances across domains. Moreover, we also plotted the channel histogram across classes, and noticed that certain channels are insensitive to class variations, motivating our loss function to maximize inter-class variance.

We will include [R8-R11] in our revision, but want to emphasize the difference between CLIPCEIL and these models. [R8] is a prompt learning-based method for domain adaptation by learning domain-agnostic tokens from multi-scale visual styles, whereas CLIPCEIL is an adapter-based method. [R9] learns a mask to explicitly filter out the domain-specific feature elements, which can be problematic as one element can contain both shared and specific information, similar to DomainDrop. [R10] uses multiple domain-specific classifier heads to remove domain-specific features. [R11] removes class-agnostic visual information by projecting all the visual embeddings onto the span of text embedding, focusing on source-free domain adaptation. In contrast, our method removes the domain-specific and class-agnostic information by minimizing the domain variance and maximizing the class variance, inspired by our channel histogram observation.

[W2] & [Q1]: Channel refinement strategies, esp. domain refinement.

Thank you for your insightful perspective. We would be happy to discuss the channel refinement strategies with the reviewer. We conducted a simple experiment to show how domain variance dynamically changes during training in Figure 1 in the rebuttal PDF. Diverse datasets (e.g. TI and DN) start with large variances, which are reduced to reasonable levels using our domain variance loss term. Figure 1 in the main text also demonstrates this. Of course, there are other options, such as adversarial learning, information theory based, etc, which are the main focus in the domain generalization area., we welcome the opportunity to discuss these further with the reviewer.

[Experiments]

[W1]: Misunderstanding of how MIRO [5] works.

Thanks for pointing it out. We will move it to the blue section.

[W2]: Fair comparison with SAGM and DomainDrop.

Thanks for the insightful suggestion. Our original purpose in listing SAGM and DomainDrop was to demonstrate that the CLIP ViT backbone can outperform the best ResNet backbone models and that CLIPCEIL is built upon this superior backbone.

To fairly compare with the ResNet backbone models, we conducted experiments on CLIPCEIL using the CLIP ResNet-50 backbone. Table 2 in the rebuttal PDF demonstrates that CLIPCEIL outperforms SAGM and DomainDrop with the ResNet-50 backbone on OfficeHome datasets.

[w3]: Comparison with additional prior works.

Thank you for your suggestions. References [R1, R4, R5] pertain to few-shot learning, while [R8, R11] address domain adaptation problems. We did not include these as comparing models focused on other tasks within the domain generalization setting, which is the primary goal of our paper, is challenging.

We have included references [R3,R6] (which are already included in main text), and [R7] with the ViT-B/16 backbone (see Table 1 in the rebuttal PDF), as well as references [R2,R9] with the ResNet-50 backbone (see Table 2 in the rebuttal PDF). Our CLIPCEIL outperforms the prior works with different backbones.

[w4]: Weights analysis for the loss terms.

Thank you for your suggestions. To avoid hyper-parameters ($\alpha$ for $L_{ref}$, $\beta$ for $L_{dir}$) searching for different datasets, we set $\alpha=1$ and $\beta=1$ as default. Following the Reviewer's suggestion, we investigated hyper-parameter sensitivity by tuning one parameter at a time while keeping the other at 1. Figure 2 in the rebuttal PDF demonstrates that $\alpha=1$ and $\beta=1$ yields the best accuracy and that CLIPCEIL achieves stable performance with $\alpha\in[0.6,1.2]$ and $\beta\in[0.6,1.2]$.

Questions

[Q1]: See [ Proposed method] [W2]

[Q2]: Can multi-scale adapter be extended to the text encoder?

Thank you for your insightful suggestion. We conducted experiments to incorporate a multi-scale adapter into text encoder. As shown in Table 3 of the rebuttal PDF, using both visual and text adapters did not perform as well as only using the visual adapter. This may be due to the increased complexity of optimizing both adapters simultaneously. It also suggests that focusing on image feature adaptation is more crucial for domain generalization tasks, since the semantic gap between visual features in pretrained and custom datasets is larger than that of text features.

Dear Reviewer,

I would like to follow up on my previous response to your review. As the discussion period is nearing its end, I am eager to address any remaining concerns you might have.

Your feedback is invaluable, and I would greatly appreciate the opportunity to discuss any aspects of the review further. Please let me know if there are any specific points you'd like to revisit or if you have any additional questions.

We appreciate the reviewer's careful comments and provide our responses below.

Weakness

[W1]: Overall novelty.

Thanks so much for your comments. First, we would like to emphasize the difference between few-shot learning and domain generalization. Although they are related, they have distinct differences. In domain generalization, the key assumption is that there are no target domain examples available during training. This means the model must learn to generalize to entirely unseen domains based solely on the information from the source domains. In contrast, few-shot learning typically focuses on task adaptation, where there are, although only a small number, target domain examples available (actually not only available but also labeled ) during training. This allows the model to fine-tune its knowledge and adapt to the new task with minimal data. This paper specifically targets domain generalization task.

Second, we want to mention that there are two main directions in fine-tuning pre-trained large vision-language models (e.g., CLIP): (text and/or visual) prompt learning and adapter techniques, and thus the adapter itself is not our contribution. Our model, along with CLIP-Adapter, Tip-Adapter, etc. all fall under the adapter technique. We want to emphasize that our contributions lie in proposing novel loss functions tailored specifically for the DG task, compared to other adapter based models that target few-shot learning or other tasks.

[W2]: Some famous adapters for CLIP are not cited, like Vt-CLIP, or cited but not compared, like Tip Adapter.

Thanks so much for your suggestions. We will add Vt-CLIP to our related work section. Since Vt-CLIP and TiP-Adapter are both for the few-shot learning setting, comparing them in the domain generalization setting is difficult.

[W3]: Performance gain compared with other methods is not significant.

Thanks so much for your comments. Our CLIPCEIL achieves the best average performance on five widely used domain generalization datasets with both the frozen encoder and fine-tuning the entire visual encoder settings. Considering the frozen encoder is a more realistic setting in practice, CLIPCEIL exceeds second-best by $2.3$% on average.

Question

[Q1]: Why do authors list the venue in the table?

Thanks for your question. Our original purpose is to make it easier for readers to identify when the comparison methods were published (highlighting the latest progress in this area) and where they were published (demonstrating that they are state-of-the-art models by referencing the top conferences or journals).