1. Feature Alignment in Large Neural Nets (e.g., CLIP)

To address your concern regarding whether large neural networks can conform to the compositional feature structure (as defined in Definition 1) using our CFA algorithm, we conducted feature visualization for a CLIP model in both its pretrained and CFA-trained versions. The comparative visualization is presented in Figure 7 in Appendix D of our revised manuscript, which you can review at this link. From this figure, it is evident that the features fine-tuned with CFA align well with a compositional feature structure, unlike the original CLIP features. We believe this visualization further justifies the ability of our proposed CFA objective to align features of large neural networks with the desired structure.

2. Feature Disengtanglement on Real-World Dataset

We believe the feature visualization for the DomainNet dataset, as mentioned above, also addresses the reviewer's concern: the CFA-finetuned CLIP model exhibits a feature structure with disentanglement similar to what is seen in the Color MNIST visualization in the paper.

3. Training Complexity

To address your concerns regarding the training stability of CFA's two stages, we conducted experiments with different training lengths for each stage. The results are provided in item 3 of our General Response above (link). The table shows that Stage-1 training is quite stable, as it involves only optimization over linear models. Additionally, the training cost of Stage-1 (linear probing) is almost negligible compared to Stage-2 (backbone finetuning). Stage-1 requires only a single forward pass over training samples for feature gathering (features are stored in CPU memory), and then the linear probing can be optimized fastly on CPU. The training cost for Stage-2 is slightly less than standard full finetuning, as the last layer remains frozen during Stage-2 of CFA. Overall, we conclude that the implementation complexity and additional training costs of CFA are mild, making it a practical method for addressing compositional generalization problems in real-world applications.

4. Performance Gain

Regarding the performance gain of CFA over reweighting, a strong baseline method, we acknowledge that the improvement is not significant. However, our main contribution lies in designing a principled algorithm for compositional generalization, a novel challenge in OOD generalization. As our theoretical analysis and feature visualization in Figures 4 and 7 demonstrate (on Color MNIST and DomainNet, respectively), our method aligns features with a compositional structure suitable for compositional generalization. In OOD generalization research, it's typical for specialized algorithms to only modestly outperform baselines like ERM, as evidenced by extensive benchmarking in studies like DomainBed [1] and WILDS [2]. Thus, we consider CFA's performance gain to be meaningful and convincing. Additionally, the mild implementation complexity and training costs (discussed in the last item) make CFA a viable method for enhancing compositional generalization.

Moreover, we want to emphasize that real-world datasets used in our paper have lots of labeling issues (for both class and domain labels), which may prevent CFA from obtaining greater performance gain. Notably, CFA employs both class and domain labels for training, leading to an increased susceptibility to label noise, particularly from domain labels. Below, we illustrate two types of labeling noise encountered:

• (i) Mislabeling: We observe that the DomainNet dataset contains many mislabeled images. Many images in the “Clipart” domain are actually real photos of a single object on a white background, which should be categorized into the “Product” domain. On the other hand, numerous class labels are also mislabeled: images in the “bush (plant)” class contain pictures of President George Bush of the USA; the “cooler” class includes electric fan images, which are incorrectly categorized; the “square (shape)” class contains table, which should be placed into the “table” class instead. Note: We only manually examined a very tiny subset of the DomainNet dataset, which comprises 0.6 million images, and have already found many mislabeled images. Therefore, overall, we believe the mislabeling ratio is not negligible.

• (ii) Ambiguous Labels: We observe that certain domains contain a large number of images that are visually similar to those in another domain. For example, in both Office-Home and DomainNet, the images in the “Product” domain are real photos of objects, making them almost indistinguishable from their counterparts in the “Real” domain. The distinguishing feature of the “Product” domain is that its images all have a white background; however, some images in the “Real” domain also share this characteristic. Additionally, in DomainNet, the “Infograph” domain contains many images stylistically similar to those in “Clipart” or “Real”; some images in the “Painting” domain are sketches, despite the presence of a separate “Sketch” domain. This ambiguity issue extends to class labels as well. In DomainNet, the classes “cup,” “coffee cup,” and “mug” lack clear stylistic distinctions.

In addition to these issues, other aspects of these datasets may affect learning. For instance, the iWildCam dataset includes a class labeled “empty,” signifying the absence of animals, which comprises a significant portion of the dataset.

5. Reweighting Not Work with WiSE-FT for DINOv2 Backbone

We did not include the results of Reweighting+WiSE-FT for DINOv2 in Table 1, as we had observed its ineffectiveness before the submission of this paper. We re-run the experiments for Reweighting for DINOv2 on DomainNet over 3 random seeds and provide the mean accuracy results in the table below.

It is evident that WiSE-FT significantly decreases the performance of Reweighting in terms of both ID and OOD accuracy. The primary reason is that, unlike CLIP that includes a text encoder capable of providing a zero-shot linear classifier on top of its image encoder, DINOv2 is solely an encoder without an initial classifier (the last linear layer). Therefore, when applying WiSE-FT to Fine-tuning and Reweighting methods on DINOv2, the initial parameters of the last linear layer are set to zeros and then interpolated with the final finetuned last linear layer. However, this interpolated image encoder may not align well with the interpolated last layer, leading to a decrease in both ID and OOD performance.

We will run Reweighting+WiSE-FT experiments for DINOv2 on more datasets, and provide results in the appendix of the next revision.

Dear Reviewer ZjmZ,

We appreciate the comments and suggestions in your latest feedback today, and we responded to it 5 hours ago [link]. However, we have just noticed that the "Readers" list for that response does not include reviewers, as the OpenReview system does not give us the option to add reviewers as "Readers" to our response. We are not sure if our previous response is visible to you. Therefore, we have copied our response below for your reference.

---

We are pleased to know that our response has addressed your concerns. We also appreciate your suggestion to integrate our discussions into the manuscript. Accordingly, we have updated the manuscript with the following new content:

1. Appendix B.1: Discussion on Training Stability and Overhead

2. Appendix B.2: Discussion of Performance Gain

3. Appendix B.3: Partial Availability of Domain Labels

As for the reweighting experiments with DINOv2, we thank you for pointing out that weight interpolation with the random initialized head might be more appropriate than our choice of zero head weights. Following your suggestion, we take weight interpolation (WiSE-FT) between the pretrained DINOv2 with the randomly initialized head and the finally trained model (from this random initialization). We repeat the evaluation over three random seeds (the head initialization is dependent on the random seed) and report the mean accuracy below. From the results shown in the table below, we do observe an improvement of WiSE-FT with this random initialized head over the zero head weights. However, the WiSE version’s ID performance still significantly underperform the original finetuned version; the OOD performance approaches that of the original version but does not surpass it. Hence, our conclusion that WiSE-FT does not work for reweighting with DINOv2 still applies.

We hope this new response and the updated manuscript can address your concerns. Please let us know if you have further questions.

Dear Reviewer ZjmZ,

Please check Paper 6129 authors's revised paper and the latest response to your questions to see if they have addressed all your concerns. If so, you might consider adjust your rating accordingly.

AC

1. Label Availability

CLIP is trained in a weakly supervised manner, and its classification performance can often be enhanced through supervised finetuning, such as on ImageNet. In our compositional generalization benchmark (CG-Bench), we observed that the zero-shot prediction accuracy of CLIP was unsatisfactory, indicating the need for further finetuning with class labels. Similarly, the DINOv2 model, a bare image encoder, has to be supervised finetuned to obtain classification capabilities. Overall, class labels are necessary for achieving effective results in CG-Bench. We recognize that the additional requirement of domain labels may not always be feasible in real-world applications. To address this, we propose two solutions to lessen the need for a fully domain-labeled dataset:

• Use partially available domain labels, e.g., only 10% training samples have domain labels

• Use zero-shot predicted domain labels by CLIP when no domain labels are provided.

The experimental results are detailed in the item-2 of our General Response above (please review at this link). The results show that our CFA method maintains most of its OOD performance advantages even with limited domain label availability, such as with only 10% of labels. In situations where no domain labels are available, employing CFA with zero-shot predicted domain labels by CLIP still shows improvement compared to the standard finetuning method. Overall, these results indicate that CFA's dependency on domain labels is quite mild.

2. ID-OOD Performance Trade-off

In the literature of OOD generalization and domain generalization, it is widely observed that methods aiming to improve OOD performance usually suffer from some mild in-distribution (ID) performance degradation [1,2]. Additionally, from the theoretical perspective of invariant feature learning [3,4], enhancing OOD performance often requires the omission of domain-specific features, which naturally leads to a decrease in ID accuracy.

In the empirical results of Table 1 of our manuscript, when comparing the two rows of CFA vs. Fine-tuning in a pairwise manner, one can observe that for the CLIP model, the ID (in-distribution) performance of Fine-tuning is always slightly better than that of CFA. However, for DINOv2, the ID performance of CFA consistently outperforms Fine-tuning across all four datasets. Overall, the difference in ID performance between CFA and other methods is quite small for both models across all datasets. We believe these findings justify that CFA maintains competitive ID performance while improving OOD performance in compositional generalization benchmarks.

References

[1] Gulrajani et al. In Search of Lost Domain Generalization. ICLR 2021

[2] Koh et al. Wilds: A benchmark of in-the-wild distribution shifts. ICML 2021

[3] Arjovsky et al. Invariant Risk Minimization. 2019

[4] Rosenfeld et al. The Risks of Invariant Risk Minimization. ICLR 2021

2. Explanation of Two Stages of CFA and Feature Visualization on Real-World Dataset

Stage-1 of CFA involves linear probing, which trains two orthogonal linear classifiers ($\mathbf{Y}$ and $\mathbf{E}$) for class and domain predictions on features $\mathbf{Z}$ of the pre-trained backbone. The backbone remains frozen in Stage-1, and the primary objective of this stage is to establish a compositional feature structure via the two trained orthogonal classifiers, $\mathbf{Z}$ and $\mathbf{E}$.

In Stage 2, CFA freezes the trained classifiers $\mathbf{Y}$ & $\mathbf{E}$, and only fine-tunes the backbone (image encoder) – this process modifies the features $\mathbf{Z}$. Drawing from research on neural collapse [3,4,5], it is understood that supervised learning with cross-entropy loss causes the features of samples from class-$i$ to collapse towards the $i$-th column vector of $\mathbf{Y}$, denoted as $y_i$ (for a visual illustration of neural collapse, refer to Fig. 1 in this ICLR 2022 paper). In our context of two orthogonal linear classifiers, our theoretical analysis indicates that for each training sample associated with class-$i$ and domain-$j$, its feature will collapse towards the composition $y_i + e_j$. Given the orthogonality constraint of classifiers $\mathbf{Y}$ and $\mathbf{E}$, the resulting collapsed features will adhere to the compositional feature structure defined in Definition 1.

As per your request, we visualized features using both the pretrained CLIP ViT model and the one fine-tuned with CFA on the DomainNet dataset. Due to limitations in 3D visualization, we selected 2 out of 6 domains and 3 out of 345 classes for representation. As figures cannot be inserted in OpenReview posts, the visualization is provided in Fig. 7 of Appendix D in the revised manuscript. Please refer to this PDF for review. This visualization clearly demonstrates that CFA can align features with the intended compositional feature structure. We are confident that this visualization effectively addresses your concern regarding the feature alignment of CFA with real data.

3. Ablation Study on Orthogonal Loss

We fixed other hyper-parameters as presented in the manuscript, only changing the orthogonal loss coefficient in Stage-1 of CFA. For each set of hyper-parameters, we repeated the experiment with three seeds. It is clear that with the coefficient set to 100, the out-of-distribution (OOD) performance peaks, leading to our choice of this value in our manuscript. From the table, we observe that a large orthogonal loss coefficient (e.g., >=100) is necessary to encourage orthogonality between heads. (The orthogonal loss coefficient of 100 was used in our paper.)

In addition to studying the orthogonal loss coefficient, we also conducted experiments on the training steps of the two stages of CFA. These results, detailed in item-3 ("Two-Stage Training Stability") of our General Response above (link), demonstrate that the final performance is relatively stable to the training steps numbers in each stage of CFA.

References

[1] Shu et al. Open Domain Generalization with Domain-Augmented Meta-Learning. CVPR 2021

[2] Zhu et al. CrossMatch: Cross-Classifier Consistency Regularization for Open-Set Single Domain Generalization. ICLR 2022

[3] Papyan et al. Prevalence of neural collapse during the terminal phase of deep learning training. PNAS. 2020

[4] Zhu et al. A Geometric Analysis of Neural Collapse with Unconstrained Features. NeurIPS 2021

[5] Kothapalli et al. Neural Collapse: A Review on Modelling Principles and Generalization. TMLR. 2023

1. Problem Setting

Compositional generalization (CG) explicitly assumes that all domains and classes are seen during training, with only certain combinations of them being unseen, which are used for OOD evaluation. As a comparison, in Open-Set Domain Generalization (OS-GD) [1,2], test samples are from a new domain, and unseen classes in the training phase can appear during the test phase. Notably, OS-DG typically requires models to assign the “unknown” class label to test samples from classes unseen in the training set, which essentially constitutes an auxiliary task of out-of-distribution (OOD) detection.

Note: while OS-DG does not explicitly state requirements on test domains, it generally poses certain implicit assumptions about the test domains. For example, the OS-DG problem setup as described in [1] implicitly places distributional assumptions on domains, implying that the test domain is related to the training domains in a specific manner (so that data augmentation techniques, such as feature-level MixUp applied to training domains, can partially encompass test domains).

Below, we use two tables to illustrate the differences between CG and OS-DG, where "✓" marks a domain-class combination seen during training.

• Compositional Generalization (CG)

  In the CG table, combinations marked with “✓” are present in the training set, while those labeled “Test” are not. The CG challenge is to determine if models trained on “✓” combinations can accurately generalize to “Test” combinations.

  	Class 1	Class 2	Class 3
  Domain 1	✓	Test	Test
  Domain 2	Test	✓	Test
  Domain 3	✓	Test	✓

• Open-Set Domain Generalization (OS-DG)

  In the OS-DG table, the symbols “✓” and “Test” have the same implications as in the CG table. Additionally, “X” denotes a domain-class combination that is unavailable during training and is also not evaluated. The class “Unknown” encompasses all classes not present during training, with models expected to predict the “Unknown” class for these.

  	Train Class 1	Train Class 2	Train Class 3	Class “Unknown”
  Train Domain 1	✓	X	X	X
  Train Domain 2	X	✓	X	X
  Train Domain 3	✓	X	✓	X
  Test Domain	Test	Test	Test	Test

In summary, while CG and OS-DG problems are similar, they aim for different outcomes in OOD generalization. CG seeks to generalize to unseen domain-class combinations composed of domains and classes seen during training. OS-DG can be viewed as domain generalization (DG) with an additional task of OOD detection.

I have understood the difference between compositional generalization (CG) and open-set domain generalization (OS-DG) in the problem setting from the two tables. However, from the first table, the problem setting of CG is somewhat similar to the compositional zero-shot learning (CZSL) [1]. It is recommended that the author can provide comparative explanations in related work.

After carefully reading the author's responses, I feel that my concerns have been resolved and I raised the score to 6.

[1] Hao et al. Learning Attention as Disentangler for Compositional Zero-shot Learning. CVPR2023.

From table above, we can observe that as the domain labeling ratio increases from 10% to 100%, the domain prediction accuracy modestly improves from 82.0% to 86.3%, indicating a relatively small enhancement. This suggests that the domain label is indeed quite easy to predict, which could explain why our CFA can work well with a partial availability of domain labels. Furthermore, the zero-shot prediction accuracy of the CLIP ViT-B/16 model for domain labels is 61.5%, significantly higher than a random guess (25%). This outcome explains why CFA with CLIP-predicted domain labels can also improve over vanilla fine-tuning.

Class Label Availability. As for class labels, we do require their full availability for the training set, since our work focuses on supervised finetuning of pretrained encoders. Meanwhile, although CLIP is not explicitly supervised by domain and class labels, the texts in the image-text pairs (used for CLIP pretraining) provide abundant class and environment information. Furthermore, CLIP's unsatisfactory zero-shot performance on our CG-bench indicates the need for supervised finetuning to improve its effectiveness. On the other hand, DINOv2 is a self-supervised pretraining method, thus it needs to be supervised finetuned before applying to downstream classification tasks. Overall, the above statement wants to justify that the class label availability cannot be waived for the compositional generalization task considered in our paper.

3. Two-Stage Training Stability

In this ablation study, we address the concerns of Reviewers H15m and ZjmZ and justify the training stability of our two-stage method. All of the following experiments are conducted for CLIP ViT-B/16 on the Office-Home dataset and are an average over 3 seeds.

We first show that our method is stable to the number of iterations in Stage-1. From the following table, we can see that as the linear heads are trained longer in Stage-1, the ID (WiSE) performance slightly increases and the OOD accuracy reaches the peak at 4000 to 6000 iterations. Except for the trade-off between ID and OOD performance, our method is stable with no drastic drop in accuracy. The 6000 iterations of Stage-1 was the hyperparameter used in our work.

We then demonstrate how training epochs in Stage-2 will affect the final performance of our model. The table below shows that for longer training epochs, the ID accuracy increases and the OOD accuracy decreases. Hence, in our paper, we used 3 epochs for the best OOD performance.