Connecting Domains and Contrasting Samples: A Ladder for Domain Generalization

We appreciate your helpful feedback. We hope the following answers can address your concerns.

Q1: It may overstate its claims regarding the theoretical analysis of why Contrastive Learning harms the performance of Domain Generalization models.

We hope to argue that our claim regarding the theoretical analysis is conservative. It's important to note that our paper's central theme is not just introducing the concept of connectivity and theoretical analysis, but rather emphasizing connectivity enforcement from both data and model perspectives.

Meanwhile, to clearly convey the main contribution of our paper to the audience, we don't present any decorative theorems/propositions in the paper.

We instead aim to propose useful DG methods for practitioners and attribute most of the theoretical analysis to the motivating theory paper [2]. We agree we leverage the theoretical motivation behind [2], as clearly indicated in Section 3.1, for the new setting of DG; if the readers are interested in the theoretical derivative, they will find the analysis in [2] can be applied to DG settings with proper adaptations and regularity conditions. We do not take that as our contribution, while as suggested by the reviewer we've added a remark to the next revision regarding the theoretical analysis.

Q2: The idea of improving the similarity of representations across different domains is proposed and utilized in the literature.

We want to emphasize the key idea of our paper is not aligning the representations across different domains, but a unified approach, combining data and model perspectives with self-supervised objectives to bolster connectivity, thereby advancing domain generalization.

From the data angle, we build upon the shortcomings of traditional methods that align augmentation views of the same input, as explained in Appendix C. We propose two direct improvements for enhancing domain connectivity through data:

1. implementing more aggressive data augmentation, and
2. broadening the range of positive samples, extending beyond self-augmented outputs to include intra-class samples across different domains. Nevertheless, this data alignment strategy merely enhances the clustering of representations within known domains, and may not effectively bridge the gap between unseen test domains and trained domains within the same class.

Therefore, we extend our focus to the model perspective. In the paper, we observed that using a pre-trained ResNet-50 model, intra-class samples from training and testing domains are scattered yet interconnected. By integrating this observation, our paper presents a holistic approach.

Q3: The proposed Generative Transformation Loss for Pre-trained Representations raises concerns.

Thanks for bringing up the issue. We'd like to clarify the role of generative transformation as a pivotal proxy objective that facilitates model anchoring. We've described the details in our general response and incorporated them in Section 3.4.

Q4: The paper fails to adequately explain the relationship between domain connectivity and classification/regression performance.

We will add the following remark to the next revision for better clarity, while we also hope to argue in the first paragraph of Section 3.1 we have clearly indicated the relationship between domain (intra-class) connectivity and the classification/regression performance is already studied in [2].

We remark that after connecting different domains in the DG settings, we can directly follow the derivation in the contrastive learning theory paper [2] and similarly conclude the classification/regression performance benefits from connectivity. Rather than the theoretical analysis, the primary aim of our paper is to thoroughly improve domain connectivity, addressing both data and model aspects, and ultimately bolstering domain generalization capabilities.

[1] Promoting Semantic Connectivity: Dual Nearest Neighbors Contrastive Learning for Unsupervised Domain Generalization. CVPR 2023.

[2] Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap. ICLR 2022.

We appreciate your helpful feedback. We hope the following answers can address your concerns.

Q1: Is this connectivity based on sample connectivity? The discussion with [1] is overlooked.

Our definition of connectivity is based on cross-domain sample connectivity within the class. In [1], the assumption is that data adheres to the stochastic block model (SBM) and they use SBM's edge probability to define connectivity. We've referenced [1] and expanded our discussion in relation to [1] to better articulate our approach. It's important to note that our paper's central theme is not just introducing the concept of connectivity, but rather emphasizing its enforcement from both data and model perspectives.

For an intuitive understanding of the connectivity in our paper, we introduced another quantitative metric to help evaluate whether the representation space is ''well-connected'' in Appendix A.4. For images within the same class, we take those images as nodes and construct a graph, only connecting two nodes when their distance on the pre-trained space is smaller than a threshold. We denote the smallest possible threshold which makes the graph connected as $\tau$, and denote the mean and the std of the pairwise distances respectively as $\mu$ and $\sigma$. We can thus use $(\tau - \mu)/\sigma$ as a metric to describe the connectivity of the representations.

Q2: What is the high-level motivation for the generative transformation loss?

Thanks for bringing up the issue. We'd like to clarify the role of generative transformation as a pivotal proxy objective that facilitates model anchoring. We've described the details in our general response and incorporated them in Section 3.4.

[1] Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation, ICML 2022

We appreciate your helpful feedback. We wish the following answers can address your concerns.

Q1: Could you elaborate on the challenges faced when implementing self-contrastive learning in DG?

Thanks for the question. Self-Contrastive Learning (SCL), which aligns the augmentation views of the same input, has achieved successful performance in unsupervised pre-training tasks.

However, it does not naturally fit the domain generalization setting since it assumes the ability to sample $x$ from the whole data distribution; in the training stage of domain generalization, we instead are only able to access partial domains. This mismatch can lead to sub-optimal performance in domain generalization if the users mechanically adopt the classical contrastive learning loss.

In Appendix C of our paper, we provide an example and detailed explanation that even attaining optimal self-contrastive learning loss cannot guarantee good performance in the domain generalization setting. To be specific, in Figure 6, we deliberately visualize the example. Slashes and spots are used to represent domains $d_1$ and $d_2$; orange and blue rectangles respectively denote classes 1 and 2. The mapping function $\varphi (\theta(x)) := (\cos (\theta), \sin(\theta))$ with $\theta(x) = (x - \mathrm{sgn}(y)) \pi$ learned on domain $d_1$ can perfectly classify the samples, and the mapping attains perfect alignment and uniformity (the objective of SCL). However, when applied to a new domain $d_2$, the classifier completely fails (0% acc). This example highlights the potential challenges in applying self-contrastive learning effectively in the domain generalization context. We have also updated Appendix C accordingly to better elaborate on the challenges.

Q2: What drove the need for more intensive data augmentation and the inclusion of cross-domain positive samples?

Building on the response to Q1, SCL in the previous example fails to obtain intra-class connectivity due to insufficient data augmentation and domain-separated (rather than class-separated) representations, which ultimately causes poor generalization performance.

Inspired by the above analysis, we thus propose two approaches to improve domain connectivity:

1. applying more aggressive data augmentation and
2. expanding the scope of positive samples, from solely self-augmented outputs $a(x)$ to the augmentation of intra-class samples across domains.

Q3: Please shed light on the rationale for model anchoring, particularly leveraging domain connectivity in pre-trained representations and its synergy with generative transformation loss.

The previous connectivity improvements mentioned in Q1 and Q2 only involve data transforms, while this data alignment strategy merely enhances the clustering of representations within known domains, and may not effectively bridge the gap between unseen test domains and trained domains within the same class.

Therefore, we extend our focus to the model perspective. In the paper, we observed that using a pre-trained ResNet-50 model, intra-class samples from training and testing domains are scattered yet interconnected.

By integrating this observation, our paper presents a unified approach, combining data and model perspectives with self-supervised objectives to bolster connectivity, thereby advancing domain generalization.

Regarding generative transformation, it's a pivotal proxy objective that facilitates model anchoring. We've described the details in our general response and incorporated them in Section 3.4.

Q4: Would it be possible to also conduct the ablation study on benchmarks that achieved a performance gain of 1.0 or less?

Thank you for your interest. We are glad to offer more empirical evidence to support our findings.

Our previous decision to conduct the ablation study exclusively on the OfficeHome dataset was primarily to reduce computational demands. Here we additionally performed an ablation study on the VLCS dataset, as shown in the table below, where the performance gain is 0.9. These results further confirm that the three components we identified contribute consistently to the effectiveness, as detailed in our paper. We've included the experimental results in Appendix A.5 of our revision.

We appreciate your helpful feedback. We hope the following answers can address your concerns.

Q1: The toy experiment of illustrating the drawbacks of directly applying SCL to DG is put in the appendix, which is a key part of the entire logic of this paper and should be emphasized. In Appendix C, paragraphs are duplicated. As stated above, I think this part is an important component of the whole paper story.

Thank you for your suggestion. We have updated Appendix C accordingly.

This example is mainly motivational (i.e., not as indispensable as parts), and thus we put it in the Appendix due to space limit. We'll incorporate this revised section into the main body of the arxiv version for coherent narrative flow.

To elaborate, in Figure 6, we deliberately visualize the example. Slashes and spots are used to represent domains $d_1$ and $d_2$; orange and blue rectangles respectively denote classes 1 and 2. The mapping function $\varphi (\theta(x)) := (\cos (\theta), \sin(\theta))$ with $\theta(x) = (x - \mathrm{sgn}(y)) \pi$ learned on domain $d_1$ can perfectly classify the samples, and the mapping attains perfect alignment and uniformity (the objective of SCL). However, when applied to a new domain $d_2$, the classifier completely fails (0% acc).

Q2: Experimental improvement is generally not large compared with the current SOTA.

We hope to argue that our method's enhancement is at least on par with, if not greater than, existing baseline methods.

Detailed examples: In the VLCS and OfficeHome datasets, our methods – DCCL, SelfReg, PCL, and MIRO – show improvements over the state-of-the-art (SOTA) benchmarks by margins of 0.4/1.1, 0.3/-0.8, -1.0/1.0, and 0.5/0.8, respectively. Furthermore, within the domain generalization community, with the same backbone, fixed data splitting, and long-term development, it's important to note that achieving significant absolute improvements is challenging. It often requires the combined efforts of multiple research papers.

Q3: The term "domain connectivity" is a little confusing. It is more like "domain-invariant intra-class connectivity", while "domain connectivity" seems like inner-domain connectivity.

Thanks for the advice. We follow the usage in [1] and have updated the term 'domain connectivity' to 'intra-class connectivity' for greater clarity and to more accurately convey our intent.

[1] Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation, ICML 2022

Q4: Question regarding whether samples come from the same domain or not, they belong to positive pairs as long as they are from the same class.

We clarify that the 'cross-domain' positive samples in our study encompass both samples from the same and different domains.

This is because we lack domain labels for the samples. In Appendix A.5 of our submitted paper, we provide a rationale for using cross-domain contrast (CDC), which shows an accuracy of 71.8% as noted in Table 2. To contrast, we conducted a baseline experiment using only within-domain positive samples and observed a substantial drop in accuracy when compared to CDC (from 71.8% to 70.4%). Furthermore, an oracle experiment exclusively using cross-domain positive pairs yielded similar results (from 71.8% to 71.9%). These results validate our approach and suggest that, if domain information is available, its effective utilization might require thoughtful design to achieve notable improvements.

Q5: why is the generative transformation loss needed when there is pretrained model anchoring already?

Thanks for bringing up the issue. We'd like to clarify the role of generative transformation as a pivotal proxy objective that facilitates model anchoring. We've described the details in our general response and incorporated them in Section 3.4.

We appreciate your helpful feedback. We hope the following answers can address your concerns.

Q1: The combination of contrastive learning and continual learning has been explored and shown effective multiple times.

Thanks for raising the question.

Previous research, as referenced in [1,2], primarily tackled the positive data alignment issue in domain generalization by developing proxy objectives. Meanwhile, studies in [3,4] advanced the alignment of training domains, utilizing specially designed image generators to create domain-transformed positive samples.

However, these approaches primarily involve data transforms. In contrast, our work diverges by exhaustively exploring the concept of connectivity from both data and model perspectives. On one hand (data), we build upon the shortcomings of traditional methods that align augmentation views of the same input, as explained in Appendix C. We propose two direct improvements for enhancing domain connectivity through data:

1. implementing more aggressive data augmentation, and
2. broadening the range of positive samples, extending beyond self-augmented outputs to include intra-class samples across different domains. Nevertheless, this data alignment strategy merely enhances the clustering of representations within known domains, and may not effectively bridge the gap between unseen test domains and trained domains within the same class.

We therefore extend our focus to the model perspective. In the paper, we observed that using a pre-trained ResNet-50 model, intra-class samples from training and testing domains are scattered yet interconnected. By integrating this observation, our paper presents a unified approach, combining data and model perspectives with self-supervised objectives to bolster connectivity, thereby advancing domain generalization.

Q2: Could the authors elaborate more on why learning an additional decoder for the generative transformation can help domain generalization?

Thanks for bringing up the issue. We'd like to clarify the role of generative transformation as a pivotal proxy objective that facilitates model anchoring. This can be understood from multiple angles:

1. Echoing the findings in [2] that directly aligning positive pairs across vastly different domains often results in poor performance, our research similarly identifies a substantial gap in the representations of pre-trained and fine-tuned models. Direct alignment using contrastive learning, as evidenced by our findings in Table 2, tends to be sub-optimal. In response, we introduce the concept of variational generative loss to comprehend the transformation process and bridge these representational gaps.
2. Additionally, the generative transformation module is designed to reconstruct the features of the pre-trained model at an intra-sample level. This complements the inter-sample level supervision provided by contrastive loss. The module, along with its associated loss function, is intended to provide a more enriched supervised signal, encapsulating crucial within-sample information. This, in turn, supports and enhances the anchoring of the pre-trained model.

Q3: Why these components can work coherently to benefit the domain generalization task is unclear.

As shown in the response to Q1, our approach combines data and model perspectives with self-supervised objectives to bolster connectivity, thereby advancing domain generalization.

[1] SelfReg: Self-supervised Contrastive Regularization for Domain Generalization. ICCV 2021

[2] PCL: Proxy-based Contrastive Learning for Domain Generalization. CVPR 2022

[3] Towards Principled Disentanglement for Domain Generalization, CVPR 2022

[4] Model-based domain generalization, NeurIPS 2021

We appreciate the suggested reference. We've cited and discussed it in the Appendix B.1 of our paper. To help differentiate our contributions from this paper, we summarize the main points thereof as follows. As the mathematical notations is non-standard in [1] (such as the expectation of distribution in Def. 1 and minimum product $\min x_i^+ x_j^+$ of two augmented samples in Eqn. (14)), there might be misunderstanding.

• The concepts of "connectivity" differ in two papers.

[2] motivates both this project and [1] (as claimed in Section 3.2 of [1]). In [2], the original definition of intra-class connectivity appears in Assumption 4.5 just means the augmentation graph (c.f. their Def. 4.4) is connected. We follow the same definition of "connectivity" in our paper.

However, [1] is based on the concepts of intra-domain, intra-class, and semantic connectivity as

Definition 1. (Semantic Connectivity) For any input $x \in \mathcal{X}$, let $\mathcal{A}(\cdot \mid x)$ be the distribution of its augmentations $A$. Let $C$ be the joint distribution on $\mathcal{X} \times \mathcal{X}$ of augmented views of images $x_i, x_j$ as $C\left(x_i^{+}, x_j^{+}\right)=\mathcal{A}\left(x_i^{+} \mid x_i\right) \mathcal{A}\left(x_j^{+} \mid x_j\right)$. Then we have intra-domain $C_\alpha$ and intra-class $C_\beta$ connectivity.

$$$$

C_\alpha:=\mathbb{E}_{d \sim P_D^S} \mathbb{E}_{x_i, x_j \sim P_d^{S_{\mathrm{UL}}}} C\left(x_i^{+}, x_j^{+}\right), \quad C_\beta:=\mathbb{E}_{y \sim P_Y^S} \mathbb{E}_{x_i, x_j \sim P_y^{S_{\mathrm{UL}}}} C\left(x_i^{+}, x_j^{+}\right)

> Then, the semantic connectivity is defined as $C_s:=C_\beta / C_\alpha$. For the reader's convenience, we remark their definition of the joint distribution $C()$ is uncommon: it is a constant multiple of the so-called minimum product (in their Eqn. (14-16))

e_A\left(x_i^{+}, x_j^{+}\right)=\min _{x_i^{+} \in A\left(x_i\right), x_j^{+} \in A\left(x_j\right)} x_i^{+} x_j^{+}.

$$We argue it is obvious that the connectivity concept based on the joint distribution in their paper is quite different from ours and [2]; therefore the theoretical motivation behind two papers are indeed different. - The methods proposed are different. Except for the common strategy of strong augmentation recommended by the contrastive learning theory paper [2], our proposed methods are different from the ones in [1]. They propose two nearest-neighbor-based methods for constructing positive pairs, while our main contribution lies in the exploitation of both the pre-trained models and the intra-class data connectivity. --- [1] Promoting Semantic Connectivity: Dual Nearest Neighbors Contrastive Learning for Unsupervised Domain Generalization. CVPR 2023. [2] Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap. ICLR 2022.$$