Towards domain-invariant Self-Supervised Learning with Batch Styles Standardization

Thank you for your time and insightful feedback. You will find below a response to your concerns/comments/questions.

In Section 5.2, the authors do not show the ablation effect of the original SimCLR.

We have conducted the same ablation experiments as detailed in Section 5.2 specifically for a SimCLR with standard image data augmentation. The results of these additional experiments have been included to the existing plots on Figure 2. In examining the experiment on average domain purity varying with the number of nearest neighbors (Figure 2a), standard SimCLR exhibits only slightly higher average domain purity than SimCLR with FA, but significantly higher than SimCLR with Batch Style Standardization (BSS) (higher domain purity is worse). This observation suggests that FA does not significantly enhance domain-invariance compared to BSS. Similarly, in Figure 2b, it is noticeable that standard SimCLR results in negatives dissimilar to the anchors. SimCLR with FA produces negatives more similar to the anchors but not to the same extent as SimCLR with BSS. Finally, in terms of performance concerning batch size, standard SimCLR yields considerably lower performances compared to SimCLR with FA or with BSS.

Perhaps there is a limitation to BSS. Assuming that domain labels are not available, there is no guarantee that a number of randomly selected images belong to different domains. [...] it may indeed create harder negative samples. However, it would also create the problem of more similarity between pairs of positive samples, potentially forcing the network to capture more similar style information than domain invariant information in the positive sample pairs.

Indeed, in the worst-case scenario, we could sample a batch of $N$ examples from a single domain (same style). Applying BSS successively $V$ times on this batch would result in $V$ batches of examples, each with its own style (as depicted in Fig 1c). As you correctly mentioned, because all examples initially came from the same domain (same style), we should end up with a single style for all examples, creating harder negatives but simpler positives. In contrastive learning, the loss tries at the same time to bring the positive pairs closer and to drive away the negative ones: since in this pathological case, all examples have the same style, the style cannot be used to do both at the same time. It should also be noted that on top of the style standardization, geometric augmentation and batch-wise color augmentations are applied (see section 3.2.2, ablation in section 5.1, table 4). This mitigates the problem since the $V$ positives have undergone different color augmentations.

What does "spurious correlations" mean?

In general, correlations or features are said to be spurious when they predict the target label without possessing a causal relationship with it. In the context of SSL, spurious features would be typically associated with non-causal attributes such as style (color, texture, etc.), while the target variable would be the ones related to the SSL task objectives. As an example, in SimCLR, when diverse styles coexist among the positive and negative examples, “style features” may be used to solve the contrastive task.

In section 4.2, "We did not use ImageNet transfer learning, except on DomainNet to allow fair comparisons with prior UDG works." Does it mean that this paper uses an ImageNet pre-trained model when conducting experiments on the DomainNet dataset?

Using Transfer Learning / pretrained models in DG/UDG is a common practice in the community but we think it is misguided. Often the pre-training dataset, like ImageNet for example, can include one or more of the target domains (e.g: photo for PACS). We have tried to limit as much as possible the usage of transfer learning in our experiments and only used it for the DomainNet dataset.

If the author-uploaded PDF is accurate, the three subfigures in Figure 2 indeed only contain SimCLR w/ FA and SimCLR w/ BSS (without a third standard SimCLR or SimCLR w/o any). Intuitively, SimCLR w/o any should perform worse than both SimCLR w/o FA and SimCLR w/o BSS because it does not use any cross-domain data augmentation. However, for rigor, it's still necessary for the authors to present the baseline SimCLR results to demonstrate the effectiveness of methods like BSS or FA for UDG.

We do not understand why you cannot see the new experiments concerning standard SimCLR on Figure 2. Normally, we had updated Figure 2 on our revised version submitted on November 16, 2023, at 11:11 UTC, including the requested experiments related to the standard SimCLR (blue color on each of the subfigures). However, the figure’s caption was not updated. Just in case, we have reuploaded the new version of the paper (with a more detailed caption). Kindly inform us if you still cannot see these new experiments.

As you correctly intuited, the SimCLR performs worse than the SimCLR with FA and SimCLR with BSS since it does not use cross-domain augmentation (FA) or normalization (BSS)  that help a lot to achieve out-of-domain generalization.

I apologize for the misunderstanding in my previous responses. I mistakenly believed that the ICLR review process restricts the uploading of new paper versions, which led me to not download the PDF again.

After re-reading the new version of the paper, the authors show the results of Standard SimCLR. As stated in the Rebuttal, the performance of SimCLR is worse than SimCLR w/ FA and SimCLR w/ BSS. Overall, the authors addressed previous Weaknesses in the newly uploaded version.

Thank you for your time and relevant feedback. You will find below a response to your concerns.

What is the performance of the ERM (empirical risk minimization) on those benchmarks?

Performances of the ERM baseline have been included in Tables 1 and 2. In our setting, ERM has a low performance since it is the only method that does not benefit from the SSL pretraining step while the amount of labeled data used for finetuning is too small to make the model generalize well.

As a small fraction of labeled data is always used, why not try some good semi-supervised learning methods such as FixMatch [3] or AdaMatch [4] (which also deals with domain shift)? [...] what is the advantage of using a framework of unsupervised pretraining + fine tuning?

This is a fair question: Indeed, in practical scenarios aimed at producing a model with robust generalization to unseen domains, a dilemma can arise regarding two primary learning strategies:

1. Unsupervised Pretraining handling domain shift (UDG) + Finetuning
2. Semi-Supervised Learning, such as FixMatch, combined with Domain Generalization techniques to handle domain shift. (Note: AdaMatch works in a Unsupervised Domain Adaptation setting not Domain Generalization) In practice, these strategies are not mutually exclusive: Combining both approaches would allow one to harness the strengths of each, potentially achieving optimal performance on unseen domains. Consequently, the suggested approach would be to (1) perform unsupervised pretraining addressing domain-shift (UDG) and subsequently (2) fine-tune the model using Semi-Supervised Learning in conjunction with Domain Generalization techniques.

However, within the context of UDG, where the primary focus is evaluating the method's capacity to learn domain-invariant features, the typical workflow involves: (1) unsupervised training on source domains, (2) finetuning on source domains via linear probing, and (3) evaluation on unseen target domains. That is why the community follows this workflow.

As CLIP models have shown very good performance on samples under distribution shift, especially in image classification, I wonder to what extent the problem can be solved by them already. [...]. It would be much more interesting to see if the proposed BSS still holds the same performance gap on top of CLIP [...] it would still be great if the authors could share their thoughts on this point.

It's indeed a very insightful remark. Vision-language models like CLIP have demonstrated impressive zero-shot capabilities in in-distribution evaluation scenarios and even in open-world evaluation scenarios. We acknowledge that VLMs have made strides in solving domain-shift problems and may serve as a strong foundation for developing new techniques for out-of-distribution (OOD) generalization.

However, there are certain considerations and potential challenges when relying solely on VLMs for domain-shift problems:

1. VLMs often necessitate a substantial amount of labeled data for pretraining, and pretrained image and text encoders might not be well-suited for specific task data. For instance, in the histopathology field, existing pretrained VLM like CLIP does not generalize well while histopathology available paired data (images, diagnoses) are too scarce to train a new VLM from scratch [1].
2. While fine-tuning these models can yield significant performance gains on a given data distribution, it has been observed to potentially diminish out-of-distribution generalization ability [2, 3]. Hence, careful fine-tuning is essential to preserve their generalization capability in OOD evaluation scenarios.
3. On top of that, fair evaluation of their OOD capabilities are hard because VLMs are often trained on very large datasets that include many unknown domains. This makes it difficult to be sure that the target domains have not been seen during training.

As suggested, it would be interesting to investigate whether BSS can enhance domain-invariance and improve performances on top of VLMs, thereby addressing the challenge of loss of OOD generalization during fine-tuning.

References:

• [1] Lu et al. "Towards a visual-language foundation model for computational pathology.", arXiv preprint 2023.
• [2] Wortsman et al. "Robust fine-tuning of zero-shot models.", CVPR 2022
• [3] Goyal, Sachin, et al. "Finetune like you pretrain: Improved finetuning of zero-shot vision models.”, CVPR 2023.

To be more self-contained, it would be great if the authors could also introduce the contrastive-based UDG methods before Section 3.2.2.

Contrastive-based UDG methods have been introduced before Section 3.2.2 (related works about UDG).

What are the source domains for PACS pretraining?

For the experiments on PACS, we followed the common practice in the UDG community employing a leave-one-out domain evaluation. In Table 1, we only specifies the target domain for evaluation which implies that all other domains serve as pretraining.

Thank you for your time and constructive feedback. You will find below a response to your concerns.

The idea of transforming the styles in a batch is deeply related to Mixstyle but authors did not compare or discuss the difference.

While MixStyle also transform the styles in the batch to improve out-of-domain generalization, significant differences exist between the proposed BSS and MixStyle:

1. The objectives of MixStyle and BSS diverge: MixStyle seeks to alter the style of examples, introducing a broad style variability at the features-level, whereas BSS strives to eliminate the style variability by enforcing a single style at the image-level (style augmentation vs style standardization).
2. To alter the style of examples at the features-level, MixStyle mixes the features maps` statistics (mean, std) with those of random examples. In contrast, BSS normalizes the style of all examples in a batch by replacing all amplitudes with those of a randomly chosen image, resulting in a batch with a single style. In practice, MixStyle could be extended to normalize the style of examples by swapping all the statistics of a batch with those of a randomly chosen image and used within BSS to replace the proposed Fourier-based normalization.
3. MixStyle relies on the assumption that style information of examples can be captured by the examples’ statistics (mean, std) of bottom layers features in CNN [1]. This assumption is not directly transferable to all architecture like Vision Transformers for example. This makes MixStyle specific to the CNN architecture, whereas BSS operating at the image-level is architecture agnostic.

References:

• [1] Huang et al.. "Arbitrary style transfer in real-time with adaptive instance normalization.", ICCV 2017

The motivation of combining BSS with self-supervised learning is not clear. [...]. I do not see the insights of such combination.

The primary motivation behind normalizing styles examples is to reinforce domain-invariance. Style normalization in a batch has the desired effect only if comparisons between examples of the batch occur during training. This is the case in the considered SSL methods but not in a standard supervised setting with a cross-entropy loss.

The intuition behind why combining BSS to SSL methods helps to achieve domain- invariance is the following: both contrastive and non-contrastive SSL methods aim to distribute batch examples over an embedding space. This distribution over the embedding space can be driven explicitly by contrastive loss like in SimCLR or implicitly by methods preventing representation collapse like Sinkhorn-Knopp (SWaV, MSN), centering (DINO), or variance regularization (VicReg). However, when diverse styles coexist within a batch, there is no inherent mechanism preventing the distribution from being influenced by these different styles. In such case, these spurious correlations may harm the performance of the fine-tuned SSL model in an out-of-distribution evaluation scenario (UDG). Therefore, our hypothesis posits that normalizing the styles of examples in a batch should help reduce the emergence of spurious correlations in SSL features thereby enhancing domain-invariance.

Empirical evidence supports this intuition, as substantial performance gaps between our method utilizing Fourier-based augmentations (FA) and BSS (styles normalization) have been observed (please refer to Table 1, 2, 3). Additionally, the experiments provided in section 5.2 clarify the underlying mechanisms that contributed to BSS's effectiveness in improving domain-invariance in SSL representations.

The BSS part, is hard to understand

We have tried to be as clear as possible by providing a description of the exact process of how BSS is performed (Section 3.2.2 - 2nd paragraph), a Figure illustrating how BSS is performed (Figure 1) and pseudo-code along with a Pytorch implementation of BSS. We will try to improve the explanation but it would help if you could explain which specific aspect you find unclear.

Authors should compare with existing Fourier methods to validate their effectiveness.

All results from Table 1, 2 and 3 compare to existing UDG methods based on FA (DiMAE, CycleMAE (new)). Comparison to a standard FA strategy is also available for all experiments. If you can think of more pertinent comparisons, could you please point them out?

Lacks theoretical support

In the UDG community, it is common practice to present motivations, intuitions, and empirical validations through performances or experiments (DARLING, BRaD, DiMAE, CycleMAE). Producing meaningful theoretical support for DG deep learning methods is hard: strong assumptions regarding the data, such as a modelization of semantic/style separation, would be required and in practice, such assumptions never hold on real datasets. Moreover, our non-reliance on domain labels makes it even harder.

Thank you for your time and constructive feedback. Your first point about novelty is treated in the global response. You will find below a response to your other remarks.

The author should compare with CycleMAE

Thank you for pointing out this recent work that we missed during our initial submission. We have updated the related works accordingly and included performances of the method in Table 1 and 2.

In examining the performances on PACS (Table 1), CycleMAE demonstrates slightly superior average performances at label fractions of 10% and 100%, compared to SimCLR with BSS. However, it should be noted that CycleMAE and our approaches differ in experimental settings:

• CycleMAE employs ImageNet pretraining, while we do not.
• Full finetuning is conducted on the 10% labeled data setting in CycleMAE, whereas we utilize linear probing as per BRaD experimental setting.
• CycleMAE uses a ViT-small architecture, whereas most compared methods, including ours, utilize a ResNet18 architecture.

Concerning the performances on the DomainNet subset (Table 2), our initial observations and comparisons remain consistent.

Thank you for your time and constructive feedback. Your main concern about novelty is treated in the global response, your others remarks are covered below.

SimCLR and SWaV should be categorized as contrastive-based SSL methods in the second paragraph of Contributions.

The categorization of SimCLR and SWaV as contrastive-based SSL methods have been clarified in an updated version of the paper in the second paragraph of Contributions.

Investigation on why BSS is sometimes outperformed (though by a small margin) by regular Fourier-based augmentation on the DomainNet dataset?

We do not have a clear understanding of why BSS does not work as well for these few cases or a specific methodology to investigate them. However, it should be noted that, on average, BSS outperforms Fourier-based augmentation (FA) with a clear margin.