Connect Later: Improving Fine-Tuning for Robustness with Targeted Augmentations

We thank the reviewer for the feedback. erbX notes that “the paper is clearly written” and “the experiment shows the effectiveness of Connect Later over several baselines on three datasets”. We address specific points below:

On iWildCam, Connect Later only marginally outperforms ERM+targeted augmentations, which indicates that if there is diverse distribution shifts, the proposed method is not quite effective.

We ran additional seeds for iWildCam to clarify these results, for a total of 15 seeds. We report 95% confidence intervals on the mean estimate in parentheses and demonstrate that the mean F1 results of ERM+targeted augs are outside of the 95% confidence intervals of Connect Later.

To further strengthen our results, we tested on another benchmark dataset, Camelyon17-WILDS, and show that Connect Later sets a new state-of-the-art on this benchmark for DenseNet121. Connect Later applied with the stain color jitter targeted augmentation achieved 94.9% +/- 0.4% accuracy OOD compared to the current state-of-the-art (93.8 +/- 0.2%, cite ICON). Overall, Connect Later achieves the state-of-the-art OOD performance on iWildCam-WILDS (for ResNet-50), Camelyon17-WILDS (for DenseNet-121), and AstroClassification.

Could the authors test the sensitivity of the performance of Connect Later with respect to the design choice of transformation distribution?

• During the rebuttal period, we verified that Connect Later (with targeted augmentations) outperforms Connect Later with generic augmentations at fine-tuning time in OOD performance, tested on AstroClassification (79.9% vs. 72.7%). The difference in ID accuracy between Connect Later and standard fine-tuning+generic augmentations is much smaller (80.54% vs 79.9%) in comparison. This suggests that the choice of augmentation is important, especially for OOD performance.
• For variations on targeted augmentations, Gao et al. 2023 tested a variant (Copy-Paste All Backgrounds) of the targeted augmentation we use for iWildCam (Copy-Paste Same Y). Copy-Paste segments out the animal from the image and pastes it onto a different background. Copy-Paste Same Y requires that the background is from a camera trap that has observed this animal species before, preventing, e.g., camels being pasted in water. Copy-Paste All Backgrounds does not leverage this requirement. Gao et al. found a 2% degradation in OOD performance with Copy-Paste All Backgrounds compared to Copy-Paste Same Y, indicating that the realism of the targeted augmentation is important.
• For pretraining augmentations, we performed an ablation on the masking percentage for the AstroClassification task (Fig 3) and demonstrated that both ID and OOD performance are relatively robust to the chosen masking percentage between 20-90%.

The proposed framework hinges on the manual design of the transformation distribution and requires knowledge about the distribution shift between source and target domains to create effective targeted augmentations. This property limits the applicability of the proposed method in scenarios where the shift is unknown or diverse.

We describe a general procedure for designing targeted augmentations from the source and target data distributions in Section 4 of the paper, as well as an example of designing the redshifting augmentation used for AstroClassification within this framework. While these augmentations do require some manual design, a relatively small amount of manual design leads to strong empirical results (SOTA on AstroClassification, iWildCam-WILDS, Camelyon17-WILDS + best performance on new dataset Redshifts).

We thank the reviewer for the feedback. atYU notes that supportive results were shown on several real-world datasets. We answer specific questions below:

If target augmentations involve knowledge of the distribution shift, why pre-training is needed? If we are able to generate the target data, where other options include using powerful generative models [1], why do we need pre-training? It seems to be more a question of when is pre-training necessary.

The ablations in the paper show that pretraining is necessary: specifically, ERM+targeted augs removes the pretraining step from Connect Later and results in worse OOD accuracy on all datasets compared to Connect Later (67.5% vs. 79.9% OOD accuracy on AstroClassification, 36.3% vs. 37.2% OOD macro F1 on iWildCam). Note that even though targeted augmentations can transform source data to more closely resemble target data, they are not perfect and do not describe the target distribution fully. Pretraining incorporates unlabeled data from the target to learn a better representation of the target data. Generative models such as Stable Diffusion could also be used to more easily design a targeted augmentation, and we view them as complementary to our framework.

the experimental results were only shown with one type of pre-training method and a different pre-training for different datasets. How sensitive are the results to the choice of pre-training method?

• Connect Later improves OOD accuracy with both masked autoencoding (AstroClassification, Redshifts) and contrastive pretraining (iWildCam-WILDS, Camelyon17-WILDS).

• During the rebuttal period, we ran the standard fine-tuning baseline with SWaV contrastive pretraining on another benchmark, Camelyon17-WILDS, and found that the choice of pretraining method is not the source of inconsistent OOD results from pretraining. On Camelyon17-WILDS, contrastive pretraining with standard fine-tuning improves accuracy over both ERM+targeted augs and ERM. This is very different from iWildCam-WILDS, where contrastive pretraining with standard fine-tuning did not even improve accuracy over ERM.

• On Camelyon17-WILDS, Connect Later also achieves the new state-of-the-art ID and OOD accuracy. Overall, Connect Later achieves the state-of-the-art OOD accuracy on iWildCam-WILDS (for ResNet-50), Camelyon17-WILDS (for DenseNet-121), and AstroClassification.

• Instead, we explain the inconsistency in OOD results of pretraining by differences in the connectivity of domains and classes in each dataset. During the rebuttal period, we investigated why pretraining was much more effective for OOD performance for AstroClassification than for iWildCam by evaluating connectivity following Shen et al., 2022 [1]. We found that one of the assumptions made in [1] that governs when contrastive pretraining works for UDA was violated for iWildCam, but not for AstroClassification.

• Specifically, we evaluate the connectivity parameters alpha, beta, and gamma (alpha: across-domain (same class, different domain) connectivity, beta: across-class (same domain, different class) connectivity, gamma: across-both (different domain, different class) connectivity) for iWildCam and AstroClassification. We use the multi-crop augmentation strategy used in SwAV contrastive pretraining for iWildCam and the masking augmentation used in AstroClassification pretraining, respectively. Shen et al., 2022 requires that min(alpha, beta) > gamma for contrastive pretraining to work as a UDA method. We find that beta (across-class) < gamma (across-both) for iWildCam, which violates Shen et al.,’s assumption, whereas the assumption holds for AstroClassification. We also observe that both across-domain connectivity and across-class connectivity are much higher for AstroClassification compared to iWildCam, verifying our reasoning for the differing OOD downstream performance with pretraining. | Dataset | across-domain | across-class | across-both | |---|---:|---:|---:| | iWildCam | 0.116 | 0.071 | 0.076 | | AstroClassification | 0.287 | 0.159 | 0.097 |

[1] Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptation, 2022.

Hello!

We were wondering if you've had a chance to look over our comments? We show that the choice of pretraining method is not the cause for inconsistent results by testing with SwAV contrastive pretraining on another benchmark dataset, Camelyon17-WILDS, where pretraining does improve over ERM. We also show that Connect Later sets a new state-of-the-art on Camelyon17-WILDS with DenseNet-121. There are also other experiments and details in the comments above!

Please let us know if you have other questions or concerns. Thank you!

Thanks for the clarifications and discussions. My biggest concern is still on the generality of the method, e.g. how would one use the proposed method on a new dataset? It seems like different datasets uses different pre-training methods/only considers one pre-training method and the target augmentations need to be carefully designed. Thus, I am inclined to keep my score.

We thank the reviewer for the feedback. vvbp notes that the paper looks at an interesting problem and contains encouraging results on two different domains. We answer specific questions below:

Why does pre-training not always lead to increased OOD performance? This question remains largely unanswered. I think this paper would be better if it provided, for example, (1) clear evidence that the assumptions from Shen et al. (2022) are routinely violated in real-world datasets and commonly used augmentations

We empirically evaluated connectivity during the rebuttal period and show that the connectivity measurements of iWildCam-WILDS violates the assumptions from Shen et al, 2022 that govern the success of contrastive pretraining for domain adaptation, while AstroClassification has much higher connectivity (details below). This validates our intuition that generic augmentations, such as the ones used in SwAV contrastive pretraining, are not enough to connect the domains for certain datasets and shifts.

Specifically, we evaluate the connectivity parameters alpha, beta, and gamma (alpha: across-domain (same class, different domain) connectivity, beta: across-class (same domain, different class) connectivity, gamma: across-both (different domain, different class) connectivity) for iWildCam and AstroClassification. We use the multi-crop augmentation strategy used in SwAV contrastive pretraining for iWildCam and the masking augmentation used in AstroClassification pretraining, respectively. Shen et al., 2022 requires that min(alpha, beta) > gamma for contrastive pretraining to work as a UDA method. We find that beta (across-class) < gamma (across-both) for iWildCam, which violates Shen et al.,’s assumption, whereas the assumption holds for AstroClassification. We also observe that both across-domain connectivity and across-class connectivity are much higher for AstroClassification compared to iWildCam, verifying our reasoning for the differing OOD downstream performance with pretraining. We thank the reviewer for this suggestion and will add this to the paper in the final revision.

We follow Shen et al., 2022 to measure connectivity. First, we train a binary classifier to distinguish between class-domain pairs (e.g. for across-domain connectivity, the classifier distinguishes between augmented examples with the same class but different domain). We compute the connectivity as the test 0-1 error of these classifiers (1 - test_accuracy), averaged over class-domain pairs.

different self-supervised methods were used for the iWildCam-WILDS dataset (SWaV contrastive learning) and the astronomical time-series (masked autoencoding). This makes it hard to draw conclusions from table 1, since it is unclear which differences can be attributed to the use of different pre-training objectives.

During the rebuttal period, we ran the standard fine-tuning baseline with SWaV contrastive pretraining on another benchmark, Camelyon17-WILDS, and found that the choice of pretraining method is not the source of inconsistent OOD results from pretraining. On Camelyon17-WILDS, contrastive pretraining with standard fine-tuning improves accuracy over both ERM+targeted augs and ERM. This is very different from iWildCam-WILDS, where contrastive pretraining with standard fine-tuning did not even improve accuracy over ERM. Our results on Camelyon17-WILDS and iWildCam-WILDS both use SwaV contrastive learning, showing that the choice of pretraining method is not the source of inconsistent OOD results. On Camelyon17-WILDS, Connect Later also achieves the new state-of-the-art ID and OOD accuracy.

DANN and CORAL results are from Sagawa et al., 2022 (10 seeds) and the previous state-of-the-art results are from ICON (https://github.com/a-tea-guy/ICON, 3 seeds). All other results are averaged over 20 seeds. We report 95% confidence intervals in parentheses.