Ask Your Distribution Shift if Pre-Training is Right for You

We thank the reviewer for the helpful feedback.

Unclear contributions

The reviewer states that “the paper suggested to use initialization with a pretrained model and training on a balanced dataset, which is a standard practice anyway.” While training on balanced data has indeed been suggested by previous work, our work (in Section 6) suggests that, as a principle, this balanced dataset can be small and even non-diverse when fine-tuning a pre-trained model. We believe that this intuition is valuable for informing dataset curation processes.

Our work also offers an understanding of why pre-training in conjunction with a method for handling bias (such as balancing a dataset) is effective for robustness, which has not been previously studied to our knowledge.

Limited theoretical contribution

Please refer to the general response.

Vague or inconsistent definition of support

As noted by the reviewer, in-support and out-of-support examples are examples inside and outside the support of the reference distribution, respectively. In other words, an example is in-support if its probability density under the reference distribution $p_\text{ref}$ is small. To divide a shifted distribution into in-support and out-of-support splits, we instead estimate the probability density ratio $p_\text{ref}/p_\text{shift}$ and find examples where this quantity is small. The reviewer was confused that this is a different quantity than $p_\text{ref}$. As we explain and justify in Appendix B.3.1, we do so because estimating $p_\text{ref}$ is difficult for high-dimensional inputs (in our case, images) and $p_\text{ref}/p_\text{shift}$ is similarly meaningful.

Presentation issues

We are surprised by the reviewer’s comment that effective robustness (ER) is not defined, as it is defined in the Section 2. To further clarify this metric, we added a longer definition in the Appendix in the revision.

Practical implications are not well argued.

Please refer to the general response.

Q1

If the training data only has pictures of indoor dogs and outdoor cats, then examples of outdoor dogs and indoor cats are out-of-support. Here, we expect that a model could suffer both from biases and from poor extrapolation to this previously unseen group.

Q2

We thank the reviewer for raising this point. In experiments that are not currently included in the paper, we found that fine-tuning a pre-trained model on a female only dataset is ineffective (the model still performs poorly on blond males). This might be because there are other attributes spuriously correlated with hair color even among just the female population, and this is why we opt for counterfactual editing. We will include this ablation in future revisions.

We would like to note that we are not claiming that counterfactual data is the only or most effective way to de-bias a dataset. The purpose of this experiment is to illustrate that de-biasing a small dataset and fine-tuning a pre-trained model on it can be effective, regardless of the de-biasing method.

We thank the reviewer for the helpful feedback.

Measuring the robustness of pre-trained models with various architectures with respect to ResNet-18 models trained from scratch (1)

The reviewer expressed concerns that we measure the robustness of pre-trained models with various architectures with respect to ResNet-18 models trained from scratch. We would like to point out that the pre-trained models include ResNet-18s (pre-trained on ImageNet) in addition to other, larger architectures, so there is already a direct comparison between models with the same architecture. That said, in future revisions we will augment this result by adding models trained from scratch with ResNet-50 and DenseNet-121 architectures (we also tried ViTs, but these tend to perform poorly when trained from scratch on small datasets [1]).

Biased datasets with zero minority sub-group probabilities (2)

We thank the reviewer for pointing out that there might also be biased datasets in which the minority group probabilities are close to zero. This is an important point that we intend to clarify in future revisions. Indeed, in this extreme case a model would suffer from the bias. However, a model could also suffer from poor extrapolation, since the minority groups represent a previously unseen setting. We do not categorize such a shift as in-support for this reason.

The reviewer says that “current analysis seems to suggest that pre-training would be able to handle the more extreme cases better, which is counter-intuitive”. We agree that this result as stated by the reviewer would be counter-intuitive; however, this is not what our current analysis suggests. Our analysis suggests that pre-training can help with extrapolation, implying that a pre-trained model might be more robust than a model trained from scratch under the same extreme case. Notably, the analysis does not suggest that a pre-trained model would perform better in this extreme case than in the case where data from the minority group is available (since, indeed, a pre-trained model would likely suffer more from biases).

Margin gain correlations (3)

In the general response, we discuss why, in our view, even uncorrelated margin gains imply that interventions are complementary.

We are confused about why the reviewer believes that the correlation of (ii) is poorly motivated and might be tautologically positive. In our understanding, for something to be tautologically true means that they mean the same thing. In this case, applying two interventions separately and summing their independent gains and applying them together and measuring the gain are not the same. While some correlation is to be expected, the high positive correlation provides evidence that applying the two interventions together yields the combined benefits of applying each individually.

Counterfactual data in the curation process (4)

The reviewer points out that “if there is no spurious correlated feature during training, then the transfer problem corresponds to an extrapolation one”. This is exactly the purpose of the dataset curation process; since our analysis suggests that pre-training can help with extrapolation, if we can set up a dataset where the problem is just extrapolation then a pre-trained and fine-tuned model might do well. While there may be other strategies besides pre-training that can help with extrapolation, we do not believe that “simple augmentations/regularization” would be an effective solution in general.

We are not claiming that counterfactual data is the only or most effective way to de-bias a dataset. The purpose of this experiment is to illustrate that de-biasing a small dataset and fine-tuning a pre-trained model on it can be effective, regardless of the de-biasing method. We intend to clarify this in future revisions.

Question about the in-support and out-of-support splits in Figure 4

We report the number of samples found to be in-support and out-of-support in Table 1 in Appendix C.2.1. As the reviewer alludes to, the accuracy on the in-support split is higher than the average accuracy under distribution shift (since the in-support split contains “easier” examples). We conduct an analysis in Appendix C.2.4 illustrating that, even accounting for this difference in difficulty between the in-support and out-of-support splits, pre-training can help substantially out-of-support but not in-support.

[1] Dosovitskiy et al. “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.” (2021).

We thank the reviewer for the helpful feedback.

Concerns about Theorem 1

The reviewer expressed confusion about $w^*_\text{ref}$ being undefined. We state that $w^*_\text{ref}$ does not depend on the initialization $w_\text{init}$. We also state that it is a property of the reference dataset. In the revision, we clarify this when we first refer to $w^*_\text{ref}$.

The reviewer correctly points out that pre-training can in fact hurt robustness, which we discuss in Appendix D.2. Indeed, this is reflected by our theoretical analysis, which suggests that pre-training can influence a model’s extrapolation, not that it will necessarily help (this is why we use the language “can help with extrapolation”). See the general response for more discussion.

While the model in Theorem 1 is deterministic, crucially, it depends on initialization (in our case, whether the model is pre-trained or randomly initialized). Even though the loss is convex, there are multiple equally good solutions and the initialization specifies which is selected. Pre-training thus can affect the learned model, even in this convex setting.

See the general response for concerns about the simplicity of the theoretical analysis.

Robustness to in-support shifts

We are confused by the reviewer’s comments about models being agnostic to in-support shifts. Consider [1], for example.

Definition of robustness

In our work, we give both an intuitive and formal definition of robustness. Intuitively, as described in our introduction, we consider the robustness of a method to be its ability to succeed on a shifted distribution that differs from the reference distribution the model is trained on.

Formally, we define and evaluate the robustness of a model via the effective robustness metric [2] (we define this metric in Section 2). To further clarify this metric, we added a longer definition in the Appendix in the revision.

Splitting Method

We are unsure why the reviewer is unable to view Appendix B.3.2. We have opened and download the pdf on multiple browsers, and in each case the Appendix is included in the file. We also note that, as we state in the main text in Section 4.2, we actually give the full details in the splitting method in Appendix B.3.1 and not B.3.2.

Using Interventions to prevent exploiting biases

We would like to point the reviewer to Appendix B.4. Here, we indeed give a description of the Deep Feature Reweighting (which we assume the author is referring to) and also carefully describe our implementation of this method. Our contribution here is not a specific approach, but rather a guiding principle of combining pre-training with an intervention that can handle biases. Thus, we did not consider the details of the specific choice of this intervention to be worth including in the main paper.

Location of the Appendix

We refer the reviewer to the ICLR 2024 call for papers and author guidelines:

• The ICLR 2024 author guide states: “We encourage authors to submit a single file (paper + supplementary text). Please mark the supplementary material clearly.”
• The ICLR 2024 call for papers states: “Authors may use as many pages of appendices (after the bibliography) as they wish.”

[1] Chaudhuri et al. “Why does Throwing Away Data Improve Worst-Group Error?” (2023).

[2] Taori et al. “Measuring Robustness to Natural Distribution Shifts in Image Classification.” (2020).

We thank the reviewer for the helpful feedback. Please see general response for a discussion on:

• Takeaways
• Cases where pre-training hurts robustness
• Correlation of margin gains

Below, we address the reviewer’s remaining concerns:

Notation in Section 3

To clarify the reviewer’s confusion about Theorem 1:

1. $proj_{W_\text{ref}}$ is the projection operator onto the subspace $W_\text{ref}$. We apologize for the confusion and clarify this in the revision.
2. Orthogonal is to be interpreted mathematically; $w_\text{init}-proj_{W_\text{ref}}w_\text{init}$ is orthogonal to $W_\text{ref}$ because we remove the component that lies within $W_\text{ref}$. This is also why this second term would not influence the model’s output on any point within $W_\text{ref}$ (because the product of this point and $w_\text{init}-proj_{W_\text{ref}}w_\text{init}$ would be zero).
3. The proof is in Appendix A and is referenced in the paper.

Overlapping text in Figure 1

We are confused by the author’s comments about the overlapping text in Figure 1. We were unable to reproduce this issue when

1. Downloading the PDF and viewing it locally.
2. Viewing the PDF in Safari, Chrome, Brave.

We suggest that the reviewer uses one of the methods above to view the PDF.

Ablations on shift intensity

We thank the reviewer for the suggestion of ablating shift intensity, and intend to include this in the Appendix in future revisions. We already have some intuition of how ablating the fraction of samples from the minority group (e.g., randomly tinted examples in the tint shift) affects the effective robustness of pre-trained models. In our existing experiments, we observe that when the number of samples available from minority groups is small, pre-trained models do have a little effective robustness, even under in-support shifts. However, in Appendix C.1.2, we provide evidence that as we increase the total number of samples (while maintaining the same distribution), these effective robustness benefits vanish. If we were to decrease the fraction of samples from the minority groups while maintaining the total dataset size, we would expect the effective robustness of pre-trained models to increase somewhat, but as we discuss in Appendix C.1.2, this robustness could be attributed to better extrapolation from a small number of samples.