When Does Bias Transfer in Transfer Learning?

We thank the reviewer for their response! We discuss their points below.

W2. Defining Bias: We define bias as a feature that the model relies on but is not causally linked with the target class. In particular, we focus on biases that the transfer learning model would not have relied on if it had been trained from scratch. We have added this definition to Section 2 in our revision.

W1, W3. Novelty and Related Work: Neyshabur et. al focuses on understanding whether low-level statistics or feature reuse is transferred during transfer learning. However, they do not investigate the problem of bias transfer: where features from the pre-training data (that are not used during downstream training) can persist during downstream deployment. Shafahi et al focus on adversarial robustness: in our paper, we focus on biases as features (and in particular, emphasize natural biases which might appear in standard.

Lubana et. al is a recent work (ICML 2023) which performs mechanistic analysis. As part of their analysis, they perform fine-tuning from a backdoored CIFAR-10 to a clean CIFAR-10, and find that fine-tuning may not eliminate spurious cues. In our work, we perform a larger study that fully investigates the landscape of bias transfer across a wide range of tasks for different fine-tuning regimes. We further discuss potential mitigation strategies, such as target dataset debiasing and weight decay. Finally, we investigate bias transfer beyond synthetic biases, and find evidence of such transfer in natural datasets such as ImageNet.

We thank the reviewer for bringing these to our attention and have added them to the related work of the final version.

Sorry for my late response.

Thank you for clarifying your definition of features and bias transfer. Now I understand that a feature is defined as a transformation in the input domain, which seems a non-standard definition of "feature" though, and the bias transfer is defined counterfactually with such transformation. With these definitions, I also understand that backdoor attacks can be considered as a bias. But I still find it difficult to understand without these definitions which is also only discussed very roughly in Appendix.

In our paper, we explore a specific vulnerability: we show that if the pre-training model learns a feature which is not used during downstream training, the fine-tuned model will still be sensitive to that feature (see the definition of bias transfer above). We find that this sensitivity can cause unexpected downstream deployment failures.

After reading this explanation, I still consider that it is indeed a special case of feature reuse and thus the findings are naturally implied from the previous results, which itself is ok but leads to the limited novelty and impact of the paper. Also the authors should discuss in more detail about the relationship to the feature reuse, including how it can be suggested from the previous work, rather than just citing the papers in Related Work.

We thank the reviewer for their response, and address their concerns below.

W1. Novelty: We want to point out that the main message of our paper is that spurious features which are present in the pre-training data and completely absent in the downstream task can unexpectedly pop up during downstream deployment. The rest of our paper (comparing fixed-feature to full-network fine tuning, varying weight decay, looking at different downstream tasks) is all in an effort to fully characterize this phenomenon, but they are not meant to be “surprising” in and of themselves.

W2. Mitigation Strategies: In our paper, we discuss several mitigation strategies. In addition to full-network fine-tuning, we find that weight decay and de-biasing can partially mitigate bias transfer (see Section 3.2). Indeed, a major take-away from our paper is that, while mitigation strategies such as weight decay and dataset debiasing can help, such post-hoc strategies typically do not fully remove bias transfer and often are not feasible in many settings (see Reviewer RsQk’s comment). As a result, we believe it is critical to prevent such biases at their source by more carefully choosing our pre-trained models in the first place.

W3. Manipulating the target dataset: In our overall framing, we consider the case where the target dataset is balanced (e.g., does not have the bias) but the source dataset is skewed. In this case, bias transfer occurs, even though the bias is not present in the downstream dataset. Furthermore, in Section 3.2 and 4.2, we consider actively debiasing the dataset to counter the bias. However, we find that debiasing the dataset only partially mitigates bias transfer (especially in the fixed transfer case).

W4. Backdoor attack is just inserting a feature. We agree that backdoors are just features, which is why we believe it is a useful starting point for studying bias transfer (since biases are also features). We use the standard conception of backdoor attacks found in the data poisoning literature, see e.g., [1, 2].

Q1. Biases in the downstream dataset: The scenario where the downstream dataset is biased falls within the standard supervised setting (where the training dataset has biases and struggles on minority groups). Since there is already a vast amount of literature on debiasing in this regime (see for example, Group DRO, Deep Feature Reweighting, Just Train Twice), we do not focus on that regime here.

Q2. What happens when the latter is balanced? As we show in Sections 3.2 and 4.2 of our paper, even if the downstream dataset is balanced or unbiased, bias can still transfer! This means that ML developers must pay attention to the pre-training models/data, and cannot rely on solely downstream balancing.

[1] https://arxiv.org/abs/1708.06733

[2] https://arxiv.org/abs/1912.02771

Thank you for your response!

[W2]

We agree with the reviewer that no mitigation strategy is fully effective, and that indeed the only foolproof way to avoid bias transfer is to ensure that the pre-training data and model are unbiased. However, we believe that this message (along with the partial mitigation strategies we discuss in our work) only strengthen the message of our paper: bias transfer is a troubling phenomenon, and has important implications for both deployment and regulation of these models.

Our paper thus advocates for further transparency for pre-training models: for example, publishers of such models might list potential biases (similar to Datasheets [1]) when releasing checkpoints. Constructing solutions for detecting and fully countering bias transfer is an exciting avenue for future work (however, our goal in this paper is to show the existence of and characterize bias transfer).

[Q1]

We apologize for misunderstanding the reviewer's question here! Since time is rather limited, we ran the following experiment, which we are happy to fully flesh out and include in the final version of the paper. We consider the backdoor experiment from Section 3: where a square is placed on the dog classes of ImageNet, and then either the biased or unbiased ImageNet is fine-tuned on CIFAR-10. This time, however, we consider the case where the downstream dataset also has the same spurious feature, by placing a square on the dog class in CIFAR-10 with varying percentages. Our results can be found in Appendix A.9 of the revision: however we summarize them in a table below. The first row is our standard bias-transfer setup (no bias in the downstream dataset).

Fixed-feature:

Full-network:

We find that when the downstream dataset is already biased, pre-training with the biased ImageNet significantly amplifies the downstream bias in both the fixed-feature and full-network settings.

[1] Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Iii, H. D., & Crawford, K. (2021). Datasheets for datasets. Communications of the ACM, 64(12), 86-92. https://arxiv.org/abs/1803.09010

We thank the reviewer for their detailed review! We address the comments below.

Impact of learning rate: We replicated our backdoor experiments in Section 3 with varying learning rates (Appendix A.8 in the revision). Similar to weight decay, we find that increasing the learning rate does not mitigate bias transfer in the fixed-feature setting (ASR actually goes up) but does help in the full-network setting. However, increasing learning rate is a fairly invasive intervention, and often has adverse effects on model performance.

SimCLR and CLIP: While we focused on supervised pre-training models in our paper, we would expect biases from SimCLR and CLIP to similarly transfer. Indeed, there has been a large body of work indicating that CLIP contains several concerning biases, especially concerning race and gender [1, 2]. Understanding how these types of biases manifest in fine-tuned models is an interesting avenue for future work.

[1] https://arxiv.org/abs/2108.02818

[2] https://arxiv.org/abs/2210.14562

I thank the authors for their response. After reading other reviewer discussions, and more thoroughly reading [1], I have decided to maintain my score. The main concern of other reviewers seems to be lack of recommendations for mitigating biased transfer, and the similarity to Wang et al [1].

While [1] tackles a similar question, I think that this paper provides significant additional insights. This paper conducts experiments across multiple architectures, fine-tuning scenarios, and types of biases whereas [1] is relatively limited along these axes. Two empirical findings I think are significant and differentiate the work from those previous are the following:

• Spurious correlations from the pretraining set can imperceptibly persist in the downstream model even after fine-tuning.
• Showing that the type of biases that previous works induce artificially, are naturally occurring at scale in ImageNet.

Additionally this work provides some theoretically plausible explanation for how over-parametrization enables biases to persist. I think this could be of interest to others and point to future directions of research. Finally, I don't think the paper should be penalized for not solving the problem of bias. The authors explore a few straight-forward mitigation strategies and show that the problem is not easily solved so researchers pretraining models should be careful.

[1] Wang and Russakovsky: Overwriting Pretrained Bias with Finetuning Data.

We thank the reviewer for their response! We address the raised points below.

W1. Wang et. al: We thank the reviewers for bringing this to our attention. Wang et. al investigates gender biases in the case where the spurious feature is present in both the pre-training dataset and the fine-tuning dataset. They then consider actively changing the fine-tuning dataset to counter the correlation in the pre-training data (this is equivalent to the “debiasing” scheme in our paper). In our work, we find that (especially in the fixed feature case) a large portion of the dataset would need to be debiased in order to mitigate bias transfer. As you mention, this level of downstream debiasing may not be feasible.

Our work further highlights a different risk: spurious features which are present in the pre-training data and completely absent in the downstream task can unexpectedly pop up during downstream deployment. These types of “backdoor” features are especially pernicious, since a practitioner might not be aware that their pre-training model has such features. We fully investigate the landscape of this phenomenon, across different types of biases (synthetic and natural), architectures, and fine-tuning regimes (fixed-feature vs. full-network). While we discuss potential mitigation strategies (e.g., weight decay), we arrive at a very different conclusion than Wang et. al: practitioners cannot solely depend on post-hoc strategies to deal with bias transfer. As a result, we believe it is critical to prevent such biases at their source by more carefully choosing our pre-trained data in the first place.

W2a. Full-network vs fixed-feature effectiveness is obvious. We agree with the reviewer that fixed-feature fine-tuning being more prone to bias transfer than full-network fine-tuning is not surprising. Our goal here is to fully investigate the discrepancy between these two fine-tuning strategies, particularly in light of the fact that fixed-feature fine-tuning is becoming increasingly popular in the age of large pre-trained (foundation) models. We thus view quantifying and ablating the gap between the two as a valuable contribution.

More generally, the main finding of our paper is that spurious features which are present in the pre-training data and completely absent in the downstream task can unexpectedly pop up during downstream deployment. The remainder of our paper (including the comparison of fixed-feature to full-network fine tuning) is all in an effort to fully characterize this phenomenon, but is not meant to be “surprising” in and of itself.

W2b. Weight Decay: In appendix A.5, we discuss a theoretical justification for why weight decay can help mitigate the bias. Specifically, we consider the logistic regression example from Section 2. The gradient of the logistic loss with L2 regularization becomes

$\nabla \ell_{\mathbf{w}} (\mathbf{x}_i, y_i) = (\sigma(\mathbf{w}^\top \mathbf{x}_i) - y_i)\cdot \mathbf{x}_i + \lambda\mathbf{w}=(\sigma (\mathbf{w}_S^\top \mathbf{x}_i) - y_i)\cdot \mathbf{x}_i + \lambda(\mathbf{w}_S +\mathbf{w}\_{S'})$

Where $\mathbf{w}\_S$ and $\mathbf{w}\_{S’}$ are the projections of $\mathbf{w}$ onto the span of the target datapoints ($S$) and its complementary subspace ($S’$). The gradient (as in Section 2) restricts the space of updates to those in $S$. However, due to regularization, $\mathbf{w}_{S’}$ is driven to 0, so any planted bias in $S’$ collapses with regularization.

W2c. Debiasing the target dataset: We fully agree with the reviewer here—in fact, we believe that (a) according to our paper, debiasing the target dataset does not completely eliminate bias transfer, and (b) “fully” debiasing a dataset is often intractable, as the reviewer notes. This trend holds more generally: while several mitigation strategies can help reduce bias transfer, post-hoc approaches typically do not fully remove it (and often are not infeasible without sacrificing something performance). We thus believe it is critical to prevent such biases at their source by more carefully choosing our pre-trained data in the first place.

Q1. Adversarial Pre-training: Adversarial pre-training might be an interesting strategy: indeed, previous work has found that adversarially robust pre-training models do seem to transfer better (see [1]). However, these biases are spurious correlations which (at their core) are just features. It’s not clear that adversarial pre-training will be able to reduce reliance on such features.

Q2. Weight Decay in the Fixed Feature Regime: In this setting, we apply regularization to the linear layer only.

[1] https://arxiv.org/abs/2007.08489

Hi, thank you for your response!

• Any $\mathbf{w}$ can be written as \mathbf{w}\_S + \mathbf{w\}_S' where $\mathbf{w}\_S$ is $\mathbf{w}$ projected into the span of the training examples $S$ and \mathbf{w\}_S' is $\mathbf{w}$ projected onto the associated null space $S'$. Then $\mathbf{w}^\top x_i = (\mathbf{w}\_S + \mathbf{w}\_{S'})^\top x_i = \mathbf{w}_S^\top x_i$ because $\mathbf{w}\_{S'}^\top x_i = 0$ (since $S'$ is the null space of the span of the training examples).
• Note that $||\mathbf{w}\_S + \mathbf{w}\_{S'}||_2^2 = \mathbf{w}\_S^\top \mathbf{w}\_S + 2\mathbf{w}\_S^\top \mathbf{w}\_{S'} + \mathbf{w}\_{S'}^\top\mathbf{w}\_{S'} = ||\mathbf{w}\_S||_2^2 + || \mathbf{w}\_{S'}||_2^2$ since $\mathbf{w}\_S^\top \mathbf{w}\_{S'} = 0$. Since $\mathbf{w}\_{S'}$ does not appear in the logistic loss, any solution $\mathbf{w}$ where the component $||\mathbf{w}\_{S'}||_2^2 > 0$ will be suboptimal. The problem is convex, so SGD will lead to an optimal solution which means $||\mathbf{w}\_{S'}||_2^2 = 0$.
• We have just added an experiment in the uploaded revision of Appendix A.5 which considers a toy logistic regression example. We show that the component of the initial weight vector projected onto the null space of the training examples (i.e., $||\mathbf{w}\_{S''}||$) degrades to 0 with weight decay (but is unchanged without weight decay). Measuring $||\mathbf{w}\_S||$ and $||\mathbf{w}_{S'}||$ for our neural networks in the main experiments is much more involved (one initial strategy might be to study the SVD of the weights with weight decay). We would be happy to pursue this for the final version.

Please let us know if you have any further questions.