On the Foundations of Shortcut Learning

Thank you for your questions and feedback!

...decreasing the “availability” of the background image by setting its entropy to 0 (black color) is directly biasing the classifier to learn from the remaining available, foreground signal where you know the label is highly predictive.

We want to highlight the fact that our availability manipulation leaves the feature-label statistics (the relationship between background and label, across the dataset) unchanged, and changes only the representation of the background feature that is present within the image. In other words, for a given predictivity setting, the mutual information of the background to label, I(Background; Y), and foreground to label, I(Foreground; Y), is constant regardless of availabilities. We independently manipulate predictivity and availability with the goal of examining the contributions of these two distinct factors on a model’s feature reliance. The general pattern we observe in the experiment you mention – that decreasing the number of patches of background decreases background preference – is indeed not unexpected, but by systematically manipulating these factors, we can characterize quantitative patterns in the way these contributions interact.

In section 3, the two main parameters that govern availability are alpha_i and eta_i = 0, so alpha_i remains...explain your intuition why alpha_i controls “how easily the feature can be extracted, or leveraged, from inputs”? ...

With our controlled data setup, we manipulate hypothesized notions of availability, and test whether models are indeed sensitive to them. We hypothesized that $\alpha_i$ (scaling factor) is a notion of availability which pushes around models’ feature use, and find empirically that this is indeed the case (hence the statement).

I don’t fully understand why your definition of reliance is useful...why taking a sign difference of z_s and z_c is reasonable.

Thank you for this comment. It led us to realize a notational mistake (missing absolute values) in our reliance definition in Section 3, which we’ve now updated (all figures and results remain correct and are unchanged). Please refer to the updated equation in the updated paper PDF. To answer your question, our "reliance" measure quantifies the extent to which a model uses feature $z_s$ as the basis for its classification decisions – intuitively, the extent to which the model decisions match those which would be generated by a classifier which relied exclusively on $z_s$ versus by a classifier which relied exclusively on $z_c$. To capture this notion, we take a difference between a template match with each of these classifiers. By construction of the Gaussian dataset, a value’s sign is its feature label.

The novelty of this work relative to related work, which I am not very familiar with, is unclear to me and not discussed in the paper. Could you please delineate your work, and clarify your novel contributions, or point me to appropriate positions in the paper which describe this, if missed?

We summarize our contributions at the end of the Introduction, and situate these contributions with respect to the literature in both the Intro and Related Work sections – see especially the paragraph beginning, “The literature examining feature use…”. Briefly: we explicitly introduce “availability’’ to refer to the factors which influence the likelihood that a model will use a feature more than a purely statistical account would predict. We systematically manipulate both predictivity and availability and give a quantitative account of their interaction, characterizing conditions giving rise to shortcut learning. In experiments with controlled and naturalistic data, we identify types of availability that drive model behavior. We find that availability-driven shortcut bias is greater for nonlinear than linear models, and that model depth amplifies bias. We also provide a novel theoretical analysis, consistent with the empirical results, indicating the inevitability of a shortcut bias for a ReLU network compared to a linear one. Further, we show that standard vision architectures used in practice are sensitive to availability manipulations. Please let us know if any further clarification would be helpful!

I would be interested in more complex synthetic scenarios than the one outlined. Did you think about these?

Yes, definitely. We focused on a setting in which we could explicitly control notions of availability which it’s reasonable to hypothesize could drive feature reliance in real models, and considered a variety of types of data, from vector inputs to synthetic images to naturalistic images. We found that across several different notions of availability, models exhibited a shortcut bias, which is not what a purely statistical account would predict. We are very interested in future work to surface even more types of availability that drive the behavior of models used in practice.

Thank you for these questions and comments!

The theoretical analysis makes approximations to obtain tractability (small covariance, quadratic approximation of ReLU kernel). It would strengthen the analysis to discuss the impact of these approximations. Are the core insights still valid without them?...The quadratic approximation for the ReLU kernel greatly simplified the theoretical analysis. Could you provide some empirical verification that this does not alter the core findings? For example, compare the availability bias for the true ReLU kernel versus the quadratic approximation.

Thank you for raising this point. As mentioned in the Global response above, based on your comment, we ran some simulations to compare the behavior of the ReLU kernel and its quadratic approximation. The simulations work with a real (as opposed to asymptotically small) covariance matrix. The results show a good match between the two kernels, as well as with the purely analytical predictions in Figure 5 of the main paper. We added these results and the Python code used to generate them to Appendix I of the updated Supplementary Material PDF. Due to the limited time we had for the rebuttal, we were able to do a minimal set of experiments to address your concern. We will expand and add more results to this section in the camera-ready version.

The proposed notion of availability is intuitively reasonable but remains a hypothesis. Additional ablation studies that directly validate the choice of manipulations affecting availability could make this more concrete. The measures of reliance and shortcut bias, though well-motivated, are indirectly quantified through alignment with idealized classifiers. More analysis could be provided to justify these metrics over alternatives.

Just to make sure we’re understanding: we do directly manipulate factors which we hypothesize affect availability. We observe an effect of these factors on shortcut learning, thereby obtaining support for our hypothesis. If the worry is that we don’t rule out other factors that affect availability, we agree and would not want to claim that we’ve identified all such factors.

Have you experimented with any architectural manipulations beyond depth and activation functions that could potentially reduce shortcut bias?

Things like skip connections, normalization layers etc. Exploring this could provide practical guidelines. We are also interested in these questions and plan to explore them in future work.

For the image experiments, it would be interesting to also show the impact of availability manipulations on a model pretrained on Imagenet. Does pretraining affect sensitivity to shortcuts?

Good question! In SI Figure B.9, we found that ImageNet-pretrained models are still sensitive to availability manipulations, though there is a difference from models trained from scratch.

The measures of reliance and bias involve probing a model in latent space. For vision experiments, could you provide some visualizations of model decisions before/after availability manipulations to build intuition?

In experiments shown in Figures 6 and B.9, we had found that the accuracy of vision models at classifying Waterbirds images by the core feature (bird) is influenced by non-core feature (background) availability. In Figure B.10, for a particular background availability manipulation, we compare attribution maps for a ResNet18 when trained on one level of availability manipulation versus another, and see an expected difference in focal point.

Thank you for your questions and comments!

The main observation that the model depth and non-linearity increase bias towards the shortcut features is intuitive. Both depth and non-linearity allows the model to learn a rich representation.

On the other hand, one might predict that the rich representation would enable models to go beyond shortcut features (rely on them less). Of course, this is not what happens, but we believe a few intuitive accounts are possible here, making the empirical verification worthwhile.

Theoretical analysis using NTK is problematic as they don't learn feature and thus cannot necessarily explain model's preference towards the shortcut feature.

Although we agree that learning has a big role in creating shortcut features, there is also opportunity/potential for shortcut features inherent in the architecture itself. Our theory focuses on the latter via the NTK.

It would be interesting to see experiments with the vision transformers.

We agree that in future work, it will be interesting to study shortcut learning in domains beyond the ones we have focused on here, and to add experiments with ViTs.

Do you think the observations will be similar for other non-linearities?

We do! In Figures 3B and B.1, we find that, empirically, models with a Tanh activation function induce a larger shortcut bias than a linear counterpart.

Thank you for your questions and feedback!

W1. The notion of availability is very closely connected with the concept of simplicity of neural networks. The simplicity bias has been pointed out as a cause of non-robust learning. The paper gives multiple references (e.g. Shah et al., 2020, etc.) for works dealing with simplicity bias, but they are not discussed in detail. The authors must clarify what is the difference between the notion of availability presented in this work, and the simplicity bias previously introduced. What new observations does the proposed framework bring?

We pointed to Shah et al. (2020), Hermann & Lampinen (2020), and related papers as including motivating examples of cases in which a network may prefer one feature over another even when the two are equally predictive. Shah et al. specifically characterized features as simple or complex in terms of their corresponding decision boundary (e.g. linear or piecewise), and found that models will learn the linear decision boundary, and can ignore complex features. Our paper is different from this earlier work in several ways:

• Our study of availability is not restricted to cases where features are linearly vs nonlinearly related to task labels. For example, we study cases in which both features are linearly related to labels, but nonetheless one is more available than the other.
• Unlike the prior studies, we study how models trade off availability with predictivity, which is critical to understanding the conditions under which shortcut learning is likely to occur.
• We study how feature reliance changes as a function of (continuous) degree of availability, not just model preference given features which differ in kind.

W3. For image datasets: “a Bayes optimal classifier is not comparably sensitive to the predictivity of the non-core features” How do we know this? What experiments give this conclusion? W3.2 Also, how is the Bayes optimal classifier created in the case of real image datasets (WaterBirds, CelebA)?

The optimal classifier is based on the assumption of ground-truth knowledge of the true bird class and the true background type. As such, it represents an upper bound on accuracy. However, our point in including the Bayes optimal classifier was to examine the small modulations in accuracy that are due to non-core feature predictivity.

Could the curves given by making an intervention on the background, be use to benchmark the robustness of different method? E.g. more robust models would have curves that are less sensitive to interventions on the background. Whould this offer any additional insight as opposed to comparing the accuracy of the model on balanced data, without foreground-background spurious correlations?

That’s a good question which would be interesting to study in future work!

Thank you for your questions and comments!

The theoretical analysis is limited to a single hidden layer MLP which does not reflect the type of architectures used in practice. It is difficult to conclude whether this also holds for other types of architectures like Transformers or CNNs.

We agree that our theoretical setup uses a simple architecture, but a precise analysis of the bias even in this setting is highly non-trivial (please check the appendix for a detailed coverage). Moreover, our work is the first that presents any theoretical analysis of the predictivity/availability tradeoff, for which we had to develop a novel proof strategy (as opposed to an incremental refinement to existing results). Our analysis delivers a precise characterization of availability and predictivity which we believe to be a novel contribution.

Experiments are limited to supervised classification settings. Self-supervised training or other tasks like object detection would be interesting to consider in the context of shortcut learning.

We agree that studying the interaction of availability and predictivity in these other settings would indeed be very interesting! For example, it’s been observed that self-supervised vision models are still texture-biased (e.g., Hermann et al. 2020), though the statistics of ImageNet are unknown and so controlled study is needed. Nevertheless, as per the response above, we believe the analysis in the supervised setting already requires a good deal of non-trivial work; we look forward to expanding to other settings in future work.

The paper points out that availability can explain (some) occurrences of shortcut learning. The availability can be determined by looking at the training data distribution, but could one also identify these shortcuts stemming from availability by directly looking at the train neural network weights? For example, if multiple filters in the same CNN layer share the same pattern?

Your suggestion is clever and a very sensible idea to try. The redundancy you describe is exactly what we might expect to see for, say, texture patches. It would be a bit challenging to quantify “share the same pattern”, but we agree this idea warrants further consideration in future work.