RaVL: Discovering and Mitigating Spurious Correlations in Fine-Tuned Vision-Language Models

We thank Reviewer ONXK for reviewing our work and providing helpful feedback.

[Q1] Assumption of region-level spurious correlations.

We thank the reviewer for raising this point. In line with prior works in vision-only and vision-language settings [1,2,3], RaVL is specifically designed to surface and mitigate fine-grained, region-level spurious features. As we acknowledge in the Appendix (Lines 779-787), we agree with the reviewer that there are some sources of spurious signal that do not manifest in this way (e.g. global, image-level features like texture or background information as mentioned in Geirhos et al.). However, we focus explicitly on region-level spurious features due to the fact that such spurious correlations have been shown to negatively affect model performance in many real-world, practical settings; some notable examples include image-level markings in dermoscopic images [8], medical devices in radiographs [9], and text markers in medical images [10]. As a result, we believe our work effectively complements work done by Geirhos et al. and others by targeting an important yet underresearched category of spurious correlations (namely, region-level features) that have been shown to contribute to significant model performance gaps.

[Q2] Visualizations.

Thank you for this suggestion. We refer the reviewer to the PDF provided with the general response. In Rebuttal Figure 2, we demonstrate the utility of our mitigation approach by providing qualitative visualizations using GradCAM. Visualizations show that after the RaVL mitigation procedure, the VLM $\mathcal{M}_{new}$ more precisely relies on core features.

We also note that in order to quantitatively evaluate the extent to which the VLM understands fine-grained region-level information, we introduced the Region Overall and Region Worst Group metrics in Section 4.2. Results in Table 3 show that RaVL contributes to improvements on these metrics, suggesting better understanding of region-level information.

[Q3] Use of feature maps.

We agree that feature maps serve as a rich source of signal and can be leveraged for identifying and addressing spurious correlations. Indeed, several works have explored this direction by identifying neural features that influence classification decisions and then using humans in the loop to annotate feature maps as core or spurious [2,11]. Although this approach was shown to work well, a key consideration is the need for human supervision in order to interpret feature maps and distinguish between core and spurious features.

In our work, we used ROIAlign and region-level embeddings rather than feature maps for the following reasons:

• Enable automated analysis: Our approach is designed to operate without the need for human supervision, enabling generalizability to datasets from diverse domains (e.g. medical image data). Using region-level embeddings enables us to perform automated detection and mitigation of spurious correlations via our novel clustering approach and our region-aware loss function. On the other hand, using feature maps will likely require human supervision, as shown by [2] and [11].
• Perform dimensionality reduction: Whereas feature maps may be large in size with many channels, region-level embeddings exhibit low dimensionality while still capturing relevant semantic information. With regards to the reviewer's point about the clustering stage, using softmax-normalized region-level embeddings can enable more efficient and accurate clustering in comparison with using large feature-maps that encompass diverse features.
• Target fine-grained spurious correlations: As discussed in [Q1], our goal is to specifically detect and mitigate fine-grained, region-level spurious correlations. Using local region-level embeddings rather than global feature maps helps us accomplish this goal.

[Q4] Clustering approach.

We used K-medoids clustering for the following reasons:

• Robustness to outliers: When compared to a standard K-Means clustering approach, K-Medoids is less sensitive to the presence of outliers due to the fact that cluster centroids are medians rather than means. Robustness to outliers is important in our setting, since region-level embeddings represent visually-diverse features and there is likely to be significant variation across samples.
• Ability to operate with custom distance functions: Whereas K-Means traditionally minimizes the Euclidean distance between the centroid and the samples in a cluster, K-Medoids can be implemented with custom distance functions. Due to the fact that the original VLM $M$ is trained with respect to cosine distance, we implement K-Medoids with a cosine distance metric in this work in order to effectively model the patterns learned by $M$.

[Q5] Batch Size.

In order to train model $M_{new}$, we utilize a combination of the CLIP loss $L_{CL}$ and our region-aware loss $L_R + L_A$. Both loss functions are contrastive in nature and require a large set of negative samples in a batch in order to learn useful representations. As a result, our framework is sensitive to batch size; this is expected behavior [12] and is consistent with a long, existing line of work on contrastive learning methods (e.g. [7]). We note that for all results reported in Table 3, we keep batch size consistent across methods in order to enable fair comparisons. Below, we provide a brief batch size ablation on a single COCO evaluation setting. As expected, significant reductions in the batch size reduce efficacy of contrastive learning.

We again thank Reviewer ONXK for their review of our manuscript and their positive overall assessment of our work. We hope that the above responses adequately address all concerns.

We thank Reviewer LC5I for reviewing our work and providing helpful feedback.

[Q1] Computational complexity. “Looking into the regional features may make the approach computationally expensive. Computational complexity analysis is missing in the paper.”

We refer the reviewer to General Response [Q1], where we address this point.

[Q2] In-the-wild experiments. “A lot of experiments were done under controlled settings. More evaluations in the wild would make the results stronger.”

Thank you for this suggestion, and we agree about the importance of real-world validation. However, we note that it is challenging to quantitatively evaluate the accuracy of discovery and mitigation approaches, since the ground-truth spurious correlations learned by a VLM are typically unknown. As a result, we opted to use controlled evaluations, which artificially induce spurious correlations and then quantitatively evaluate the extent to which the correlation can be detected and mitigated. Importantly, using controlled evaluations enables us to perform large-scale, quantitative evaluations of RaVL without using humans in the loop. We believe that this procedure is essential before scaling to unlabeled real-world datasets.

In addition to these controlled quantitative evaluations, our original manuscript provided qualitative results from several in-the-wild evaluations (Section 3.3 and Figure 3). In response to the reviewer's suggestion for expanding these analyses, we have extended this analysis with additional qualitative evaluations. We refer the reviewer to the PDF provided with the general response above. In Rebuttal Figure 1, we show the following:

• For the OpenCLIP ViT-L/14 model, RaVL surfaces a feature cluster consisting of green plants and fences. We observe a performance gap of 18.3 points between images with class label "outdoor chicken coop" that contain the RaVL-identified feature and those that do not contain the feature. This suggests that the OpenCLIP ViT-L/14 model can better classify an "outdoor chicken coop" scene when green plants and fences are present.
• For the CLIP ViT-B/32 model, RaVL surfaces a feature cluster consisting of people. We observe a performance gap of 24.3 points between images with class label "pub (indoor)" that contain the RaVL-identified feature and those that do not contain the feature. This suggests that the CLIP ViT-B/32 model can better classify a "pub (indoor)" scene when people are present.
• For the OpenCLIP ResNet-101 model, RaVL surfaces a feature cluster consisting of chairs. We observe a performance gap of 23.3 points between images with class label "restaurant patio" that contain the RaVL-identified feature and those that do not contain the feature. This suggests that the OpenCLIP ResNet-101 model can better classify "restaurant patio" scenes when chairs are present.

[Q3] References. “A small error in writing, line 259 says “are designed for unimodal settings” - ref [50] is designed for multi-modal settings.”

Thank you for raising this point. In Line 259, the phrase "designed for unimodal settings" is intended to refer solely to Distilling Failures, George, and Domino (referenced in Line 257). We agree with the reviewer that Spurious-Aware Detection (reference [50]) is not unimodal, and we will be sure to improve the clarity of this sentence in the final version of our manuscript.

[Q4] Extending beyond VLMs. “Have you tried the approach on other settings, other than fine-tuned VLM? For example, spurious correlations can happen in other vision classification models as well.”

Thank you for this suggestion, and we agree that spurious correlations can occur in many types of models. In this work, we focus specifically on the setting of fine-tuned vision-language models because fine-tuned VLMs (i) are becoming increasingly commonplace, particularly in domain-specific applications like medicine, (ii) are likely to be trained on small domain-specific datasets, preventing models from gaining the robustness benefits associated with web-scale training and increasing the likelihood of learning spurious correlations, and (iii) are an underresearched yet important problem setting. In order to address spurious correlations in the fine-tuned VLM setting, RaVL assumes the existence of paired language; consequently, RaVL utilizes text embeddings for both discovery (RaVL Stage 1) and mitigation (RaVL Stage 2).

In the context of vision-only classification models where the language modality is not present, our discovery approach can be modified by computing the image score distribution vector $s_{I_i}$ and the region score distribution matrix $S_{R_i}$ using softmax-normalized class logits. The mitigation approach can be modified by framing the region-aware contrastive loss function as a cross entropy loss between region-level logits and class labels. We aim to explore these directions in future work.

We again thank Reviewer LC5I for their review of our manuscript and their positive overall assessment of our work. We hope that the above responses adequately address all concerns.

Thanks again for the rebuttal.

This paper is based on the assumption that, “a model M that has learned a spurious correlation between an image feature e_a and a textual attribute y will demonstrate low zero-shot performance on (i) images in D_V with label y without the feature e_a and (ii) images in D_V with other labels Y \ {y} with the feature e_a. ” (e.g., see lines 124 - 127 in the paper)

I have one more question on the paper. For some “true” image features, when the following are true, they may also appear to have the property (e.g., see lines 124 - 127 in the paper) mentioned above.

1. The “true” image features only appear in a subset of the class y. This can happen because not all images in the same class have the same features.
2. The training data and test data are sampled differently: the “true” features only appear (or appear more often) in the training data with class y, and appear less often in the test data with class y.

How do you distinguish between spurious image features and the “true” image features with different appearing frequencies in training and test data?

Thanks for the response. My concerns have been well addressed.

We thank Reviewer ACX1 for reviewing our work and providing helpful feedback.

[Q1] Computational complexity. “There is a lack of computational complexity analysis. To detect fine-grained spurious correlations, RAVL needs to segment images into multiple regions, cluster all the regions per class, and calculate the contributions of each cluster to mispredictions. Therefore, the computational complexity seems high. It would be better to analyze the time cost of the proposed method.”

We refer the reviewer to General Response [Q1], where we address this point.

[Q2] Region-level labels for evaluation. “How to obtain region-level labels when constructing the evaluation dataset (Line 220)?”

In order to evaluate RaVL, we create 654 evaluation settings from two domains: (1) synthetic data (MNIST and FashionMNIST) and (2) real-world data (COCO). Each evaluation setting includes an evaluation dataset with images $I_i$, class labels $y_i$, region bounding boxes $R_i$, and region-level labels $L_i$. We obtain region-level labels as follows:

• For our synthetic data settings, we construct each image $I_i$ to include four equally-sized quadrant regions $\mathcal{R}_i$. We randomly select one quadrant to include the core object (i.e. digit or fashion item) associated with the class label $y_i$. In some images, we randomly select another quadrant to include the pre-defined spurious feature $e^{eval}$, which is a red rectangle in this case. Since these images are synthetically generated, we have a priori knowledge of the features in each quadrant, which we use to generate the region-level label set $L_i$.
• For our real-world settings, the set $\mathcal{R}_i$ consists of the ground-truth bounding boxes annotated in COCO and $L_i$ consists of the associated object labels. We note that recent advances in open-set object detection (e.g. Recognize Anything) can enable this procedure to be extended to other datasets even if ground-truth bounding boxes and labels are not available.

We additionally emphasize here that the RaVL discovery and mitigation approaches do not require region-level labels, and users who wish to use RaVL do not need access to region-level labels. Here, we exclusively use region-level labels within our evaluation framework in order to quantitatively assess performance.

[Q3] Hyperparameter selection. “How to determine the hyperparameters used in the spurious correlation discovery and mitigation stages?”

Hyperparameters for the discovery stage (RaVL Stage 1) are selected as follows:

• Number of clusters: RaVL includes a clustering step to identify groups of visually-similar regions. We do not manually set this hyperparameter; rather, we leverage an automated approach for selecting the optimal number of clusters. For each evaluation setting, we sweep all cluster sizes ranging between $|\mathcal{Y}| * 2$ and $|\mathcal{Y}| * 5$ where $|\mathcal{Y}|$ represents the size of the class label set; we then select the optimal number of clusters using Silhouette scores. We select these bounds to be larger than the class label set size by several multiples in order to ensure that clusters adequately separate distinct features. Prior works have also utilized overclustering approaches for this objective [5,6]. Users can adjust the bounds based on their composition of their dataset; for instance, complex datasets with diverse features may require a larger range.
• Threshold for pruning cluster influence scores ($\tau_l$): In order to identify candidate image features that directly contribute to classification errors, we prune all clusters with low cluster influence scores below a threshold $\tau_l$. We set $\tau_l$ to 0.25 in this work. This hyperparameter was determined empirically based on experiments on a small set of validation data.

Hyperparameters for the mitigation stage (RaVL Stage 2) are selected as follows:

• Contrastive loss temperature $\tau$: We set the contrastive loss temperature to a default value of 0.07 based on prior work [7].
• Loss weighting factor $\lambda$: The loss weighting factor $\lambda$ balances the tradeoff between the original CLIP loss function $L_{CL}$ and our novel region-aware loss function $L_{R} + L_{A}$. We set $\lambda$ to 0.8 in this work, which upweights the CLIP loss function $L_{CL}$. This weighting factor ensures that the model $\mathcal{M}_{new}$ learns accurate global image-text relationships without capturing fine-grained spurious correlations. We determined this hyperparameter based on experiments on a small set of validation data. We additionally note here that the "Upsampled FT" baseline in Table 3 is an ablation where $\lambda$ is set to 1 (and the region-aware loss function is not utilized); this demonstrates the efficacy of our loss function.

We note that like all hyperparameters, these values could likely be further optimized; however, even without extensive hyperparameter tuning, we achieve competitive results. Additional details on hyperparameters are provided in Appendix B, C, and D.

We again thank Reviewer ACX1 for their review of our manuscript and their positive overall assessment of our work. We hope that the above responses adequately address all concerns.