Trained Models Tell Us How to Make Them Robust to Spurious Correlation without Group Annotation

Dear Reviewer 9oxU,

Thank you for your valuable feedback and acknowledgment of the effectiveness of our work. We appreciate your attention to various aspects of the framework. Below, we aim to clarify confusing points and provide a comprehensive discussion and analysis on the questions you have raised.

---

Weakness 1

Compared to Sampling Method in JTT

Please refer to Section F.2, Comparison of High-Loss and Misclassified-Sample Selection.

Having a hyperparameter ($k$) to control the number of selected samples from high loss samples provides more flexibility to LS compared to JTT. There is a tradeoff between the purity and the number of selected high-loss samples: our observations (Figure 2 (3 in the revision)) demonstrate that minority samples are more commonly found among those with high loss in the ERM model. As the number of selected high-loss samples increases, the proportion of minority samples among them decreases. However, selecting more samples could improve the overall training for those samples. The flexibility to choose among various numbers of selected samples allows EVaLS to find an optimal point in this tradeoff. Sensitivity to the selection of $k$ is shown in Figure 9 of the paper. Choosing only misclassified samples cannot handle this tradeoff effectively, particularly when the number of misclassified samples is either too high or too low. See Table 11 for performance comparison.

Compared to SELF

SELF uses a different criterion for selecting minority samples. It relies on the disagreement between the outputs of a trained model and an early-stopped model, leveraging the observation that models behave differently with respect to spurious correlations during the final and early stages of training. In contrast, we base our approach on the observation that minority samples are more prevalent among high-loss samples than low-loss ones. Both approaches yield effective results. However, loss-based sampling (EVaLS-GL) outperforms disagreement-based sample selection by SELF in 4 out of 5 benchmarks, under a similar level of group supervision. Moreover, as stated in L371–372 (390-91 in the revision), SELF requires additional training information (an early-stopped model), whereas loss-based sample selection does not (see methods with $\star$ in Table 1). This makes SELF less feasible in many cases where the model or training data is large, and early-stopped checkpoints are unavailable.

Before proceeding, we want to clarify some points that may have been misunderstood:

1. EVaLS-GL, which utilizes group labels, achieved better results than EVaLS

This is logical and not a contradiction. For every spurious attribute that its exact group annotations are available, using the annotations to create a balanced (re)training dataset or for model selection is the optimal approach to ensure robustness to correlation shifts in that spurious attribute. This can be observed by comparing DFR and GDRO with other methods in Table 1. In other words, it is not expected that EVaLS, which does not use group-level data, will surpass models that leverage this additional information for robustness to those groups. Nonetheless, as we state in our Abstract (L32–34), the fact that EVaLS achieves near-optimal results under these constraints demonstrates its robustness and practical utility, especially since it can be used in scenarios where the aforementioned group annotations are not available.

This property makes EVaLS applicable to scenarios where the spurious correlations a trained model relies on are unknown during (re)training. Since no solution before us has solved this problem—prevalent across machine learning models in real-world applications—EVaLS represents a significant advancement in improving model robustness against spurious correlations. Achieving near-optimal accuracy without group annotations, while also addressing robustness to unknown spurious correlations, makes EVaLS an effective solution for real-world scenarios where some spurious correlations are known and others are not.

To more effectively illustrate this point, we proposed Dominoes-CMF. In Figure 4(b), you can see that EVaLS achieves higher worst-group accuracy across all 8 groups (as demonstrated in Figure 4(a) and explained in Section 2.2, Robustness of a Trained Model to Unknown Shortcuts) compared to EVaLS-GL, which requires group annotations for model selection, and DFR, which uses group annotations for both retraining and model selection.

2. LS appears to require group labels for hyperparameter tuning, while EV utilizes environment labels generated through Environment Inference for Invariant Learning (EIIL) as a stand-in for group labels

Environments are not considered equivalent to groups. They are diverse subsets of the data. In our framework, what we require from environments is that they depict group shifts (see Section 3.2, L214–215 (259-260 in the revision) and L244–258 (263-269 and 285-292 in the revision); Abstract, L26–30; Introduction, L73–74 and L77–80). For example, majority samples of a class may be more prevalent in two environments, but this is not problematic as long as there is a notable shift in the percentage of minority samples between them (see Table 2 for group shifts in the obtained environments across datasets). Consequently, the performance of EVaLS demonstrates that LS does not necessarily require group labels for hyperparameter tuning. Instead, combining it with worst-environment accuracy, derived from environments with the above properties, can achieve near-optimal accuracy.

Weakness 2

First, please refer to point 2 above for clarification. Results for GDRO+EIIL (Creager et al., 2021) are shown in Table 1. It can be seen that it underperforms other methods in the table in most comparisons, indicating that their approach is not as effective as others. Specifically, GDRO+EIIL underperforms GDRO, which utilizes group annotations, as expected. You can find more information about results on spurious correlation datasets below.

---

Question 1

First, please refer to point 1 above for clarification. When group annotations are available, we expect methods that utilize them to outperform others. So, it is expected that EVaLS-GL outperforms EVaLS when exact group annotations are available.

---

Question 2

Let’s first review datasets with spurious correlations in Table 1 and 6, and then compare the results of EVaLS and EVaLS-GL.

On Waterbirds with available group annotations, EVaLS-GL outperforms EVaLS by $1$%.

Regarding CelebA, EVaLS achieves a slightly higher WGA ($0.7$%). It is known (L1449-1453 (1455-1457 and 1470-1471 in the revision)) that annotation noise is present in this dataset, as evidenced also by previous research (e.g., [*]). Consequently, group annotations in CelebA are not completely accurate. EVaLS bypasses this issue by utilizing inferred environments instead of relying on noisy group annotations.

On datasets with two spurious attributes—one known and one unknown (UrbanCars and Dominoes-CMF)—utilizing inferred information rather than available annotations provides a solution for fairly achieving robustness to all spurious attributes. Based on the results of both experiments, EVaLS outperforms EVaLS-GL by an average margin of $1.89$%.

Conclusion

If group annotations for a known spurious correlation are available, using them is more effective for robustness to the spurious correlation than utilizing environments. However, environment-based validation can also achieve near-optimal accuracy (L517–518 (520-521 in the revision)). If group annotations are noisy, this may result in lower performance when utilizing them. In such cases, there is a trade-off between using noisy group annotations and inferred environments.

If a spurious correlation that a trained model relies on is unknown during training, using inferred environments is useful for robustness, whereas methods that require explicit annotations are not applicable.

Combining the above points, for model selection:

• If the goal is to ensure robustness to a known spurious correlation with ground-truth annotations, using group annotations is an effective solution (depending on the quality of the ground-truth annotations).
• If the goal is to generally improve the robustness of a trained model to both known and unknown spurious correlations, using inferred environments is an effective solution.

[*] Speth et al., Automated Label Noise Identification for Facial Attribute Recognition, CVPR, 2019.

Dear Reviewer 3v9E,

First, we appreciate your acknowledgment of the importance and impact of results that demonstrate robustness in scenarios where some or all spurious correlations are unknown. We believe this is a highly influential step in the field.

---

Questiion 1

WGA of AFR on Dominoes-CMF are as follows:

A similar improvement from EVaLS to EVaLS-GL is observable between AFR adn AFR+EIIL.

---

Question 2

A dataset that has multiple spurious shortcuts, each highly correlated with the target independently, requires a large amount of data to capture both spurious correlations. Constructing such a dataset with proper quality, along with training and evaluating models on it, requires some time to accomplish. We are working on it and will inform you.

---

Minor points

Thank you for pointing out the slips and other aspects that may cause potential confusion in our paper. We will correct the slips and try to address your suggested improvements in the revision.

Thank you for the reply. Clearly EVaLS outperforms AFR and AFR+EIIL in the Dominoes-CMF, and it would be a great addition to Figure 4 (b) and Table 6. Is it possible to have them updated in the pdf?

Regarding the dataset, I understand that it is not a trivial task for a rebuttal process.

Dear Reviewer 3i2u,

We appreciate your positive feedback on the effectiveness of our scheme and the clarity of our paper.

---

Weakness 1

EVaLS is a scheme designed to ensure annotation-free robustness of trained models against spurious correlations (L74–76). In this scheme, modules could be replaced while completely maintaining the overall framework and contribution. As shown in Appendix F, we have conducted comprehensive ablation studies to evaluate these components:

• Loss-based sampling alternatives: Examined in Sec. F.1 and F.2.
• Replacing EIIL with other environment inference techniques: Detailed in Sec. F.3.

It is important to clarify that our work has not proposed using or combining previous methods as a novelty. Instead, our contributions lie in demonstrating the following points (L106–120):

1. Contrary to prior approaches (Appendix A, L728–750), we show that exact group annotations are not a prerequisite for robustness to spurious correlations, and identifying environments with group shifts (Sec. 3.2) proves effective for model selection.
2. When ground-truth annotations are unavailable or spurious correlations are unknonw, leveraging the trained model’s learned representations can effectively enhance robustness to spurious correlations (Figure 4(b) and Table 1—UrbanCars).
3. Loss-based sampling (Sec. 3.1) is not only an effective method for ensuring robustness to group shifts (compared to other methods such as those in JTT and SELF; see EVaLS and EVaLS-GL in Table 1), but it is also supported by a theory of data balancing with general assumptions (Sec. 3.3 - Theoretical Analysis).

---

Weakness 2

Theoretical Analysis

We do not assume that the losses on the majority and minority samples follow Gaussian distributions. As we state in L291-292 (311-312 in the revision), “We assume a general assumption that in feature space (output of $g_\theta$), samples from the minority and majority of a class are derived from Gaussian distributions,” where $g_\theta$ is the feature extractor (L183-184 (208-209 in the revision)). To our knowledge, this is one of the most general assumptions in the theoretical analysis of majority-minority dynamics. It does not even assume different dimensions for core and spurious features (as [6] or [7] do to formulate spurious correlation for other goals and analysis). By this assumption, the Gaussian distribution of our logits (not losses) is derived (Lemma D.1). As mentioned in L294-296 (314-316 in the revision), the order of samples in logit space and loss space is monotonic within each class, and thus the tails are equivalent in these two spaces. Thus, loss-based sampling is backed by our proposed theoretical analysis.

Moreover, please note that, as we state in L285-287 (306-308 in the revision) and as observed in Proposition D.1, it is not obvious and always the case that loss-based sampling could create two group-balanced sets of data. Condition (i) (Eq. (5) in L924) and Condition (ii) (L926-943) are necessary and sufficient conditions for this purpose. Whether these conditions are met depends on the classifier, distance between distributions and their variances in feature space (and consequently in logit space), and the amount of spurious correlation.

Additionally, as stated in L310-311 (330-331 in the revision), we have added a practical justification in Appendix D.2 to show that the conditions of Proposition 3.1 (and more completely in D.1) are met in practice.

Comparison to [2]

[2] proposes SELF, which has been compared to our approach in Table 1. It uses a different criterion for sample selection, which, based on results in Table 1, is less effective than our loss-based sampling. More precisely, loss-based sampling (EVaLS-GL) outperforms disagreement-based sample selection by SELF with a similar level of group supervision in 4 of 5 benchmarks. Moreover, As it is stated in L371-372 (390-391 in the revision), SELF requires training information (an early-stopped model), in contrast to our approach and methods like DFR and AFR (indictated by $\star$ in Table 1).

[6] Nagarajan et al., Understanding the failure modes of out-of-distribution, ICLR, 2021.

[7] Sagawa et al., An investigation of why overparameterization exacerbates spurious correlations, PMLR, 2020.

Weakness 3

Note that for last layer retraining and model selection data, we follow the same scheme as in DFR [1] (Please see their Sec. 6 “Feature Reweighting Improves Robustness,” Hyper-parameters paragraph), which is also used in future works (see Appendix D Training Details in SELF [2]). The important point here is that last layer retraining data is not used for feature learning. It is used for reweighting the features that are learned by the feature extractor $g_\theta$ (L183 (208 in the revision)) during training, which is a relaxed form of feature selection (see Figure 1 in DFR [1]).

Regarding BAM [4], we noticed there are evidences of unreliable results as you can see below. Before stating them, note that as we review in Sec. 2. Problem Setting (L139-149), there exist several types of subpopulation shifts with different sources, including class-imbalance (L140-141) and spurious correlation (L142-157). Following Yang et al. (2023b): Given input $x = (x_{c}, x_{s}) \in \mathcal{X}$, which consists of core pattern $x_{c}$ and spurious pattern $x_{s}$, and label $y \in \mathcal{Y}$, we can write the classification model: $P(y|x) = \frac{P(x|y)}{P(x)}P(y) = \frac{P(x_{c}, x_{s}|y)}{P(x_{c}, x_{s})}P(y) = \frac{P(x_{c}|y)}{P(x_{c})}\frac{P(x_{s}|x_{c}, y)}{P(x_{s}|x_{c})}P(y)$ Spurious correlation occurs when $P(x_{s}|x_{c}, y) \gg P(x_{s}|x_{c})$, while class imbalance represents the scenario where $P(y) \gg P(y')$ for $y, y' \in \mathcal{Y}$. Thus, such shifts can occur independently in datasets (see Section 2 Preliminaries L139-149), .

The model selection criterion used by BAM [4], ClassDiff, is a criterion for enhancing robustness to class-imbalance, and it was not reasonable to use it for spurious correlation. To better illustrate why, note that if you have a completely random classifier, w.h.p. it achieves near zero ClassDiff, while its WGA w.h.p. would not be much higher than random. In a more concrete example, setting their auxiliary coefficient ($\lambda$) to 0 and #Epochs in Stage 1 ($T$) to 0, and upweight factor ($\mu$) to 1 in BAM, is equivalent to simply training an ERM model. Using such hyperparameters to train a model on a dataset results in the same WGA as ERM. If the dataset contains a spurious correlation but has no class imbalance, both ClassDiff and WGA will be low.

We reviewed their Appendix B Training details and found that they used a limited range of hyperparameters (see BAM [4] Table 6). By avoiding scenarios that lead to lower (better) ClassDiff but lower (worse) WGA, the results are reported for settings that have a minimum level of bias amplification during initial training and a proper version of the initially trained model. Note that the advantage of bias amplification (Nam et al., 2021 in the paper) and using an early-stopped version for identifying spurious correlation (Zhang et al. in the paper, JTT) are known, and it is expected that for a set of hyperparameters the method works. However, the effectiveness of ClassDiff was questionable, and we did not find enough evidence of its effectiveness (because the BAM was tested on a narrow range of hyperparameters). To further justify our thoughts, for UrbanCars, we tested BAM [4] on a set of hyperparameters outside the reported range, and in comparison to in-range hyperparameters, we observed that while the ClassDiff became lower, WGA did not show improvement compared to that of ERM.

Thus, we concluded that there is not enough evidence of effectiveness to benchmark BAM [4] and decided to exclude it from the comparison.

---

Weakness 4

The worst group accuracy and average accuracy (in parentheses) results you have requested are as follows:

By comparing EVaLS results in Table 1 and Table 4, we can see that in most setups, both criteria underperform compared to EV. Worst-class accuracy achieves a $0.7$% higher WGA on Waterbirds and minimum class difference shows similar WGA but lower average accuracy on UrbanCars compared to EV. On CelebA, both criteria underperform EV by a margin of more than $10$%. Minimum class difference results in a $7.7$% lower WGA on Waterbirds, and worst-class accuracy underperforms EV by $0.5$% on UrbanCars.

Question 2

As stated in Sec. 2.1 (L139-L149), there exist multiple types of subpopulation shifts (see response to weakness 4). Refer to Yang et al. (2023b) in the paper for a categorization of subpopulation shifts and their formulation (Table 1 in their paper). EVaLS is designed for robustness to spurious correlation, but CivilComments and MultiNLI are examples of other forms of subpopulation shifts (class imbalance and attribute imbalance). It is stated in Abstract L100-101, Experiments L348-350 (368-370 in the revision), and Appendix E.3 L1274-1276 (1277-1279 in the revision), and reported also by Yang et al. (2023b) (see Table 2 in their paper).

As explained in L458-466, patterns distinguishing groups in CivilComments and MultiNLI are not predictive for the target class (see tables below). This reduces their visibility in the model’s final layers (see Lee et al. (2023) for the role of various layers of neural networks in different types of shifts). Since environment inference algorithms like EIIL (see Appendix F.3 for further investigation) depend on the last layers of a trained model, they cannot infer environments with notable group shifts (defined in L253-254 (257-259 in the revision)) in CivilComments and MultiNLI. The group shifts in CivilComments and MultiNLI (L464-466 (467-469 in the revision)) are significantly lower than those of the datasets reported in Table 2. Thus, the focus of environment-based validation is on datasets with spurious correlations (see limitations (L528-529 (530-531 in the revision)) and future works (L534-535 (537-539 in the revision)) in the Discussion section).

Nevertheless, loss-based sampling (LS) is effective for CivilComments and MultiNLI. Our EVaLS-GL, using ground-truth group labels for model selection and loss-based sampling for retraining, outperforms other methods with a similar level of group supervision on MultiNLI. Also, its WGA on CivilComments is also only $2.1$% lower than the state-of-the-art method (by DFR [1]).

Proportion of attributes in each class for CivilComments dataset (Table 9 in the paper)

MultiNLI training sets statistics

---

Question 3

Note that for last-layer retraining and model selection data, we follow the same scheme as in DFR [1] (please see their Sec. 6, “Feature Reweighting Improves Robustness,” Hyper-parameters paragraph), which is also used in other works (see Appendix D: Training Details in SELF [2]). If the number of minority samples $n$ is larger than ~20 in the validation set, which is the case in all the datasets we have, the number of samples in $D_{MS}$ can be approximated by a normal distribution $\mathcal{N}(\mu = \frac{n}{2}, \sigma^2 = \frac{n}{4})$. By calculating a confidence interval, the probability that fewer than $\frac{n}{4}$ of the samples are in $D_{MS}$ or $D_{LL}$ is approximately $2\Phi(-\frac{\sqrt{n}}{2})$, which would be very low. Thus, like every other case in machine learning, and specifically in our datasets and settings similar to DFR [1], random splits work properly and place an acceptable percentage of samples in each of $D_{MS}$ and $D_{LL}$ with a probability of almost 1. However, in the rare cases you described or other cases with a limited number of data points (as we state in Section 5: Discussion, L526-528 (529-530 in the revision), as a limitation), EVaLS will struggle if not enough minority samples exist in their retraining and model selection dataset.

Dear Reviewer jr76,

We appreciate the feedback and the opportunity to clarify our work's contributions. We are delighted that you find our scheme novel and interesting.

---

Weakness 1: Issue with High-Loss Minority Samples

Note that EVaLs aims to enhance a trained model's robustness against spurious correlations it relies on (L523-524 (525-526 in the revision)). For a grouping based on an attribute $a$, if the loss distribution for groups within a class—distinguished by the presence or absence of attribute $a$—does not reflect distinct populations, it indicates that the model does not rely on attribute $a$ for its predictions.

The loss distribution of a trained model across different groups has certain properties. We would like to provide additional clarification and evidence on this matter:

1. Our Empirical Observations: As stated in Section 3.1 (L205-208 (249-252 in the revision)), our observations (Figure 2 (3 in the revision)) strongly support the assumption that minority samples are overrepresented among high-loss examples.
2. Loss Dynamics Across Minority and Majority Groups: In the majority groups, both the core and spurious patterns are aligned and predictive of the target. In these scenarios, both patterns lead to lower loss. Conversely, in minority groups, core and spurious patterns exhibit contradictory signs for predicting labels. As a result, if the model relies on spurious correlations, the loss for minority groups will be higher.
3. Empirical Risk Minimization (ERM) Principles: In an ERM framework, the learning process prioritizes minimizing the overall loss across the dataset. In the context of spurious correlation (Sec 2.1, L143-L156), majority groups dominate the data and contribute more the overall loss. Thus, a model which is trained on the data with ERM shoud exhibit lower loss on these groups. Conversely, minority groups are underrepresented, may incur higher losses. If a majority group consistently exhibits high loss, it suggests a failure in the learning process to converge, contradicting ERM principles.
4. Evidence in Literature: As stated in L61-63, “the loss value of the model, or its alternatives, are popular signals for recognizing minority groups,” which has been utilized in previous works (Liu et al., 2021a; Qiu et al., 2023; Nam et al., 2020; Noohdani et al., 2024).

---

Weakness 2: Environment Inference:

As detailed in Appendix F.3, EVaLS could be used with other environment inference methods. As we stated in Section 1 (Introduction, L81-83) and demonstrated in Table 12, EVaLS-RC, which employs a random linear classifier for environment creation, outperforms EIIL on some datasets. These results further highlight that the effectiveness of EVaLS does not depend exclusively on EIIL.

What EVaLS requires for model selection is a set of diverse environments that exhibit correlation shift (Abstract, L26-30; Introduction, L73-74 and L77-80; Sec 3.2, L214-215 (259-260 in the revision) and L244-258 (263-269 and 285-292 in the revision)). As discussed in L244-258 (263-269 and 285-292 in the revision) and analyzed in Table 2 of the main paper, modest group shifts in the environments are sufficient for EVaLS to perform well. It does not rely on finding "perfect" environments.

Also, as discussed in Appendix A (Related Work, L722-727), environment inference is a different area of research and a broader problem addressed in invariant learning (L713-727).

We should emphasize that EVaLS is a group annotation-free framework that its performance across various inference methods underscores its flexibility.

---

Weakness 3: EVaLS Consistency:

Note that EVaLS does not utilize group annotations information regarding spurious correlation for model selection or (re)training, and thus, as we clarify in our Abstract L32-34, its near-optimal accuracy showcases its effectiveness. This is a significant advancement, particularly in scenarios where group labels are unavailable or unreliable (a common and challenging real-world scenario).

In other words, it is not expected that EVaLS, which does not utilize group-level data, will surpass all models that use this additional information. Nonetheless, the fact that EVaLS achieves comparable results under these constraints demonstrates its robustness and practical utility.

As you can see in Table 1, when group annotations are available for model selection, EVaLS-GL outperforms other methods with a similar level of group supervision on most datasets. However, it underperforms but achieves near-optimal worst group accuracy compared to DFR and GDRO with higher levels of group supervision in most datasets, as expected. It also outperforms DFR and GDRO in the case of the UrbanCars dataset, where annotations for one of its spurious attributes are not available during training (see L454-457 (457-460 in the revision)).

Finally, the claim that EVaLS's better performance on certain datasets is due to chance contradicts our findings. Across diverse datasets with spurious correlations, EVaLS consistently performs well, supported by statistical measures to minimize random variance. Its comparable performance to methods utilizing group information underscores its strength.