Thank you for highlighting the relevant works and for your insightful requests regarding the method’s sensitivity and the compatibility of its theoretical and practical aspects.

Weakness 1

Please refer to the general response in the Author Rebuttal for a review of the contributions of this work. We also note that the modules in the method can be replaced while completely maintaining the overall contribution and framework. In Appendix E, we substitute loss-based sampling with other methods (Sec. E.1) and provide an ablation study on replacing EIIL with other simple environment inference techniques (Sec. E.2).

Weakness 2

Thank you for pointing out relevant works.

As stated in Appendix A (L552-555), SELF introduces CB last-layer retraining that does not require group annotations for retraining or model selection. Another method proposed by SELF, ES disagreement, requires group annotations for model selection but not for the retraining (see Appendix A, L547-549). It also needs an early-stopped version of the trained ERM along with the final model.

BAM is a two-stage training method that works with or without group annotations for model selection. Without group annotations, BAM uses ClassDiff to select hyperparameters, ensuring balanced performance across all classes.

BPA is a clustering-based method (see L206). While BAM claims BPA doesn’t need group annotations for model selection, it is important to note that our investigation shows that it does.

All comparisons use similar splits and architectures as the benchmarked methods in the paper.

Coamparison with class-balanced schemes

Following Yang et al. (reference [4] in the paper): Given input $x=(x_{c}, x_{s})\in\mathcal{X}$, which consists of core feature $x_{c}$ and spurious feature $x_{s}$, and label $y\in\mathcal{Y}$, we can write the classification model:$P(y|x)=\frac{P(x|y)}{P(x)}P(y)=\frac{P(x_{c},x_{s}|y)}{P(x_{c},x_{s})}P(y)=\frac{P(x_{c}|y)}{P(x_{c})}\frac{P(x_{s}|x_{c}, y)}{P(x_{s}|x_{c})}P(y)$Spurious correlation occurs when $P(x_{s}|x_{c}, y)\gg P(x_{s}|x_{c})$, while class imbalance represents the scenario where $P(y)\gg P(y')$ for $y, y'\in\mathcal{Y}$. Thus, as stated in Preliminaries section (L112-123), such shifts can occur independently in datasets.

Yang et al. observes in Figure 2 of the paper that Waterbirds, CelebA, and CivilComments have significant class imbalance, but MultiNLI does not. It can be seen that robustness improvements in class-balanced schemes in SELF and BAM relate to the amount of class imbalance in the dataset. For example, as emphasized in various parts of the paper (L79, L257, L313-315), the CivilComments dataset does not show spurious correlation but does exhibit class imbalance. Consequently, class-balanced schemes significantly improve worst-group accuracy on this dataset. However, results for MultiNLI (which exhibits attribute imbalance but no class imbalance (see L114-116)) are just slightly higher than ERM (for BAM) or even lower than it (for SELF).

Class-balanced schemes are helpful for handling class-imbalance-based subpopulation shifts but fail in addressing spurious correlations. As the authors of SELF note, these schemes can’t match state-of-the-art methods. This is due to their inability to manage subpopulation shifts caused by spurious correlations. Our approach, which involves selecting an equal number of samples from each group through a loss-based sampling scheme, creates a class-balanced retraining dataset. However, our paper clarifies (L316-325) that environment-based validation doesn’t address other types of subpopulation shifts beyond spurious correlations.

Worst group and average accuracy (in parentheses) for CB last-layer retraining (SELF) and BAM on datasets with spurious correlations is as follows.

Although we couldn’t provide BAM results for UrbanCars due to high training computational costs, it’s evident that on this benchmark, which lacks class imbalance but has spurious correlations, the reported class-balanced scheme shows no improvement compared to ERM.

Discussing the results of the suggested methods

Regarding UrbanCars: we were unable to provide results for BAM during the rebuttal phase due to its significant computational cost. For ES disagreemnet SELF, WGA and AA are $82.1_{\pm1.8}$ and $89.3_{\pm0.8}$ respecitvely. For BPA, WGA is $62.4$.

For other datasets, results are obtained from the respective papers.

For Waterbirds, ES disagreement SELF and BAM have higher WGA at the cost of lower average accuracy. EVaLS-GL outperform these methods with similar level of group supervision on all other datasets.

Weakness 3

Figure 2 in the paper provides exactly the information you’re looking for! Please refer to the general response.

Q1

Please see the general response.

Q2

You are absolutely correct. The $F^2(\beta)$ should change to $F^2(\alpha)$ as $tail_L$ is a function of $\alpha$.

Q3

Due to the technical difficulty of dealing with arbitrary distributions, our proof relies on the Gaussian nature of the logit distribution. In a case study (class 2 of CelebA with 94% spurious ratio on 20000 datapoints), this assumption almost holds (see qqplot in Figure 4 in the attached PDF). Evaluating our theorem's conditions, the empirical variance of the minority is higher than the majority (see “Distribution of logits” in the general rebuttal), satisfying the necessary and sufficient condition (L619) for Proposition C.1. The theorem proves the existence of suitable $\alpha, \beta$ (see L223-225). To empirically justify our tail-based method, we tested all possible tails $\alpha, \beta$ and found 6 value pairs that make the dataset group balance. Thus, our practical results align with our theoretical work.

We deeply appreciate your consideration of our rebuttal. Thank you for suggesting the inclusion of the discussions in the final version of the paper; we will certainly do so.

We have carefully addressed all your concerns based on your feedback. If there are any specific areas where you feel further clarification or justification is needed, we are eager to address them promptly. Your guidance helps us achieve the highest standards, and we hope this will be reflected in a higher score.

Thank you for your constructive and insightful feedback. Our response to the mentioned weaknesses is as follows:

Weakness 1

1. We must emphasize that the optimal number of selected high/low loss samples for retraining the last layer is chosen from a set of various values, based on the worst validation group/environment accuracy. Hence, the proposed method could be considered a more general solution that encompasses selecting misclassified samples (If selecting misclassified samples yields the best validation results, this configuration is chosen automatically by our method).
2. Having a hyperparameter ($k$) for controlling the number of selected samples from loss tails gives us flexibility to handle some challenges. For example,

• There is a tradeoff between the purity and the number of selected high-loss samples: as observed in Figures 2 and 7 in the paper, minority samples are more commonly seen among samples on which the ERM model has a high loss. As we increase the number of high-loss selected samples, the proportion of minority samples among them decreases. On the other hand, retraining on a larger number of samples could improve the overall training process. Choosing flexibly among various numbers of selected samples grants EVaLS the advantage of finding an optimal point in this tradeoff. You can see how sensitive the worst validation group accuracy is to selecting $k$ in Figure 5 in the attached PDF in the general response. Choosing only misclassified samples could not handle this trade-off especially when the number of misclassified samples is too high or low.
• There are situations in which the dataset contains corrupted data or samples with label noise on which the ERM model has a high loss or may misclassify. Using such samples for retraining the last layer degrades its performance. However, choosing the number of selected samples based on the worst validation group/environment accuracy instead of using a specific number allows our method to mitigate this issue to an extent.

We also conducted an experiment in which we replaced the loss-based sampling in EVaLS with the selection of misclassified samples and an equal number of randomly selected correctly classified samples from each class. As can be seen in the results, the performance is degraded compared to EVaLS on the Waterbirds and Urbancars datasets, with a slight improvement but a higher standard deviation on the CelebA dataset.

Weakness 2

As requested, the balance levels of samples selected based on loss and misclassification are presented in the following table for the Waterbirds, CelebA, and UrbanCars datasets. The ratio $\frac{minority}{k}$​ for each class is reported for three seeds, with the average across all seeds reported in parentheses. The results for the UrbanCars dataset are reported for the minority group with the lowest number of training samples.

Weakness 3

Thank you for requesting such an interesting comparison! We believe that your intuition is correct. When the unknown spurious attribute is shape instead of color, methods that use the known group annotations show higher worst group accuracy compared to the previous scenario. In other words, when the unknown spurious attribute is a weaker shortcut, group robustness becomes a simpler task. Note that since EVaLS does not rely on any information regarding spurious attributes, it does not matter which spurious attribute is known and which is unknown; the results remain the same as reported in Table 2 of the paper. You can see the results for other methods for the new settings in the table below.

We found that utilizing group annotations for a simpler attribute (a stronger shortcut) enhances results compared to the previous setting where the unknown attribute was simpler. Regarding DFR, we believe that part of its performance is due to the larger number of retraining samples compared to EVaLS and EVaLS-GL (as stated in l355), rather than the provided group annotation. To verify this, we evaluated the performance of DFR when its number of retraining samples is reduced to the number of samples used for EVaLS. In this case, the performance of DFR drops to $66.3_{\pm1.15}$, which is lower than EVaLS.

Additionally, the performance gap between AFR and AFR + EIIL, as well as between EVaLS-GL and EVaLS (using environment-based validation instead of ground truth validation group labels of the known spurious attribute), decreases.

Thank you for your detailed reviews and insightful questions.

Weakness 1

Figure 2 in the paper does not show the proportion of minority (majority) samples that have high (low) loss. Instead, it depicts the proportion of minority/majority samples among the top x% of samples with the highest/lowest loss values. At the 100% mark on the x-axis, it shows the proportion of minority/majority groups in the entire dataset.

For example:

• In the second plot from the left, about 80% of the 50% samples with lowest loss in both class 1 and 2 of the Waterbirds dataset belong to majority groups.
• In the second plot from the right, over 30% of the highest-loss samples (top 1%) in the unbalanced class (class 2) of CelebA belong to minority groups.

When n% of samples in the top x% of samples with the highest/lowest loss belong to minority/majority groups, the remaining (100-n)% belong to majority/minority groups. To clarify, we will update the x-axis title to “% of selected samples from samples with highest/lowest loss values”.

As requested, Figure 1 in the attached PDF of the general author’s rebuttal shows the % of both majority and minority groups in the same plot, and Figure 3 illustrates the loss value distribution for these groups.

Please let us know if any further information or clarification is required.

Weakness 2

We appreciate the suggestion to include pseudocode. We present a pseudocode in the “General Response”.

Weakness 3

The scenario you described, involving multiple dimensions for spurious and core features, already aligns perfectly with our framework. As noted in L217-219 (and elaborated further in L585-589 in the Appendix C), we assume that both minority and majority groups follow a Gaussian distribution in the feature space. This assumption leads to Lemma C.1 (line 590), where the Gaussian distribution for our logits (not losses) is derived. Using a Gaussian distribution in the feature space is a common practice due to the technical challenges associated with other distributions. The paper you referenced also assumes a Gaussian distribution for both core and spurious dimensions. So, we believe that our assumption contains the one you have mentioned.

As mentioned in L215-216, the order of samples in logit space and loss space is monotonic. Therefore, it is unnecessary to focus on the distribution of the loss or assume any specific distribution (Gaussian or anything else) for it.

Weakness 4

”results are missing for datasets such as UrbanCars and MultiNLI.”

The missing results (depicted by - in Table 1 in the paper) are as follows:

“Only a few baselines are compared for the new dataset in Table 2.”

Table 2 shows that loss-based sampling and environment-based validation effectively handle unknown spurious correlations, focusing on methods with similar setups (last-layer retraining). We also include the worst group accuracy for a new method (AFR) and its environment-based validation version. The worst group accuracies are:

AFR: $54.27_{\pm2.73}$

AFR + EIIL: $61.54_{\pm1.85}$

“There is also no convincing and fine-grained analysis (e.g., ablation study) to understand how the proposed approach ensures data balancing and improves group robustness.”

Regarding data balancing: Figure 2 in the paper shows the proportion of minority/majority samples among those with the highest/lowest loss at various thresholds. The balance level for chosen $k$ is detailed in the response to Q2 in this rebuttal.

Regarding group robustness: We conducted several ablation studies and analyses to understand how EVaLS work:

• Group shifts under environment inference (Appendix-Table 3)
• Ablations on environment-based validation methods (Sec. E.2) and alternatives to loss-based sampling (Sec. E.1)
• Experiments on benchmarks with single and multiple spurious correlations (e.g., UrbanCars)
• A new dataset (Dominoes-CMF) for studying multiple independent spurious correlations

For further analysis, please let us know.

Weakness 5

Regarding multiple spurious correlation dataset, we use UrbanCars (as outlined in L308-312), and also propose Dominoes-CMF dataset with two independent spurious correlations (Sec. 2.2). Table 2 shows that EVaLS, which doesn’t require group annotation, improves model robustness to unknown spurious correlations compared to more supervised solutions.

Finally, note that EVaLS is a robustification method to unknown spurious correlations that are learnt by a trained model. So, as illustrated in the results of the paper, it improves robustness for various benchmarks with single or multiple spurious correlations.

Q1

Please refer to the “Distribution of logits for training datasets” section in the general response.

Q2

The optimal number of balancing samples, $k$, is determined by the worst accuracy on inferred environments. The tables below report the proportion of minority group in selected high-loss samples, as well as the proportion of majority group within the selected low-loss samples for a sample seed:

Waterbirds

Class 2 of CelebA

Q3

Refer to L313-315. The CivilComments dataset has class imbalance, so fine-tuning/retraining on class-balanced data can significantly improve group robustness. EVaLS-GL’s effectiveness is due to using a class-balanced dataset for retraining, as also noted in reference [2] (see L553-554).

Q4 and Q5

Thanks for informing us about the typos. You are right. We will correct them in the final version.

Dear Reviewer RE7s,

We’re glad our clarification was helpful. It’s good to know that the pseudo-code and highlights have been effective in preventing confusion. We appreciate your suggestion to include them in the final version and will incorporate them.

Regarding the limitations that you have stated

As you mentioned, we state in the Discussion section (L356-357) that environment-based validation is limited to spurious correlation and not other types of subpopulation shifts.

Regarding SELF [2]

Please refer to the response to Reviewer 7E4z. As we described there and also in Appendix A-Related Work (L547-554), reference [2] in the paper proposes two methods:

Class-balanced (CB) last layer retraining: See L552-554. As we discuss comprehensively in the response to Reviewer 7E4z, this method is not helpful in targeting robustness to spurious correlation, but it improves robustness to class-imbalance, another type of subpopulation shift which is independent of spurious correlation. For details, refer to the formulations in the response to Reviewer 7E4z.

Early-stop (ES) disagreement SELF: As noted in the response to Reviewer 7E4z, ES disagreement requires group annotations for model selection, but not for retraining (L547-551). It also needs an early-stopped version of the trained ERM and the final model.

In the following table, we restate the worst group accuracy for our methods and those of SELF on spurious correlation benchmarks. We also add the results of CB last-layer retraining and ES disagreement SELF for Dominoes-CMF.

Waterbirds: SELF’s methods outperform EVaLS. As SELF illustrates in their Sec. 3 (Preliminaries), since CB last-layer retraining is performed on validation data, which is group-balanced for Waterbirds, CB last-layer retraining is also group-balanced for Waterbirds. When the dataset for is not group-balanced, CB last-layer retraining achieves a worst group accuracy of $77.4_{\pm0.3}$ (Table 7 in reference [2] of the paper (SELF)).

CelebA: As you can see, EVaLS and EVaLS-GL outperform both CB last-layer retraining and ES disagreement. We believe there may have been an oversight in your response, as it incorrectly suggests their method outperforms ours. CB last-layer retraining underperforms all other benchmarked methods in Table 1.

UrbanCars: EVaLS-GL outperforms all other methods. Both EVaLS and ES disagreement (with higher group supervision) show similar worst group accuracy. CB last-layer retraining shows no significant improvement over ERM. This is expected, as UrbanCars’ training data is class-balanced, unlike Waterbirds and CelebA. Therefore, further class-balancing for CB last-layer retraining makes no difference.

Dominoes-CMF: EVaLS outperforms other methods. EVaLS-GL also outperforms methods in SELF [2]. CB last-layer retraining does not show any significant improvement compared to ERM. This is expected, as Dominoes-CMF also has class-balanced training data like UrbanCars.

Conclusion:

CB last-layer retraining targets class-imbalance shifts and is ineffective for spurious correlation. Except for the Waterbirds dataset, our methods ٍ(EVaLS and EVaLS-GL) outperform those of SELF [2] with similar group supervision on all other datasets. EVaLS shows comparable or better results (e.g., in CelebA) compared to methods with higher group supervision. See Table 1 in the paper for more comparisons.