Like Oil and Water: Group Robustness Methods and Poisoning Defenses May Be at Odds

The definitions in Section 2.2 are unclear and should be improved [...] The definitions in Section 3.3 are unclear and should be improved.

We agree with the reviewer. We included more clear definitions for groups, as well for the identification model I(x) in the paper.

[Comments about the bibtex, styling errors, colors in the figures, confusing phrases, and other typos]

Thanks for pointing these out, we fixed all of them!

It would be helpful for reference to include a table with the proportions of each group and class in each benchmark dataset, as well as the proportions of each group and class in the 10% sample of CelebA used in the experiments.

We agree, we will include it in an updated version of the paper.

[1] Simple and Fast Group Robustness by Automatic Feature Reweighting, Qiu et al. (ICML'23)

We thank the reviewer for their very constructive and detailed feedback! Please find below our response to your comments:

The main weakness of the paper is its limited dataset evaluation, which consists only of Waterbirds and a 10% subset of CelebA. While the experiments are indeed comprehensive with different techniques and metrics, the size and composition of the datasets questions whether the results hold more generally. [...] Finally, while CelebA is a more complex dataset than Waterbirds, and the 10% subset was likely necessary due to computational restrictions, the fact remains that both datasets are small, belong to the vision domain, and only utilize binary-valued spurious features.

To address the reviewer's concern, we ran the experiments from Table 3 on different sizes of CelebA (all the other experiments in the paper were already on full CelebA). Below, we show the effect of EPIc (poisoning defense) on the WGA (results are over 3 runs):

This shows that the trends we observed (i.e., EPIc hurts the WGA by eliminating minority groups) are consistent as the ideal WGA (when we intervene to prevent EPIc from eliminating any minority samples) is higher than the standard WGA (applying EPIc w/o any intervention).

Additionally, to show the amplification due to group robustness methods, we ran experiments with the Subpopulation Attack (SA) on full CelebA for JTT and ERM (the baseline suggested by Reviewer Shsu).

In the first table, we observe that the amplification is generally high (a large ASR gap between JTT and ERM). The second table shows that JTT still works as intended by improving the WGA over ERM. In conclusion, our observations remain consistent for the full CelebA as well.

Some suggestions may include [...] MultiNLI [4] for the language domain (which would also introduce a non-binary-valued spurious feature). [...] There have been significant contributions since 2021, particularly in methods which pseudo-label the minority group (with or without an explicit identification model), e.g., [2,5,6,7,8,9]

Thank you for suggesting these experiments! To ensure the validity of our observations for newer group robustness methods and more advanced datasets, we applied AFR [1] (referenced as [8] by the reviewer) on MultiNLI. As in our previous experiment, we consider SA as the attack, ERM as a baseline (no group robustness method), and report both the ASR and WGA.

We observe the same tension in these experiments: while AFR improves the WGA over the baseline (as expected), it also inadvertently amplifies the ASR (e.g., from 5.3% to 14.4%, a significant boost for a weak attacker).

Some suggestions may include Spawrious [3] for the vision domain [...].

We agree with the reviewer that Spawrious would be a good addition to our experiments. Unfortunately, due to time limitations, we were unable to conclude the experiments on this benchmark before the rebuttal deadline. However, we are committed to include them in the final version (we will include them in the Appendix).

I would encourage the authors to perform a more detailed literature review [...]

We will update our paper's appendix with the discussion we shared with Reviewer Shsu on the different (more recent) group robustness methods (some of them are referenced as [5,7,8,9] by the reviewer), and the parallels among the heuristics they rely on.

We thank the reviewer for their constructive feedback! Please find below our response to each of your comments:

…the authors only consider group robustness methods which use a loss-based heuristic in lieu of validation group annotations. This significantly reduces the scope of authors' contributions…

Although we experimented with methods that specifically use a loss-based heuristic, we believe that many other group robustness methods rely on similar heuristics, perhaps less directly. Here, we discuss four methods from recent literature that implemented seemingly different heuristics that will align with loss-based heuristics:

• Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlations, Zhang et al. (ICLR’22): This method applies contrastive learning to push together the representations of training samples that are labeled into the same class but predicted differently. Since, the low-budget poisoning attacks we consider often generate hard-to-learn, misclassified samples, we believe this method will also suffer from the tension we identify, e.g., misclassified poison samples of class $y$ will be represented similarly to correctly classified samples of class $y$, which will amplify their effectiveness as the model becomes less likely to misclassify them.

• Dropout Disagreement: A Recipe for Group Robustness with Fewer Annotations, LaBonte et al. (NeurIPS’22 Workshop): This method uses the disagreements between different forward passes of the model with dropout to find which training samples to amplify. Different dropout forward passes are known to disagree on samples where the model has high uncertainty [1], which are often hard-to-learn, higher-loss samples. As a result, we believe this method will also suffer from the same tension we identified, e.g., the hard-to-learn poison samples we generated will create dropout disagreements and will be amplified.

• Learning Debiased Classifier with Biased Committee, Kim et al. (NeurIPS’22): This method relies on disagreements among the members of an ensemble of models to identify the “biased” samples (i.e., the samples that contain spurious correlations). If a sample has low ensemble agreement, it will be amplified by this method. Ensemble disagreement is a known uncertainty metric [2] in the literature. As a result, we believe that this method will also suffer from the same tension we identified, e.g., the hard-to-learn poison samples will be misclassified by more members of the ensemble, and will be amplified.

• Towards Last-layer Retraining for Group Robustness with Fewer Annotations, LaBonte et al. (NeurIPS’23): This method constructs a reweighting set based on either (1) ERM model’s misclassifications, or (2) the misclassifications of an early-stopping model, or (3) dropout disagreement (similar to the Dropout Disagreement paper) or (4) the disagreements between the ERM and early-stopping models. All these heuristics to identify minority group samples will end up identifying the hard-to-learn poisoning samples as well. The authors identified (4) as the most promising heuristic. It is known that hard-to-classify samples are learned at later iterations of model training [3]. As a result, we believe that method will also suffer from the same tension we identified, e.g., the hard-to-learn poison samples will be classified differently by the early-stopping and regular models, and will be amplified.

These common threads show that, despite not using the loss heuristic directly, many methods in this line of work rely on related ideas that will amplify uncertain, misclassified, high-loss, or hard-to-learn samples. Moreover, it is known that many example difficulty metrics are highly correlated with one another [3,4]. Ultimately, we believe that these related heuristics cannot avoid the tension we identified because low-budget poisoning attacks generate hard-to-learn samples. To address the reviewer's concern and communicate that our findings have wider implications than JTT/GEORGE, we will add this discussion to our paper (we plan to add it in the Appendix). Also, please find below our new experiments on AFR [7], a state-of-the-art group robustness method.

We thank the reviewer for their constructive feedback! Please find below our response to each of your comments:

The results on CelebA (Section 4) only uses a randomly sampled 10% of the dataset which is concerning. Can the authors also consider the full CelebA setup, so that the results are more reliable?

To address the reviewer's concern, we ran additional experiments with 20%, 50% and 100% of the CelebA. The results, over 3 runs, showing the WGA in each scenario are in the following table:

The persistent gap between the Ideal and Standard cases demonstrates the negative impact of EPIc on group robustness (i.e., our findings are consistent).

Why are the ASR values negative in Table 3 (Waterbirds / SA setup).

For the SA, we define the ASR as the relative accuracy drop of the attack on the target group over a non-poisoned model. So, in case the defense not only prevents the accuracy drop by removing the poisons but also slightly increases the accuracy on the targeted group, then the ASR would be negative. We will clarify this in the updated paper.

Could the authors elaborate on whether any experiments were conducted involving the other two types of poisoning defenses mentioned in Section 2.1?

Indeed, in our paper, we only consider defenses that assume that poisons are hard to learn as more realistic, weak, attackers are the ones who would benefit from the amplification offered by group robustness methods. However, we believe that poisoning defenses designed against easy-to-learn poisons (e.g., [1]) would indeed not eliminate legitimate minority samples and hence not hurt the worst-group accuracy (though, such defenses struggle against attacks that craft hard-to-learn poisons [2]).

A promising future direction could be the defenses that assume poisons and clean samples follow different distributions, so they create a small, poison-free set of samples for each group to erase poisons (similar to [3] but with a group-balanced clean data to preserve group robustness).

The broad focus implied by the current title, "Poisoning Defenses," does not accurately reflect the specific emphasis of the work on poisons that are challenging to learn. A more precise title would set clearer expectations for readers.

We agree with the reviewer's concern that our title might be too broad. We will specify that we study weaker poisoning attacks, e.g., "Resilience to Weak Poisoning Attacks and Group Robustness are at Odds". Please let us know if you have a suggestion!

[1] Anti-Backdoor Learning: Training Clean Models on Poisoned Data, Li et al. (NeurIPS'21)

[2] Narcissus: A Practical Clean-Label Backdoor Attack with Limited Information, Zeng et al. (CCS'23)

[3] Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks, Li et al. (ICLR'21)

We thank the reviewer for their constructive feedback! Please find below our response to each of your comments:

…poisoning attacks also do not reflect what the attacker in practice may really care to do. For example, many ASR are quite low as presented in Table 2 and I am not sure if it is because the poisoning attacks are configured in some undesirable ways…

An attacker in practice aims to achieve the highest ASR possible within their limited budget. In the case of a poisoning adversary, the number of injected samples determines the real-world budget. The lower the budget is, the more realistic an attack becomes (e.g., the attacker needs to compromise fewer websites to inject images [1]).

A main takeaway from our experiments is that low-budget attacks where the adversaries struggle to achieve high ASR (hence the results in Table 2) can benefit from the defender’s use of group robustness methods such as JTT. The same cannot be said for high-budget attacks that can already achieve high ASR (thus, amplification is less critical). Therefore, the low ASR of the attacks we performed (Table 2) is not due to any specific configuration of the attacks (e.g., hyper-parameters) but due to the considered attacker budgets. This allows us to expose how low-budget attacks (more realistic but weaker) can be inadvertently amplified into stronger ones, essentially giving the attacker “more bang for their buck”.

Furthermore, we would like to clarify that for the Subpopulation Attack (SA) in Table 2, ASR indicates a relative accuracy drop on the attack's target group over a non-poisoned model. Therefore, some of our experiments already consider more powerful attacks (e.g., when the ASR is 31%), in which our findings are consistent.

I am not sure if it is because the poisoning attacks are configured in some undesirable ways (with respect to the attack objective), who may choose a different approach to achieve higher ASR —----------- In page 4, what is the intuition behind configuring the poisoning attacks as described?

To address the reviewer’s concerns and questions, we considered attackers (DLBD) selecting poison base samples from different groups of CelebA (the results are over 3 runs):

The maximum ASR holds for Gr1, which is the one we chose in our experiments as a base group for crafting the samples. Since the attacker changes the label of the samples they poison, the label/attribute configuration will become 1/1 which matches the one for LRG-1 (as stated in the paper).

For a dirty-label attack, if the attacker is aware of the groups that are easiest to learn, they can take advantage of it and create poisons from those groups as base samples (this matches the setting from our theoretical result). Also, we show below the average loss of each group when there are no poisons in the dataset:

We observe that the easier a group is to learn (i.e., less average loss), the higher the ASR will be if the attacker crafts a DLBD attack with base samples from that group.