Compositional Risk Minimization

We thank the reviewer for their insightful and constructive feedback. Please see our responses below and let us know if you have further questions.

Constraints on Feature Representations

The additive energy distributions, which are at the foundation of our approach, are inspired by the independent mechanisms principle. These additive energies appear in our classifier and constrain the feature representations. It is this inductive bias that leads to improved robustness as demonstrated in our experiments. The energy components in our approach are semantically meaningful and "disentangled", as each component corresponds to different attribute. Building disentangled and semantically meaningful representations based on additive energy distribution is a future work we are excited about.

Pre-specifying Attribute $a$

To pre-specify the attribute $a$, we use whatever is standard for that dataset in the literature. For instance, in the Waterbirds dataset it is common to choose $y$ to be the bird and $a$ to be the background. However, our training is independent of which attribute is spurious as we learn to predict the joint distribution over all the attributes.

Additional Baselines

Based on your suggestion, we have added IRM and VREx, and we are in the process of benchmarking Fishr as well. The results are presented below, with more detailed results in Table 14, Appendix E.2 for the datasets Waterbirds, CelebA, MetaShift, NICO++, and MultiNLI. We will soon add results for CivilComments as well, since this dataset is much larger than the others (16 different compositional shift scenarios) it will take longer to finish experiments on them.

Our primary comparison baseline has been with GroupDRO, as it is the most effective method for addressing subpopulation shifts, hence it was the main focus of our work. We also included baselines like LA (Logit Adjustment) that share conceptual similarities with our approach. Baselines such as IRM and VREx were initially excluded because they are less suited for tackling subpopulation shifts, we have now incorporated them into our evaluation.

We find that CRM significantly outperforms IRM and VREx w.r.t worst group accuracy. In fact, CRM also obtains better performance than IRM and VREx w.r.t average accuracy and group balanced accuracy as well.

Invariance of p(x|z)

There are two types of data generation mechanisms in the literature on causally inspired domain generalization. In the former one, the assumption is the invariance of $p(y|z)$ and $p(z)$ varies [1] (with $z=x$ in the case of covariate shift [5]). In the latter one, the assumption is the invariance of $p(x|z)$ and $p(z)$ varies [2][3] (with $z=y$ in the case of label shift [4]). Our assumptions are closer to the latter line of work.

References

[1] Arjovsky, Martin, et al. "Invariant risk minimization." arXiv preprint arXiv:1907.02893 (2019).

[2] Rosenfeld, Elan, Pradeep Ravikumar, and Andrej Risteski. "The risks of invariant risk minimization." arXiv preprint arXiv:2010.05761 (2020).

[3] Wang, Haoxiang, et al. "Invariant-Feature Subspace Recovery: A New Class of Provable Domain Generalization Algorithms." arXiv preprint arXiv:2311.00966 (2023).

[4] Garg, Saurabh, et al. "A unified view of label shift estimation." Advances in Neural Information Processing Systems 33 (2020): 3290-3300.

[5] Bickel, Steffen, Michael Brückner, and Tobias Scheffer. "Discriminative learning under covariate shift." Journal of Machine Learning Research 10.9 (2009).

What is c?

$c$ controls the probability of success given by $1-1/c$.

Complexity Analysis as a Function of $d$

We really appreciate this question and indeed believe its an interesting analysis. We did experiments on a synthetic dataset to show the group complexity $s$, i.e., number of groups as a function of $d$.

We consider the case of two attributes $z= (z_1, z_2)$, each can take $d$ possible categorical values. We sample data from the following (additive) energy function. $E(x, z)= || x - \mu(z_1) ||^2 + || x - \mu(z_2) ||^2$

where $x, \mu(z_1), \mu(z_2) \in \mathbb{R}^{n}$. We fix the data dimension as $n=100$ and vary the total categories per attribute $d$ in the following range $[10, 20, 30, 40, 50]$. Then for each scenario $(n, d)$ we analyze how the performance of CRM degrades as we discard more groups from the training dataset. Please refer to Appendix E.3 for more details regarding the setup.

Figure 6 (Appendix E.3) presents the results of our analysis. We find that CRM trained with $20$ % of the total groups still shows good generalization ($\sim 90\%$ test accuracy), and the drop in test accuracy as compared to the oracle case of no groups dropped is within $10$ % .

Thank you for your insightful and constructive feedback. Please see our responses below, and reconsider if we addressed your concerns. Also let us know if you have further questions.

Uniform vs Imbalanced Prior

This is indeed an important point to better highlight and discuss. Thank you for suggesting this. Note that we had already briefly explored the use of different test priors in Table 10 of the Appendix. We are renaming that section of the appendix into "Choice of test prior and importance of extrapolated bias" and adding the following expanded discussion regarding the choice of test prior:

Our algorithm allows the flexibility at test time to choose the prior over test groups as we see fit. That choice should be informed by what we might know or guess about the distribution over test groups, as well as what the metric of interest is. If we assume we can know nothing about the test group distribution, and we care about robust metrics invariant to changes in that distribution, such as balanced-group accuracy or worst-group-accuracy (WGA), then it makes sense to use a uniform prior over groups. This is what we did in most of our experiments. On the other extreme, if we can estimate the test group distribution and what we care about is average test accuracy, then we should use that estimate (which we call empirical prior) as the test prior. We explore these alternatives and show the results in Table 10. As expected from the theory, the empirical prior achieves better average test accuracy, while the uniform prior performs better in terms of worst group accuracy.

Other choices of test prior are possible, depending on what we know and care about. For e.g. if we know some attribute combinations are impossible at test time, we can set their prior probability to 0, while e.g. keeping it uniform over all other possible combinations. Or if we assume that the marginal test distribution of each attribute will be close to its marginal train distributions, we can estimate these marginals on the training set and define an independent test prior as their product. Exploration of the practical usefulness of such alternatives is left as future work.

Multi-Attribute Experiments

Following your suggestion to include empirical results with more attributes we have carried out a new experiment with CelebA using three attributes. This new results table (Table 13 in Appendix E.1) is also reproduced in the general response above, for convenience. CRM continues to be the more robust approach.

Thank you for your insightful and constructive feedback. Please see our responses below, and reconsider if we addressed your concerns. Also, let us know if you have further questions.

Multi-Attribute Experiments

Following your suggestion to include empirical results with more attributes we have carried out a new experiment with CelebA using three attributes. This new results table (Table 13 in Appendix E.1) is reproduced below for convenience. CRM continues to be the more robust approach.

Evaluation in the Traditional Setting

In the paper, we had already carried out a comparison in the traditional setting where all attribute combinations have some representation. These were presented in the rightmost column in Table 1, "WGA (no groups dropped)", as well as in Table 11 in Appendix which contains additional metrics. CRM remains competitive with the baselines in this scenario. This confirms that the method performs well also without requiring group-dropping.

Results: Baselines, Metrics, and Macro-F1 scores

Could you please clarify the work you had in mind as the "original benchmark paper [1]", as you did not provide the actual citation. Baseline performances therein may have been obtained in differing setups (e.g. without fine-tuning), and/or with different hyper-parameter selection strategies. Also, they were most likely not dropping groups, which is the more challenging setting we focus on.

Regarding metrics that account for group imbalances, while average test accuracy does not, we reported group balanced accuracies in Appendix D.1 (Tables 5, 6, 7, 8, 9 containing the detailed results over all of the different discarded group scenarios). Group-balanced accuracy accounts for group imbalances as it essentially considers a uniform distribution over groups (see Appendix C.2 for details about the metrics). We had chosen to focus the main paper table on Worst Group Accuracy (WGA) -- which is also invariant under test-group rebalancing -- and is typically the metric of choice for robustness under subpopulation shifts. To not overload the main paper table we did not include balanced group accuracy therein.

Following your suggestion, we have computed macro F1 scores and added these results in Table 14, Appendix E.2. The F1 scores are computed per group and then averaged across the groups. Macro F1 is akin to the group-balanced accuracy metric in our paper, except that it computes F1 score instead of accuracy per group. The results show that CRM is either better than or competitive with baselines w.r.t both macro F1 and group balanced accuracy.

To summarize these findings, we also compute rankings of the method w.r.t a test metric and then report the average ranking across the different datasets (and their corresponding compositional shift scenarios). We share the average rank of each method below, as well as in Table 15, Appendix E.2

CRM obtains the best rank (lower the better) w.r.t balanced accuracy, macro F1, and worst group accuracy. Note that average accuracy is not the best indicator of generalization under spurious correlations. However, we can adapt CRM to utilize the empirical prior on test distribution to improve its average accuracy, as shown before in ablation studies with CRM (Table 10, Appendix D.2). In fact, as shown above, CRM with empirical prior obtains the best rank w.r.t average accuracy (at the cost of other metrics).

We thank the reviewer for their response to our rebuttal. Please take our responses below into consideration as we believe it should address your unresolved concerns.

Mixup Baseline

We are actively working on your suggestion to add the Mixup baseline. Below, we've provided results with Mixup for the Waterbirds, CelebA, MetaShift, and NICO++ datasets; please check Table 16 in Appendix E.4 for more details. We find that CRM outperforms Mixup across the board w.r.t. the worst group accuracy. That said, we would like to note that group DRO and logit shifting-based methods are considered strong baselines, which we had already compared within our work.

Why only report the average rankings of different methods?

Our paper already has comprehensive results per dataset; we reported the average ranking of methods just to offer a concise summary. The detailed evidence (Table 1, 14, and others) in our paper overwhelmingly supports that under compositional shifts CRM systematically outperforms all other baselines across all the datasets w.r.t worst group accuracy, which is the metric of choice for robustness under spurious correlations.

For comprehensive results we point you to the main results in Table 1, the ablations of CRM in Table 10, results on the original benchmark without dropping groups in Table 11, the multi-attribute experiment in Table 13, and results with additional baselines and Macro F1 score in Table 14. Further, we have provided results conditioned on both the dataset and the discarded group scenario for that dataset, please check Appendix D.1, Tables 5-9.

Please let us know if our response above has clarified your unresolved concerns, we have benchmarked CRM against Mixup (Table 16, Appendix E.4) and clarified your concern regarding average rankings as well. Hence, we believe we have thoroughly addressed both of your concerns, and we kindly ask you to consider revisiting your score.

As the rebuttal period is now coming to an end, we wish to remind the reviewer to please check our response above. We believe we have addressed both of your unresolved concerns. Regarding the Mixup baseline, we have also benchmarked it on MultiNLI and CivilComments, where our finding remains consistent: CRM outperforms Mixup, especially regarding the worst group accuracy. We provide these results below, and for the reader's convenience, we state results on other benchmarks again.

We are very thankful of your response and score increase. To address your concern about the multi-attribute case experiment, we we have further augmented the CelebA dataset to include a total of 5 attributes. In addition to the (Gender, EyeGlasses) spurious attributes in the 3-attribute CelebA dataset (Appendix E.1), we further include (Hat, Earrings) attributes to result in a total of 32 groups with 5 attributes.

The results are presented below, where we find that CRM still outperforms the baselines for generalizing under compositional shifts, especially w.r.t the worst group accuracy.