Do causal predictors generalize better to new domains?

We thank the reviewer for the thoughtful feedback. Below, we aim to answer the reviewer’s concerns:

(Quality in terms of writing)

We will revise our work and adopt a more precise writing style, as well as reference in greater detail to what we are referring to.

(Hand-selected causal features)

it is also very likely the case that these chosen features are not actually causal

We agree with the reviewer that the chosen features are likely misspecified. We even state a similar concern in the discussion section (“We likely made some mistakes in this classification", line 271f.). Nevertheless, we tried out every conceivable way to obtain causal features (domain knowledge, remove single features from the ones obtained by domain knowledge, causal discovery algorithms, causal machine learning methods). If the reviewer knows another way to obtain the causal features in an empirical study, we would be happy to include it in our analysis.

assuming these algorithms can accurately identify causal features, any inconsistency would likely be due to incorrect hand-selected features

This is true. That’s why we tried out both approaches in our study: hand-selecting based on domain knowledge and applying causal discovery algorithms. The results don’t change though.

(Synthetic dataset)

I suggest that the authors create a synthetic dataset [...] to strengthen their claims.

Following your suggestion, we conducted synthetic experiments. The setup is depicted in the PDF of the author’s rebuttal. Our code is based on the synthetic study conducted by Montagna, et al., 2023. We refer the reviewer to their results for a detailed performance analysis of the causal discovery methods.

The synthetic experiments confirm our empirical findings. Using all features achieves best out-of-domain prediction accuracy. The one exception is if the distribution shift is exclusively on the anti-causal features and even in this case, a strong shift is needed before causal features achieve best out-of-domain accuracy.

(Robustness tests)

if the original causal features are incorrect, any subset chosen from these features is unlikely to improve generalization

Yes, the robustness test on the causal features merely tests for misclassifying one feature as causal although it is not. To enhance our robustness test, we also tested 500 random subsets for each task. The subsets are randomly drawn from all features. See Appendix C.5. for details. We welcome any suggestions to further test the robustness of our results.

(Empirical evaluation)

It is uncertain whether the comparison of accuracies [..] is a fair comparison. [...] Such differences in prediction performance might come [...] from the different hypothesis classes and model complexities.

We follow the example of preceding empirical studies when comparing standard machine learning methods, causal machine learning methods and domain robustness methods via accuracy (Gulrajani and Lopez-Paz, 2020, Miller, et al., 2021, Gardener, et al., 2023). The objective is the applicability in practice. It is of separate interest to explore and disentangle the reasons for the inferior performance of certain methods, e.g., different hypothesis classes and model complexities.

For example, one could use the same causal features but achieve better domain generalization by employing domain robustness methods

We train all feature sets in the main analysis with the same methods. In particular, we also train on the causal features on domain robustness methods (DRO, GroupDRO, an adversarial label robustness method).

address how their work might have potential limitations or potential negative societal impacts.

We will improve our limitation section and explicitly state our limitations and potential negative societal impacts.

References:
Montagna, F., Mastakouri, A., Eulig, E., Noceti, N., Rosasco, L., Janzing, D., ... & Locatello, F. (2024). Assumption violations in causal discovery and the robustness of score matching. Advances in Neural Information Processing Systems, 36.
Gulrajani, I., & Lopez-Paz, D. (2020). In search of lost domain generalization. arXiv preprint arXiv:2007.01434.
Miller, J. P., Taori, R., Raghunathan, A., Sagawa, S., Koh, P. W., Shankar, V., ... & Schmidt, L. (2021, July). Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In International conference on machine learning (pp. 7721-7735). PMLR.
Gardner, J., Popovic, Z., & Schmidt, L. (2024). Benchmarking distribution shift in tabular data with tableshift. Advances in Neural Information Processing Systems, 36.

We thank the reviewer for the thoughtful feedback. To answer the reviewer’s concerns and questions:

(Distribution Shift)

unclear what type of distribution shift

We are considering natural distribution shifts. For example, the distribution shift induced by switching between geographic regions or demographic groups. Therefore, we are likely to suffer from all three forms of distribution shift at once. A recent study by Liu, et al., 2023, shows that conditional shifts are prevalent in tabular settings.

precisely numerically mention how much the test distribution changed

Following your suggestion, we added a table with details on the observed distribution shifts in the PDF of the author’s rebuttal. We adapted the metrics for target shift, conditional shift and covariate shift from Gardener, et al, 2023. See Appendix E.2 of their paper for the detailed definitions.

Maybe the irrelevant/non-causal features are helping because the test distributions are not significantly different

We agree with the reviewer that one explanation of the empirical results is that the test distribution may not differ from the training distribution in a substantial way. In support of this argument is the fact that the training and testing domains are, for instance, different geographic regions within the U.S. bordering each other, or population groups with different education levels or racial identity.

(Datasets)

Please read the introduction section of [2], [5] for details. How do the authors plan to deal with such domain dependence and achieve generalization if they use all features

Lee, et al, 2023, and Subbaswamy, et al, 2019 propose proactive algorithms, anticipating potential sources of unreliability in the data. In that context, our study should be interpreted as testing how unreliable variables in common tabular data are. In other words, do we need to act proactive when dealing with tabular data encountering natural distribution shifts? Are there domain dependences that limit the generalization?

We didn’t find empirical evidence for that within our datasets. All datasets we studied are commonly used in empirical evaluations, and most of them are currently part of a benchmark for out-of-domain generalization. Hence, we conclude to utilize all features for prediction in common tabular datasets. Note that our conclusion does not extend to, in that sense, uncommon datasets with known high domain dependence.

They should also show their results on the datasets where causal predictors claim to perform well

We looked for tabular datasets with (i) public or open credentialized access and (ii) interpretable features that can be classified in ‘causal’ and ‘non-causal’. We haven’t found any dataset matching these criteria where the causal predictors performed well. We’d welcome any suggestions from the reviewer, and are happy to include these datasets in our study.

(Ablation Study)

perform an ablation study by removing one non-causal feature at a time [...] Why are the non-causal features helping the model to perform well?

We conducted an ablation study and provided the results in the PDF of the author’s rebuttal. We remove anti-causal and non-causal features one at a time and measure the corresponding out-of-domain accuracy. In a later comment, we discuss in detail the non-causal features whose removal significantly dropped the out-of-domain performance.

Are completely irrelevant features helping as well?

Our datasets are carefully curated. They are based on surveys that collect information experts in that field deem relevant. In some cases, features are selected from a multitude of variables, e.g., 23 features from the US Census data for predicting income. In addition, our datasets cover applications dealing with complex social dynamics (health, employment, politics,...). Therefore, it is hard to declare any feature as completely irrelevant.

(Confounders)

Are the authors considering the possibility of confounders?

We strongly believe that there are confounders within our datasets. Their existence is another plausible explanation for the inferior performance of the causal features. We do not model the data generating processes precisely enough to employ methods proposed by Lee, et al, 2023, and Subbaswamy, et al, 2019.

(PC Algorithm)

How do the authors obtain causal parents from PC (Line 240)?

The PC algorithm estimates a completed partially directed acyclic graph (CPDAG), see Figure 33 in Appendix C.4 as an example. We use the implementation from the R package ‘pcalg’. The edges connecting the target with other features are directed for the tasks ‘Food Stamps’, ‘Income’ and ‘Unemployment’. Therefore, we can readily identify these features as estimated causal parents. In the task ‘Diabetes’, the target is only connected to high blood pressure via an undirected edge. We treat high blood pressure as potential causal parent. If deemed necessary, we can also include all skeletons as figures in our appendix.

References:
Liu, J., Wang, T., Cui, P., & Namkoong, H. (2024). On the need for a language describing distribution shifts: Illustrations on tabular datasets. Advances in Neural Information Processing Systems, 36.

We discuss the non-causal features whose removal significantly dropped the out-of-domain performance. We split by task.

Food Stamps (food stamp recipiency in past year for households with child across geographic region):

• Relationship to reference person: There could be a stable and informative correlation within the survey of US Census between kind of household members (encoded in relationship to the reference person/head of the household, e.g., multiple generation household vs roommates) and food stamp recipiency. We didn’t classify this variable as causal, as it’s survey related.

Income (income level across geographic regions)

• Relationship to reference person: same argument applies.
• Marital status: Marital status and personal income are both intricately linked with socio-economic status, although we haven’t found any research of causally linking them together.
• Insurance through a current or former employer or union / Medicare for people 65 or older, or people with certain disabilities: These insurances are benefits not tied to income, but rather the person’s employer or age and medical condition. They are however indicative of the economic and social environment of the individual, which helps to classify the income level.
• Year: The year, e.g., 2018, encodes information about the economic status, which may be predictive across geographic regions.

Public Coverage (public coverage of non-Medicare eligible low-income individuals across disability status):

• State/year: The current state of living and year encode information about the economic status.

Voting (voted in the US presidential elections across geographic regions):

• Party preference on specific topics, e.g. pollution
• Opinion on party inclinations, e.g., which party favors stronger government
• Opinion on sensitive topics, e.g., abortion, religion, gun control

The opinions/preferences of an individual may sort them to specific sub-groups of the populations, wherein civil duty is or is not prominent. It is fathomable that similar sub-groups form across geographic regions.

Hypertension (High blood pressure across BMI categories):

• State: The current state of living encodes information about the socio-economic status, which research linked to hypertension in several studies (Leng, 2015).

Sepsis (sepsis across length of stay in ICU):

• Hospital: Hospitals serve different groups of the populations, which may be correlated with different risks of attaining sepsis.

References:
Leng, B., Jin, Y., Li, G., Chen, L., & Jin, N. (2015). Socioeconomic status and hypertension: a meta-analysis. Journal of hypertension, 33(2), 221-229.

Thanks to the authors for their efforts in their submitted manuscript and their detailed rebuttal. I apologize for replying late. Below I share my concerns based on the authors’ responses.

The authors mentioned that their considered datasets are likely to suffer from all three forms of distribution shift at once. They provided covariate shift, concept shift and label shift for all datasets.

However, in my opinion, these shifts should be measures for specific variables (weakness 1) instead of the whole dataset. For example: conditional distribution for which variable changes the most compared to other variables when you consider a new domain. That would explain the source of shift, and hint us why a predictor performs good/ bad. These information are more insightful instead of seeing a number for the whole dataset.

The authors mentioned, “do we need to act proactive when dealing with tabular data encountering natural distribution shifts? Are there domain dependences that limit the generalization?”

I understand the authors’ arguments about causal predictors performing bad. However, my perspective is that if a predictor using all the features does not perform bad in their considered new domain even in the presence of different types of shifts, should we consider that a new domain at all? Does it contain significant amount of shift to be considered as a new domain?

Hence, we conclude to utilize all features for prediction in common tabular datasets.

Again, I understand the authors’ claim about causal predictor performing bad in the considered common datasets and the fact that using all features gave better accuracy. However, there are many existing works, which show how the prediction gets domain specific or highly dependent on sensitive variables if we condition on all variables. Are we proposing a method with new failures modes?

We strongly believe that there are confounders within our datasets.

To me, the role of confounders was not explicitly mentioned in the main paper. It should be explicitly mentioned as one of the possible reasons of why causal features do not perform well.

5. A question to the authors: How does a predictor perform better in new domains if the prediction is dependent on “State/Year/hospitals”? Aren’t these one type variables that create new domains? This means that the trained predictor would be domain specific and fail when deployed in different state/year/hospitals.

6. A final message to the authors is that their empirical results are very impressive. However, these results should not mislead readers about using causal features for prediction tasks. Different assumptions and possibilities (ex: presence of counfounders) should be explicitly mentioned.

Based on authors’ responses and the discussion with other reviewers, I would consider changing my scores.

We thank the reviewer for their helpful comments and positive feedback! To answer the reviewer’s comments:

how the domain split is done [...] can be described more clearly

Thank you for drawing our attention to that. We will state how we obtain the domain splits more clearly.

Anti-causal improves out-of-domain performance of arguably causal features (ll 221): Do you have an explanation/hypothesis?

A reasonable explanation is that the relationship of the anti-causal feature and target does not change strongly between domains. For example, patients with diabetes have a considerably increased risk of cardiovascular disease. Their relationship is partly explained by biomedical mechanisms, e.g., dyslipidemia (abnormal levels of lipids in the bloodstream) is a common feature of diabetes and poses a significant risk factor for cardiovascular diseases (Schofield, et al., 2016). It is conceivable that these mechanisms are stable across domains (population groups with different racial identity).

The study only provides empirical results [...] hard to make any more general claims.

To address your concerns, we conducted synthetic experiments to test how far our claims generalize. The setup is depicted in the PDF of the author’s rebuttal. Our code is based on the synthetic study conducted by Montagna, et al., 2023.

The synthetic experiments confirm our empirical findings. Using all features achieves best out-of-domain prediction accuracy. The one exception is if the distribution shift is exclusively on the anti-causal features and even in this case, a strong shift is needed before causal features achieve best out-of-domain accuracy.

References:
Schofield, J. D., Liu, Y., Rao-Balakrishna, P., Malik, R. A., & Soran, H. (2016). Diabetes dyslipidemia. Diabetes therapy, 7, 203-219.

We thank the reviewer for their helpful comments and amazing feedback! To answer the reviewer’s question:

add an experiment with simulated data, with varying degrees of difference in the out-of-distribution data [...] (something similar to what is done in anchor-regression literature)

Following your suggestion, we conducted synthetic experiments. The setup is depicted in the PDF of the author’s rebuttal. Similar to Rothenhäusler, et al., 2021, we vary the degree of difference in the out-of-domain data using shift intervention on target, features and confounders.

The synthetic experiments confirm our empirical findings. Using all features achieves best out-of-domain prediction accuracy. The one exception is if the distribution shift is exclusively on the anti-causal features and even in this case, a strong shift is needed before causal features achieve best out-of-domain accuracy.

References:
Rothenhäusler, D., Meinshausen, N., Bühlmann, P., & Peters, J. (2021). Anchor regression: Heterogeneous data meet causality. Journal of the Royal Statistical Society Series B: Statistical Methodology, 83(2), 215-246.

We thank the reviewer for their helpful comments and positive feedback! We will fix the typos, thanks for pointing them out to us. To answer the reviewer’s questions:

add the sample size to Table 1?

Yes, we will add the sample sizes to Table 1. Note that they are currently provided in Table 6 in the Appendix.

elaborate a bit more and interpret the results wrt to the strength of the domain shift

We added a table with details on the strength of the domain shifts in the PDF of the author’s rebuttal. We adapted the metrics for target shift, conditional shift and covariate shift from Gardener, et al, 2023.

We will explain and interpret the results in more detail, taking into account the strengths of the domain shifts.

these findings also translate to genomics [...] Have you considered this?

No, we have focused our attention on datasets with easily interpretable features so far. If the reviewer has a suggestion on a plausible task based on genomics data and an appropriate natural distribution shift, we’d be happy to include that in our analysis.