SpaCE: The Spatial Confounding Environment

We are extremely thankful for your positive feedback. We have made revisions following your suggestions, marked with magenta color in the revised text. We address your comments in order.

1. We experimented with variations of Fig. 2 but found it the most convenient solution to clarify in the caption that the correlations are stronger as s and s’ get closer.

2. It is true that we had space limitations and had preferred to expand on clarity at the expense of the analysis of the results. Nonetheless, based on your suggestion, we moved some of the description of data collections in Section 3 to the appendix to make more space for the analysis of the experiments. We thus added two new analysis paragraphs in Section 5. The first of these paragraphs presents the results by environment in Fig. 10 of the appendix, providing some new insights, for instance, about the case when the graph neural network outperforms other models. The second paragraph presents a new analysis of the performance by confounding and smoothness score. We use mixed effects regression model adjusting by random effects by environment, allowing us to conclude that higher errors are associated with lower smoothness scores and higher confounding scores as desired. We hope that these new paragraphs increase the impact of the results, as suggested by the reviewer.

3. Thank you for pointing out the two minor typos. These have been fixed in the revision.

We are thankful for your positive remarks on the structure of the paper and your accurate summary. We have made improvements in Section 4 of the revision based on your comments, marked with red color.

Perhaps there is still some confusion about the counterfactual generation procedure. The process can be summarized in four steps: First, we learn $f$ that best predicts $Y_s$ from $(X_s, A_s)$ using AutoML. Second, we estimate the empirical additive errors $\hat{R}\_s = Y_s - f(X_s, A_s)$ and their joint (spatial) distribution $(\hat{R}\_s)\_{s\in\mathbb{S}} \sim P_R$. Third, we replace these endogenous empirical residuals with an independent similarly distributed exogenous noise $(R\_s)\_{s\in\mathbb{S}} \sim P_R$. Finally, we obtain counterfactuals by varying the treatment while holding constant the confounders and the exogenous noise. We added this summary in Section 4 for additional clarity.

We answer your questions directly. As quick remarks, $R_s$ is indeed the exogenous noise (Q3), and the abduction step is indeed what the reviewer suggests as the typical thing to do (Q2).

The reviewer is concerned about the limitations of additive noise models (ANMs) (Q1, Q3). First, notice that while eq. (1) is additive, SpaCE's benchmark datasets can exhibit complex forms of non-additive noise due to the interactions with the missing confounder, which becomes part of the noise when masked. The post-masking causal model can be written as $\tilde{Y}_s=f(X^\text{obs}_s, A_s, U_s)$ where the effect of the noise $U_s=(R_s, X_s^\text{miss})$ is likely not additive due to interactions of $X_s^\text{miss}$. Second, it is worth noticing that ANMs remain the focus and leading open challenge in the spatial confounding literature that our paper builds upon (c.f., [2,3,4,5]). Are the reviewer's concerns motivated by the use and limitations of ANMs in causal discovery tasks [1, 6]? The spatial confounding literature we build upon mainly concerns causal effect estimation, not discovery. While there can be challenges for cause-effect estimation under non-additive noise models too, simple techniques complementing spatial confounding methods for ANMs have been used; for example, with generalized linear models (e.g., Poisson regression) or outcome transformations (e.g., logarithms) for count-based or binary outcomes [5]. Thus, SpaCE's benchmark datasets are highly relevant to the current state of the art in spatial confounding adjustment.

Regarding potential failures (Q1), the ANM will be a sensible model when the empirical residuals are approximately symmetric and continuously supported, as exemplified in Fig. 3c. We have documented the outcome transformations and stored histograms of the empirical and synthetic residuals with each environment in the Harvard Dataverse platform to ensure validity. We have also emphasized this in the revision.

References

• [1] Janzing & Steudel, 2010. “Justifying additive noise model-based causal discovery via algorithmic information theory”
• [2] Kamal et al. 2023. “Spatial Causality: A Systematic Review on Spatial Causal Inference”. Geographic Analysis.
• [3] Dupont et al. 2022. “Spatial+: a novel approach to spatial confounding”
• [4] Paciorek, 2010. “The importance of scale for spatial-confounding bias and precision of spatial regression estimators”
• [5] Urdangarin et al. 2023. “Evaluating recent methods to overcome spatial confounding”
• [6] Peters et al. 2004. "Causal Discovery with Continuous Additive Noise Models."

Thank you for your positive notes on the paper's strengths. We now address your other comments and questions. The text in the revision addressing these comments will be highlighted in blue.

We included your recommended citations for works related to causal evaluation (outside spatial confounding) in the introduction.

Regarding the deferral of interference to future work, this decision came after careful consideration. The three main reasons are the lack of ensembling methods allowing for spill-overs, the need for spill-over-specific metrics, and the different dataset domains typically considered in the literature. We clarify below:

Lack of ensembling and ML techniques for interference. The literature has yet to significantly develop ML models for counterfactual generation under interference. Most existing approaches typically rely on limited manually constructed spill-over exposure maps characterizing interference (e.g., assuming that the nature of interference depends only on the average treatment of neighbors) [1,4]. Graph neural networks (GNNs) are promising to reduce the dependency on manually constructed features [2], but extensive evaluation and research justifying their correct utilization is still needed [3]. Even if considering GNNs, a core design element in SpaCE uses AutoML (including boosting, random forests, NNs, etc.) to avoid artificially favoring a specific machine learning model. Currently, generated synthetic datasets do not exhibit spill-overs since, conditional on the confounders, AutoML baselines do not allow local outcomes to depend on non-local treatments.

Contrasting estimands and metrics. The focus of interference research is on estimating spill-over/indirect effects vs direct effects [1]. Spill-over effects, as typically defined [4], are specified in terms of assumptions in changes in the distribution of the treatment of neighbors. These estimands and associated metrics must be defined and explained in detail, deviating from those used for confounding. Tests and evaluations for interference would also require different algorithmic baselines. Addressing these differences within the space limitations of a single conference paper would have required a significant deviation at the expense of clarity and focus on the increasingly important topic of spatial confounding.

Different focus on the nature of the data. It is true that, as pointed out by the reviewer, some of our raw data collections could exhibit spillovers. However, the interference literature has more frequently focused on other types of networked data, such as social networks [1, 5], which have traditionally been outside the main concern of the spatial confounding literature. Based on the reference recommended by the reviewer, Cheng et al. (2022), it can also be pointed out that some datasets for interference already exist based on social networks, while no current solution using realistic representative data exists for spatial confounding, which is the important gap filled by SpaCE.

We have included a summarized version of this discussion in section 6.

References

• [1] Forastiere et al., 2021. “Forastiere, Laura, Edoardo M. Airoldi, and Fabrizia Mealli. "Identification and estimation of treatment and interference effects in observational studies on networks."
• [2] Ma & Volker., 2021. “Causal inference under networked interference and intervention policy enhancement”
• [3] Jian & Sun., 2022. “Estimating Causal Effects on Networked Observational Data via Representation Learning”.
• [4] Hudgens & Halloran, 2008. “Toward causal inference with interference.”
• [5] Cheng et al. 2022. “Evaluation methods and measures for causal learning algorithms.”

Dear reviewer, as the author/reviewer discussion period ends today, we would be very grateful if you could indicate if you have additional questions. We are understanding that the deadline is very tight. We also highlight to the reviewer the additional analysis and experiments indicated in color text in the revision and summarized in the comment to all reviewers here.

Thank you again for your thoughtful comments and suggestions that improved our work during the revision. Answers to all questions raised by the reviewers were posted a few days ago. The revised text indicates the changes with colored lettering.

If the reviewers have remaining questions, we would be very grateful if they could indicate so before the author/reviewer discussion period ends tomorrow.

For your convenience, we summarize our main revisions here. Please refer to the individual responses for additional explanation and other comments not mentioned here.

• Citations about evaluation protocols in causal inference outside the spatial confounding literature have been added to the introduction and metrics section (R1).
• A more complete discussion in Section 6 of the additional orthogonal challenges required for spill-over/interference analysis was provided to justify the deferral to future work (R1).
• We have provided additional justification in Section 4 for using a generative non-linear additive noise model and explaining why with this choice SpaCE's benchmark datasets represent the current state of the art and concerns in the spatial confounding adjustment literature we build upon (R2).
• Expanded and improved the explanation of the data-generating model in Section 4, particularly the role of the exogenous noise $R_s$ (R2).
• Significantly expanded the analysis of experiment results in Section 5, adding two new paragraphs with an analysis of the performance by environment and confounding/spatial scores (R3).
• Fixed minor typos and improved the caption of Fig. 2 (R3).