Balancing Interference and Correlation in Spatial Experimental Designs: A Causal Graph Cut Approach

Applications to other data examples

We sincerely appreciate your valuable feedback. In response, we conducted extensive investigations during the rebuttal, including: (1) empirical validation on additional data, (2) methodological extensions, (3) theoretical investigations. The methodological investigation is elaborated in our response to Reviewer VLbx.

Empirical validation on environmental data. During rebuttal, we found a paper that studies the causal effect of wind speed on PM10 levels (Zhang et al., 2024, arXiv:2409.04836, Section 6). As the raw dataset cannot be directly used for evaluation (refer to our first response to Reviewer 96AW), we utilized the simulator described in their paper for comparison. The results, shown in Figure, demonstrate our proposal achieves smaller MSEs compared with the benchmarks, confirming its applicability to environmental applications.

Methodological extensions. Our primary focus is Example 1, but we have extended our design to handle two additional settings: (i) a single-experiment setting without repeated measurements and (ii) a multi-experiment setting allowing the carryover effects over time. For the single-experiment setting, our approach remains effective given prior knowledge of the covariance structure, as confirmed by our newly conducted numerical study. For multi-experiment settings with carryover effects, we have outlined the methodology while reserving its implementation for future work, as detailed in our response to Reviewer VLbx. Collectively, these extensions enable our framework to accommodate the applications described in Examples 2 and 3.

Theoretical investigation. We have identified crucial assumptions that guarantee the validity of our proposed extension in the multi-experiment setting (ii). Specifically, we would require a Markov assumption (Puterman, 2014, John Wiley & Sons) and a temporal mixing condition (Bradley, 2005, Probab. Surv.) to maintain the covariance estimator's consistency in the presence of temporal dependencies. Similar conditions are widely employed in the RL literature for consistent estimation (Kallus and Uehara, arXiv:1909.05850).

Covariance assumption

We have comprehensively addressed your concern in the following three ways: First, we conducted extensive empirical validation under violations of the covariance assumption. Second, we clarified the role of the non-negativity assumption. Third, we developed methodological extensions for scenarios where Assumption 2 fails to hold.

Numerical experiments: We conducted another new experiment during rebuttal to illustrate the robustness of our proposal. We design a periodic covariance function $\Sigma_{ij} = 1 - \rho \times ((i - j)\mod 3)$. Under this choice, the decaying covariance assumption no longer holds and the non-negativity assumption is violated when $\rho>0.5$. The results, shown here, demonstrate that our methods still outperform baselines.

We also remark that in both our ridesharing simulator and the PM10 dataset (see our first response), we find violations of the two assumptions. Nonetheless, our designs remain highly competitive. These results consistently demonstrate our proposal's robustness.

Non-negativity: The non-negative assumption is primarily employed in Prop. 1 & 2 to demonstrate the trade-off between interference and correlation. It is not required to ensure the proposed surrogate function forms a valid upper bound (Prop. 3).

Methodological extension: While the decaying covariance assumption (Assumption 2) is common in spatial statistics (Cressie, 2015), our method remains flexible when this condition is violated. In such cases, we propose a simple fix: scaling the first term in loss function (2) by a constant $C > 1$ to ensure it remains an upper bound. The hyperparameter can be optimally determined through simulation studies based on experimental data evaluating the performance of the resulting estimator across different $C$ values.

MSE of ID in Figure 5(b)

Your are right that the asymptotic MSE of ID shall remain constant regardless of the number of repetitions $N$. However, in finite sample, the MSE actually improves with $N$, due to the estimation of the $g$ function.

Specifically, in theory, Neyman orthogonality guarantees that the asymptotic MSE will match that with an oracle g given sufficiently large $N$. In practice, we estimate $g$ by incorporating all prior data at each experiment. This leads to an initial period where the MSE decreases as $N$ increases, reflecting the improvements in the estimation of $g$. As the estimated $g$ approaches its oracle value, the MSE then stabilizes. The trend shown in Figure 5(b) clearly demonstrates this pattern.

We sincerely appreciate your thoughtful review and valuable feedback on our work. Your positive assessment is particularly encouraging to us. Below, we provide detailed responses to each of your comments.

Real data analyses

While we do have a real-world dataset, it cannot be directly used for evaluation. To properly assess a given design, one would need to: (1) implement the design to generate corresponding data, and (2) compute the MSE from this generated data. This is precisely why we developed our real-data-based simulator, which enables adaptive data generation tailored to different experimental designs for comparison.

During the rebuttal, we identified another publicly available real-world dataset that examines the causal effect wind speed on PM10 levels [1]. However, as with our ridesharing dataset, this raw observational data cannot be directly used for evaluation. We therefore employed the simulation model from [1] to generate synthetic data under different designs. Our simulation results, visualized in Figure, demonstrate that our method achieves significantly lower MSEs compared to existing designs.

Rationality of the synthetic outcome model (line 906)

Our outcome regression function follows the nonparametric model from [2] (Page 24), which studied the spatial A/B testing problem as well. We selected this model for two reasons: First, the outcome is a complex nonlinear function of observations, treatments, and spillover effects. Second, it naturally incorporates spatial heterogeneity through latitude/longitude coordinates, allowing for geographic variation. We consider these features to approximate the complex dynamics present in real-world settings.

Limitations

There are a couple of limitations of our proposal:

• Theoretically, our characterization of the interference-correlation trade-off relies on the assumption of non-negative covariance functions. While in practice we expect the presence of some negative covariances would not alter our conclusions, it remains unclear how to elegantly relax this mathematical constraint while preserving our theoretical findings.

• Methodologically, we omit the second-order interference term in the objective function to facilitate the optimization. We acknowledge that such higher-order effects may be non-negligible. However, its inclusion would significantly increase the optimization complexity. Developing a computationally tractable solution that properly accounts for this term remains a practical challenge for future work.

• In terms of applications, our methodology primarily targets experimental settings where independent experiments can be repeatedly conducted over time. While this framework aligns well with our ridesharing application for spatial A/B testing (the focus of this paper), it may require adaptation for other settings. We have discussed modifications for scenarios with either a single experiment or multiple experiments with carryover effects in our response to Reviewer VLbx. Extending our approach to more general experimental settings would be a valuable direction for future research.

Typos

Thanks for pointing them out. They will be corrected.

[1] Zhang, W., et al. Spatial Interference Detection in Treatment Effect Model.

[2] Yang, Y., et al. Spatially Randomized Designs Can Enhance Policy Evaluation.

We greatly appreciate your constructive comments and your positive assessment of our paper. We focus on your major comments below.

Comparison with Viviano et al. (2023)

Difference in objective: One of the key difference lies in the choice of the estimator. Specifically, our estimator explicitly accounts for interference and remains unbiased regardless of the interference pattern (we will make this clearer in the paper), whereas Viviano et al.'s IS estimator is biased under interference. As such, although both minimize MSE, our optimization focuses exclusively on variance reduction whereas they must jointly consider both bias and variance.

Gain from repeated experiments: Having repeated experiments allows us to accurately estimate the covariance function, which is crucial for our approach to achieve adaptivity. In comparison, such estimation is not feasible in Viviano et al.'s setting. As such, they adopted a minimax formulation that considers the worse case across all covariance functions.

Following your suggestion, we conducted an ablation study during the rebuttal to distinguish the improvements from having repeated experiments. We consider single-experiment settings with certain prior knowledge regarding the covariance function, e.g., having access to a noisy covariance matrix. We kindly refer you to our response to Reviewer VLbx for the experimental results (see Extensions to Single-Experiment Settings).

Theoretical claim 1

The inequality holds due to the presence of $R$ in (2) and the indicator function $\mathbb{I}(\mathcal{N}\_{i'} \cap \mathcal{C} \neq \emptyset)$ in $I_1$. The key step in establishing this inequality lies in upper bounding the indicator by $\sum_{\ell \in \mathcal{C}} W_{\ell i'}$. This inequality holds because when $i'$ is adjacent to $C$, there exists at least one $i\in \mathcal{C}$ such that $W_{ii'}=1$; see this Figure for a graphical illustration. When restricting to two clusters, this leads to bounding $I_1$ by two triple sums $\sum_{i\in \mathcal{C}\_1}\sum_{i'\in \mathcal{C}\_2}\Sigma_{ii'}^+\sum_{\ell'\in \mathcal{C_1}}W_{\ell i'}$ and $\sum_{i\in \mathcal{C}\_2}\sum_{i'\in \mathcal{C}\_1}\Sigma_{ii'}^+\sum_{\ell'\in \mathcal{C_2}}W_{\ell i'}$. The outermost summation in each triple (over $i$) produces terms proportional to the cluster sizes $|\mathcal{C}\_1|$ and $|\mathcal{C}\_2|$, whose sum adds up to $R$ (the number of regions). As such, the sum of all units in the other clusters is accounted by the factor $R$ rather than terms that depend only on the boundary regions.

Theoretical claim 2 & Question 2

Three conditions are needed to consistently estimate covariance:

• Each $g_i$ can be consistently estimated;
• Each error $e_i$ is additive, i.e., independent of the covariates and treatment;
• The error-covariates pairs are independent across experiments.

Note: The independence condition (3) can be relaxed to certain mixing condition that allows for temporal dependence, provided the dependence decays sufficiently quickly over time (Bradley, 2005, Probability Surveys).

Theoretical Claims 3

There are some differences between the two approaches, which we clarify below. In our proposal, we remove only the high-order interference term, and keep the first-order interference term in the objective function. To the contrary, Viviano et al imposes a weak interference assumption that removes all relevant interference terms when calculating the worst-case variance (see their proof of Lemma 3.2).

New experiments

The baselines' inferior performance partially stems from their reliance on IS, while our method, ID and GD employ DR. To mitigate this effect, we conducted a new experiment by increasing variance of the random error, so as to reduce the impact of variance reduction by DR. Results show that the modification reduces (but does not close) the performance gap between our method and the baselines compared to Figs 5 & 6.

Meanwhile, we conducted another experiment under a combination of (i) various covariance structures, (ii) the magnitude of correlation $\rho$, and (iii) the number of grids $R$ to report the contributions of SC, $I_1$, $I_2$. Results suggest that $I_2$ can be proportional to $I_1$. This is because the plots visualize the MSE of our estimated optimal design rather than the oracle optimal design. Since our objective function involves only $I_1$ and not $I_2$, it reduces $I_1$ by increasing $I_2$, making $I_2$ comparable to $I_1$.

Question 1

Will revise the definition as an average over $R$ regions to maintain consistency with the literature.

Methods And Evaluation Criteria

We appreciate your thoughtful comments, which are mainly about our settings with repeated and independent experiments. We believe your concerns may arise from certain misunderstandings of our paper, and we appreciate the opportunity to clarify them. We address them in three ways: (1) clarifying our focus on spatial A/B testing; (2) demonstrating that our approach is readily applicable to single-experiment settings, supported by promising numerical results; (iii) showing that our approach can be extended to handle carryover effects (the delayed treatment effect you mentioned).

(1) Focus of this paper. We clarify that this paper focuses on the spatial setting -- specifically, Example 1 (A/B testing in marketplaces) -- and not network A/B testing. While many designs from network experimentation can be adapted to our setting (and we compared them numerically), our primary focus, as stated on Page 1 (the last sentence), remains Example 1.

In applications like ridesharing, experiments can be conducted daily, with data across days treated as independent. This is due to the drop in demand early in the morning (1 -- 5 AM), ensuring each day’s data represents an independent realization. Such settings are well-adopted in prior work (Li et al., 2023, NeurIPS; Li et al., 2024, ICML; Luo et al., 2024, JRSSB). Another application occurs in marketing auctions, where daily budget resets eliminate carryover effects, making the independence assumption plausible (Basse, 2016, AISTATS; Liu, 2020, arXiv:2012.08724).

(2) Extensions to single-experiment settings. Our core methodology does not rely on this assumption of repeated experiments. It remains applicable to single-experiment settings when we have certain prior knowledge regarding the underlying covariance matrix, either from a pilot study or based on historical data.

To satisfy the practical requirement, we utilized a proxy covariance matrix, obtained by inserting noises into the true covariance matrix, and conducted additional experiments during the rebuttal. The results demonstrate that our estimator: (i) maintains optimality against existing methods, (ii) achieves near-oracle performance (comparable to the oracle method with the true covariance matrix), (iii) remains robust to the approximation errors.

We also remark that while cross-fitting helps simplify the theory (by avoiding the need for imposing VC-class conditions on $g$ to establish the asymptotics of the ATE estimator), our method remains effective without it - numerical results above were obtained without cross-fitting.

(3) Extensions with carryover effects. The target here becomes the cumulative ATE aggregated over time. To handle carryover effects, we can use existing doubly robust estimators from the RL literature (Kallus & Uehara, 2022, OR); see also first equation on Page 18 of Yang et al. (2024). The resulting estimator maintains a similar form to ours, with two modifications: (i) the function g is replaced by a Q-function, and (ii) the IS ratio is replaced by its marginalized counterpart (Liu et al., 2018, NeurIPS). Treatments over time can be assigned using switchback designs, which are widely adopted (Bojinov, 2023, MS; Xiong et al., 2024, arXiv:2406.06768). As shown in Theorem 5 of Yang et al. (2024), the MSE of this ATE estimator has a similar closed-form expression, enabling us to apply the same decomposition and design a similar surrogate loss for optimization.

Experimental Designs or Analyses

In the rebuttal, we considered other topologies and conducted additional experiments. Results reported here demonstrate the advantages of our proposal over baselines.

Essential References

We are happy to include these additional references and discuss their findings as you required. However, as we discussed, we primarily focus on the spatial setting, and references on network experimentation might not be essential.

Bernoulli randomization

The Bernoulli randomization is the same as our individual design. While Kallus (2018) established its minimax optimality without additional assumptions, we also proved its optimality in Proposition 2 without spatial interference. However, in settings with spatial interference, such a design is no longer guaranteed to be optimal. Specifically, Theorem 3.4 in Viviano et al. 2023 shows that cluster randomization outperforms Bernoulli randomization when the interference is not too weak. We also provide theoretical (Proposition 1) and numerical (extensive experiments) evidence that Bernoulli randomization becomes suboptimal in such cases.