A/B testing under Identity Fragmentation

We thank you for your review and feedback on our paper. We address your questions and concerns below.

Dataset and Experiment:

• Since the use case is motivated by application in online experimentation, data for these settings is usually private or unavailable. As such we have to rely on simulations and semi-synthetic data. Furthermore, unlike standard problems like regression, the underlying ground truth is unavailable, necessitating use of simulated data to be able to verify the predictions. This is also the approach used in related works [5]. We would also like to draw your attention to the AirbNb simulation has been generally used for a variety of papers dealing with experimentation on digital marketplace [2,3]. However, to incorporate the reviewers suggestion, we have evaluated our method on a real observational air-pollution dataset used in [1]. Its setting is slightly different from ours, as it focuses on spatial interference from neighbouring power plants. But we can adapt our method to the use case. We have added these results in the Sec 5.4, and added details in the supplementary material.

Contribution:

• The novelty of this work is in demonstrating identifiability of GATE under identity fragmentation which has not been done before. The second novelty comes in considering the case of imperfect knowledge of the device graph which has also not been done. We do so via transforming this problem into a noisy regression problem. The proposition is a novel claim of our work. Its derivation relies on a theorem (stated in Theorem1) from earlier work [4].

Questions:

• Q1) Our model is linear in computation of exposure but non-linear in exposure itself. We draw your attention to the function $g$ in A2 which is a potentially non-linear function applied to the exposure values.

• Q2) Our contribution is in highlighting how identity fragmentation invalidates standard estimators used in A/B testing, and how one can still identify the treatment effects by suitable transformation of the problem. This is important as without identifiability, learning model parameters need not be valid in deriving treatment effects. Next we used the VAE-based method to estimate the treatment effect. In principle other methods for variational inference like normalizing flows etc can also be used for learning the model parameters.

• Q3) The noise distribution as long as independent should be tractable in our setting. While in principle one can posit a model for time-series outcomes as well, it is not clear that identifiability of the mean function holds with non-independent errors. Analysing that is an important direction of future research.

References

[1] Adjusting for unmeasured spatial confounding with distance adjusted propensity score matching, Papadogeorgou et al. Biostatistic, 2019

[2] Cluster Randomized Designs for One-Sided Bipartite Experiments, Brennan et al. Neurips 2022

[3] Interference, bias, and variance in two-sided marketplace experimentation: Guidance for platforms, Li et al, WWW 2022

[4] Advances in the measurement error literature, Susanne M Schennach. Annual Review of Economics, 2016

[5] Graph agnostic estimators with staggered rollout designs under network interference. Cortez et al, Neurips 2022

Thank you for your thoughtful review and positive feedback on our paper. We are pleased to hear that you appreciate the idea, write-up, and the overall presentation of our proposed solution. We take this chance to address the shortcomings and questions you have raised.

Would it be better to use a probabilistic matrix and rely on a threshold to decide the neighborhood?

• We draw the reviewers attention to Remark 1 in the paper where we mentioned something along this line of using confidence in an edge to determine neighbourhoods. Conservatively choosing a threshold for inclusion of an edge, will likely result in including a number of extraneous edges, which motivated assumption A7. The problem comes from missing an influence which can lead to biased estimates. From Figure 5.3 we can see that there is a potential bias-variance tradeoff with including few edges increasing bias while adding more edges adding variance. One can try to choose the threshold for including an edge based on the importance of these two factors.

the actions determining Y are completely different on each device in N ?

• Assumption A2,A3 support heterogeneity i.e. behaviour on the nodes belonging to the same user need not be similar. Specifically, even for the same treatment allocation z and the same neighbourhoods, two devices can have different true exposures (as $\phi$ depends on $X_i$). However, since our method does not compute true exposures, it cannot determine which nodes are actually influencing the outcomes.

Would adding the temporal (time) aspect of the model improve treatment estimates?

• This is an interesting question, which is a potential future direction of research. While time-series will provide more information, lack of independent errors mean that the result in this work does not ensure identification of the model.

Other issues

• We have added information about this in the Experimental Section, with greater details added in the Appendix. We would also like to draw your attention to the fact that the purpose of this paper is precisely to estimate treatment effects without resorting to identity linking. The superset assumption is motivated exactly by the failure to do identity linking. Since, coarse information can be used under certain contexts, one can create these supersets with such coarse information. Our method maintains privacy, while allowing for estimation of GATE.

Typos and Conclusion

• We have fixed the typo issues and slightly reformatted the paper to address some other points raised by reviewers (such as adding conclusion)

We thank the reviewers for their valuable feedback. We have revised the draft to address your concerns, and give summary answers below.

W1-a

• The novelty of this work is in demonstrating identifiability of GATE under identity fragmentation which has not been done before. The second novelty comes in considering the case of imperfect knowledge of the device graph which has also not been done. We are agnostic to how neighbourhood $M(i)$ is obtained. As such in the experiments for 5.1 we randomly added extra edges in the true-graph to obtain the neighbourhoods. This was hinted in footnote 4 but we see that we can make this clearer. We have revised the draft to make this explicit.

• Regarding the relationship between the covariate X and the randomly generated "random device graphs," X plays no role in determining neighbourhoods $M(i)$. We see that it might not have been sufficiently clear in the manuscript, but our approach is agnostic to how $M(i)$ is obtained. In our experiments,X pertains to device-level covariates influencing outcomes conditional on treatments, such as location, phone type, demographic information, or other relevant details available to marketers at the device level. Although X could theoretically be used to determine neighborhoods M(i), our approach remains agnostic to the method of obtaining M, treating it as a given parameter. We have explained this better on Page4 of the revised version.

W1-b

• $\tau$ is defined via Equation 1. One the variational model p_\theta is estimated, $Y_i$ is directly specified by p_\theta(Y_i|X_i,z). We draw your attention to Sec 4.2, where the exact probabilistic model is mentioned. Since p_theta directly specifies the means $\mu_Y$, One then estimates $\tau$ as $1/n \sum \mu(Y|\vec{1},X_i) - \mu(Y|\vec{0},X_i)$, which is just equation 1, where the potential outcomes are replaced by the estimate $\mu$ obtained from p_\theta. We have now added these details in Sec 4.2 and provided more information in the supplementary material.

W-2 a

• The experiments in Sec 5.1, are to demonstrate the correctness of our method when all its assumptions hold. The similarity between the generative model in 5.1 and A2, is so that the probabilistic model is correctly specified. We would also like to highlight that A2 and A3, are general enough to cover many sort of exposure models used in literature [1,2,3]. We have made this clear on Page 5. Furthermore, in Section 5.2, we experiment with the airbnb model which does not satisfy A2. (In fact it does not satisfy any exposure assumption). In such a case one can get biased estimates, but as can be seen from Figure 4, all other estimates (except HT) are also biased, and HT requires knowledge of the true graph.

W-2 b

• DM and Poly-regression in our experiments stems from their common usage as estimators in the literature. Poly-regression fits a polynomial between exposure (typically the number or fraction of treated neighbors) and observed outcomes, while DM represents the classic difference in mean estimate discussed in Section 3 under SUTVA. We have incorporated this information into the supplementary material, providing additional experimental details for clarity.

Questions:

1 We used single layer MLPs for all the functions. We have added this in the revision (Remark 3)

2a X plays no role in device graph generation. We independently create a device graph and populate covariates X at the nodes. While in principle one can use the node level X to guess the device graph, we simply assume the existence of the device graph. This is because, as long as A1-7 hold, the model remains identifiable, irrespective of how the estimated device graph $M$ was created. That said practically, having poorer/dense graphs adds significant variance to the estimates. Since we assume $M$ to be given in all our experiments, covariates X are simply used as additional covariates in specifying the mean function parameterizing the outcomes.

2b Bias is computed as the estimated value $\hat{\tau} - \tau$, where $\tau$ is the true treatment effect. In both experiments, we have access to the underlying generative model, which is used to produce data.As such the true treatment effect can be directly obtained. In the figures we plot the relative bias i.e. $(\hat{\tau} - \tau)/\tau$. Relative RMSE is computed in a similar way as root mean square error between $\hat{\tau}$ and $\tau$, and then normalized by $\tau$

References

[1] Estimation of causal peer influence effects, Toulis and Kao, ICML 2013

[2] Cluster Randomized Designs for One-Sided Bipartite Experiments, Brennan et al. Neurips 2022

[3] Graph cluster randomization: Network exposure to multiple universes, Ugander et al, KDD 2013

[4] Graph agnostic estimators with staggered rollout designs under network interference. Cortez et al, Neurips 2022

We thank the reviewer for giving us valuable feedback. We answer your questions and concerns below.

Regarding M(i) in Section 5.1

• a) We had trimmed some content out to meet the page-limit requirements. In Footnote 4 in Section 5.1, we mention that there were up to 100 extraneous nodes, but we see that the exact details could be made clearer. For each node i in the graph, we randomly select between 10 and 100 additional non-neighboring nodes to include in M(i). We have edited Section 5.1 and gurther details have been incorporated into the Appendix. Since we did not explicitly control the size of $M$, experiments in 5.1 do not incorporate the how size of $M$ effects the estimate, and are to demonstrate the validity of our approach across different model parameters when all the assumptions A1-6 are satisfied.
• b) While the size of $M(i)$ has no direct relation to the strength of interference $r$, you have correctly noted that the extent of those supersets might depend on the strength of interference. The dependence of model behaviour on their interplay is not analyzed in Section 5, because, all the baseline interference aware estimators need the exact neighbourhood information, which precludes analyzing the varying coverage of $M$ . As such in those experiments we have not considered analyzing behaviour with $M$. The interplay between r and superset size is instead considered in Section 5.3. Figure 5a, shows estimation by our model with different values of r. One can see that lower $r$ can support larger (and hence less accurate $M$). We have edited the paper draft to make this clearer, and provided greater details in the supplementary material.
• c) $r$ quantifies the ratio of the effect of the ego-node (or the node itself) vs the impact of other nodes in the true neighbourhood $N_i$ on the outcome. We have also added details of what $r$ is on Page 8 (Sec 5.1).

Regarding Conclusions:

• We have fixed the ending of Section 5, and at the reviewers request we have also added a concluding section.