A Cautionary Tale on Integrating Studies with Disparate Outcome Measures for Causal Inference

• Thank you very much for your constructive comments -- we appreciate it. Following your suggestion, we have revised the introduction to place greater emphasis on our real data case study. Specifically, we now use the case study to motivate our central question: When can data integration with disparate outcomes yield benefits? We also preview our empirical findings in the introduction, highlighting the fundamental tradeoff between gains in precision and the strength of assumptions required to achieve them.

“In our case study, we assess whether XT-NTX is more effective than BUP-NX in reducing withdrawal symptom severity. The point estimate from the primary study is slightly negative, suggesting a marginal advantage for XT-NTX, but the 95% confidence interval includes zero—indicating no statistically significant difference between the two treatments (see Figure 1(b); estimate $\hat{\theta}_0$). To improve estimation precision, we explore leveraging auxiliary data. Under the strong assumption our combined analysis yields a statistically significant result favoring BUP-NX over XT-NTX (see Figure 1(b); estimate $\hat{\theta}_a$).. While such findings may appear actionable, they hinge critically on an untestable assumption linking outcomes Y and W. If this assumption is violated, the resulting estimates may be misleadingly precise. Our case study thus serves as a cautionary example: although integrating auxiliary data can improve precision, it must be done with careful scrutiny of the underlying assumptions, which—if invalid—can lead to confidently incorrect conclusions."

• Regarding the theoretical results: Data integration is gaining significant traction in causal inference and machine learning, particularly for enhancing statistical efficiency in applications across public health, public policy, and healthcare. In practice, disparate outcomes across datasets are quite common—often arising due to differences in study design, logistics, or measurement protocols. While practitioners routinely rely on heuristics to integrate such datasets, to the best of our knowledge, the literature lacks formal investigation into when and under what conditions such integration leads to real benefits. Our paper addresses this gap with a set of novel theoretical results that:

1. Quantify the conditions under which asymptotic efficiency gains are possible, and measure their magnitude;
2. Characterize the finite-sample benefits and limitations—an aspect that is rarely studied, yet highly relevant for practical applications;
3. Identify and quantify the risks of integrating datasets with non-equivalent outcome measures, especially under violated assumptions.

We are confident these results provide a principled foundation for our overarching cautionary message: while data integration holds promise, its success depends critically on strong and often untestable assumptions. Our work offers both a theoretical framework and empirical motivation to guide responsible and assumption-aware integration of datasets with disparate outcomes.

We sincerely thank the reviewer for their detailed and thoughtful feedback. We appreciate the opportunity to improve our manuscript. Below, we address the main concerns raised.

1. Purpose and Central Contribution

Thank you so much for your thoughtful question. We would like to emphasize that the central focus of our paper is not the proposal of a new estimator, but rather a general cautionary message about the challenges and limitations of data integration in the presence of disparate outcome measures.

In fields such as public health, social sciences, and healthcare—where high-quality, causally informative data are often limited—data integration has emerged as a promising strategy to improve precision and support evidence-based decision-making. However, due to variations in study design, implementation logistics, and measurement protocols, different studies frequently assess the same underlying construct using different outcome measures.

To the best of our knowledge, the existing causal inference literature does not offer principled guidance on when such data integration is beneficial, or even valid. Our paper is among the first to formally characterize the conditions under which integrating data with disparate outcomes can lead to efficiency gains—and when it cannot.

Our theoretical results show that:

• Asymptotic efficiency gains are only achievable under strong, often untestable assumptions about the relationship between outcomes across datasets;
• Finite-sample gains may be possible under milder assumptions, but we precisely quantify when and to what extent. These gains are often modest and tend to vanish as sample size increases;
• There is a significant risk of bias if the required assumptions are violated, making the integration not just unhelpful but potentially misleading.

To illustrate these findings, we include a real data case study that mirrors and reinforces our theoretical conclusions. We have updated the introduction to better emphasize this central message and added the following summary statement:

“This paper provides a cautionary framework for data integration in the presence of disparate outcomes, demonstrating that while such integration may offer marginal gains under ideal conditions, it carries a high risk of bias when assumptions are violated. To the best of our knowledge, we offer the first formal quantification of this tradeoff and emphasize the need for careful scrutiny before applying such methods in practice.”

2. Assumption Clarifications and Structural Model Updates

Error Distribution: Thank you so much for pointing this out. You are correct—we do not require the errors to follow a normal distribution. Our results only rely on the assumption that the errors have finite variance. We have updated the structural model descriptions and clarified the corresponding assumptions in the theoretical results to reflect this more precisely. Importantly, this correction has no impact on the conclusions of the paper, as none of our derivations or results depend on normality.

Partial Linearity and Assumption Strength: Thank you for highlighting this important point. First, we want to clarify that our results are derived under a partial linear relationship between $Y$ and $W$: while the dependence between $Y$ and $W$ may be nonlinear, it is moderated through $X$. This structure is chosen for parsimony, to make our argument as transparent and interpretable as possible.

The central point we aim to make is that even under this strong structural assumption, efficiency gains from incorporating auxiliary data are not guaranteed, unless the dependency between outcomes is almost fully known. By demonstrating the lack of gain under the partial linear assumption (A.5), we are making a stronger statement: that if efficiency gains are not possible even in this idealized setting, they are even less likely under more general, nonlinear relationships.

On Untestable Assumptions: Second, all causal inference relies on untestable assumptions. This is also true about data integration for the causal inference literature. If we do not have data of both $Y$ and $W$ measured for the same individuals, then we need to rely on untestable assumptions. The key point of this paper is that such untestable assumptions are very common in the literature, but there is hardly any scrutiny on the cost vs benefit. Our work is one of the first to try it, principally for the scenario described in the paper.

Generalizing the Model Structure: Third, in response to your suggestion, we have extended our analysis by relaxing the linearity assumption. Specifically, we consider a more general setting in which:

where $\zeta$ is an arbitrary (possibly nonlinear) function of $W$. We find that even in this more flexible setup, efficiency gains are only achievable when both $\alpha(X)$ and $\zeta(W)$ are known, mirroring our original findings. This reinforces our core message: without strong, often untestable assumptions about the functional relationship between disparate outcomes, integrating auxiliary data does not yield reliable efficiency gains.

Bias Cannot Be Removed Without New Untestable Assumptions: Unfortunately, there is no way to debias the bias due to misspecification of $\alpha$ without making additional untestable assumptions. This is indeed the key point of our paper—that there is no free lunch.

3. Finite-Sample Gains and the Role of Auxiliary Sample Size

Thank you for the opportunity to clarify this point further. A central motivation for our work is the setting where the auxiliary dataset is much larger than the primary dataset, common in public health, social science, and healthcare applications.

We have expanded the discussion following Theorem 6 to articulate when and why finite-sample efficiency gains may arise in such settings.

To explore this in depth, our simulation study (presented in the Appendix) systematically examines how results vary with the ratio of sample sizes ($n_1/n_0$), the dimensionality of covariates, and the overall sample size. These empirical results reinforce the theoretical insights described above.

We have added the following clarifications to the discussion of Theorem 6:

“Finite-sample efficiency gains from incorporating auxiliary data arise when the function $\alpha(X)$, capturing the dependence between $Y$ and $W$, is simpler to estimate than $\mu_Y(X)$. In such cases, one may first estimate $\mu_W(X)$ using auxiliary data, and then use the combined data to estimate $\alpha(X)$. However, the existence and extent of these gains depend on the relative complexity of the function classes and the sample sizes involved. As $n_0$ increases, the benefit of this two-stage strategy diminishes, and asymptotically, both the direct and the auxiliary-based estimators converge to the same efficiency.”

“The first term captures the gain from replacing the full function class $M_Y$ with a lower-complexity class $\mathcal{A}$, and the second term reflects the accuracy of estimating $\mu_W(X)$ from the auxiliary data. Gains are most pronounced when $\omega_\alpha \ll \omega$ (i.e., when $\alpha(X)$ is much simpler than $\mu_Y(X)$) and when $n_1 \gg n_0$ (i.e., when we have a large auxiliary dataset relative to the primary dataset). This characterization formally supports the intuition that structural assumptions, when coupled with abundant auxiliary data, can yield meaningful finite-sample gains—even though such gains vanish asymptotically.”

4. Real Data Case Study

Regarding the suggestion to include an additional real data benchmark case study, we respectfully disagree with the reviewer's interpretation. This paper does not propose a new method, and the goal of the case study is not to demonstrate the effectiveness of any particular estimator. Rather, the case study serves a different and important purpose: it illustrates how both point estimates and uncertainty quantification evolve as progressively stronger assumptions are imposed, highlighting the inherent tradeoff between precision and validity in data integration with disparate outcomes.

The central message of the case study—and the paper as a whole—is that there is no free lunch in data integration. Gains in efficiency may come at the cost of relying on strong, untestable assumptions, which can, in turn, introduce bias. This example underscores the broader cautionary tale of our work: data integration can be tempting, but without critical evaluation of underlying assumptions, it may lead to misleading conclusions.

Furthermore, we wish to emphasize that, unlike in machine learning or deep learning, data integration in causal inference with disparate outcomes lacks benchmark datasets with known ground truths. This fundamental difference limits the applicability of empirical validation strategies commonly used in other disciplines. In causal research, the evaluation of assumptions—and understanding the consequences of their violations—is often the most principled way to assess a method’s utility.

6. Noise Variance

Thanks for this nuanced question. For the sake of the parsimony of the theory section, we assume $\sigma_Y$ and $\sigma_W$ are known. However, one can consistently estimate the variance of noise terms in $Y$ and $W$ (under the homoskedasticity assumption) by calculating the variance of $Y - \hat{Y}$ and $W - \hat{W}$. Regarding the efficiency bound, this will add two terms regarding the uncertainty due to the estimation of $\sigma_Y$ and $\sigma_W$. However, note that our no-free-lunch conclusion does not change.

We hope this thorough response addresses your concerns and clarifies the broader motivation and contribution of our paper. We sincerely appreciate the opportunity to improve the clarity and rigor of our work.

Dear Reviewer, we would be happy to clarify any further questions or concerns you may have. Please let us know. Thank you so much!

We sincerely thank the reviewer for their thoughtful and constructive feedback. We greatly appreciate the opportunity to revise our manuscript and have carefully addressed each of the concerns raised. Below, we summarize the key changes made in response to the reviewers’ comments and questions:

1. Lack of a “bridge” dataset with both outcomes (COWS and SOWS)

We agree that including a dataset in which both outcomes are simultaneously measured would significantly strengthen the empirical component of the paper. However, to the best of our knowledge, no such dataset currently exists for the MOUD case study. Importantly, one of the key prospective implications of our work is to motivate the design of future studies that collect both outcomes in tandem, enabling more assumption-lean data integration. We have added a discussion of this point in the revised manuscript.

2. Simulation study exploring bias due to misspecification of $\alpha$

In response to the reviewer’s suggestion, we added a new simulation study to examine the bias introduced when $\alpha(X)$ is misspecified. The simulation results closely align with our theoretical findings and serve to further motivate the cautionary conclusions drawn from our real data case study. This addition reinforces the broader message that violations of the assumptions underlying data integration can lead to bias.

3. Merging of Theorems 3 and 4

We agree that merging these results improves clarity and conciseness. We have accordingly updated the manuscript to reflect this revision.

4. Generalizing the relationship between Y and W

Our original results were derived under a partial linear assumption: conditional on X, Y and W are linearly dependent. This assumption was chosen for clarity and parsimony. However, as suggested, we have extended our analysis to a more flexible setting where: $Y = \alpha(X) \cdot \zeta(W) + \beta(X) + \text{noise}$ with $\zeta$ being an arbitrary (possibly nonlinear) function. Our analysis shows that efficiency gains are only achievable when both $\alpha(X)$ and $\zeta(W)$ are known, consistent with our original conclusion. This generalization strengthens our central message: efficiency gains through data integration are only possible under strong, often untestable, assumptions about the relationship between outcomes.

5. Additional theoretical considerations

We have clarified that our paper does not address the case where \alpha and \beta are random functions. We view this extension as outside the current scope and have noted it as a limitation and potential direction for future research. We believe that relaxing this assumption would only further constrain the potential for efficiency gains under even the strongest conditions.

6. Interpretation of “primary” vs. “auxiliary” data labels

We appreciate the reviewer’s question regarding the asymmetry between primary and auxiliary datasets. We clarify in the revised manuscript that these labels are somewhat arbitrary and the framework is symmetric in structure.

7. Clarification on the sample size of the auxiliary dataset

The auxiliary dataset used in our case study contains 324 individuals. We have now explicitly clarified this in the main text to avoid confusion.

We are grateful for the opportunity to revise our work and believe that these changes have significantly strengthened the manuscript. We hope the revised version addresses the reviewers' concerns and communicates the contribution of our work.

Simulation Study for Misspecified $\alpha$

Data Generating Process

• $X_1 \sim \text{Uniform}[1,2]$
• $X_2 \sim \text{Uniform}[0,1]$
• $W = \phi T + X_1 + \text{noise}$
• $Y = \alpha(X) W + \text{noise}$ where $\alpha(X) = X_1 + a X^2_2$
• $\alpha_{\text{mis}}(X) = X_1$ (misspecified model)
• $\zeta \in \{0,1,2\}$
• $\phi \in \{-1,0,1\}$

The results demonstrate a clear pattern:

1. When $a = 0$ (no misspecification), bias is negligible
2. As $a$ increases (greater misspecification), the magnitude of bias increases proportionally for non-zero $\phi$
3. When $\phi = 0$, bias remains consistently negligible regardless of $a$

Regarding Linearity

Thank you for your question about (non)linearity. We would like to emphasize that the linear case is the most insightful scenario in our analysis. If efficiency gains are unattainable in the linear case, they will necessarily be unattainable in the non-linear case as well.

It's worth noting that when $\zeta$ (the correct basis) is known, even an arbitrary non-linear scenario reduces to the linear case. However, when $\zeta$ is unknown (even if $\alpha$ is known), we return to the fundamental limitation where asymptotic gains are not possible.

Gaussian Noise

Thank you for the clarification regarding additive Gaussian noise. In the current version of our paper, we do leverage mean-zero Gaussian properties and homoskedastic noise to derive the efficient influence function. However, this was primarily for parsimony and is not fundamental to our main claims about efficiency gains.

A more general result on the derivation of efficient influence functions (EIF) follows directly from Section 2 of Chernozhukov et al. (2018) "Double/Debiased Machine Learning for Treatment and Causal Parameters," which derives EIFs under mild regularity conditions including log-differentiability and finite variance. The literature on semiparametrically efficient estimation generally requires only additive mean-zero noise with finite non-zero variance, not specifically Gaussian noise.

For clarity, consider the derivation of $R^*_0(O; \theta_0, \eta_0)$. Following equation 2.5 in Chernozhukov et al. (2018), let $\mathcal{L}(O; \theta,\eta)$ be a criterion function maximized when $\theta=\theta_0$ and $\eta=\eta_0$. Under mild regularity conditions, we can use $\mathcal{L}$ to characterize the Neyman orthogonal score function.

Following their Example 2.1, we can derive $R^*_0(O; \theta_0, \eta_0)$ using negative mean squared error (quasi log-likelihood) as our $\mathcal{L}$:

$\mathcal{L} = -\frac{(Y - \mu_Y(X) - \theta(X)(T-\mu_T(X)))^2}{2}$

Using the concentrating-out approach from Newey (1994), we obtain:

$R^*(O; \theta_0, \eta_0) = R_\theta - \Pi(R_\theta | \Lambda_{\eta}) = (Y - \mu_Y(X) - \theta(X)(T-\mu_T(X)))(T-\mu_T(X))$

We sincerely thank the reviewer for their thoughtful and constructive comments. Below, we address the key concerns raised and summarize the corresponding updates made to the manuscript:

1. Clarifying the Use of Auxiliary Data Despite the Absence of XT-NTX

Thank you for this thoughtful and nuanced observation. You are absolutely right—the XT-NTX arm is not present in the auxiliary study. However, since BUP-NX is common to both the primary and auxiliary datasets, our goal is to assess whether and under what conditions information on BUP-NX from the auxiliary study can be leveraged to improve the efficiency of estimating $\mathbb{E}[Y(t=\text{BUP-NX}) \mid S=0]$ in the primary study population. We have revised the manuscript to clarify this point. The updated text reads:

“Although the XT-NTX arm is absent in the auxiliary dataset, the BUP-NX treatment is shared across both studies. We aim to evaluate if and when information on outcome response to BUP-NX from auxiliary study can be used to improve the efficiency of estimating the outcome under BUP-NX in the primary population, $\mathbb{E}[Y(t=\text{BUP-NX}) \mid S=0]$, while carefully considering the assumptions required for valid data fusion.”

2. Main Takeaways from the Case Study

Thank you so much. We have added the following to describe the main takeaways of our case study:

“In our case study, we assess whether XT-NTX is more effective than BUP-NX in reducing withdrawal symptom severity. The point estimate from the primary study is slightly negative, suggesting a marginal advantage for XT-NTX, but the 95% confidence interval includes zero, indicating no statistically significant difference between the two treatments (see Figure 1(b); estimate $\hat{\theta}_0$). To improve estimation precision, we explore leveraging auxiliary data. Under the strong assumption, our combined analysis yields a statistically significant result favoring BUP-NX over XT-NTX (see Figure 1(b); estimate $\hat{\theta}_a$). While such findings may appear actionable, they hinge critically on an untestable assumption linking outcomes $Y$ and $W$. If this assumption is violated, the resulting estimates may be misleadingly precise. Our case study thus serves as a cautionary example: although integrating auxiliary data can improve precision, it must be done with careful scrutiny of the underlying assumptions, which—if invalid—can lead to confidently incorrect conclusions.”

3. Placement of the Simulation Study

Thank you so much for your helpful suggestion. Due to space constraints, we initially moved the simulation studies to the appendix. However, following your recommendation—and now having an extra page available for the camera-ready version—we have moved the simulation study back into the main text. We believe this change strengthens the paper by making the empirical insights more accessible and better integrated with the theoretical results.

4. Summary of Theoretical Takeaways

Thank you so much for the extremely constructive input. We have added Subsection 4.5, which summarizes the main theoretical and technical takeaways from our results.

5. Clarifying Notation: $(P_n - P)(\psi)$

Thank you for the question. The expression $(P_n - P)(\psi)$ denotes the difference between the empirical average $P_n(\psi)$ and the population expectation $P(\psi)$ of the influence function $\psi$. This is a standard shorthand in semiparametric statistics and causal inference literature, used frequently to express asymptotic linearity and derive variance for regular estimators (see, for example, Kennedy 2016; Tsiatis 2006). To avoid confusion, we have clarified the notation in the revised manuscript. The updated text reads:

“Here, $P_n(\psi)$ denotes the empirical average and $P(\psi)$ the population expectation. The notation $(P_n - P)(\psi)$ is shorthand for $P_n(\psi) - P(\psi)$, as commonly used in semiparametric theory.”

6. Necessity of Shared Treatment

Thank you for raising this important point. The goal of our paper is to understand whether, and under what assumptions, auxiliary data can be used to improve efficiency when the outcomes differ across studies. In this context, at least one treatment arm must be shared between the primary and auxiliary studies. Without a common treatment and in the presence of disparate outcomes, it becomes fundamentally impossible to leverage auxiliary data for valid causal inference. We have updated the discussion section to clarify this point. The revised text reads:

“Our framework requires that at least one treatment arm be shared across the primary and auxiliary studies. In the absence of such overlap, and when outcomes differ across studies, there is no basis for linking the potential outcome distributions across datasets, rendering auxiliary data less usable for improving efficiency gains. This highlights a key structural limitation in data fusion settings with disparate outcomes.”