Thank you for highlighting the importance and clarity of the paper. We hope this response addresses your questions, but please let us know if you have any remaining concerns that would prevent an “accept score”.

Relation to shrinkage estimators. We agree that the idea of combining multiple estimators to reduce variance is conceptually close to classical shrinkage estimators. However, we would like to emphasize a key distinction: unlike traditional shrinkage approaches, our method only includes unbiased estimators in the combination. We believe one of the contributions of our work is to show that it is possible to achieve variance reduction without introducing bias. This is a crucial property for practical adoption, especially in high-stakes applications. That said, we appreciate the connection and will expand our discussion of this relationship in the related work section of the camera-ready version.

Relation to ensemble methods and model averaging. Ensemble methods such as bagging, random forests, boosting, and the Super Learner have long been used in semiparametric inference, typically combining models trained on the same (often small) dataset. In contrast, our approach integrates predictive models trained on vast amount of external data, independent of the dataset at hand. Further, our goal here is not to boost predictive accuracy per se, but to enable valid and more precise inference for treatment effects by leveraging these externally trained models.

Re foundation models. Our framework is model-agnostic, and any externally trained prediction model can, in principle, be incorporated - it does not need to be a foundation model. We focus on foundation models in this paper because all our experiments are conducted using LLMs. Moreover, we expect foundation models to be the biggest use case of our general methodology, particularly within the social sciences. We will make this clearer in the camera-ready version to avoid any misunderstanding that our method is limited to foundation models.

That’s a great question.

TLDR: The combined estimator will have strictly lower variance than any of the individual estimators, except for a specific edge case where the correlation among estimators is equal to a critical threshold.

---

For simplicity, let's consider two unbiased estimators, and form a linear combination:

$$T_\lambda = \lambda T_1 + (1 - \lambda) T_2, \quad \lambda \in \mathbb R.$$

The variance of this combined estimator is:

$$\operatorname{Var}(T_\lambda) = \lambda^2 \sigma_1^2 + (1 - \lambda)^2 \sigma_2^2 + 2\lambda(1 - \lambda) \sigma_{12}.$$

The weight that minimizes this variance is:

$$\lambda^\star = \frac{\sigma_2^2 - \sigma_{12}}{\sigma_1^2 + \sigma_2^2 - 2\sigma_{12}}.$$

And the variance of the combined estimator is:

$$\operatorname{Var}(T_{\lambda^\star}) = \frac{\sigma_1^2 \sigma_2^2 - \sigma_{12}^2}{\sigma_1^2 + \sigma_2^2 - 2\sigma_{12}}.$$

---

Independent estimators

Now, if $T_1$ and $T_2$ are independent, then $\sigma_{12} = 0$, and the variance simplifies to:

$$\operatorname{Var}(T_{\lambda^\star}) = \frac{\sigma_1^2 \sigma_2^2}{\sigma_1^2 + \sigma_2^2}$$

This value is always strictly less than both $\sigma_1^2$ and $\sigma_2^2$, meaning that the combined estimator improves on both $T_1$ and $T_2$.

---

Correlated estimators

For correlated estimators, the improvement depends on the degree of correlation.

Assume without loss of generality that $\sigma_1^2 \le \sigma_2^2$, i.e. $T_1$ has the lower variance.

Define the correlation coefficient:

$$\rho = \frac{\sigma_{12}}{\sigma_1 \sigma_2}.$$

Then the difference between the smaller individual variance and the combined variance is:

$$\sigma_1^{2} - \operatorname{Var}(T_{\lambda^\star}) = \frac{\sigma_1^2 \left(\sigma_1 - \rho \sigma_2\right)^2}{\sigma_1^2 + \sigma_2^2 - 2\rho \sigma_1 \sigma_2}.$$

Assuming the two estimators are not perfectly correlated, the expression above is always non-negative, and is strictly positive unless: $\rho = \frac{\sigma_1}{\sigma_2}.$

The intuition is that for this specific correlation level $\operatorname{Cov}\bigl(T_1,T_1-T_2\bigr)=0$, and the difference $T_1-T_2$ is orthogonal to $T_1$ (carries no information about $T_1$’s random error).

Therefore, the combined estimator has smaller variance unless the correlation is equal to the ratio of the standard deviations of the two estimators (or the two estimators are perfectly correlated).

We hope this response addresses your questions, but let us know if you have any additional concerns.

Why not use very small sample sizes in experiments (e.g. n=10,n=20,n=30)? We appreciate the interest in understanding the limits of our method under small-sample regimes. However, we believe it is important to emphasize that n = 100 represents a far more realistic and practically meaningful setting than n = 10 or n = 20. Well-designed clinical trials rarely operate with fewer than 100 total participants. Even Phase  2 trials typically enroll 130 to 300 participants on average, depending on the disease area (see, e.g., Table 2 in [1]). Our choice of sample sizes in simulation studies is consistent with this real-world setting.

Therefore, we do not believe ablations with n = 10, 20, or 30 would provide useful insight. These sample sizes lie far outside the domain of established practice and produce conditions where even the simplest estimators (e.g., difference in means) fail to achieve nominal coverage due to severe data scarcity. In such regimes, inference becomes fundamentally unreliable - not just for our method, but for the broader class of semiparametric estimators.

Finally, we note that Proposition 1 is built on standard and widely accepted assumptions in the semiparametric literature, adapted to our setting. As discussed above, small sample sizes like n=10 fall well outside the range of well-designed experiments and beyond the scope where any statistical estimator remains reliable.

Where do you see the promise of the approach in social sciences? The papers cited are technical reports without any clear evidence of promising results. We respectfully disagree. The claim that [3] is a technical report is factually incorrect - the paper is published and has collected over 800 citations. As the authors state: "We suggest that language models with sufficient algorithmic fidelity thus constitute a novel and powerful tool to advance understanding of humans and society across a variety of disciplines." This reflects growing recognition of foundation models as a transformative tool in the social sciences.

Further, we disagree with the claim that [4] offers no clear evidence of promising results. On the contrary, the paper presents one of the most comprehensive and rigorous evaluations to date of large language models' ability to predict the outcomes of social science experiments. They use GPT-4 to forecast 476 treatment effects across 70 nationally representative experiments. Overall, the authors report a correlation of r = 0.85 between GPT-4’s predicted and actual treatment effects.

It might be that using the best model actually works much better than having multiple models? Our framework is flexible enough that it may ultimately rely on a single best-performing model. However, when different models capture complementary signals, a weighted combination can outperform any individual model, making the combination strictly better in such cases.

Nevertheless, we want to clarify that the best single model cannot outperform the combined estimator in terms of efficiency. This is guaranteed by Theorem 1, which establishes that the hybrid estimator’s variance is no worse than that of any individual component estimator. Thus, combining models is never harmful and often beneficial.

The use of multiple models is a flexible feature of our framework, not a requirement. It can be particularly valuable in high-stakes settings like clinical trials, where pre-specifying the analysis plan is often necessary and post hoc model selection is not allowed (as it can affect the validity of the inference).

Can you provide more detailed results for the benefits of multiple models vs providing the same amount of compute to the best model? Figure 4(b) shows that performance plateaus after averaging the prediction over 5 different prompts. In Table 1, we use 10 prompts per model, meaning the total compute budget is already well beyond the point where additional sampling would lead to meaningful improvements. Therefore, we believe that reallocating the same compute to a single model - e.g. using 30 prompts - would offer no added benefit.

Effective sample size in Table 1. We agree that reporting effective sample size could also be informative. However, they are dual of each other, and hence the relative ranking between methods would remain unchanged.

Lack of limitations. We agree that the limitations deserve more thorough discussion. Due to space constraints, we only briefly addressed them in the conclusion. In the camera-ready version, we will use the additional available space to include a dedicated limitations section covering model alignment.

[1] Costs of Drug Development and Research and Development Intensity in the US, 2000-2018. Aylin Sertkaya; Trinidad Beleche; Amber Jessup; et al.

[3] Out of One, Many: Using Language Models to Simulate Human Samples. Argyle et al.

[4] Predicting Results of Social Science Experiments Using Large Language Models. Luke Hewitt, Ashwini Ashokkumar, Isaias Ghezae, Robb Willer

Thank you for the time dedicated to your initial review.

We’d appreciate your engagement with our rebuttal, especially as the NeurIPS policy requires reviewers to actively participate in the discussion phase before submitting a “Mandatory Acknowledgement.”

We believe a discussion is particularly important here because several key points in the review appear to reflect misunderstandings or factual inaccuracies:

• The claim that [3] is merely a technical report is factually incorrect - it is a published paper with over 800 citations.

• The statement that [4] provides no promising evidence does not reflect the content of the paper, which reports a strong correlation between GPT-4’s forecasts and experimental outcomes across 70 social science experiments.

• The suggestion that sample sizes like n = 10 are appropriate benchmarks ignores the context of real randomized trials, which our method is designed to support - even small Phase 2 clinical trials enroll over 130 participants.

• The concern that combining models may underperform compared to selecting the best one overlooks Theorem 1, which formally proves that the combined estimator is at least as efficient as any single component estimator.

Given the above, is there a specific concern you think remains unaddressed?

Thank you for your request for clarification.

We agree that the current framing in the introduction and abstract could be misinterpreted. In particular, we do not simulate entirely new data tuples. Instead, for given covariates $X_i$ from the experimental data, we use the LLM to predict the counterfactual outcome, i.e. $f(X_i,1-A_i)$. These predictions are then pluged into $\psi(h)$, see equation between line 115 and 116.

We regret that Figure 2 did not clearly convey how the LLMs are used. To clarify, LLMs serve as the functions $f_1,\ldots, f_k$ that appear in Algorithm 1.

We will revise the introduction and abstract in the camera-ready version to make this clearer.

Thank you for the constructive feedback. We hope this response addresses your questions, but please let us know if you have any additional concerns.

All experiments are drawn from social science surveys. Our evaluation focuses on survey experiments due to the wide availability of high-quality, publicly accessible randomized datasets. This setting allows rigorous benchmarking across multiple studies. Further, we made the scope of our paper clear early on in the introduction (lines 54 and 55):

While our methodology applies broadly, we focus our empirical results on social science survey experiments, where large language models (LLMs) can provide rich predictive signals.

In contrast, clinical trials typically involve protected and proprietary data, making evaluation challenging. Further, the goal in our paper is to introduce a novel estimator with strong theoretical guarantees and demonstrate its promise within a well-controlled experimental domain. While we are actively working on applying our method in clinical trial settings, we consider that a separate, more application-driven effort (outside the scope of this paper and not well-suited to the NeurIPS venue).

Foundation models predictions may carry biases. We want to clarify that our method is robust against any potential bias from the foundation models. Intuitively, this follows from the AIPW estimator being unbiased regardless of the choice of outcome regression $f$ (for any fixed function $f$, even if derived from a language model).

For clarity, we demonstrate this below for the treated arm (a symmetric argument holds for the control arm):

$$\mathbb{E} \left[ \frac{A}{\pi} (Y - f(X)) + f(X) \right] = \frac{\pi}{\pi} \mathbb{E}[Y \mid A = 1] - \mathbb{E}[f(X)] + \mathbb{E}[f(X)] = \mathbb{E}[Y \mid A = 1],$$

where we use the fact that in randomized experiments $A \sim \text{Ber}(\pi)$ and$f$ is an arbitrary fixed function.

Privacy considerations. This is an important point, and we appreciate the reviewer raising it. ​​In a practical setting where privacy is important, the foundation models can be open source ones that are run locally (e.g., LLaMA, DeepSeek and Qwen). Nevertheless, we will discuss this issue carefully in the limitations section, emphasizing that rigorous privacy safeguards may be necessary when applying the method in practice, in particular when used in the medical setting where foundation models may not be as easily run locally. Future work could explore injecting calibrated noise into inputs or applying structured anonymization techniques (e.g., coarsening age or income). While these directions fall outside the scope of our current paper, they open up a promising research area at the intersection of privacy and causal inference.

Trade-off between sample size and ensemble size. There is a clear trade-off in selecting the number of models to combine in small-sample regimes. As shown in Appendix B.2, empirical undercoverage can occur when too many models are ensembled on very limited data. This is a finite-sample effect: increasing the number of models in the ensemble increases the flexibility of the combined outcome model, akin to fitting a linear regression with too many covariates. As a result, asymptotic guarantees still hold, but require larger sample sizes to kick in.

Importantly, these coverage issues vanish completely with more realistic sample sizes (e.g., n = 200), where our method consistently achieves nominal coverage. Indeed, most well-designed randomized experiments do not operate in extremely low-sample regimes (e.g., n = 50), which fall outside the regime where statistical inference remains reliable.

We will clarify these points and offer guidance for practitioners in the camera-ready version.

We hope this response addresses your questions, but let us know if you have any additional concerns.

Model assumption dependence. First, we want to clarify that our approach does not risk bias amplification under model errors. For example, even if a completely incorrect outcome model is used within the AIPW estimator, the estimator remains unbiased. However, variance reduction depends on model quality - when the predictive model is poor, our method offers reduced benefits. Indeed, we provide a simple result in Appendix A.3 (Lemma 1), showing that the variance depends on the l2-error of the model.

We provide here some additional experiments to understand when the benefits from our method (i.e. reduction in variance) disappear. In particular, we ablate over the correlation between the external model predictions and the true outcome labels.

We simulated $n=100$ observations with $d=5$ Gaussian covariates $X\sim N(0,I)$. Treatments were assigned at random with probability 0.5. The potential outcome functions are generated via $g_a(X_i)= \mathbb{I} [a + \beta^\top X_i + 0.3 X_{i,0}X_{i,1} + 0.1 X_{i,2}^2 > 0 ].$

We compute the standard AIPW estimator (using a cross‑fitted logistic regression outcome model on the experimental data) and the H‑AIPW estimator, combining the standard AIPW with the external model obtained by flipping the ground truth outcomes to achieve the desired correlation.

As expected, when the external model is highly correlated with the outcome, the variance of the H‑AIPW estimator is substantially lower than that of the baseline. For moderate correlation, the variance reduction is modest. When the external model is only weakly correlated, the H‑AIPW variance remains slightly lower than that of the baseline, demonstrating the estimator’s robustness. These results indicate that even with small to moderate correlation, our method continues to offer benefits and, importantly, does no harm.

Cost of using foundation models. Thank you for raising this point. In our setting, inference costs are modest, as most datasets contain only 100 to 200 datapoints. Specifically, the cost of all the experiments reported in our paper was approximately $200 in API credits. We consider this cost negligible compared to the expenses associated with human experimentation.

Schematic diagram. We’d be more than happy to adjust Figure 2 in the current manuscript, if the reviewer has suggestions for any particular edits that could clarify the illustration more.