Enhancing Statistical Validity and Power in Hybrid Controlled Trials: A Randomization Inference Approach with Conformal Selective Borrowing

We sincerely appreciate your insightful and constructive comments. Below, we have provided our detailed, point-by-point responses.

1. Connection between MSE of $\hat{\tau}_\gamma$ and power of FRT

(i) Variance and power: Theorem 2.4 (power analysis for consistent test statistics) shows that a lower variance of a consistent test statistic leads to a higher power of FRT.

(ii) Bias and power: Simulation-based power analysis demonstrates that borrowing biased ECs can severely reduce FRT power.

(iii) MSE and power: Taking both variance and bias into account, we observe that a lower MSE of the test statistic is associated with higher FRT power. While it remains theoretically challenging to link MSE and power when the test statistic is irregular, our experiments show that MSE-guided $\gamma$ selection offers a unified solution that performs well for both estimation accuracy and power.

We also revised our storyline as follows:

Our primary goal is to improve the power of the Fisher Randomization Test (FRT) for testing the null hypothesis of no treatment effect by borrowing external controls (ECs), compared to relying solely on RCT data (No Borrowing). This is guided by three key insights: (i) Theorem 2.4 shows that the power of FRT can be increased by reducing the variance of a consistent test statistic, which can be achieved by borrowing unbiased ECs to augment the small RCT control sample; (ii) borrowing biased ECs leads to inconsistency and significantly reduces the power of FRT; (iii) these insights motivate our use of Conformal Selective Borrowing to enhance power by borrowing unbiased ECs and discarding biased ones from a larger EC pool.

To this end, we introduce an intermediate objective: testing the exchangeability of each EC and selectively borrowing those deemed unbiased. We recognize that the power of this intermediate testing step is limited by the size of the RCT control group. To address this, we propose tuning the selection threshold $\gamma$ (the significance level of the exchangeability test) to directly target our primary objective. Although tuning based on empirical FRT power would be ideal, it is impractical because (i) it requires specifying an alternative hypothesis and (ii) it is computationally expensive to compute power across a grid of $\gamma$ values.

As a practical alternative, we use the MSE of the Conformal Selective Borrowing estimator as a proxy to guide $\gamma$ selection. This MSE-guided approach offers several benefits: (i) experiments show it improves FRT power over RCT-only analysis, even though the theoretical connection between MSE and FRT power is challenging due to the irregular nature of the test statistic; (ii) it yields strong selection performance and supports our intermediate objective (see Fig. 2 and Fig. 14); (iii) with MSE-guided $\gamma$, CSB serves as both a powerful test statistic and a accurate ATE estimator; (iv) empirical MSE can be approximated leveraging the No Borrowing estimator, and we provide a non-asymptotic excess risk bound for the adaptive procedure.

2. Conformal selective borrowing (CSB) as a powerful test statistic and a reliable, efficient estimator

Although our primary motivation is to improve the power of FRT, the final method, CSB with MSE-guided $\gamma$ selection, applies to both estimation and testing:

(i) Theorems 3.7 and 3.8 provide non-asymptotic excess risk bounds for the proposed estimator.

(ii) In simulations, CSB shows better estimation performance than adaptive lasso selective borrowing (ALSB) in terms of MSE under small and moderate sample sizes. This is demonstrated in Figures 6(C) and 11(C), where both methods are compared purely on estimation accuracy without involving inference procedures.

(iii) In real-world experiments, CSB improves both robustness and efficiency over No Borrowing (NB) and Full Borrowing (FB). Specifically, CSB reduces the standard error by 20% compared to NB and mitigates the bias of FB. The ATE estimate under CSB (0.138) is close to NB (0.142), while FB overestimates it (0.241).

These results suggest that CSB is not only effective for testing under sharp nulls but also serves as a reliable and efficient estimator of treatment effects.

3. Presentation

Following the reviewer’s suggestion, we moved the full conformal $p$-value (which is computationally infeasible) and jackknife+ (a special case of CV+) to the appendix. We kept the split conformal $p$-value and CV+ $p$-value in the main text, as both are used in our experiments.

We sincerely appreciate your insightful and constructive comments. Below, we have provided our detailed, point-by-point responses.

1. Clarification on sharp null hypothesis

The sharp null hypothesis $Y_i(1) = Y_i(0)$ for all $i \in$ RCT states that there is no individual treatment effect for any unit. Another common form of the sharp null is conditional independence, $Y_i \perp A_i \mid X_i$, meaning the observed outcome is independent of treatment assignment given covariates. In contrast, the hypothesis that the average treatment effect (ATE) is zero is often called the "weak null."

In a finite-sample exact sense, FRT guarantees Type I error control under the sharp null but cannot guarantee it under the weak null. However, recent work has shown that FRT can also asymptotically control the Type I error under the weak null using studentized or pre-pivoted test statistics (Wu & Ding, 2021; Cohen & Fogarty, 2022). The RCT sample size is typically small in our context, such as trials for rare diseases. Asymptotic approximations may be unreliable, so we rely on the sharp null for a finite-sample exact test.

While this reliance on the sharp null rather than the weak null may be viewed as a limitation of FRT, to our knowledge, there is currently no testing procedure that controls Type I error in a finite-sample exact sense under the weak null without additional distributional assumptions. Section 6 also discusses possible future directions beyond the sharp null.

2. Significance of the Conformal Selective Borrowing (CSB) and comparison to Adaptive Lasso Selective Borrowing (ALSB)

Compared to existing selective borrowing approaches such as ALSB, the significance of CSB can be summarized as follows:

(i) Model-free flexibility: CSB is a model-free approach that allows flexible choice of conformal scores depending on data characteristics. For example, in our real-world application where the outcome exhibits heavy tails and heteroscedasticity (see Figure 14), we use conformalized quantile regression (Romano et al., 2019) for selection. Distance-based scores such as nearest-neighbor conformity scores (Shafer & Vovk, 2008) can be used for binary outcomes. This flexibility is difficult to achieve with model-based methods like ALSB.

(ii) Estimation: CSB performs better than ALSB in terms of MSE under small and moderate sample sizes. This is demonstrated in Figures 6(C) and 11(C), where both methods are compared purely on estimation accuracy without involving inference procedures.

(iii) Computation: CSB is compatible with the FRT, while ALSB is not readily applicable with FRT due to its computational complexity. This highlights an advantage of CSB when exact finite-sample inference is desired.

(iv) Apples-to-apples comparison under asymptotic inference (Asym): We conducted a comparison between CSB+Asym and ALSB+Asym (see here). CSB+Asym generally achieves better Type I error control than ALSB+Asym and performs comparably when $b=1$.

3. Novelty and implications of theoretical results

Theorem 2.3 builds on classical FRT results (e.g., Lehmann and Romano, 2005, Theorem 15.2.1), but to our knowledge, this is the first formal result establishing the validity of FRT in the context of hybrid controlled trials. It confirms that FRT can be applied in this setting and clarifies how it should be applied to ensure validity. Specifically, Theorem 2.3 provides the following practical guidance and important caveats:

(i) Type I error control is guaranteed only when we permute assignments according to the actual experimental design, that is, permuting treatment assignments only within the RCT while keeping the EC assignments fixed. An important caveat is that permuting across all treated and control units, including ECs, would invalidate the theorem.

(ii) To maintain validity, the test statistic should vary with the permuted treatment vector $A$, meaning the selected set of ECs should also be updated under each permutation to account for selection uncertainty. Fixing the EC selection across permutations would also invalidate the theorem.

Theorems 3.7 and 3.8 provide novel non-asymptotic MSE bounds that guide the design of the adaptive selection threshold in finite samples. Please also see our response to Reviewer f66S, Point 1.

4. Post-selection inference

The reviewer's understanding is correct. The selection uncertainty is fully incorporated into the reference distribution by allowing the selected set to vary with the resampled treatment vector $A$ in the FRT. Therefore, there is no post-selection inference concern when following this principle. However, as noted in the previous point, a key caveat is that the selected EC set should not be fixed during permutation, as this would ignore selection uncertainty and invalidate the test.

5. Presentation

We moved the related work to the main text, revised Line 056, and enlarged Figure 1.

We sincerely appreciate your insightful and constructive comments. Below, we have provided our detailed, point-by-point responses.

1. Bound in Theorems 3.7 and 3.8

We explain the key terms in Theorems 3.7 and 3.8 and their practical implications as follows:

(i) The term $c\Delta|\delta_1|$ involves $\delta_1$, the bias of the consistent No Borrowing estimator, which is of order $1/n$, and $\Delta$, the maximum bias across all $\gamma \in \Gamma$. This motivates us to apply pre-propensity-score matching on all ECs to prevent potentially large bias during the data preparation phase.

(ii) The term $c\Delta\Phi\sqrt{\log (|\Gamma|/\iota)}$ involves $\Phi$, the largest standard deviation proxy $\phi$ across $\gamma \in \Gamma$, which is of order $O(1/\sqrt{n})$. The grid size $|\Gamma|$ is fixed (e.g., 10 in our experiments), making this term well-controlled.

(iii) The term $\\max (c\Phi^2\sqrt{\log (|\Gamma|/\iota)}, c\Phi^2\log (|\Gamma|/\iota))$ arises from sub-exponential tail bounds and scales similarly to (ii).

(iv) The terms $\\max\_{\gamma\in\Gamma} |\hat{V}(\hat{\tau}\_{\gamma} - \hat{\tau}\_1) -\kappa_{\gamma}^2|$ and $\\max\_{\gamma\in\Gamma} |\hat{V}(\hat{\tau}\_{\gamma}) - \sigma^2_{\gamma}|$ corresponds to the estimation errors for $\kappa^2_{\gamma}$ and $\sigma^2_{\gamma}$. This motivates the use of a sufficiently large number of bootstrap replicates to ensure accurate variance estimation.

Overall, Theorems 3.7 and 3.8 provide concrete guidance for implementing $\gamma$ selection in practice and justify the design of our MSE-guided adaptive procedure, even if the bounds themselves are conservative due to their non-asymptotic and worst-case nature.

2. Clarifying the connection to super-efficiency-type behavior

The reviewer questions the super efficiency of $\hat{\tau}_{\hat{\gamma}}$. We acknowledge that this was a misstatement; our intent was not to claim that it is super-efficient, but rather that it exhibits behavior similar to the Hodges estimator. We have revised the sentence as follows:

"This phenomenon highlights that $\hat{\tau}_{\hat{\gamma}}$ behaves similarly to the Hodges estimator (Le Cam, 1953) and to integrated estimators in data fusion (Yang et al., 2023; Oberst et al., 2022): improving upon the baseline estimator (here, the No Borrow estimator) in certain regions of the parameter space (where there is no bias in ECs) inevitably leads to worse performance in other regions (where the bias in ECs is difficult to detect)."

3. Additional simulations

We additionally consider $p = 5$ and $X \sim N(0, \Sigma)$, where $\Sigma$ is a Toeplitz matrix with $(i,j)$-th entry $\Sigma_{ij} = \rho^{|i-j|}$ and $\rho = 0.3$, to introduce dependence among the coordinates of $X$. We did not consider larger $p$ since the sample size is small, with only 25 RCT controls. The simulation results (see here) show similar patterns and demonstrate the robustness of our method.

4. $\Phi$ and $\epsilon_\gamma$

$\Phi$ is the largest standard deviation proxy $\phi$ over $\gamma \in \Gamma$. If $\epsilon_\gamma$ is Gaussian, then $\phi$ equals its standard deviation. We previously mislabeled $\phi$ as a variance proxy and have corrected this.

5. Value and performance of $\hat{\gamma}$

Since $\hat{\gamma}$ is adaptive and varies across simulation replicates, we separately report its values in Figure 4 and analyze its behavior in Section C.2.

We do not expect $\hat{\gamma}$ to outperform all fixed $\gamma$ values uniformly. Prior work has shown that no method can uniformly outperform No Borrowing (corresponding to $\gamma = 1$) without additional assumptions (Oberst et al., 2022; Lin et al., 2024). Instead, our proposed $\hat{\gamma}$ is designed to improve power when the bias of ECs is either absent or detectable. When the bias is difficult to detect, it is reasonable that our method may incur some inevitable power loss, though within an acceptable range. Importantly, Conformal Selective Borrowing+FRT always controls the Type I error, even in such challenging cases.

6. Presentation

We removed the sentence "Based on...", corrected the missing division symbols, and zoomed in Figure 2(D).

7. Ethical issue

As per ICML guidelines, the Impact Statement is not counted toward the 8-page limit. We initially understood this to apply to the Software and Data section and had no intention of violating the page limit.

We did not disclose any identities or URLs in the submission. We mentioned the availability of the R package solely to highlight the applicability and reproducibility of our work. We sincerely apologize if our identity may have been inadvertently revealed in the package documentation, which is external to the submission; this was entirely unintentional and not meant to compromise anonymity. To avoid any potential concern, we have removed the Software and Data section from the manuscript.

We sincerely appreciate your insightful and constructive comments. Below, we have provided our detailed, point-by-point responses.

1. Clarifying the objective and the role of MSE minimization in improving FRT power

Our primary objective is to improve the power of the Fisher Randomization Test (FRT), which is limited when using only RCT data (No Borrowing). To achieve this, we rely on three key insights:

(i) Variance and power: Theorem 2.4 (power analysis for consistent test statistics) shows that power can be improved by reducing the variance of a consistent test statistic, which can be achieved by borrowing unbiased ECs to augment the limited sample size of the RCT control arm;

(ii) Bias and power: borrowing biased ECs renders the full borrowing estimator inconsistent, severely reducing the power of FRT;

(iii) Bias-variance trade-off and power: these motivate the use of Conformal Selective Borrowing to improve FRT power of RCT-only analysis by borrowing unbiased ECs and discarding biased ones from a large EC pool.

Based on above insights, we introduce our intermediate objective: testing the exchangeability of each EC. We acknowledge that the number of RCT controls statistically limits the power of this exchangeability testing. Therefore, we propose tuning the selection threshold $\gamma$ (i.e., the significance level of the EC exchangeability test) to optimize our primary objective directly. While using empirical FRT power to tune $\gamma$ would be ideal, this approach (i) requires specifying an alternative hypothesis and (ii) is computationally intensive, as computing FRT power across a grid of $\gamma$ values is costly.

This leads to our final choice: using the MSE of the Conformal Selective Borrowing estimator as a proxy to optimize $\gamma$. Our MSE-guided $\gamma$ selection offers several advantages: (i) experiments show it achieves our primary goal of improving FRT power compared to RCT-only analysis, though we acknowledge that the theoretical link between MSE and FRT power is challenging due to the irregularity of the test statistic; (ii) it yields strong selection performance and supports our intermediate objective (see Fig. 2 and Fig. 14); (iii) with MSE-guided adaptive $\gamma$, Conformal Selective Borrowing serves as both a powerful test statistic and an accurate ATE estimator; (iv) empirical MSE can be approximated by leveraging the No Borrowing estimator, and we provide a non-asymptotic excess risk bound for the adaptive procedure.

2. Finite-sample validity of FRT

For the validity of FRT (Type I error control), we do not rely on asymptotic arguments such as estimator consistency, as shown in Theorem 2.3. This is a key advantage of FRT over existing integrative methods whose validity depends on asymptotic theory. For power improvement, the optimality of the selected $\gamma$ does rely on the consistency of the No Borrowing estimator, as shown in Theorem 3.8. However, our experiments show that Conformal Selective Borrowing with adaptive $\gamma$ improves FRT power even in small-sample settings.

3. Significance in real-world experiments

The real-world experiments show that Conformal Selective Borrowing (CSB) improves both robustness and efficiency over No Borrowing (NB) and Full Borrowing (FB): (i) Given that CSB+FRT theoretically controls Type I error in finite samples, which existing integrative methods do not, it yields a significant result (p < 0.05) compared to the borderline NB result (p = 0.055), addressing the underpower issue in the original study. (ii) CSB reduces the standard error by 20% compared to NB. (iii) CSB mitigates the bias of FB; the ATE estimate under CSB (0.138) is close to NB (0.142), while FB overestimates it (0.241). We acknowledge that the extent of improvement depends on the quality of the real data at hand. To further evaluate our method in finite samples, our simulations show that CSB can improve FRT power by up to 45% compared to NB.

4. Related work on historical controls

Due to the manuscript's space limitation at the original submission, we included the comprehensive literature review on historical control borrowing in Appendix A. We acknowledge the value of such a review in the main text and will move the related work on historical controls to the Introduction section at the resubmission. Please let us know if we have missed any relevant literature.