Towards the Efficient Inference by Incorporating Automated Computational Phenotypes under Covariate Shift

Q1: [Claims and Evidence]

Response:

• Thank you for this insightful comment! To understand the benefits of incorporating ACP $\hat Y$, we introduce an assumption with a format similar to covariate shift, as presented in Lines 181-192. In real world scenarios, this assumption can be tested when $Y$ is available in the unlabeled data (e.g., in the income, politeness, wine datasets) but not when $Y$ is unavailable (e.g., in the diabetes dataset). In cases where the assumption cannot be tested, we agree that conducting a sensitivity analysis could be valuable and of help.
• We also agree that evaluations on more datasets are worthwhile. We provided a more comprehensive response in our answer to Q2 of reviewer mNmv and to Q4 of reviewer 6g7c. Please kindly refer to that section for details. Briefly we newly analyze three datasets. For each of them, we present the comparison results of confidence interval length for regression coefficients in Table 1 of More Results. The proposed method is generally a lot more efficient than PPI, PPI++, and RePPI, with the largest reduction reaching approximately 60%.

Q2: [Methods and Evaluation Criteria]

Response:

• This is definitely an excellent point that warrants further discussion in our revised version. In the presence of covariate shift, one must account for two sources of nuisance functions: the density ratio model and the regression model. Estimating both sources of nuisance functions is computationally more complex than estimating just one, as in some alternative methods. However, the inclusion of both sources forms the basis of double robustness, which is essential for achieving efficient estimation. This represents a tradeoff between estimation efficiency and computational complexity. In cases where the mechanism for selecting labels is known, such as in certain design-based studies, the proposed method only needs to estimate one source of nuisance function.

Q3: [Theoretical Claims]

Response:

• The regularity conditions on nuisance estimators are standard and widely advocated in double/debiased ML literature, e.g., van der Laan (2011), Chernozhukov et al. (2018), Kennedy (2024), Chernozhukov et al. (2024). The convergence rate is achievable for many ML methods such as in regression trees and random forests (Wager and Walther 2015) and a class of neural nets (Chen and White 1999). One can refer to hernozhukov et al. (2018) for more examples.
• As we briefly mentioned in our answer to Q1, the independence assumption for ACP generation can be tested when $Y$ is available in the unlabeled data but cannot if otherwise. When it cannot be tested, one can decompose the assumption as $p(\hat y|x) = q(\hat y|x)$ and $p(y|x, \hat y) = q(y|x,\hat y)$. It is clear that, the untestable part $p(y|x, \hat y)=q(y|x,\hat y)$ is something similar to the covariate assumption $p(y|x) = q(y|x)$. Note that this assumption is popularly adopted in semi-supervised learning, and distribution shift.

Q4: [Experimental Designs or Analyses]

Response:

• We implemented our proposed method and compared with some alternatives on three new datasets (income, politeness, wine). We provided a more comprehensive response in our answer to Q2 of reviewer mNmv, to Q4 of reviewer 6g7c, and to Q1 above. Briefly, we present the comparison results of confidence interval length for regression coefficients in Table 1 of More Results. The proposed method is generally a lot more efficient than PPI, PPI++, and RePPI, with the largest reduction reaching approximately 60%.
• We also expanded our simulation studies by comparing the proposed method with PPI, PPI++ and RePPI. We provided a more comprehensive response in our answer to Q2 of reviewer 6g7c. Briefly, the MSE results are presented in Tables 2 and 3 of More Results. Across all considered scenarios, the proposed method consistently outperforms the three alternatives.

Q5: [Relation to Broader Scientific Literature]

Response:

• As in our answers to Q1 and Q4 above, we extensively compare the proposed method with PPI, PPI++ and RePPI in both real data and simulated data.

Q6: [Essential References Not Discussed]

Response:

• Thank you for pointing out this important reference! Indeed, Yang et al. (2024) studies how to calibrate prediction sets in the presence of covariate shift and proposes a doubly robust approach to enhance the reliability and coverage of predictions. We also realized that there is work, such as Qiu et al. (2023), that explores prediction sets adaptable to unknown covariate shift. In the revised version, we will be sure to include these relevant, important, and exciting works!

Q7: [Other Strengths And Weaknesses]

Response:

• Same as Q3 above.

Q1: [Methods and Evaluation Criteria]

Response:

• Thank you. We provided a more comprehensive response in our answer to Q1 of reviewer mNmv. Please kindly refer to that section for details.

Q2: The simulations...[Experimental Designs or Analyses]

Response:

• Yes. We conducted comparisons between the proposed method and PPI, PPI++, RePPI, across a variety of settings with different values of $n$, $N$, $\alpha$ and $\zeta$. The MSE results are presented in Tables 2 and 3 of More Results. As you can see, across all considered scenarios, the proposed method consistently outperforms the three alternatives.

Q3: Why SuperLearner...

Response:

• We use Super Learner in our implementation, which integrates a library of flexible statistical learning tools to ensure consistent estimation of relevant nuisance functions. While these functions can be estimated using simple parametric models, such models are prone to misspecification, potentially introducing bias and reducing efficiency. More flexible machine learning methods mitigate this issue by avoiding strict parametric assumptions and enabling consistent estimation across a broader range of function classes. Instead of relying on a single estimation method, Super Learner leverages the strengths of multiple algorithms by constructing an optimally weighted combination, offering greater robustness and flexibility. Moreover, van der Laan et al. (2007) established a theoretical guarantee that the estimation error of Super Learner converges to that of the best-performing learner in the ensemble.
• Alternatively, we may use individual learning algorithms, such as random forests or XGBoost. Neural networks are also a potential option, though the size of labeled data in our dataset---such as the diabetes data---is relatively small compared to typical deep learning applications.

Q4: For real dataset...

Response:

• Yes we completely agree with you. We provided a more comprehensive response in our answer to Q2 of reviewer mNmv. Please kindly refer to that section for details. Briefly we newly analyze three datasets. For each of them, we present the comparison results of confidence interval length for regression coefficients in Table 1 of More Results. The proposed method is generally a lot more efficient than PPI, PPI++, and RePPI, with the largest reduction reaching approximately 60%.

Q5: For the diabetes dataset...

Response:

• Figure 2 in our submitted paper demonstrates that the distribution of inpatient visit count indeed shifts between labeled and unlabeled data. To provide a more comprehensive comparison, Table 4 of More Results presents the summary statistics of each variable in $X$. Overall, the covariate shift assumption appears reasonable in this dataset.

Q6: For the diabetes dataset...

Response:

• Same as Q1 above.

Q7: [Supplementary Material]

Response:

• Thank you for the suggestion, and we apologize for overlooking this issue. In the new version, we will make sure to appropriately reference the corresponding details in the supplementary material within the main paper.

Q8: [Essential References Not Discussed]

Response:

• Thank you for pointing out these references! Both are highly exciting works. In the new version, we will be sure to include them and conduct a more thorough literature review, particularly on the topics of covariate shift and pseudo-labeling.

Q9: [Questions for Authors]

Response:

• In the diabetes dataset, the scientific goal is to understand the relationship between $Y$ (diabetes status) and $X$ (7 variables representing patient characteristics). Since $Y$ is binary, we use logistic regression, which is arguably the most commonly used model for this type of analysis. More generally, other suitable regression or classification models, such as probit regression or support vector machines, could also be applied in this context.
• In a broader sense, as long as the parameter of interest can be defined as the minimizer of a loss function, as presented in Lines 144-146, the proposed method can be used. This includes the case with MLP-based regressors or classifiers, high-dimensional models, or large deep learning models in general. To address this question more explicitly, we explore the implementation of the proposed method on the benchmark dataset MNIST where $X$ is image, $\hat Y$ is generated using ResNet, and the comparison with the standard PPI method, for estimating the probability of the outcome being a certain label. In terms of the length of 95% confidence interval (the shorter, the more efficiency gain), the proposed method shortens about 15% (from 0.0448 to 0.0376 when $Y$=1/7, and from 0.0440 to 0.0370 when $Y$=6/8).

Q1: However, it is a bit unclear...[from Methods and Evaluation Criteria]

Response:

• Thanks for raising this question. In general, the outcome $Y$ and covariate $X$ are variables of scientific interest. The variable $Z$ represents additional information that may not be of direct scientific interest but is highly relevant to $Y$.
• For example, in our diabetes dataset, $Y$ indicates whether a patient has diabetes, and $X$ includes 7 variables measured at the year of diagnosis: age, type of insurance (self-pay vs others), counts of inpatient and outpatients visits, BMI, congestive heart failure (yes vs no), and the Charlson Comorbidity Index (CCI, larger than 2 or not). Our primary scientific interest is in understanding the relationship between $Y$ and $X$, which captures patient characteristics. Meanwhile, ACP $\hat Y$ is generated by a tree-based algorithm using diagnosis codes, medication history, and HbA1c lab test results from electronic health records---these variables form $Z$.
• In another example using newly analyzed income data, we fit a least squares model to study the relationship between $Y$ (log-income) and $X$ (age, sex). The ACP $\hat Y$ is obtained from an XGBoost model predicting log-income based on 14 variables, including education, marital status, citizenship, and race, among others---these variables consititute $Z$.
• In some cases where $\hat Y$ is generated by LLMs, defining $Z$ precisely is more challenging. For instance, in our politeness dataset, the goal is to study the relationship between $Y$ (politeness score) and $X$ (a binary indicator of hedging in requests), while $\hat Y$ is generated by OpenAI's GPT-4o mini model. Here, $Z$ represents some latent information used to produce $\hat Y$ but is not as explicitly defined as in previous examples. Similarly, in our wine dataset, the goal is to examine the relationship between $Y$ (rating) and $X$ (price and region), with $\hat Y$ also generated by GPT-4o mini.
• Finally, in a synthetic data example, consider the following data generating process $Y = \xi^T X + \alpha Z + \varepsilon$, where $\varepsilon \sim N(0,1)$, $\alpha>0$, as in our simulation studies. Suppose the parameter of interest is $\xi$, and $\hat Y = Z$. The magnitude of $\alpha$ determines whether efficiency gain occurs: as long as $\alpha\neq 0$, $Z$ correlates with $Y$, leading to an efficiency gain.

Q2: However, I would appreciate...[from Experimental Designs or Analyses]

Response:

• Yes. Thank you for this question. We implemented the proposed method on three new data sets, and compared with some existing methods: PPI (Angelopoulos et al. 2023a), PPI++ (Angelopoulos et al. 2023b) and RePPI (Ji et al. 2025).
• Income data: We analyze the relationship between wage (measured by log-income) and age, confounded by sex, based on US census data, under covariate shift. The ACP $\hat Y$ is generated by fitting XGBoost of log-income with 14 variables, including education, marital status, citizenship, and race. For covariate shift, we partition the labeled and unlabeled datasets using the probability $\exp(\alpha^T X)/\{1+\exp(\alpha^T X)\}$ where $\alpha=(0,1,0)$ and $X =(1,X_1,X_2)$ with $X_1$ age and $X_2$ sex, resulting in a ratio of 2:8.
• Politeness data: Using data that comprises texts from 5,512 online requests posted on Stack Exchange and Wikipedia, we understand the association between politeness score (range from 1 to 25) and a binary indicator for hedging within the request. The ACP $\hat Y$ is generated using OpenAI's GPT-4o mini-model that has the same range as the politeness score. For covariate shift, we split the labeled and unlabeled data in a 1:9 ratio, following the same procedure as above, where $X=(1,X_1)$ with $X_1$ hedge and $\alpha=(0,1)$.
• Wine data: Using the Wine Enthusiast review dataset, we investigate the association between wine rating (range from 80 to 100) and wine price, adjusted by wine region. Similar to the politeness data, the ACP $\hat Y$ is also generated by employing OpenAI's GPT-4o mini-model that produces predicted ratings with the same scale. To assess covariate shift, we follow the same procedure as in the previous experiments, splitting the labeled and unlabeled data in a 3:7 ratio. Here, $X=(1,X_1,X_2,X_3,X_4,X_5)$, where $X_1$ represents price, $X_2$ to $X_5$ represents California, Washington, Oregon and New York, respectively, with $\alpha=(0,1,0,0,0,0)$.
• For each of these three data sets, we present the comparison results of confidence interval length (the shorter, the more efficient, the better) for regression coefficients in Table 1 of More Results. As you can see, while there are some cases that the proposed method is tiny slightly less efficient than PPI++ or RePPI, it is generally a lot more efficient than PPI, PPI++, and RePPI, with the largest reduction in confidence interval length reaching approximately 60%.