From Narratives to Numbers: Valid Inference Using Language Model Predictions from Verbal Autopsies

We thank the reviewer for raising these concerns and providing constructive feedback. We agree that in our review of prior work we should be more thorough in citing the confluence of VA and NLP. We intend to include more specific examples such as Manaka et al. 2022a and 2022b. These authors use BERT and ELMo for feature extraction from VA narratives to enhance COD classification tasks but do not comprehensively address the COD class imbalance challenge. We will note that this is a difficult problem in this domain that our method helps to address.

Regarding the gap between BoW methods and GPT-4, it is our intent to show that our method multiPPI++ is model-agnostic, and applicable across a range of NLP models. This is an important point because practitioners could feasibly use any number of NLP methods, sophisticated or archaic, and our method remains robust and adaptable to models of varying performance.

Echoing our response to Reviewer 2 regarding class imbalance and overfitting, we point out that we conducted a sensitivity analysis (see Figures 29-54 in A.6), where we used one approach to correct for class imbalance by training the model on stratified data, with an 80/20 split for each class to achieve balance w.r.t COD. The results were not substantively different from our main results, so we showcase an example where the NLP model is not perfect, but multiPPI++ still obtains valid statistical estimates. To clarify our attention to this important concern, we will incorporate more discussion, as well as comparisons to state of the art methods (e.g., SMOTE).

To expand on the multiPPI++ complexity and tradeoffs, we will add clarification to emphasize that MultiPPI++ does not change NLP prediction performance, it only affects the accuracy of the downstream statistical estimates and appropriately widens associated confidence intervals to account for the additional originating from the NLP model. We will take care to emphasize this point more clearly.

Data concerns are always paramount. We are fortunate to be working with de-identified data, however, when working with more sensitive data there are a number of open-source LLMs which can be run locally (Llama-3, meditron, etc) to ensure safe and ethical research practices. Additionally, many institutions, public and private, buy access to local OpenAI servers. Our use of these methods is illustrative, aiming to lay a foundational understanding of how to perform valid inference with NLP predictions.

We thank the reviewer for this feedback and agree that we can do more to emphasize why this work is an important contribution to advancing the field of LLMs. While it is true that we are not proposing new NLP algorithms, NLP methods are used in an extremely wide range of settings, often with the goal of performing downstream inference. Our multiPPI++ method enables valid inference when outcomes are predicted using an NLP model and, thus, enhances the utility of NLP predictions. We believe this is a contribution to the field as it is an important step in the process of leveraging NLP to make sound, data-informed decisions.

We thank the reviewer for bringing this to our attention and agree that these claims are not sufficiently supported in our initial submission. We will elaborate on how our parameterization overcomes the issues which arise in Angelopoulos 2023b. We will also include citations to the leading prediction accuracies in the existing literature and provide a direct comparison to our NLP prediction accuracies using text narratives. These clarifications will specify and substantiate our claims you identified above.

We thank the reviewer for their valuable comments highlighting all of the above ways we can improve the clarity, scope, and motivation of this work. In addition to what we have outlined above, we will include a Related Works section to our discussion to emphasize what motivates our work and showing how our contribution is situated in the language modeling literature.

We thank Reviewer 2 for their detailed review and helpful comments. Regarding domain transfer, a primary aim of this work is to illustrate how predicted outcomes from an NLP model can be adjusted to yield accurate statistical parameter estimates even when a domain transfer has occurred. Additionally, an alternative view of this problem would be to account for ‘covariate shift,’ i.e., when the underlying distribution of our predictors differs across datasets. There has been discussion of this in Angelopoulos et al. (2023), which can readily be adapted to this work.

In section A2, we will direct readers to the specific section of Angelopolulos et al rather than quoting their description.

We will add ICD-10 Codes to the table in A.1.

We compute f1-score using the standard sklearn implementation and show confusion matrices in Figure 5.

While we include our prompt in Figure 3, we agree that we do not sufficiently discuss our strategy. We will elaborate on the prompt engineering and associated error analysis in that section.

Regarding class imbalance, in a sensitivity analysis (see Figures 29-54 in A.6), we used one approach to correct for class imbalance by training the model on stratified data, with an 80/20 split for each class to achieve balance w.r.t COD. The results were not substantively different from our main results, so we showcase an example where the NLP model is not perfect, but multiPPI++ still obtains valid statistical estimates. To clarify our attention to this important concern, we will incorporate more discussion, as well as comparisons to state of the art methods (e.g., SMOTE).

As for the performance, Figure 6 demonstrates that multiPPI++ improves the accuracy of statistical estimates using a post-facto correction term. MultiPPI++ does not change prediction accuracy, it only affects the accuracy of the statistical estimates and appropriately widens the confidence intervals to account for the additional uncertainty. We will take care to emphasize this point more clearly.

Regarding multinomial classification, Agelopoulos et al. 2023 provide a formula. We take that formula, reparameterize it, and develop an optimization algorithm. We will clarify that this does not affect the statistical properties of the estimates (the stochastic smoothness condition is satisfied, as $\nabla \theta$ is locally Lipschitz near $\theta^*$ for the problem.)

We thank the reviewer for this feedback and recognize the importance of demonstrating the versatility and robustness of our contribution. We will highlight that our primary goal was to develop and validate an initial framework that could be further generalized and adapted to various tasks within the domain of cause-of-death classification. In particular, in our discussion about future work, we will add more details about how we plan to validate the approach with external data, as well as extend to other application areas such as in analyzing clinical text.

We agree that the ethical considerations related to our research need to be more clearly articulated. We will revise the ethics statement to (1) highlight the potential for biases in human-provided narratives (e.g, inaccuracies, incomplete information, or cultural, social, or personal perspectives of the individuals providing the narratives, (2) model-induced biases (e.g, bias from the methods which may embed biases present in the training data, and (3) misclassification (e.g., in particular from class imbalance, which we address in the appendix using an output-stratified bootstrap sample for our leave-one-out experiment. (See figure 10, for example)). Further, we will clearly state that in practice, ensuring the privacy and confidentiality of the individuals whose narratives are used is critical. We will reiterate that all of the data used in this study were anonymized, but other sources of narratives may preclude the use of pre-trained LLMs such as ChatGPT due to ethical concerns surrounding data use and privacy. Here, we wanted to highlight the performance of a range of NLP techniques, and were fortunate to have a rich, de-identified dataset to do so.

Regarding your question about models using logistic / cross-entropy loss, yes, both our method and PPI++ in general are designed to be agnostic to how the prediction rule was trained and uses data-adaptive weights to incorporate information from ‘black box’ predictions into the overall procedure. We will make this more explicit in the main text.

Lastly, we agree that including threshold-free metrics such as the ROC curve, ROC-AUC, PR curve, and PR-AUC would provide a more comprehensive evaluation of how our various NLP models perform. We will gladly revise Section 4.2 to include these metrics, as well as provide a more robust discussion on the performance of the method in light of these changes. Thank you for all of these valuable suggestions.