MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning

We appreciate the reviewer’s valuable feedback and insights. Thank you for highlighting the strengths of MediQ, including the novel design of the Patient and Expert two-agent conversation system, proactive information seeking feature in the Expert, and the robust Patient system to access patient records. We hope to address any questions below:

Need to strengthen the explainability of the proposed framework.

We agree with the reviewer that ChatGPT is essentially a black-box, which is a shared concern in most LLM resarch. We ensure reproducibility by releasing all prompts, the static API versions (line 778-784), and also evaluating open-source models such as the Llama family. We also updated the Patient system with an open-source model (Llama-3) to reduce reliance on ChatGPT, and we will add the results to the camera ready version; the overall trends stay consistent.

We use the expected calibration error to quantitatively evaluate the confidence estimators (line 352-354, Figure 8(b)). Our evaluation of each part of the pipeline component do not solely rely on ChatGPT or other LLMs. For example, factuality of the Patient system relies on semantic similarity and the overall Expert performance is evaluated with the diagnostic accuracy of MedQA and Craft-MD multiple choice questions.

Need to add medical conversation dataset for better evaluation.

Most existing medical conversation datasets, such as ChatDoctor [1], are in the form of patients asking questions and doctors responding with answers in a single turn. We also considered KaMed [2], but since it lacks well-formulated multiple choice questions, it cannot be used for the current MediQ evaluation setup, where diagnostic accuracy is highlighted. Thus, given the scarce existing medical datasets, we did the best we could to operationalize the MediQ framework with MedQA and Craft-MD. Curating a high-quality clinical interaction dataset is a research direction that would greatly benefit research in proactive information seeking.

Information such as symptoms and medical testing results are already in the datasets we use. As stated in the limitations section, one of the requirements of the datasets is to include sufficient contextual information about the patient. We present an example below.

A 5-year-old girl is brought to the emergency department by her mother because of multiple episodes of nausea and vomiting… She has been hospitalized 2 times during the past 6 months… Her immunizations are up-to-date… She appears emaciated. Her temperature is 36.8°C (98.8°F), pulse is 99/min, and blood pressure is 82/52 mm Hg. Examination shows dry mucous membranes. The lungs are clear to auscultation…

If the reviewer meant that when the information requested by the Expert is not present in the context, our current Patient system will refuse to answer since we assume that it will not aid in the diagnostic process. But we agree that in order to create an even more realistic scenario, we can have the Patient system answer out-of-scope questions by creating more complete personas and augmenting them with medically consistent details. We leave this for future work and will clarify in the updated version.

How is the performance compared with Medical Q&A LLMs, such as MedPalm2?

We agree with the reviewer that medical LLMs can potentially outperform general purpose models. There are a few reasons why we did not include medical LLMs in our experiments.

First, general-purpose LLMs such as GPT-4 can outperform domain-specific models on tasks such as MMLU and MedQA [4-7]. Second, most of the medical LLMs were fine-tuned on the training set of MedQA [5,6], which might give it shortcuts in predicitng the letter choice rather than perform explicit clinical reasoning as what the MediQ framework is designed to accomplish. Third, medical LLMs lack ability to follow instructions for the diverse modules in MediQ–confidence estimation, abstention, asking questions–due to the fact that they are fine-tuned on instructions for QA tasks only [3].

We will report Meditron (fine-tuned on Llama-2 with continued pre-training on medical texts then supervised fine-tuning on QA tasks) and Alpacare (Llama-2 based medical model instruction-tuned on synthetic medical data) on the BASIC and BEST setups and update the results in the paper, though based on our preliminary experiments, we expect poorer performance because these models are not as good at following general instructions. Overall, our main contribution is introducing the framework to study question-asking, using different base models for the Expert system will not impact the framework.

We are grateful to the reviewer for their valuable feedback and believe these revisions will strengthen our paper. Thank you again for your valuable insights. We kindly request that the reviewer consider these points when assigning the final scores. We are happy to answer any further questions.

[1] Li, Y., … & Zhang, Y. (2023). Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge. Cureus, 15(6).

[2] Li, D., … & de Rijke, M. (2021, July). Semi-supervised variational reasoning for medical dialogue generation. 44th SIGIR (pp. 544-554).

[3] Xie, Q., ... & Bian, J. (2024). Me llama: Foundation large language models for medical applications. arXiv:2402.12749.

[4] Nori, H., … & Horvitz, E. (2023). Capabilities of gpt-4 on medical challenge problems. arXiv:2303.13375.

[5] Chen, Z., ... & Bosselut, A. (2023). Meditron-70b: Scaling medical pretraining for large language models. arXiv:2311.16079.

[6] Han, T., ... & Bressem, K. K. (2023). MedAlpaca--an open-source collection of medical conversational AI models and training data. arXiv:2304.08247.

[7] Zhang, X., … & Petzold, L. R. (2023). Alpacare: Instruction-tuned large language models for medical application. arXiv:2310.14558.

We thank the reviewer for their thorough review and feedback. We agree with the reviewer that our paper proposes a more realistic scenario for clinical interactions, and we conduct extensive experiments to validate each component of the framework. We hope to address the reviewer’s concerns below:

While the paper demonstrates its effectiveness via automatic evaluation metrics, it lacks human evaluation of the model's performance

Thank you for highlighting this point. We will emphasize in our limitations section that extending MediQ to real human patients is not trivial due to factors such as data sensitivity, privacy risks, and linguistic and psychological adaptations to build trust with patients. As noted in the Ethics section, MediQ serves as an initial framework to enable development and evaluation of Expert systems rather than direct interaction with users (lines 415-416). We will clarify that the proposed interactive framework is “relatively more realistic” compared to static single-turn QA benchmarks, rather than deployable in real-life scenarios.

We agree on the importance of human evaluations to further assess the model’s performance in such complex tasks. We conducted case-level interviews with an expert in clinical and medical education to get a sanity check on the model’s reasoning outputs. The expert noted that the model asks reasonable questions, but pointed out that questions in realistic clinical interactions can often be even more open-ended to elicit more detailed patient responses. We will include these insights and comments in the updated manuscript. While large-scale human annotations and testing on real patients to evaluate the framework’s real-life efficacy would require randomized control trials and present privacy and ethics concerns. However, conducting such detailed studies is beyond the scope of this paper due to the significant time and resources required. We will address this as a point for future work.

2. Parametric Knowledge:

The response of the Expert system relies more on the parametric knowledge of LLMs. Although LLMs have learned knowledge during pre-training, their accuracy and stability cannot be guaranteed.

We agree with the reviewer that the accuracy and stability of LLMs rely on parametric knowledge learned during pre-training, which is a shared concern among most works on LLMs. We will add this to the limitations section. However, we took several measures in our paper to reduce the effect of parametric knowledge:

(1) Our abstention and rationale generation modules are explicitly designed to eliminate some of these effects by breaking down the complex clinical reasoning process into modularized components and providing more explainability. Rationale generation aims to provide more explainability into the knowledge gaps of the models compared to the standard QA setting where the model is only asked to select an option.

(2) We also attempt to disentangle potential confounding variables by examining different combinations of model sizes (varying stability and knowledge), model families (potentially different pre-training data distribution), and abstention strategies. We found similar trends in the effects of different abstention strategies among different sizes and families, and observed statistical significant differences among the abstention strategies using the same model, thereby isolating the effects of confidence estimation from parametric knowledge.

We also posit that models that can better rely on elicited knowledge rather than parametric would get better results in our interactive setting. Nonetheless, it's still a concern for real-deployment. Future work can explore how to augment relevant medical knowledge to ground the expert's questions and we will include the potential influence of pre-training in the limitations section.

We are grateful to the reviewer for their valuable feedback and believe these revisions will strengthen our paper. Thank you again for your valuable insights. We kindly request that the reviewer consider these points when assigning the final scores. We are happy to answer any further questions.

Thank you for your detailed response. I have raised my score accordingly.

We appreciate the reviewer’s feedback and for highlighting our strengths, including the relevance of the topic, the efficacy of the proposed interactive system, and our thorough evaluation and analysis.

Is the purpose then to evaluate other approaches through the simulation system rather than presenting a new abstention module?

Thank you for the question. The main contribution of our paper is both the simulation/evaluation framework and insights from comparing different abstention modules, rather than proposing a state-of-the-art abstention method.

Our goal is to study and improve LLMs in medical diagnostics, where eliciting information via interactions with patients are crucial. We introduce a two-agent interactive setup to evaluate the “information seeking” ability of LLMs (ln.5-6,34-40,43,106,393-396) and then focus on improving this behavior by enabling the model to know when to ask questions.

• Our evaluation framework is a paradigm shift from the standard single-turn QA setup where all information is given upfront. Instead, the Expert only starts with some initial information, and elicits more info as it continues to interact. We posit that this is more realistic and challenging and is essential in building medical AI assistants. We will better motivate the interactive setup in the paper.

• Second, we focus on abstention, investigating how knowing when to ask questions impacts information-seeking (ln.54,149-152). We combine established confidence estimation techniques in novel ways (ln.155-177). While the techniques are not exhaustive, we found that better confidence estimation leads to better diagnostic performance. We close the gap between the more realistic information-seeking paradigm to the full information setup by 45.5% (ln.296-297), providing a starting point for research in proactive information seeking. Our fine-grained analysis offers insights regarding the conversation length (§4.3), relevance of the questions (ln.307-316), format of the information (ln.317-321), confidence thresholding (ln.337-348), and rationale generation (ln.349-365) to guide future research.

related works wrt why the proposed approaches are sufficiently different / have a different goal / do not need to be evaluated against them

We acknowledge the importance of [2, 18, 24, 45, 13] and clarify that our goal is not to propose a new confidence estimation approach but to show that different abstention methods can work with our interactive framework. We now explain our comparisons and why some were not included.

• [24] elicits user preferences on common tasks but lacks abstention. Our interactive baseline--Basic (ln.155-156)--adapts [24] into MediQ, which we show is not sufficient for clinical reasoning. In MediQ, we adapt some ideas and prompts from [24] but in addition incorporated abstention, as well as complex clinical reasoning tasks and domain knowledge prompts, making our framework more relevant for medical reasoning.

• [2] focuses on how to ask good questions via RL-based training; [18] studies how to select questions. They both differ from our focus on when to ask questions via abstention at inference time. We don’t compare to [2] and [18] because they study a different problem than ours and require training [2] or yes/no questions [18].

• [45] lacks an abstention module as noted by the reviewer. [45] involves fine-tuning a closed-source model on a closed-source dataset, making it impossible to reproduce. Future work could integrate [45]’s modules into MediQ to enhance the Expert system, but we can’t compare it in our paper due to the need for training and lack of access.

• [13] presents confidence estimation methods for abstention in general tasks without interaction. We adapted some of these prompt-based methods as baselines (Numerical Abstention in MediQ corresponds to “Ask for Calibration” in [13]).

• [1] is a confidence estimation method focusing on selective prediction, different from our work on question-asking and information-seeking in interactive medical dialogues. While we can see [1] being adopted as an Abstention strategy, we don’t compare it in MediQ due to the need for training and model probabilities to calculate confidence scores.

Our work, while acknowledging and building on [2, 18, 24, 45, 13], introduces novel contributions focused on when to ask questions via abstention at inference time to ensure safe and effective medical dialogue systems. Our novelty also lies in adapting existing confidence estimation methods into the medical domain, which requires more complex reasoning pathways and domain knowledge and was not explored by the above works. We will clarify these points in the paper and include [1] in the related work section.

Why does the full non-interactive setting reach best overall accuracy even though hallucination takes place?

The full non-interactive setup gives all information upfront, so the Expert only has to process the already sufficient information and produce a diagnosis (ln.20). This setup is unrealistic in practice (ln.19,30-33), so we propose the paradigm shift to only provide partial information initially (ln.9,76-77). The Expert is now tasked with 1) deciding when to ask questions, 2) elicit patient information, and 3) integrate information to produce diagnosis, which is inherently a harder task. Starting from only the chief complaint, the interactive Expert aims to elicit patient information to match the full non-interactive upper bound (ln.20,57-58). The reviewer is correct that the best interactive Expert still lags behind the full non-interactive upper bound, but this gap highlights the need for further development in proactive information-seeking LLMs (ln.19-20,296-298).

Thank you again for the detailed suggestions and hope we cleared up potential confusions. We kindly request that these points be considered when assigning the final scores. We are happy to answer any further questions.

We thank the reviewer for the insightful feedback and thorough review for our paper. We appreciate the reviewer highlighting the strengths of our paper, including adapting existing datasets into conversation format, clarity, interesting findings and thorough experiments and models. We address the reviewer’s concerns and questions below:

the evaluation is not detailed enough to understand whether performance on MedIQ would generalize to performance for human patients.

Thank you for raising this concern. Generalization to human patients is indeed important and non-trivial, so we leave it for future work. We will include a discussion in the updated manuscript on the challenges with extending MediQ to human patients as well as important areas for future extensions of this framework. We briefly outline a few ideas here:

• a) Incorporating complex human and social factors into the Patient system to simulate real-world interactions accurately. For instance, patients might omit information regarding stigmatized diseases, and symptoms might present differently in different patients.

• b) Exploring how to interact with human patients using MediQ from linguistic and pragmatics perspectives. This includes, studying conversational nuances, trust building, communication style, and culturally specific norms to bring the framework closer to real-world application.

• c) Train a patient system on actual conversation data from real patients to better simulate interactions with a real human patient. Collection of such clinical interaction datasets, although tricky and potentially containing privacy risks, would greatly benefit the community.

• c) Conducting randomized control trials with patients to test the real-life efficacy of the framework. However, conducting such detailed and careful studies is not feasible within the scope of this paper, as it requires significant time, resources, and medical expertise.

Additionally, we agree with the reviewer that the Patient system is critical for evaluating the Expert system. Our preliminary experiments with different variants of a Patient showed that irrelevant responses or factually inaccurate responses from the patient led to lower diagnostic accuracy. Therefore, we focused on evaluating the factuality and relevance of a Patient system's responses, to ensure high quality, relevant responses to an Expert's question (Section 3.1).

it would be better to try to streamline the main body of the paper and add a few more pages to the appendices

Thank you for the feedback on space and organization of the paper. We will incorporate your suggestions and shorten sections 2 and 3 (framework description and methods), move the analysis of the abstention threshold (line 337-348, Figure 8(a)) to the appendix, and make sure all figures are legible and highlight important takeaways.

We appreciate your constructive feedback and believe these revisions will strengthen our paper. Thank you again for your valuable insights.