MedFuzz: Exploring the Robustness of Large Language Models in Medical Question Answering

We appreciate the reviewer’s detailed evaluation and thoughtful feedback on our manuscript. Below, we address each of the concerns and questions raised.

1. Definition of Robustness vs. Generalization

Robustness in the context of MedFuzz refers to the resilience of a model’s performance statistic (e.g., accuracy) when assumptions underlying the benchmark are violated in real-world settings. This includes maintaining performance when diagnostically irrelevant details are introduced. By contrast, generalization in statistics refers to the ability of a model to perform well on unseen data sampled from the same distribution as the training data.

We will revise the manuscript to clarify this distinction and emphasize that robustness here is specifically tied to the benchmark’s assumptions and the model’s ability to handle clinically irrelevant or misleading details.

2. Patient Characteristics and Bias

We regret that we weren't more clear about how the use of patient characteristics (PC) in MedFuzz does not introduce or reinforce bias. Rather, it aims to surface biases already implicit in the target model. MedFuzz is a diagnostic tool to evaluate LLMs before they are deployed in clinical decision-making scenarios.

Importantly, MedFuzz itself does not serve answers to questions in clinical settings—it evaluates the robustness of models that do. In that evaluation, it does not change or modify the target model. This distinction will be made clearer in the revised manuscript.

3. Scale of Perturbations

We did not constrain the proportion of added text during perturbations because, in our experience, the length of added text was still well within the length of the context windows for the target LLMs. We agree with the reviewer that analyzing how varying amounts of irrelevant information impact target model performance would be valuable. We will include this as a suggestion for future work.

4. Chain-of-Thought Fidelity

The CoT analysis focused on successful attaks to demonstrate that inspecting CoT explanations alone is insufficient to reveal the vulnerabilities surfaced by MedFuzz. We will expand the analysis to include unsuccessful attacks as well.

5. Examples of Benchmark Assumption Errors

The manuscript cites examples of errors that are not caught by traditional benchmark evaluation due to simplifying assumptions in those benchmarks. For example, we cite references showing GPT-3 demonstrating biases toward certain patient groups. We will expand on these examples in the revised manuscript to better illustrate the risks posed by such assumptions.

6. Ethical Concerns Regarding Bias

We address the ethical concerns raised by clarifying that MedFuzz is designed to surface biases in the target model, not to introduce or reinforce them. MedFuzz operates as an evaluation tool, diagnosing vulnerabilities in LLMs that may be deployed in clinical settings.

We explicitly state that failure to surface such biases does not imply their absence. Furthermore, MedFuzz is not intended to answer medical questions but rather to assess the robustness of models that do. We will revise the manuscript to better highlight these points and allay concerns about bias reinforcement.

7. Perturbation Iteration Count $K$

The results for different values of $K$ are shown in Figure 2. We demonstrate how performance changes as the number of perturbation iterations increases, providing empirical support for the choice of $K=5$ as a practical balance between computational cost and perturbation effectiveness. We will ensure that this explanation is clearly referenced in the manuscript.

Revisions to the Manuscript

To address the reviewer’s feedback, we will:

1. Clarify the distinction between robustness and generalization, explicitly tying robustness to real-world violations of benchmark assumptions.
2. Emphasize that MedFuzz evaluates models to surface implicit biases, rather than introducing or reinforcing them.
3. Expand on examples of errors caused by traditional benchmark assumptions to strengthen the motivation for MedFuzz.
4. Expand the analysis of CoT fidelity to cover questions where attacks were unsuccessful, to establish a baseline for the analysis.
5. Ensure the ethical role of MedFuzz as an evaluation tool is clearly communicated.
6. Expand upon the discussion of $K$ and iteration counts and how to select ideal values of $K$.

We appreciate the reviewer’s constructive feedback, which has helped us identify areas to strengthen the manuscript and address concerns. These revisions will further clarify MedFuzz’s methodology, ethical considerations, and contributions to LLM robustness evaluation.

We thank the reviewer for their detailed and thoughtful feedback on our manuscript. Below, we address each of the points raised and clarify our methodological choices. Firstly, we will correct the \citep citation format throughout the manuscript.

1. Empirical Validation of Semantically Coherent Fuzzes

Our qualitative evaluation of the fuzzed questions relies on feedback from medical expert users who review successful attacks and assess their plausibility. While this approach is effective in surfacing interesting cases, we recognize the need for more systematic and quantitative validation to empirically verify that clinicians would consistently provide correct answers to fuzzed questions. This limitation will be addressed in future work as part of broader medical expert evaluation efforts.

2. Use in Other Domains

The approach used in MedFuzz would apply in other domains. The approach relies on a domain expert who design the attacks and evaluate the results. The domain should also face serious robustness challenges in deploying in real-world settings. We chose medicine because of our experience with challenges in this domain.

3. Choice of GPT-4 as the Attacker LLM

We selected GPT-4 as the attacker LLM due to its exceptional performance on MedQA. The attacker LLM must perform at least at a human level on the benchmark to effectively generate attacks that preserve the correct answer while introducing subtle, diagnostically irrelevant distractors. GPT-4 has also demonstrated performed well on theory of mind tests (Strachan et al., 2024), suggesting it would be good at generating ways to "trick" a test taker (the target LLM).

We recognize the potential value of exploring fine-tuned open-source models like OpenBioLLM-70B as attackers. However, current fine-tuned models lack performance on this benchmark and demonstrated generalist reasoning abilities in other settings. In future work, we aim to investigate whether fine-tuning open-source models can achieve similar attacker capabilities at a lower computational cost.

4. Use of MedQA Dataset

We selected MedQA because it remained a challenging benchmark for state-of-the-art language models until GPT-4 and its direct competitors achieved near-human performance. To demonstrate MedFuzz’s value, the target LLM needed perform well enough on the benchmark to reveal meaningful vulnerabilities beyond just not understanding the questions.

Expanding MedFuzz to other datasets like MedMCQA, PubMedQA, or MMLU Clinical Topics is an exciting direction for future work. The challenge with these datasets is that their variety in answer format and topic makes it challenging to identifying assumptions to violate that don't hold up in clinical settings. Relative to MedQA, they do not align as closely with our specific focus on robustness to real-world assumptions.

5. Scalability of the Statistical Test

The computational expense of the statistical test arises primarily from generating control fuzzes. For multiple-choice benchmarks, we recommend generating at least 30 control fuzzes per attack to ensure granularity in p-values that align with conventional significance thresholds. In future work, we plan to extend this methodology to open-ended answers by embedding generated responses and deriving p-values from the embeddings. We leave this to future work because it will require theoretical treatment as well as a much larger number of control fuzzes it will improve applicability to a wider range of benchmarks.

6. Iterative Workflow

The iterative workflow is not specific to GPT-4 and can be applied to other high-performing models like Claude Sonnet.

Iterative turns are necessary to refine fuzzes, leveraging feedback from the target LLM to ensure attacks are semantically coherent and effective. While single-shot attacks are simpler, they often fail to exploit the nuanced vulnerabilities in advanced LLMs, as demonstrated by our initial experiments with single-turn methods (these were negative results will be added to the appendix for transparency).

7. Presentation and Intuition

We appreciate the reviewer’s suggestion to improve readability. In the revised manuscript, we will:

• Add a simpler case study early in the text to illustrate the method.
• Reorganize the introduction of the test statistic to introduce notation first, followed by an explanation of how it captures the significance of successful attacks.

Summary

We are grateful for the reviewer’s feedback and have outlined revisions to enhance the clarity, scalability, and rigor of our work. These include:

1. Adding negative results from single-shot attacks to the appendix.
2. Revising sections for improved readability and presentation.
3. Expanding the discussion of iterative workflows and generalizability to other datasets and models.

We thank the reviewer for their thoughtful suggestions and are confident that these updates will strengthen the manuscript.

We appreciate the reviewer’s thoughtful and constructive feedback on our manuscript. Below, we address each of the points raised and clarify aspects of our approach.

1. Single-Turn Attacks

We initially explored single-shot attacks as a baseline approach. For example, with GPT-4 achieving 88.5% accuracy on the MedQA benchmark, we created several modified datasets that added diagnostically irrelevant patient characteristics. These datasets included patients characterized by varying socioeconomic statuses (e.g., affluent or low-income) and different racial or ethnic groups (Asian, Black, Hispanic, Native American, White), while excluding questions where race was clinically relevant. Across these datasets, no statistically significant change in accuracy was observed, indicating that such single-turn perturbations were too “easy” for advanced models like GPT-4. These findings inspired MedFuzz's multi-turn approach. We can include these negative results in the appendix to demonstrate the progression of our method.

2. Expanding Statistical Validation

We strongly adhere to the conventional statistical approach of having the end user evaluate interesting results and then using significance tests to validate that these findings contain signal. Expanding significance testing to a broader set of results would necessitate multiple comparisons corrections, which we leave for future work. Furthermore, ranking results based on p-values invites the risk of p-hacking, which we aim to avoid.

3. Faithfulness of Responses

The faithfulness analysis is not intended as a core contribution but rather as a supplementary finding to highlight that vulnerabilities revealed by MedFuzz cannot simply be detected by inspecting CoT explanations. We agree this distinction can be emphasized more clearly in the manuscript.

4. Comparison to Misinformation Attacks in Han et al. (2024)

MedFuzz fundamentally differs from the approach in Han et al. (2024). The attacks described in that work aim to poison target LLMs by injecting falsehoods during model updates, requiring access to gradients and training data. In contrast, MedFuzz does not poison, it detects “poison”, and does so without access to gradients or the data used for model updates.

We will clarify this distinction in the manuscript to address the reviewer’s concern.

5. Guidelines for Expanding MedFuzz

MedFuzz’s approach is generalizable to any domain where benchmarks rely on performance statistics (e.g., accuracy) that are contingent on assumptions not robust in real-world settings. While our study focuses on medical datasets requiring clinical expertise, domain experts in other fields can evaluate MedFuzz outputs for their respective use cases.

For medical benchmarks like MedMCQA and PubMedQA, the key challenge is identifying assumptions analogous to those violated in MedQA. We can provide broad guidelines for extending MedFuzz, such as focusing on domain-specific biases, assumptions, or oversights that simplify real-world complexity.

6. Ensuring Correct Answers in Fuzzed Questions

We rely on the attacker LLM’s high performance on MedQA to generate effective attacks while preserving the correct answer. Medically experienced users validate successful attacks by inspecting outputs and running significance tests. When the attacker fails to “fuzz” the question well, this is discovered during that human evaluation step. We plan for extensive human medical expert evaluation in future work.

7. Value of $K$ in Algorithm 1

The ideal value of $K$ (number of iterations) depends on the target model’s capabilities on a given benchmark. We will update the manuscript to suggest tuning $K$ on a pilot subset of the data, increasing it incrementally until the marginal gains from additional iterations are no longer worth the computational expense.

8. Smaller Models as Attackers

We believe that the attacker model must have reached human-level performance on the benchmark to identify effective attacks. This ensures that the attacker LLM can leverage its understanding of the benchmark and the correct answer to generate meaningful perturbations. Smaller models effectiveness would be limited by their lower performance on the benchmark. Exploring this tradeoff is a promising direction for future work.

9. Redundancy in Algorithm 1

Thank you for pointing out the Algorithm 1, we will clarify this in the revised manuscript.

Summary

We appreciate the reviewer’s positive assessment of the manuscript’s clarity, originality, quality, and significance. We have provided clarifications and plan to incorporate additional results and updates in the revised manuscript, including:

1. Negative results from single-shot attacks in the appendix.
2. Broader guidelines for applying MedFuzz to other domains.
3. Refinements to Algorithm 1 and expanded discussion on parameter tuning.

We thank the reviewer for their valuable feedback.

We thank the reviewer for their thorough evaluation and constructive feedback on our manuscript. Below, we address each point raised:

Faithfulness of Reformulated Questions

We acknowledge the concern about the reliance on the attacker LLM (GPT-4) to maintain the medically correct answer while generating fuzzed questions. In our approach, the attacker LLM is explicitly prompted to preserve the correct answer, which is provided during the fuzzing process. This ensures that the fuzzes remain anchored to the original question's intent. Furthermore, we rely on the attacker LLM’s demonstrated human-level accuracy on the benchmark as an assumption for generating high-quality fuzzes at scale.

Given the large scale of MedFuzz experiments, manual quality assurance for every fuzzed question is infeasible. However, our workflow incorporates user inspection for particularly interesting or insightful cases, with the final judgment of whether an attack is “fair” being left to human reviewers. While this assumption introduces some dependence on the attacker LLM’s capabilities, we believe it is reasonable for achieving scalability.

Distribution of P-Values

To address the request for an impression of the p-value distribution, we ran an analysis on a run where GPT-4 was the target model, resulting in 85 successful attacks. Below are summary statistics of the p-values:

• Min: 0.0, 5%: 0.0, 25%: 0.10, Median: 0.40, Mean: 0.408, 75%: 0.63, Max: 1.0

To explore trends, we categorized successful attacks into two groups: (1) significant attacks ($p < 0.01$) and (0) insignificant attacks. We calculated an odds ratio for being in group 1 vs group 0. Analysis revealed that topics like “rash,” “substance abuse,” and “ultrasound” were more than twice as likely to fall into the significant group, while others like “HIV,” “breastfeeding,” and “chronic kidney disease” were also overrepresented. However, we recognize that calculating p-values for these odds ratios would conflate p-values used as thresholds, resulting in unsound statistical inference (i.e., p-hacking).

A more robust approach, which we plan to explore in future work, would involve estimating the success probability for specific topics using repeated attacks on individual questions.

Evaluation of Chain of Thought (CoT) Faithfulness

The reviewer highlights an important point about the assessment of CoT faithfulness. In our study, we manually evaluated CoTs from successful attacks, focusing on whether the added fuzz content was explicitly referenced. This process was conducted by inspecting each CoT explanation and verifying its alignment with the fuzzed information that caused the incorrect response.

Human Performance Comparison and Quality Control

We recognize the value of including human medical experts to evaluate the quality of fuzzed questions. However, due to resource constraints, this was not feasible in the current study. We plan to include human evaluation in future work to provide an additional layer of validation for the attacker LLM’s performance and the robustness of fuzzed questions.

Regarding quality control, we rely on the attacker LLM’s prompt-engineered constraints to ensure that generated fuzzes are medically plausible and consistent with the original question’s correct answer. This reliance on a high-performing LLM is a tradeoff we make to scale the MedFuzz method across a large dataset like MedQA.

Analysis of Errors and Robustness to Specific Problem Types

The reviewer’s request for a more granular analysis of errors and model vulnerabilities is well-taken. Our exploratory analysis of attack outcomes highlighted certain topics (e.g., “rash,” “substance abuse,” “ultrasound”) that appear more susceptible to significant attacks. However, as noted above, a more rigorous approach to topic-level success probability estimation is necessary for conclusive insights. We plan to develop a framework for repeated attacks on specific topics, allowing us to model robustness conditional on problem type.

Future Directions

The reviewer’s suggestions align with our broader vision for improving MedFuzz. Specifically, we aim to:

1. Incorporate human evaluation for assessing question quality and attack outcomes.
2. Develop Monte Carlo-based topic-specific estimates of probability of attack success to give insight into which topics are vulnerable.

We believe these improvements will address the limitations of the current study and enhance the utility of MedFuzz for evaluating medical LLMs.

Summary We appreciate the reviewer’s insightful feedback and have outlined both responses to the identified weaknesses and concrete steps for future work. While some limitations remain, we are confident that MedFuzz provides valuable insights into LLM robustness and look forward to building on this foundation.