We appreciate your insightful comments and hope to further address your concerns.

Response 1 - Experimental Designs Or Analyses

1.1 Evaluation on Different TTS Methods

Thank you for the valuable suggestion. We will include these comparisons in the next versions of the paper. At the same time, the main motivation of our work is the high-quality benchmark itself, and dedicated works that focus on comparing inference-time methods (MedPrompt [1]) are better references for researchers interested in this direction. We also note that the inference scheme we use, i.e. zero-shot CoT, aligns with mainstream consensus for evaluating foundation models. We hope the above points address your concerns.

1.2 Specialist Model Results

Your suggestions are very valuable. We agree with the need to include medical specialist models in our evaluation. We did consider it, but only left it out due to time constraints. As preliminary experiments, we evaluated a specialist text model UltraMedical-70B [2]: UltraMedical-70B Results.

We provide the results of o1 for comparison, showing that the specialist model still falls behind the most advanced general reasoning model.

We plan to more thoroughly survey domain-specific models and present the full results in the next version of our paper. We will also systematically compare the performances of generalist models, specialist models, and humans to gain further insights.

Response 2 - Essential References Not Discussed

We totally agree that comparisons with other demanding benchmarks will be informative. We provide the dataset statistics of Humanity’s Last Exam (Biology / Medicine): Humanity’s Last Exam Dataset.

We note obvious discrepancies in relative model performances between the two benchmarks, which demonstrate the informative value of diverse benchmarks.

Regarding benchmark difficulty, we do recognize that the absolute scores for the Biology / Medicine subset of HLE are lower than MedXpertQA. However, this does not take away from the quality and value of MedXpertQA for two reasons:

• First, most of HLE's biology / medical questions focus on biology instead of clinically relevant medical tasks, for instance: Biology Question Example.

• Second, questions in HLE covering comprehensive patient information, thus supporting realistic clinical reasoning, as do most questions in MedXpertQA, are scarce (~35 in total). Many questions related to medical tasks only cover single steps within the complex reasoning process required to form a clinical decision for a patient. For example, the following question represents the step of interpreting a single statistic (R-R interval) from a single piece of patient information (ECG results):

  What is the longest R-R interval in seconds on this rhythm strip ECG?

  In contrast, questions in MedXpertQA that involve ECGs typically include the image as one piece of information within a multifaceted, realistic patient profile. The answerer not only interprets the ECG but also needs to consider the role of this information within a more complex decision, e.g. proposing a diagnosis or treatment. The example illustrated in Figure 2 in our paper precisely shows this scenario. Therefore, while medical questions in HLE effectively pinpoint challenging individual tasks, they are less informative of models' holistic abilities and less clinically relevant. As Reviewer y9fU happened to mention, "reasoning-heavy" and "difficult" are two different concepts, and we believe MedXpertQA has an advantage over HLE in terms of the former.

• Third, MedXpertQA and HLE are fundamentally different types of benchmarks. The construction of HLE was extremely labor-intensive, requiring experts to manually design individual questions, whereas the construction process of MedXpertQA is scalable and systematic, enabling large-scale evaluations over questions that are clinically relevant and diverse. Moreover, the difference in benchmark scale is crucial. It is in fact possible for us to achieve a similar level of difficulty by further stringent filtering, reducing the dataset size — we simply need to retain questions that stump current models. This will, however, contradict the need for systematic and comprehensive coverage in medical evaluation, as we mentioned in Section 3.1. We believe MedXpertQA strikes a good balance.

Response 3 - Other Comments Or Suggestions

Yes, thanks for your suggestion! We will clarify this in the caption later.

References:

[1] https://arxiv.org/abs/2311.16452

[2] https://arxiv.org/abs/2406.03949

Thank you very much for your recognition of our work and your valuable suggestions!

Response 1 - Other Strengths And Weaknesses

You make a great point. Please refer to Response 1.2 (Specialist Model Results) to Reviewer vM2L. Thank you for your understanding!

Response 2 - Other Comments Or Suggestions

2.1 o1 Performance on USMLE

• We want to clarify that the difficulty of MedXpertQA is not due to additional question sources or countries. During dataset construction, we found USMLE questions to be the most difficult, even compared to specialist board exams. Specialty assessments were included to enhance clinical relevance, not to increase difficulty.

• To explain your observation of o1's high scores: USMLE questions in our benchmark and those in MedQA [1] are markedly different. USMLE does not publicize exam questions, so "USMLE questions" usually refer to mock questions devised by third-party experts, with varying difficulties. For MedXpertQA, we collected questions from high-quality, diverse sources and performed rigorous dataset construction steps, e.g. question filtering. This ensured the difficulty of MedXpertQA, accounting for o1’s lower score compared to other benchmarks.

In summary, the difficulty of MedXpertQA is primarily due to the dataset construction process.

Note: EDiR questions only make up a small percentage of MedXpertQA, and the large majority of questions still represent US exams.

2.2 Real-World Representative

Point 1: First, we would like to emphasize that even if exam questions do not completely simulate real-world clinical scenarios, this is a common issue faced by all existing medical benchmarks (every baseline in Table 1, 2), not a new challenge introduced by our work. However, these benchmarks still represent the most important and impactful way to evaluate medical AI. For example, MedQA has already been cited over 100 times this year alone and has been used to evaluate frontier medical AI models such as MedPaLM 2 [2] and Med-Gemini [3]. We believe MedXpertQA can greatly contribute to medical AI progress, and that its impact is amplified by the widespread use of similar, yet less clinically relevant alternatives.

Moreover, your concerns are totally valid - we are aware of some works calling into question whether exam questions are real-world representatives. However, full representation of clinical relevance seems to be an elusive and distant topic that no existing benchmark can achieve, nor is it a claim of our work. We intend to convey in good faith that new work should be recognized primarily for the significant improvements it introduces over existing works, rather than for pursuing overarching goals.

Point 2: Secondly, we need to clarify that we do NOT claim that MedXpertQA completely simulates real-world scenarios. Our claim has always been that it significantly improve (in Abstract) clinical relevance compared with previous, widely adopted benchmarks, which we believe is sufficiently proved and represents an important contribution in itself. Compared with previous benchmarks, MedXpertQA improves clinical relevance through fundamental improvements for both subsets:

• For Text, our addition of medical specialist evaluations is certain to improve relevance, since realistic medical tasks are highly specialized and assigned to different departments. A single, general evaluation suite is evidently inadequate for evaluating the full spectrum of clinical tasks. GMAI-MMBench [4] raised a similar claim. Our discussions with medical expert collaborators further verified this point.

• For MM, current benchmarks commonly design surface-level questions without realistic patient information, leading to extremely limited relevance. MedXpertQA's questions were constructed by human experts and intended to demonstrate how well a medical student will likely perform facing real patients. These expert-designed questions far surpass those constructed through fixed templates or automatic LLM generation of existing benchmarks in quality and relevance. The corresponding images are also realistic and diverse.

Response 3 - Questions for Authors

Again, your concerns are reasonable, and we were also worried about this during our work. This is the main reason we did not publicize the sources of our questions. We also note that this issue is ubiquitous for benchmarks across different domains, such as the AIME datasets in mathematics. While omitting sources may not solve the problem completely, there's not more we can feasibly do for now, and we hope you understand the difficulty behind this issue.

References:

[1] https://arxiv.org/abs/2009.13081

[2] https://arxiv.org/abs/2305.09617

[3] https://arxiv.org/abs/2404.18416

[4] https://arxiv.org/abs/2408.03361

Thanks for your thoughtful comments!

Response 1 - Claims And Evidence

1.1 Leakage Prevention

First, we note that data leakage prevention is an extra precaution we took on top of our main contribution, a challenging, clinically relevant benchmark. Models' subpar performance on MedXpertQA already reflects that its questions haven't been well learned during pretraining.

Our literature review found no effective method for reducing leakage risk of benchmarks, so we used the intuitive LLM rewriting strategy. We detail our extensive efforts here: https://postimg.cc/D4w1hw7g.

Changing questions drastically tends to lower their quality and relevance, and our method balances general quality and low leakage risk. Our method exceeds "simple paraphrasing" - it combines meticulously designed instructions with strict multi-round human reviews and error correction (AppendixD).

1.2 Leakage Risk Evaluation

• We intend to solidly show MedXpertQA's low leakage risk, not devise a new method.
• We highlight the validity of the metrics we used (Please reply for further clarification. Thanks!).
• No previous publication in medical AI benchmarking covered this.
• A recent work compared leakage risks of different benchmarks, showing the superiority of MedXpertQA: https://postimg.cc/yDrvtTB3.

1.3 Reasoning-Oriented Evaluation

• We initially considered having experts label Reasoning (R)/Understanding (U), but we realized that this task is quite straightforward for LLMs given clear prompt guidelines. We also provide expert-written answers and explanations collected from sources (Table13). This dense guidance enables a simplified form of annotation under expert supervision.
• We sufficiently considered the distinction between reasoning complexity and general difficulty when designing our labeling prompt (Table13).
• Human reviewing: For 10% sampled questions (Text-490, MM-400), reviewers found 28 and 11 questions incorrectly labeled as R. We think this error rate (~4.3%) is acceptable.
• Empirical results (Section 5.2) reflect the validity of annotations. LRMs perform much better than their backbones on R, and this does not hold on U. We even note the opposite trend for Qwen-series models on U, which would not hold if U questions were merely easier. For single models, R scores aren't consistently lower than U.

Response 2 - Methods And Evaluation Criteria

2.1 Sequential Reasoning

Two of our question sources included many sequential questions. We found that each question had sufficient context and wasn't dependent on answering previous ones, so we used individual questions instead of preserving the sequence. Though MedXpertQA dissects the multistep process into separate questions, its coverage of all stages of clinical decision-making (Figure3) ensures that it tests wide-ranging abilities needed for realistic multistep tasks. Recent works, e.g. MedChain[1], have focused on this, and we'll add discussions of these works in the next version of the paper.

2.2 Statistics on Individual Filtering Steps

Please see Response 2.1 to Reviewer Rrtx

2.3 Difficulty Quantification and Calibration

We combine two metrics: https://postimg.cc/RNrP4tWF.

Response 3 - Experimental Designs Or Analyses

Please see Response 1

Response 4 - Essential References Not Discussed

We'll add these works in the next version, omitting it here due to space limitations.

Response 5 - Other Strengths And Weaknesses

1 & 2: Please see Response 1

3: We show examples in F.1, F.3 (with human analyses of model errors). Models handle complex clinical information and make nuanced decisions between different possibilities. In addition, fine-grained analysis of model reasoning will be more informative once models achieve better performance.

Response 6 - Other Comments Or Suggestions

• Please see Response 2.2 to Reviewer Rrtx
• Valuable suggestion! We will consider adding it in the next version.
• In Page 4 paragraph 2, we used GPT-4o-2024-11-20. We'll clarify this in revisions.

Response 7 - Questions For Authors

1. Ground truth answers were collected from question sources, whose QA pairs were designed by medical experts.
2. Difficulty: -> Response 2.3
3. Leakage Prevention: -> Response 1.1
4. Risk Evaluation: -> Response 1.2
5. Sequential decision-making: -> Response 2.1
6. While MCQA is not tailored for interactive chat models, MedXpertQA covers diverse tasks, some of which touch on clinical questioning: https://postimg.cc/ZCVYzGXs.
7. Reasoning: -> Response 1.3

---

While responding to your comments, we noticed aspects where our current paper did not sufficiently reflect our efforts. That being said, we believe the drastic improvement in benchmark difficulty is a convincing indication of our extensive efforts. We hope our response addresses your concerns and look forward to incorporating relevant information into the paper.

References:

[1] https://arxiv.org/abs/2412.01605

We hope that our clarifications fully address your concerns!

Response 1 - Claims And Evidence

1.1 Target Audience

• We agree with the value of expert insight. We worked closely with medical practitioners when designing and reviewing MedXpertQA.
• Audience from an AI background is irreplaceable. Since the goal of MedXpertQA is to help researchers understand and improve model limitations, our most important target audience is medical AI researchers.

1.2 Real-World Representation

Please see Response 2.2 to Reviewer gh7D.

1.3 Concern on Patient Data Collection

• This is only one type of question in MedXpertQA, whose subtasks span different stages of medical decision-making (Figure 3). It also contains questions requiring data processing and organizing: https://postimg.cc/ZCVYzGXs.
• We acknowledge that MedXpertQA doesn't explicitly model the multi-step process of single patients. This is an inherent limitation of MCQA, and relevant works, e.g. MedChain [1], tackle the issue. Nevertheless, MedXpertQA's diverse task coverage ensures that it adequately tests wide-ranging capabilities.
• Finally, patient data collection may not be most challenging. Poor results of sota models reflect the difficulty of downstream clinical decision-making, the focus of MedXpertQA.

Response 2 - Methods And Evaluation Criteria

2.1 Benchmark Construction Details

We've attempted to present the construction process concretely and in detail. We provide metric formulas, hyperparameters, etc. This level of detail exceeds many prominent benchmarks, e.g. MMLU-Pro, MMMU. If any aspect remains unclear, we can provide further clarification.

• Unfortunately, we are unable to provide the initial dataset distribution, as labeling 37k questions for multiple attributes would be too costly. We appreciate your understanding.
• Statistics on questions remaining after each step: Dataset Filtering Stats.
• Our dataset construction is tailored to the medical domain, thus not intended for direct application to general AI. Our main contribution is the benchmark itself, especially MM, which fills crucial gaps in current medical multimodal evals.

2.2 Human Performance Evaluation

We already have some preliminary human performance results: https://postimg.cc/fkkJCjy0. The details:

• Expert (Junior) Score: Response distributions collected from question sources mostly come from medical students and reflect human performance. Rewriting has little impact on human performance, since it retains question information. Thus, human performance on original questions can be compared with final model results. The large response number of each question (up to 238k) makes these stats highly representative. Fewer than 200 final questions lack response data, and we'll hire humans of similar caliber to answer them and complete the human performance evaluation. We'll incorporate this information before 4.8 AoE.
• Expert (Senior) Score: We will assess human experts with medical licenses/MDs for a separate expert performance measurement. These experiments will take longer and will be added in future versions.

2.3 Model Biases

• MedXpertQA primarily aligns with practices from the US. We'll cover this in the revised paper.
• Model bias analysis, while important, exceeds the scope of this work. Bias mitigation should be conducted by model developers, and works such as [2,3,4] would be better references for practitioners interested in these issues.
• We provide MedXpertQA's coverage of patient demographics. Gender (from keyword matching):
  • Text: Male 1025, Female 903
  • MM: Male 874, Female 550
  • Total: Male 1899, Female 1453

For ethnicity, we re-examined 100 questions and found no mentions.

Response 3 - Experimental Designs Or Analyses

See Response 2 (R2) for stats.

Table D was from expert evaluations. Experts not only reviewed each question, but also did multiple rounds of editing.

Figure 5 results for all models: https://www.hostize.com/zh/v/TH3ddN2Bz3.

Response 4 - Essential References Not Discussed

Please see R2 to Reviewer vM2L.

For MedXpertQA specifically, an interesting result is the noticeable gap between R1 and o1, which would be unexpected if we directly extrapolated from other reasoning benchmarks.

Response 5 - Questions For Authors

1. -> R2.1
2. -> R2.2
3. Figure 4 requires data on paired backbone LLMs and LRMs, thus can't be done for all models. Please refer to the main results table for results on more models.
4. -> R3
5. -> R2.3
6. -> R4
7. -> R1.1-1.3 MedXpertQA's main goal is evaluating models' fundamental medical abilities. These abilities provide crucial support for clinical use, but downstream applications are not our focus.

References:

[1] https://arxiv.org/abs/2412.01605

[2] https://www.nature.com/articles/s41591-024-03113-4

[3] https://www.nature.com/articles/s41746-025-01503-7

[4] https://www.nature.com/articles/s43856-024-00601-z