Evaluating LLMs Across Multi-Cognitive Levels: From Medical Knowledge Mastery to Scenario-Based Problem Solving

We sincerely appreciate your thoughtful and constructive feedback. Below, we address each of the concerns you raised.

1. Benchmark design issue : We highly appreciate your kind comments. Indeed, the cognitive difficulty of a question is affected by multiple factors, including task type and question complexity. However, the difficulty of evaluation questions is typically constrained by the task format. For example, while the MedQA dataset is designed to evaluate LLMs’ ability to analyze clinical cases, its multiple-choice format somewhat limits its ability to fully reflect how well LLMs request and integrate diagnostic information in real clinical settings. In fact, the task formats in our benchmark inherently reflects variations in cognitive difficulty: Mid-level tasks reflect the impact of increasing reasoning steps and expanding decision space on performance, while high-level tasks further evaluating LLMs in requesting and integrating critical information for decision-making in clinical scenarios. While there is indeed some variation in difficulty among questions within the same task format, the constructed benchmark generally meets our design expectations.

   We also conducted a clinician evaluation to validate the proposed benchmark. Specifically, we first randomly sampled 20 questions from each task to form a subset of 100 questions. Then, we recruited four licensed clinicians with 3 to 8 years of experience to evaluate this subset from two perspectives: (1) their accuracy in answering the questions and (2) their subjective difficulty ratings to the questions (1=Easy, 10=Hard). The evaluation results are as follows:

   Cognitive Level	Clinician Accuracy	Clinician Subjective Difficulty
   Low	68.8	Mean:5.0 Median: 5.5
   Mid	54.2	Mean:6.0 Median: 5.8
   High	23.5	Mean:7.5 Median: 7.5

   The experimental results show that clinicians’ accuracy decreases as the cognitive level increases, aligning with their subjective difficulty rating. This indicates that the designed tasks successfully achieved the intended difficulty levels and can effectively reflect how far LLMs are from effectively solving real clinical problems. Thank you again for your kind suggestions, and we will add this clinician evaluation into the revised paper.

2. Evaluation of reasoning LLMs: Thank you for your constructive suggestions. Indeed, since our paper submission, reasoning LLMs have been receiving increasing attention. Following your kind suggestions, we primarily evaluated two typical reasoning LLMs, DeepSeek-R1 and o3-mini, and compare them with SOTA chat LLMs in corresponding parameter scales. The evaluation results are as follows:

   Model	Low-Level	Mid-Level	High-Level
   DeepSeek-V3	78.3	44.8	19.4
   DeepSeek-R1	89.3	73.1	25.9
   GPT-4o-mini	62.7	43.0	13.8
   o3-mini	88.1	75.2	15.5

   Results show that reasoning LLMs outperform chat LLMs across all cognitive levels, though the performance gap narrows on high-level tasks due to their significantly increased difficulty. Note that we have not evaluated other reasoning LLMs due to time and cost constraints (as the price of o1 is quite high). We plan to add this evaluation and include additional evaluations in our revised paper.

3. Insights of medical LLM development: Thank you for your thoughtful comments. This work provides two key insights into the development of medical LLMs:

   (1) Although the performance of smaller LLMs (<10B) on medical benchmarks is gradually approaching that of larger LLMs, our study indicates that increasing parameter size remains crucial for tackling tasks in higher cognitive levels.

   (2) While medical post-training (used in current medical LLMs) and inference-time scaling (applied in reasoning LLMs) strategies work on low- and mid-level tasks, effectively solving high-level tasks demand further advancements in clinical reasoning abilities—particularly in the retrieval and integration of key information for decision-making in real-world scenarios.

   Again, we appreciate your comments and will further highlight our insights in the revised paper.

4. Discussion with more benchmarks: We are sincerely grateful for your kind suggestions. Wu et al. investigate LLMs’ ability to handle diverse medical tasks by constructing MedS-Bench, a large-scale benchmark covering 11 types of clinical tasks. Johri et al. study LLMs' ability to obtain the necessary patient information for diagnosis through diaglogue by proposing CRAFT-MD, an evaluation framework that simulates multi-turn doctor-patient interactions using a multi-agent approach. Meanwhile, our work focuses on evaluating how close LLMs are to effectively solving real-world clinical tasks by designing evaluations with progressively increasing cognitive difficulty and systematically analyzing existing LLMs. We will further enhance the discussion of other medical LLM benchmarks in our revised paper.

We sincerely appreciate your thoughtful and constructive feedback. Below, we address each of the concerns you raised.

1. Comparability between different cognitive levels: Thank you for your thoughtful concern. Indeed, comparing LLMs’ performance across different cognitive levels is crucial for analyzing their medical capabilities. Considering this, we have implemented a metric alignment strategy in the "Performance Metrics" section (see Equation 4 on page 5) to eliminate interference caused by random guessing and ensure the comparability of tasks across different cognitive levels.

   In fact, we have carefully considered the cognitive difficulties when designing the tasks in our benchmark. Compared to MCQs, mid-level tasks increase difficulty by removing clues, increasing reasoning steps, and expanding the decision space. High-level tasks go further by requiring LLMs to request necessary information instead of receiving it directly from the question. To ensure that the constructed benchmark meets our expectations, we followed your kind suggestions and conducted a clinician evaluation. Specifically, we randomly sampled 20 questions from each task, resulting in a clinician evaluation subset of 100 questions. Four licensed clinicians with 3 to 8 years of experience were recruited to assess the benchmark’s difficulty from two perspectives: (1) their accuracy in answering the questions and (2) their subjective difficulty ratings to the questions on a scale from 1 (Easy) to 10 (Hard). The evaluation results are as follows:

   Cognitive Level	Clinician Accuracy	Clinician Subjective Difficulty
   Low	68.8	Mean:5.0 Median: 5.5
   Mid	54.2	Mean:6.0 Median: 5.8
   High	23.5	Mean:7.5 Median: 7.5

   We found that clinicians’ accuracy also decreases as the cognitive level increases across three levels. Moreover, their subjective difficulty ratings align with this trend, further demonstrating the validity of our benchmark. Once again, we appreciate your kind suggestions and will incorporate this clinician evaluation into the revised paper.

2. LLM scale effect issue: Thank you for your thoughtful comments. Indeed, the scaling law indicates that larger models generally perform better, but it lacks a detailed analysis of the impact of parameter size on tasks across different cognitive levels (difficulties). Considering this, we conducted a systematic evaluation to investigate parameter size effects at varying cognitive levels and found that larger LLMs significantly outperform smaller ones on harder medical tasks, demonstrating the critical role of parameter scale in real-world clinical problem-solving. Moreover, our analysis of medical-specific LLMs reveals that existing post-training methods fail to enhance high-level task performance, offering key insights for the development of medical LLMs.

3. Evaluation setting & parameter issues: We sincerely appreciate your thoughtful comments.

   (1) Evaluation setting: For low- and mid-level tasks, we adopted the few-shot learning (widely used in LLM benchmarks), as it introduces minimal subjective bias by only guiding LLMs with examples, while prompt engineering and CoT techniques may introduce subjective biases through prompt design, potentially affecting the fairness across LLMs. For high-level tasks, regarding the reasoning difficulty, we adopted an agent-based setting [1], guiding the model to generate a rationale before producing the corresponding action.

   (2) Decoding parameters: We chose the decoding parameters to align with task characteristics. For low- and mid-level tasks, we set temperature = 0 (greedy decoding) as these tasks have well-defined answer formats, and greedy decoding selects the most confident answer. For high-level tasks, we set the temperature=0.8 to balance the reasoning diversity and output stability of LLM responses. We also verified through preliminary experiments that LLMs produce stable outputs with this parameter setting.

   [1] Hager P et al. Evaluation and mitigation of the limitations of large language models in clinical decision-making. Nature medicine, 2024.

4. Examples of LLM answer: Thank you for your constructive advice. We will include more examples of LLM answers in the appendix to better illustrate our evaluation process.

5. Clarification of the Ambiguous Statement: Thank you for your careful reading. We sincerely apologize for the ambiguity—here, "training samples" refer to the demonstrative examples used in few-shot in-context learning. Regarding few-shot learning, we have briefly introduced the process in Section 4.1 ("Evaluation Setting") and illustrated the corresponding input format in Figure 9. Notably, our evaluation does not involve any model training. Once again, we appreciate your careful reading. We will conduct thorough proofreading to avoid similar ambiguities in the revised manuscript.

We sincerely appreciate your kind feedback as well as your recognition of our work. Below are our responses to each of the concerns you raised.

1. Discussion with other LLM benchmark: We sincerely appreciate your kind suggestions. Zhou et al. [1] proposed a medical evaluation framework to generate reliable and diverse test samples based on knowledge bases, addressing the issues of low reliability and diversity in automatic test sample generation. Meanwhile, our work aims to explore the limitations of LLMs in solving real-world medical problems by constructing a benchmark with medical tasks of varying cognitive levels and systematically evaluating existing LLMs on this benchmark, offering insights for developing LLMs suited to real-world medical applications. Once again, thank you for your kind suggestions. We will strengthen discussions with other medical LLM benchmarks in the revised paper.

   [1] Zhou, Yuxuan, et al. "Reliable and diverse evaluation of LLM medical knowledge mastery." arXiv preprint arXiv:2409.14302 (2024).

2. Name of proposed benchmark: Thank you for your kind suggestions. Indeed, a good benchmark name could facilitate reference and discussion. We plan to name the benchmark MulCogEval (Multiple-Cognitive-Level Evaluation) and will include relevant annotations in the paper.