Revisiting the Scaling Effects of LLMs on Medical Reasoning Capabilities

We are sincerely grateful for your detailed and constructive feedback. Below, we provide our responses to each of the concerns you raised.

1. Novelty of Our Work: Thank you very much for your meticulous review. It is important to emphasize that the core contribution of our work is not the construction of a benchmark for evaluating LLM medical reasoning capabilities, but rather the systematic analysis of the scaling effects of LLMs on tasks with varying reasoning difficulties. In fact, prior to conducting this study, we observed that some of the latest small-scale LLMs (e.g., LLaMA3-8B) achieved comparable or even superior performance to larger-scale LLMs across various benchmarks, including famous medical benchmarks like MedQA. The remarkable performance of these small-scale LLMs has led some researchers to believe that small-scale LLMs trained on larger-scale datasets can rival the performance of larger-scale LLMs. The primary goal of our study is to empirically analyze the effects of training data size $N$ and model parameter size $D$ on LLMs’ reasoning abilities by comparing the performance of a series of LLMs on tasks with varying reasoning difficulty. Our evaluation results reveal that while the latest small-scale LLMs achieve performance comparable to larger-scale LLMs on simpler tasks, their performance falls significantly behind on more challenging reasoning tasks. This indicates that for complex reasoning tasks, merely scaling up the training data size is insufficient; scaling up the model parameter size is also essential. The conclusions drawn from this study provide valuable guidance for the future development and application of LLMs in reasoning tasks.

   • The Modified Dataset Still Contains Only MCQs: In our work, we considered that existing MCQs often provide extra reasoning clues, have a relatively limited decision space, and require fewer reasoning steps. To address this, we modified MCQs from existing medical benchmarks as follows: (1) For reducing reasoning clues, we combined the questions and options of existing MCQs into statement verification tasks, preventing LLMs from leveraging comparisons between options to gain additional reasoning clues; (2) For expanding decision space, we replaced the correct options in MCQs with distractors, requiring LLMs to determine whether the correct answer lies in the options or not; (3) For increasing reasoning steps, we added an additional reasoning step where LLMs must first judge whether the given answer is correct for the original MCQ and, if not, provide the correct answer. While the modified questions still include some form of options (e.g., true/false, answer exists/does not exist), these modifications do enhance the reasoning complexity of the original tasks from three aspects, supporting our analysis of the scaling effects of LLMs on medical reasoning capabilities.
   • Why Not Use Existing Reasoning-Complex Datasets: Thank you for your question. Actually, in our early research, we also considered directly using existing reasoning-complex medical benchmarks for analysis. However, it is crucial to note that the primary goal of this study is not to evaluate the medical reasoning capabilities of current LLMs but rather to analyze the scaling effects of LLMs on medical reasoning. For this purpose, we need to compare the performance of LLMs on both simple and complex reasoning tasks. Directly comparing results from existing medical benchmarks and reasoning-complex benchmarks would be unfair, as these benchmarks assess different medical knowledge areas, and the evaluation results would be significantly influenced by the LLMs’ level of medical knowledge mastery. Therefore, in our implementation, we modified a single medical benchmark while controlling for the medical knowledge being tested. By addressing three key factors affecting reasoning difficulty, we constructed medical benchmarks tailored to our research needs.

2. GPT-generated CoT Issue Thank you for your thoughtful feedback. In this study, we followed [1] to generate “Chain of Thought” (CoT) prompting using GPT-4, aiming to maximize the reasoning capabilities of LLMs. Indeed, CoT testing based on GPT-4 diverges from realistic medical scenarios. However, the primary objective of this research is not to evaluate LLMs’ reasoning performance in real-world medical contexts but to study the scaling effects of LLMs on medical reasoning capabilities. Given that inferencing using GPT-4-generated CoT represents a strong baseline method on current medical benchmarks, we primarily adopted this approach for our experiments. Based on your valuable suggestions, we plan to further validate our findings in the future using alternative settings (e.g., direct 5-shot learning).

   [1] Nori H, Lee Y T, Zhang S, et al. Can generalist foundation models outcompete special-purpose tuning? case study in medicine[J]. arXiv preprint arXiv:2311.16452, 2023.

We are sincerely grateful for your detailed and constructive feedback. Below, we provide our responses to each of the concerns you raised.

1. Generalizability: We sincerely appreciate your insightful suggestions. In fact, prior to conducting this study, we observed that some of the latest small-scale LLMs (e.g., LLaMA3-8B) achieved comparable or even superior performance to larger-scale LLMs across various benchmarks, including famous medical benchmarks like MedQA. The remarkable performance of these small-scale LLMs has led some researchers to believe that small-scale LLMs trained on larger-scale datasets can rival the performance of larger-scale LLMs. The primary goal of our study is to empirically analyze the effects of training data size $N$ and model parameter size $D$ on LLMs’ reasoning abilities by comparing the performance of a series of LLMs on tasks with varying reasoning difficulty. Our evaluation results reveal that while the latest small-scale LLMs achieve performance comparable to larger-scale LLMs on simpler tasks, their performance falls significantly behind on more challenging reasoning tasks. This indicates that for complex reasoning tasks, merely scaling up the training data size is insufficient; scaling up the model parameter size is also essential. The conclusions drawn from this study provide valuable guidance for the future development and application of LLMs in reasoning tasks.

   Considering that medicine is a key application domain for large language models, and solving problems in this field requires expert-level reasoning and decision-making, we have primarily conducted a case study on the medical domain. While we acknowledge that findings in our paper need further validation to generalize to other domains, the underlying principles of scaling effects and reasoning complexity explored in our work are likely applicable to other reasoning-intensive fields. In the revised paper, we will add a discussion to highlight these points and clarify the potential scope of generalizability.

2. Data Diversity: We are very grateful for your constructive suggestion. Our work focuses on studying the scaling effects of LLMs across tasks with varying reasoning complexities, and MedResEval—derived from the widely recognized MedQA benchmark—was specifically designed for this purpose. The reformulations we introduced significantly increased reasoning difficulty from three key aspects, making it sufficient to investigate the scaling effects of LLMs in medical reasoning. While the proposed MedResEval is sufficient for demonstrating our claims, we plan to further incorporate other medical datasets in the future to further validate the generalizability of our claims.

3. More Complex Tasks are Not Expanded Enough: Thank you for your thoughtful suggestions. The primary goal of this work is to empirically analyze the effects of training data size ($N$) and model parameter size ($D$) on LLMs’ reasoning abilities by comparing the performance of a series of LLMs on tasks with varying reasoning difficulties. To control for other variables as much as possible (e.g., directly adopting other reasoning-complex medical datasets might introduce variations in medical knowledge being tested, potentially affecting the fairness of the study), we derived our evaluation dataset by reformulating existing medical benchmarks based on three key factors to create new tasks with increased reasoning difficulty. Indeed, investigating LLMs’ performance on more complex medical scenario problems is also crucial. Considering this, we also designed a diagnostic simulation task in this work, where we perturbed key diagnostic information and required LLMs to reconstruct the correct diagnostic details and complete the corresponding diagnostic task. While our current MedResEval dataset is sufficient for studying the scaling effects of LLMs across medical tasks with varying reasoning complexities, we also plan to extend our evaluation to include more complex medical scenario problems to further evaluate the medical reasoning capabilities of current LLMs.

We are sincerely grateful for your detailed and constructive feedback. Below, we provide our responses to each of the concerns you raised.

1. Data Availablity: Thank you for your constructive comments. We will make all the datasets and codes involved in this paper publicly available to facilitate the related research.

2. Analysis on Medical LLMs: Thank you very much for your insightful suggestions. The primary goal of our work is to revisit the scaling effects of LLMs by analyzing how training corpus size and parameter size influence model performance on problems of varying reasoning difficulty. Given that medicine is a critical application domain for LLMs and solving medical problems heavily relies on reasoning abilities, we conducted a case study in the medical domain. Considering the current limited number and variety of medical LLMs, as well as the differences in their domain-specific fine-tuning procedures, our paper primarily focuses on evaluating and analyzing general-domain LLMs. Based on your constructive feedback, we also conducted a preliminary study by evaluating the Meditron series models (since MedPALM has not been open-sourced, we are currently unable to evaluate it but will do so when possible). Meditron-7B/70B are medical LLMs fine-tuned from Llama2, while Meditron3-8B/70B are fine-tuned from Llama3. The experimental results are as follows:

   Model	Simple	Complex
   Meditron-7B	11.6	-2.4
   Meditron-70B	40.8	12.8
   Meditron3-8B	44.8	10.5
   Meditron3-70B	65.9	33.9

   We found that medical LLMs exhibited trends similar to general-domain LLMs across tasks of varying reasoning difficulty. On simpler tasks, Meditron3-8B outperformed Meditron-70B by 4%. However, on more complex tasks, Meditron-70B still surpassed Meditron3-8B by over 2%. This suggests that medical LLMs are also consistent with the conclusions presented in our paper.

3. Involving More Medical Datasets: We sincerely appreciate the suggestion to incorporate other datasets like MedMCQA or PubMedQA. Our work focuses on studying the scaling effects of LLMs across tasks with varying reasoning complexities, and MedResEval—derived from the widely recognized MedQA benchmark—was specifically designed for this purpose. The reformulations we introduced significantly increased reasoning difficulty from three key aspects, making it sufficient to investigate the scaling effects of LLMs in medical reasoning. Once again, we thank you for the valuable suggestion and plan to explore the inclusion of additional datasets (e.g., MedMCQA) in the future to further validate the extensibility of our evaluation framework.

4. Intuition Behind Eq3: Thank you for your valuable suggestion. The intuition behind Eq3 primarily stems from observations of Eq1, which indicates that when the parameter size N and computational cost C are fixed, the training data size D and test loss L exhibit a power-law relationship. Considering that downstream task performance P is negatively correlated with test loss L, we hypothesize that P may also share a similar power-law relationship with D, where P improves with increasing D before eventually saturating. We further empirically validated this hypothesis through experiments. We will elaborate further on this intuition in the revised version of the paper.

5. Difficulty Dependent Aspect of Eq3: We greatly appreciate your constructive suggestion. The difficulty-dependent aspect in Eq3 is not solely captured by the separation into tasks. It is also reflected in the parameter $\alpha_{N,i}$, which varies with both the model size $N$ and the specific downstream task $i$. This parameter encapsulates the inherent complexity of the task and how a model with a specific parameter size $N$ handles it, thus directly influencing the performance scaling behavior. Thank you for your suggestion, and we will elaborate this in our revised paper.

6. Highlighting why the 3 aspects mentioned were the way to increasing difficulty: Thank you for your constructive feedback. We have provided a detailed explanation of how these three aspects influence reasoning difficulty in Section 3.2 (line 181- line 196). Based on your valuable suggestions, we will further refine the relevant descriptions to enhance clarity in our revised paper.

2. Complexity of Questions in MedResEval: Thank you for your thoughtful comments. In this study, we designed three tasks targeting the three key factors you mentioned, generating three types of questions based on existing medical benchmarks. The issues you pointed out are indeed critical, and we have carefully considered them during the dataset creation process, implementing corresponding designs to address them:

   1. Reducing Available Clues: As you pointed out, the difficulty of multiple-choice questions (MCQs) lies in selecting the correct answer from several closely related options. Considering this, we constructed the evaluation samples by taking each original MCQ and pairing the correct option with a selected wrong option to generate two statement verification questions with opposite labels. LLMs must correctly evaluate the validity of both statements independently to perform better than random guessing, without relying on hints by comparing between options. This ensures that LLMs evaluate the correctness of both the true and misleading options without contextual clues from comparing choices.

   2. Expanding Decision Space: Indeed, including an easily dismissible incorrect answer does not necessarily increase the complexity of the question. To tackle this, we replaced the original correct answer with a distractor, creating another MCQ that does not have a correct answer in the options. Then, we provide both the original and modified questions to LLMs, requiring them to determine whether a correct answer exists among the given options. This design significantly expands the decision space, as LLMs must not only evaluate the given options but also consider the possibility that none of them is correct.

   3. Increasing Reasoning Steps: Of course, evaluating the correctness of a single provided option may reduce reasoning complexity. We have also considered this important point and retained the original MCQ question along with all its options. Actually, we introduced a two-step reasoning task: LLMs are first asked to determine whether a given option is correct for the MCQ. If the answer is incorrect, they must then identify the correct option from the rest options. For example:

      Question: A 25-year-old man comes to the office because of pain in his left shoulder… Which of the following enzymes is most likely deficient in this patient?

      Options:

      A: Branched-chain alpha-ketoacid dehydrogenase

      B: Cystathionine synthase deficiency

      C: Homogentisic acid oxidase

      D: Phenylalanine hydroxylase

      E: Propionyl-CoA carboxylase

      Alice chose answer D. Please verify if this is correct, and if not, provide the correct answer.

      Answer: incorrect, answer C is the correct answer

   To answer such questions, LLMs need to not only evaluate whether Alice’s choice is correct but also identify the correct answer if it is not. Similar to the reducing-available-clues task, we also generated two questions for each original MCQ by pairing the correct option with a distractor. This ensures a more comprehensive evaluation of the LLMs’ reasoning abilities.

Thank you once again for your valuable feedback. We will further refine our paper to enhance the clarity of our implementation details.

We are sincerely grateful for your detailed and constructive feedback and very sorry for the late response. Below, we provide our responses to each of the concerns you raised.

1. Clinical Relevance of Questions in MedResEval: We sincerely appreciate your insightful comments. In fact, prior to conducting this study, we observed that some of the latest small-scale LLMs (e.g., LLaMA3-8B) achieved comparable or even superior performance to larger-scale LLMs across various benchmarks, including famous medical benchmarks like MedQA. The remarkable performance of these small-scale LLMs has led some researchers to believe that small-scale LLMs trained on larger-scale datasets can rival the performance of larger-scale LLMs. The primary goal of our study is to empirically analyze the effects of training data size $N$ and model parameter size $D$ on LLMs’ reasoning abilities by comparing the performance of a series of LLMs on tasks with varying reasoning difficulty. Our evaluation results reveal that while the latest small-scale LLMs achieve performance comparable to larger-scale LLMs on simpler tasks, their performance falls significantly behind on more challenging reasoning tasks. This indicates that for complex reasoning tasks, merely scaling up the training data size is insufficient; scaling up the model parameter size is also essential. The conclusions drawn from this study provide valuable guidance for the future development and application of LLMs in reasoning tasks.

   Considering that medicine is a key application domain for large language models, and solving problems in this field requires expert-level reasoning and decision-making, we have primarily conducted a case study on the medical domain. We found that real-world medical reasoning tasks are often far more complex than questions in LLM benchmarks. In this work, we specifically focus on three key factors that influence reasoning difficulty:

   1. Available Clues: Questions in medical benchmarks often provide additional reasoning clues (e.g., for multiple-choice questions (MCQs), LLMs can leverage elimination strategies to rule out incorrect multiple-choice options). In contrast, real-world medical tasks typically involve less available clues. For example, in clinical diagnosis tasks, doctors must derive the diagnosis based solely on the given patient information, without relying on options to guide reasoning.

   2. Decision Space: Questions in medical benchmarks usually have a small decision space (e.g., 4–6 options for MCQ), whereas real-world medical tasks often require reasoning within a much larger decision space. For instance, in clinical diagnosis, potential decision options may include hundreds or more candidate diseases.

   3. Reasoning Steps: Medical benchmark questions typically require single-step reasoning, while real-world medical tasks often involve multi-step reasoning. For example, a doctor must first arrive at a clinical diagnosis and then determine the appropriate treatment plan based on the diagnosis.

   Indeed, the three key factors discussed in this study are not only closely tied to medical reasoning but are also highly relevant to other reasoning-intensive domains. Therefore, our findings have the potential to be extended to these areas, providing insights for LLM reasoning research. Thank you for your feedback; we will further clarify the motivation and clinical relevance of our work in the paper.

We are sincerely grateful for your detailed and constructive feedback and very sorry for the late response. Below, we provide our responses to each of the concerns you raised.

1. Lack of limitations and future work in Conclusion part: We sincerely appreciate your constructive feedback. While this study provides a detailed evaluation of existing LLMs and reveals the impact of training corpus sizes and parameter sizes on the medical reasoning capabilities of these models, it has the following limitations: (1) Although this study includes 10+ LLMs from two popular LLM families (Llama and Qwen), due to the current availability of open-source LLMs, our evaluation focuses primarily on models with parameter sizes around ~7B and ~70B and pays less attention on intermediate-sized models, as such open-source models are less readily available; (2) While this study conducts a case study in the medical domain and investigates the difficulty-dependent scaling law of LLM reasoning capabilities, this methodology has the potential to be extended to other reasoning-intensive domains. In future study, we plan to include more models of other sizes when research conditions permit, and conduct experiments in other reasoning-intensive domains to further validate the generalizability of our conclusions. We will incorporate these limitations and future work into the revised version of our paper.

2. Bar plots are a little bit hard to illustrate the performance changes by various Ns & Ds: Thank you very much for your constructive suggestions. In the paper, we primarily used bar plots to present the experimental results, aiming to intuitively compare the performance gaps between the latest small-scale LLMs and larger-scale LLMs across tasks with varying reasoning difficulties. Based on your suggestions, we will revise the presentation of this section in the updated manuscript to provide a clearer comparison of performance changes across different Ns and Ds.

3. Indicating both x-axis and y-axis are in log-scale in Figure 6 caption：We are very grateful for your detailed suggestions. We will revise the caption of Figure 6 to help readers better understand the experimental results.

4. Significant Analysis Issue: We are very grateful for your constructive feedback. To ensure the robustness of our conclusions, we employed the self-consistency method during model evaluation. Specifically, we prompted the LLMs to generate answers to the same question five times and then determined the final model prediction using a majority vote. This approach effectively enhances the performance of LLMs and contributes to the robustness of the evaluation results. Based on your kind suggestions, we also conducted the significant analysis by conducting two additional repeated experiments. Given the high computational cost of evaluating all models, we prioritized supplementary experiments on the four models from the Llama family. The results of these experiments, with 95% confidence intervals, are as follows:

   Task	llama2-7B	llama2-70B	llama3-8B	llama3-70B
   Original	16.2±0.3	41.8±1.3	44.8±0.5	64.9±0.0
   Reducing Available Clues	0.8±0.1	17.9±0.2	13.4±0.7	29.2±0.8
   Expanding Decision Space	0.5±0.3	6.9±0.6	0.9±0.2	28.7±0.6
   Increasing Reasoning Steps	-1.3±0.1	26.0±1.2	-1.2±0.2	37.5±0.4

   We observed that the confidence intervals obtained from multiple independent experiments are relatively small since our testing setting inherently enhances robustness. Based on your valuable feedback, we will incorporate this significance analysis into the results presented in the paper to further demonstrate the robustness of our findings.