We thank the reviewers for their thoughtful and positive feedback. We are encouraged that they found our motivation and idea clear (R2), novel (R1, R2), and important (R3, R4). We are particularly grateful that our experiments were unanimously found to be comprehensive and thorough (R1, R2, R3, R4). The paper was also praised for being well-written and structured (R2, R3), and the code release was noted as a major plus (R3). We are pleased that reviewers recognized that our work provides valuable guidance for resource allocation in training reasoning models for out-of-distribution (OOD) tasks (R1, R3, R4). Below, we address the specific concerns and suggestions raised by the reviewer.

1. Mechanistic explanation or validation on various findings from our study

We thank the reviewer for highlighting these insightful observations and agree that a deeper mechanistic investigation into each would be highly valuable. In this work, we aimed to strike a balance between the depth and breadth of our exploration. Our primary objective was to conduct a large-scale empirical study on the optimal resource allocation between SFT and RL for a challenging OOD task, a focus which required comprehensive ablations and experiments that consumed the majority of our computational resources. This necessarily limited the scope for dedicated studies into all of the secondary phenomena we observed. Consequently, we present findings—such as the completion length contraction, the surprising efficacy of the 'Answer-First' pattern, and the hypothesis that denser rewards lead to local optima—as important, empirically-grounded early observations. While a full mechanistic validation of each was beyond the scope of this paper and would deserve a dedicated study, we believe they are valuable contributions in their own right, revealing unique dynamics of RL in this domain and providing a strong foundation for future exploration.

Nevertheless, we provide a brief discussion on some of the findings as below, in combination with some very recent work from general domains that share similar observations.

• Length Contraction: For DRG assignment, the reasoning process is structurally straightforward: identify the principal diagnosis → extract relevant secondary diagnoses → assess CC/MCC status → assign the DRG (Sec. 3.1). While domain knowledge is nontrivial, the sequence of steps is generally fixed, unlike mathematical problems where solution strategies vary considerably. Hence, it is intuitive that a well-trained LLM may converge toward shorter, more structured outputs. Indeed, Cheng et al. (2025) found that response length increases correlate with performance gains only in certain domains; notably, length contraction is observed in Code, Logic, and Tabular tasks [1]. Moreover, another study shows that RL-trained reasoning models can learn to bypass verbose CoT by internalizing the reasoning process while still producing correct answers [2].
• Preference for Answer-First: Our observation that answer-first prompting yields superior performance likely reflects how LLMs encode and retrieve latent knowledge. These findings align with recent studies, which suggest that CoT and extended reasoning may not always be necessary for reasoning models, and a “no-thinking” pattern can sometimes yield better performance [3, 4]. In addition, presenting the answer early may help the model condition its explanation generation on a confident anchor, avoiding hallucinated or self-justifying CoT paths. This hypothesis aligns with the work showing that answer-first prompting can expose inconsistencies in reasoning and even outperform conventional CoT-first strategies in certain settings [5].

Reference:

1. Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective.
2. Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models.
3. Reasoning models can be effective without thinking.
4. Reasoning models don’t always say what they think.
5. Order Matters in Hallucination: Reasoning Order as Benchmark and Reflexive Prompting for Large-Language-Models.

2. Formatting inconsistency

Thank you for your sharp eye and for pointing this out. We will fix the "Kl Decay" typo and ensure all formatting is consistent throughout the final camera-ready version.

3. Necessity of cross-domain validation

We thank the reviewer for the valuable feedback on generalizability, and we agree that broader validation is an important next step. Our study was intentionally designed as a deep, empirical investigation into a single, highly complex OOD task—DRG coding—to enable a rigorous analysis that would be difficult to replicate with the same depth across multiple domains. This focused approach allowed for the extensive experiments on SFT-RL data trade-offs (Sec 5.2), detailed ablations (Sec 5.3), and investigation of RL prerequisites (Sec 5.4) that led to our central finding: a clear, log-linear scaling relationship between the volume of SFT data and subsequent RL performance. While based on one task, this result provides strong, empirical evidence for our hypothesis that for complex OOD tasks, RL effectiveness is fundamentally constrained by the domain knowledge infused via SFT. We therefore position our claim as an important finding that provides strong motivation for future work to test this hypothesis across other domains, as noted in our limitations (Sec E).

Below, we provide a brief discussion on some very recent work from other tasks and domains that resonates with our conclusion:

• No New "Knowledge" from RL?: An ongoing and pertinent debate is whether RL truly incentivizes new reasoning capabilities beyond those already encoded in the base model. Yue et al. (2025) conducted a rigorous analysis across mathematical, coding, and visual reasoning benchmarks, concluding that RL does not induce novel reasoning patterns beyond those already present in the pretrained model [1].
• Hybrid SFT and RL Training is Needed: Lu et al. (2025) analyzed training dynamics on complex reasoning tasks, observing that RL excels at maintaining and improving performance within a model’s existing capabilities, whereas SFT is more effective for making progress on questions beyond the model’s current scope [2].
• Medical Domain Evidence: Zuo et al. (2025), analyzing Qwen-7B models across medical question-answering and mathematical tasks, noted that medical tasks demand much richer factual/domain knowledge compared to mathematical tasks, which emphasize logical reasoning [3]. They concluded that in medical domains, SFT remains important due to the necessity of detailed domain knowledge, while RL enhances reasoning accuracy by pruning inaccurate or irrelevant reasoning paths.

In summary, our core conclusion is reinforced by these very recent cross-domain studies. While these papers provide corroborating evidence, our work is distinguished by its systematic and comprehensive investigation of the SFT-RL trade-off, offering a granular, quantitative analysis of this critical relationship.

Reference:

1. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
2. Learning What Reinforcement Learning Can’t: Interleaved Online Fine-Tuning for Hardest Questions.
3. Knowledge or Reasoning? A Close Look at How LLMs Think Across Domains.

We thank the reviewers for their thoughtful and positive feedback. We are encouraged that they found our motivation and idea clear (R2), novel (R1, R2), and important (R3, R4). We are particularly grateful that our experiments were unanimously found to be comprehensive and thorough (R1, R2, R3, R4). The paper was also praised for being well-written and structured (R2, R3), and the code release was noted as a major plus (R3). We are pleased that reviewers recognized that our work provides valuable guidance for resource allocation in training reasoning models for out-of-distribution tasks (R1, R3, R4). Below, we address the specific concerns and suggestions raised by the reviewer.

1. Question about the insight relates to the “Bitter lesson”

• Thank you for your insightful feedback. While our conclusion aligns with Sutton’s "bitter lesson," we hope our work offers a timely and empirically grounded clarification for the domain of complex, out-of-distribution (OOD) tasks.
• Following successes of reasoning models like DeepSeek-R1, a prevailing narrative has been to "scale RL," leaving a critical question unanswered: what, precisely, should be scaled? Our work addresses this for complex, OOD tasks where the answer is not straightforward. We demonstrate that naively scaling RL (in terms of data and compute) on a base model fails entirely for DRG coding, as vanilla models were unable to produce correct outputs with RL alone.
• Through rigorous experimentation, we conclude that for OOD challenges, knowledge infusion is the critical component to scale. Our findings suggest that scaling SFT is often more effective and computationally efficient than scaling RL alone. In this light, our work encourages the community to more carefully consider what is being scaled when applying RL. We hope this clarification serves as a meaningful and timely contribution, offering an evidence-based explanation and a clear path forward for the many practitioners working to apply RL to domain-specific OOD tasks.

2. Additional literatures

Thank you for suggesting these insightful references. We would be happy to include both papers as related work in the camera-ready version. The first paper from OpenAI presents an effective approach that uses fine-grained, composable rules and an LLM grader to create a hybrid reward signal, balancing model safety and usefulness [1]. We will also highlight the novel framework in the second paper, which employs symbolic representations of clinical reasoning to ground vision-language models (VLMs) in medical knowledge and uses a symbolic reward function to evaluate VLM responses for individual correctness and clinical validity [2].

Reference:

1. Rule based rewards for language model safety.
2. Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Grounding.

3. Missing limitations section

• We thank the reviewer for the helpful suggestions and wish to clarify that a limitations section is provided in Appendix Sec E. In the camera-ready version, we will expand this section to incorporate your valuable points on scalability and SFT data.
• Of note, following another reviewer’s kind suggestion and with IRB approval, we conducted an additional evaluation using an internal dataset of 2,500 real-world cases from a healthcare institution. DRG-Sapphire achieved an accuracy of 53.6% on this internal dataset, compared to 54.8% on MIMIC, demonstrating its robustness beyond the MIMIC corpus. We also submitted an IRB application for validating and implementing our model in real-world clinical settings at large scale.

We appreciate the reviewer's positive feedback. We are pleased that we could address your concerns and would be happy to answer any further questions.

We thank the reviewers for their thoughtful and positive feedback. We are encouraged that they found our motivation and idea clear (R2), novel (R1, R2), and important (R3, R4). We are particularly grateful that our experiments were unanimously found to be comprehensive and thorough (R1, R2, R3, R4). The paper was also praised for being well-written and structured (R2, R3), and the code release was noted as a major plus (R3). We are pleased that reviewers recognized that our work provides valuable guidance for resource allocation in training reasoning models for out-of-distribution (OOD) tasks (R1, R3, R4). Below, we address the specific concerns and suggestions raised by the reviewer.

1. Question about in-context learning examples

• Due to the large DRG search space (over 700 distinct codes) and the base model's lack of DRG-specific knowledge (Fig. 1A), we found in-context learning (ICL) examples insufficient for effective knowledge injection. It is not feasible to cover the full code space and their associated clinical notes within a few ICL examples. While increasing the number of ICL examples could improve coverage, it would also lead to a quadratic increase in inference cost.
• A RAG-based approach for selecting examples would also be problematic for two reasons: (1) Clinical notes with similar semantic features can map to very different DRG codes, making retrieval unreliable. (2) The retrieved examples would constrain the model's output to a small subset of DRG codes, limiting its ability to reason over the entire code space. For these reasons, we chose to apply SFT to infuse domain-specific knowledge before RL.

2. Suggestion to add error analysis

We conducted additional error analyses as kindly suggested by the reviewer.

• We identified 9,537 cases consistently misclassified by all three checkpoints from different training stages: the cold-start SFT (SFT model in Fig. 5A), the small-scale GRPO (GRPO model in Fig. 5A), and the final GRPO-Sapphire model. As shown in the table below, even in these challenging cases, the model improved progressively during training, becoming notably "less wrong."

• Additionally, a physician expert conducted a detailed manual error analysis on 100 randomly selected cases misclassified by the final GRPO-Sapphire model, applying the same error taxonomy used in DRG-LLaMA.

• The physician expert also marked 15% of these cases as ambiguous, noting that the correct DRG was unclear based on DRG guidelines, for example due to multiple potential principal diagnoses. Collectively, these results underscore the inherent complexity of DRG assignment. Interestingly, the identification of potentially incorrect DRG labels highlights a promising potential use case for our model in detecting medical coding errors.

• When reviewing cases where other proprietary LLMs fail, we observed that most failures stem from a lack of DRG-specific knowledge, highlighting the OOD nature of the task. For example, these models may output invalid DRG codes or struggle to determine whether a secondary condition qualifies as a CC or MCC.

3. Question about impact of erroneous DRG code assignment

We thank the reviewer for raising this important point. After consulting with physician leaders, we agree that the impact of errors is highly use case-dependent. For high-stakes applications like automated billing, the error tolerance must be extremely low due to potential financial consequences. In contrast, for operational or educational tools—such as the one in Appendix D.1 that helps physicians understand DRG assignments—a higher degree of fault tolerance may be acceptable. To rigorously assess these factors in practice, we have submitted an IRB application to prospectively validate the model's performance and impact within a healthcare system. This will enable a formal evaluation from a healthcare delivery perspective.

4. Question about the reason of completion length contraction

We hypothesize that the completion length contracts because DRG assignment is a highly structured reasoning task, allowing the model to learn a more concise and efficient solution path. Due to word limits, we refer the reviewer to our response to RJgS (Q1) for a full analysis with supporting references.

5. Question about cross-domain generalization of the “Bitter-lesson”

Our conclusion is grounded in comprehensive experiments on the OOD task of DRG assignment, and we suggest these findings are likely to generalize to other OOD tasks where the model has minimal pre-training exposure. We provide a detailed discussion with supporting literature in our response to Reviewer RJgS (Q3).

6. Performance when training on SFT dataset produced by 14B model

This is a great thought, and we have conducted additional experiments as the reviewer suggested.

Interestingly, RL training based on SFT data produced by the 14B model did not improve performance. We suspect this may be due to a teacher–student capacity gap: the smaller 7B student model struggled to imitate the complex style and output distribution of the larger model, resulting in suboptimal policy initialization that hindered RL optimization.

7. Question about SFT dataset

We are happy to clarify this important detail. The SFT dataset contains two parts: (1) the true DRG label from MIMIC, which is always correct, and (2) the CoT reasoning trace. For the CoT trace, we differentiate between its factuality (e.g., whether a condition is CC vs. MCC) and its cognitive pattern (the reasoning logic). We do not expect the initial CoT traces to be factually perfect, as the Qwen base model lacks DRG-specific knowledge. Instead, it is crucial that the cognitive pattern is correct. Through prompt engineering (Sec. H), we introduce diverse and valid cognitive patterns for the model to learn during SFT (lines 162–165). This provides a sound structure for the RL phase to build upon and learn factuality.

8. Question about iterative training

We agree that training can be done iteratively, either via repeated SFT/RL cycles as the reviewer suggests or through multiple policy updates per generation. However, there are two key considerations for such a process:(1) RL training risks entropy collapse as training progresses [1], necessitating careful entropy management to ensure training success [2, 3]. (2) If the SFT dataset from the latest checkpoint does not introduce new knowledge, iterative learning may be ineffective. A recent study proposes a compelling approach, introducing partial solutions during iterative training to simplify problem difficulty and reshape the learnability landscape [4]. This increases the likelihood of discovering correct trajectories, leading to improved RL training outcomes.

Reference:

1. The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models.
2. Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning.
3. Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs.
4. QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation.

9. Question about LoRA

We did not test LoRA because the primary bottleneck in our RL work, as in many RL with verifiable rewards studies, was the computational cost (inference time) of generating rollouts, not GPU memory. Since memory was not the constraint, we opted for full-parameter training to obtain a cleaner evaluation of our method, thereby avoiding the complexity and potential confounding variables of LoRA-specific hyperparameters (e.g., rank selection).

10. Question about pure-online GRPO

We chose the pure-online variant of GRPO for several reasons.

• Our approach aligns with the original GRPO implementation from DeepSeek-Math [1] which is pure-online.
• This decision is supported by growing evidence that on-policy RL can yield better performance than off-policy settings [2, 3]. Specifically, off-policy learning has been shown to accelerate entropy collapse [2]. Furthermore, recent work by Hao et al. (2025) identifies potential overfitting issues associated with off-policy learning [3].
• From an experimental design perspective, this choice reduces the number of hyperparameters to tune, simplifying the setup and leading to a cleaner analysis.

Reference:

1. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.
2. Skywork Open Reasoner 1 Technical Report.
3. On-Policy RL with Optimal Reward Baseline.

I thank the authors for their detailed responses and the additional experiments. Most of my questions/concerns have been addressed. I am not sure about the conclusions derived by the authors with respect to the observations pertaining to experiments in "Performance when training on SFT dataset produced by 14B model". However, I do not have any alternate hypothesis as well in this regard.

We thank the reviewers for their thoughtful and positive feedback. We are encouraged that they found our motivation and idea clear (R2), novel (R1, R2), and important (R3, R4). We are particularly grateful that our experiments were unanimously found to be comprehensive and thorough (R1, R2, R3, R4). The paper was also praised for being well-written and structured (R2, R3), and the code release was noted as a major plus (R3). We are pleased that reviewers recognized that our work provides valuable guidance for resource allocation in training reasoning models for out-of-distribution (OOD) tasks (R1, R3, R4). Below, we address the specific concerns and suggestions raised by the reviewer.

1. Question on the exact source of the performance improvement

We address this important question as follows:

• TL;DR: Initial SFT (or “adaptation,” as the reviewer puts it) is necessary but NOT sufficient for improving LLM performance on OOD tasks. Reinforcement learning with verifiable rewards (RLVR) further enhances reasoning and generalization beyond what SFT alone provides.
• We agree with the reviewer that the baseline models indeed lack relevant knowledge about medical coding. This is precisely the main focus of our paper: investigating how RLVR can be effectively applied to OOD tasks. We demonstrate that RLVR alone is insufficient for injecting domain-specific knowledge (Fig 10A); thus, initial SFT ("domain adaptation," as noted by the reviewer) is essential. However, after this warming-up phase, RLVR further enhances performance, indicating that RLVR offers benefits beyond simple domain adaptation.
• We also agree with the reviewer that the "domain adaptation," such as memorizing DRG vocabulary and adapting to the MIMIC corpus, is important but represents only a partial view of knowledge infusion in our complex task. Medical coding tasks vary in complexity; for instance, ICD coding may straightforwardly match terms like "bladder infection" to "cystitis" when MEAT criteria (Monitoring, Evaluating, Assessing, and Treating) are satisfied. However, DRG coding is significantly more intricate. A clinical note mentioning "cystitis" may or may not correspond to the DRG code "KIDNEY AND URINARY TRACT INFECTIONS," depending on principal diagnosis designation and competing conditions or procedures. Moreover, the severity and complications (e.g., simple infection versus septic shock) influence whether "cystitis" is categorized as a CC or MCC. As detailed in lines 42-45, this complexity highlights the importance of reasoning capabilities in our model.
• Throughout the RLVR process, we clearly observed advancements in the critical reasoning capabilities required for accurate DRG assignment. Specifically, the model learned to identify the correct principal diagnosis and assess whether secondary conditions meet CC/MCC criteria based on clinical complexity (see Sec I for examples). Additional error analysis presented below (Q3) further support this observation. Crucially, these reasoning capabilities emerged during the RLVR stage, after domain adaptation via SFT was complete, thus reflecting learning beyond mere supervised memorization.
• When discussing memorization and overfitting to a data distribution, it is important to distinguish between RL and SFT. Recent work has demonstrated that RL in LLMs can achieve much better generalization than SFT and moves beyond memorization, with findings suggesting that “SFT Memorizes, RL Generalizes” [1]. Additionally, RL-tuned math reasoning models generalize well across diverse domains and tasks [2].
• Additionally, below we provide robust DRG-Sapphire performance on our internal dataset, extending beyond the MIMIC corpus. Collectively, these findings strongly suggest our model has achieved deeper learning than simply memorizing coding vocabulary and adapting to the MIMIC corpus distribution.

Reference:

1. SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training.
2. Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning.

2. Only one train and test set is used

• We appreciate the reviewer’s valuable suggestion and, following IRB approval, conducted an additional evaluation using an internal dataset from a healthcare institution. We constructed this dataset by stratified random sampling of 2,500 real-world cases, drawing each DRG code in proportion to its frequency in the MIMIC test set to ensure comparability. DRG-Sapphire achieved an accuracy of 53.6% on this internal dataset, compared to 54.8% on MIMIC, demonstrating its robustness beyond the MIMIC corpus. We also submitted an IRB application for validating and implementing our model in real-world clinical settings at large scale.
• To the best of our knowledge, MIMIC is the only publicly available dataset containing both clinical notes and DRG labels, and was the sole dataset used in prior work such as DRG-LLaMA. While we acknowledge the limitation of a single source, MIMIC’s training and test sets are large (230k and 28k cases, respectively) and reflect real-world patient distributions in North America, as they span over a decade of ICU and ED admissions at Beth Israel Deaconess Medical Center in Boston, Massachusetts.

3. A lack of error analysis

We conducted additional qualitative and quantitative error analyses as kindly suggested by the reviewer.

• We identified 9,537 cases consistently misclassified by all three checkpoints from different training stages: the cold-start SFT (SFT model in Fig. 5A), the small-scale GRPO (GRPO model in Fig. 5A), and the final GRPO-Sapphire model. As shown in the table below, even in these challenging cases, the model improved progressively during training, becoming notably "less wrong."

• Additionally, a physician expert conducted a detailed manual error analysis on 100 randomly selected cases misclassified by the final GRPO-Sapphire model, applying the same error taxonomy used in DRG-LLaMA.

• The physician expert also marked 15% of these cases as ambiguous, noting that the correct DRG was unclear based on DRG guidelines, for example due to multiple potential principal diagnoses. Collectively, these results underscore the inherent complexity of DRG assignment. Interestingly, the identification of potentially incorrect DRG labels highlights a promising potential use case for our model in detecting medical coding errors.

4. Question about reward determination

We detailed our rule-based reward modeling in Sec A.2 (lines 460–468). For the accuracy reward, we used exact string matching. This stringent rule-based approach follows the standard practice in recent RLVR work, beginning with DeepSeek-R1. Such minimalism is key to mitigating the well-known issue of reward hacking in RL and forms a cornerstone of RLVR methodology. In early experiments, we explored more relaxed reward regimens (e.g., partial string matching). However, consistent with prior work, we observed frequent reward hacking—for example, the model would generate incorrect but verbose answers to game the reward function through partial matches. Finally, we wish to clarify that while we used string match, we assigned different scores for full versus partial DRG matches (e.g., correct principal diagnosis only or CC/MCC status only), thereby providing denser and more informative reward signals.

5. Novelty about dynamic resampling

We emphasize that this work is not about introducing yet another SFT/RLVR trick. Rather, our goal is to understand how SFT and RLVR influence LLM learning in OOD tasks. To our knowledge, we are among the first to highlight several practical insights, particularly the training cost analysis in Fig. 7C, which we hope will be valuable to the community.

6. Question about quality of SFT dataset constructed by Qwen2.5-7B

We are happy to clarify this important detail. The SFT dataset contains two parts: (1) the true DRG label from MIMIC, which is always correct, and (2) the CoT reasoning trace. For the CoT trace, we differentiate between its factuality (e.g., whether a condition is CC vs. MCC) and its cognitive pattern (the reasoning logic). We do not expect the initial CoT traces to be factually perfect, as the Qwen base model lacks DRG-specific knowledge. Instead, it is crucial that the cognitive pattern is correct. Through prompt engineering (Sec. H), we introduce diverse and valid cognitive patterns for the model to learn during SFT (lines 162–165). This provides a sound structure for the RLVR phase to build upon and learn factuality.

7. Performance improvement

Compared to prior classification-based methods, DRG-Sapphire offers the key advantage of generating high-quality explanations for DRG assignments (Fig 4), while also achieving SOTA performance. In the medical domain, such interpretability is essential for trust and transparency. As emphasized by physician leaders (Sec D.1), these interpretable rationales are foundational for real-world deployment, enabling actionable clinical insights.

Dear Reviewer,

With the discussion period concluding soon, we hope our rebuttal has addressed your concerns, including the one about the source of performance improvement (Q1).

Please let us know if you have any remaining comments; we are happy to elaborate further. We deeply appreciate the effort you have dedicated to reviewing our work.