MedRAX: Medical Reasoning Agent for Chest X-ray

We would like to thank the reviewer for the insightful comments. We have worked hard to answer their concerns and think the suggestions have helped improve the clarity of our work.

Key Methodoloy. The full details of Algorithm 1 are provided in response to Reviewer KTkS. Additionally, an anonymous GitHub repo is prepared at https://github.com/syaro1383/medrax

Prior Work. We have added M4CXR and AgentClinic to the prior work section of the revised manuscript.

Need for ChestAgentBench. The justification for making a new benchmark is provided in response to Reviewer DGwp.

Further Evals. Thank you for suggesting an expanded evaluation with more benchmarks. We've conducted additional benchmarking to compare MedRAX with state-of-the-art models on two benchmarks. Model performances were obtained from Park et al. (2024)

1. MIMIC-CXR Radiology Report Generation

This benchmark evaluates single-image chest X-ray radiology report generation on the MIMIC-CXR test set, which includes 3,858 images. It assesses the clinical accuracy of generated reports by analyzing the presence or absence of 14 observations of medical conditions using CheXbert.

• mF1-14: Micro-averaged F1 score for all 14 CheXbert observation labels
• mF1-5: Micro-averaged F1 score for 5 key findings (cardiomegaly, edema, consolidation, atelectasis, pleural effusion)
• MF1-14: Macro-averaged F1 score for all 14 labels
• MF1-5: Macro-averaged F1 score for 5 key findings

Table A: Single-image performance on MIMIC-CXR test set

2. SLAKE VQA Benchmark

The SLAKE benchmark evaluates medical visual question answering using 114 chest X-ray test samples with close-ended questions in English. These questions typically focus on the presence or absence of abnormalities, anatomical identifications, and medical condition assessments.

• Accuracy: Percentage of exact matches between model predictions and ground truth answers
• Recall: Proportion of ground truth words present in the generated responses

Table B: Medical VQA performance

Other Comments. We have corrected "Llava-Med" to "LLaVA-Med" and added metric descriptions to all table captions.

Questions. 1-9. Discussed above.

10. Following the initial generation of questions by GPT-4o based on Eurorad cases, we performed an automated quality verification, also utilizing GPT-4o. Specifically, the automated check evaluated each generated question and answer set for structural consistency (e.g., six-choice question with one correct answer), explicit grounding in the provided clinical and radiological context, and clear verifiability of the correct answer from the original Eurorad source case material. Any questions failing these criteria were automatically identified and excluded. We have provided more details on this verification procedure in the revised manuscript.

11. During development, we observed that heavily favouring only tool use could lead to rigid or incorrect outputs if a tool failed or misinterpreted, while discouraging tool use missed opportunities for leveraging specialized analysis. Finding a balance, where the agent reasons first but uses tools judiciously to complement and tune its reasoning, appeared to yield better results empirically. A formal quantitative analysis exploring this balance was outside the scope of this paper's experiments but is noted as valuable future work.

12. Yes, the prompt for MedRAX is as follows:

"Answer this question correctly using the chain of thought reasoning and carefully evaluating choices. Solve using your own vision and reasoning and then use tools to complement your reasoning."

This prompt encourages the agent to combine its own reasoning ability and external tools to complement or refine initial assessments. The prompt worked better empirically during development. A formal quantitative comparison of prompting strategies is planned for future work.

We would like to thank the reviewer for the insightful feedback.

ChestAgentBench Representativeness. We appreciate the reviewer's question on benchmark representativeness. We designed ChestAgentBench to be both representative and capable of assessing advanced reasoning, addressing limitations in prior benchmarks:

• Addresses Evaluation Gaps in Complex Reasoning: Existing benchmarks often focus on simpler, single-step VQA or isolated tasks, which are insufficient for evaluating sophisticated AI agents designed for clinical practice. These simpler tasks do not capture the multi-step diagnostic reasoning, evidence integration, and tool use inherent in real-world radiology workflows. ChestAgentBench was explicitly created to fill this critical evaluation gap by presenting complex challenges that require agents to demonstrate deeper, sequential reasoning and integrated analytical capabilities, truly assessing their readiness for clinical application support.
• Ensures Broad Clinical Scope and Realistic Data Distribution: The benchmark's representativeness is supported by its diverse foundation and statistically verified distributions. Derived from 675 real clinical cases, it includes scenarios from various hospital settings and covers a wide spectrum of common and important chest X-ray findings.

Distribution by Department:

• Emergency Room (ER): 19.7%
• Intensive Care Unit (ICU): 4.9% (Note: The remaining cases originate from other hospital settings, including general wards.)

Frequency of Common Chest X-ray Findings:

• Mass: 26.3%
• Effusion: 24.6%
• Pleural Effusion: 21.5%
• Consolidation: 21.3%
• Nodule: 17.2%
• Calcification: 10.7%
• Pneumothorax: 7.6%
• Lymphadenopathy: 7.6%
• Pneumonia: 7.2%
• Emphysema: 7.1%
• Interstitial findings: 6.9%
• Bronchiectasis: 5.9%
• Atelectasis: 4.9%
• Fibrosis: 4.1%
• Edema: 3.9%
• Cavitation: 3.9%
• Fracture: 3.0%
• Tuberculosis: 2.6%
• Metastasis: 2.6%
• Cardiomegaly: 1.5%

This broad distribution across 53 anatomical areas, varied patient demographics, and numerous pathologies ensures agents are tested on a realistic range of clinical challenges.

• Grounded in Expert Cases and Assesses Integrated Clinical Competencies: The benchmark's questions are directly derived from and verifiable against detailed findings and expert discussions within 675 authentic, expert-curated clinical cases, ensuring clinical validity and grounding in real-world knowledge. Furthermore, ChestAgentBench systematically evaluates the integration of seven core clinical competencies (such as detection, localization, comparison, relationship analysis, and diagnosis) through complex question types. This design forces agents to demonstrate multifaceted reasoning akin to a clinician synthesizing diverse information, rather than just performing isolated tasks, thereby providing a comprehensive assessment of their true diagnostic reasoning abilities grounded in expert practice.

Benchmark Quality Check. We thank the reviewer for highlighting the need for clarification on our quality check procedure. Following the initial generation of questions by GPT-4o based on Eurorad cases, we performed an automated quality verification, also utilizing GPT-4o. Specifically, the automated check evaluated each generated question and answer set for structural consistency (e.g., six-choice question with one correct answer), explicit grounding in the provided clinical and radiological context, and clear verifiability of the correct answer from the original Eurorad source case material. Any questions failing these criteria were automatically identified and excluded. We have provided more details on this verification procedure in the revised manuscript.

We want to thank the reviewer for their thorough and thoughtful comments.

Statistical Significance. We appreciate the reviewer's point about statistical measures. All experiments were run deterministically (LLMs with temperature 0 and deterministic tools), thus eliminating variability in model outputs for identical inputs. Additionally, since MedRAX integrates pre-trained models without any further training or fine-tuning, there is no randomness from training procedures. This approach ensures full reproducibility and makes traditional confidence intervals less applicable in this context.

Failure Cases. For MedRAX failure cases, we have observed instances where the LLM becomes overconfident in its own reasoning and neglects to effectively utilize available tools, leading to incorrect conclusions that could have been corrected with proper tool integration. Regarding baseline failures, models like LLaVA-MED often struggle due to limited training data, resulting in poor generalization to diverse clinical scenarios. MedRAX mitigates this limitation by combining specialized tools with a general-purpose LLM that has broader knowledge and reasoning capabilities. We have included specific examples demonstrating these failure patterns in the revised manuscript.

Experiment Probing Spurious Correlations. We appreciate the reviewer's thoughtful suggestion on investigating spurious correlations within the pretrained models used by MedRAX. The concern about biased predictions is addressed through diverse tools that provide cross-validation capabilities. Specifically, MedRAX integrates grounding models that highlight disease regions, allowing the LLM to validate classification predictions against visual evidence from multiple sources. This multi-tool approach helps mitigate reliance on any single potentially biased component. We agree that the reviewer's suggested approach represents a valuable direction for our future work.

Related Work. We have added "RaDialog: A Large Vision-Language Model for Radiology Report Generation and Conversational Assistance" to the related work section.

Strengths and Weaknesses.

1. Thank you for raising this important question. MedRAX integrates into clinical workflows as an interactive radiologist copilot. Clinicians can ask questions about CXRs (e.g., for disease classification, localization, VQA) via the user-friendly interface to assist in diagnosis. Its flexible deployment options (local or cloud) and modular design address practical IT and privacy barriers to adoption.

2. Discussed above.

3. Table 1 serves as a partial ablation, demonstrating the performance differential between standalone VQA tools (e.g., CheXagent), general-purpose models (GPT-4o), and our integrated MedRAX framework. However, we acknowledge that a more granular ablation study selectively disabling specific tools would provide deeper insights into which tools are most critical for different categories. We will include this analysis in the revised manuscript.

4. The final response is processed through GPT-4o. Our framework is designed to minimize hallucinations by providing comprehensive context from multiple specialized tools, creating an information-rich environment for the LLM. This multi-tool approach helps ground the model's responses in concrete findings from validated medical AI systems rather than relying solely on the LLM's internal knowledge.

5. MedRAX utilizes LangChain to manage the reasoning process dynamically. The Reason(state,M) function relies on: (1) an initial system prompt defining the agent's role, available tools with descriptions, and required output structure, and (2) the continuously updated conversation history (memory M) containing past thoughts, actions, and tool results. The agent framework routes execution based on the LLM's output: structured tool calls trigger tool execution, while a final response concludes the process. We have included the agent system prompt and tool descriptions in the revised manuscript.

6. MedRAX is flexible: while our evaluation used GPT-4o vision for performance, it can operate without vision LLMs by relying on specialized visual tools. The benchmark evaluation cost using GPT-4o vision involved 2,500 questions with on average 1.85 images per question (~512x512px, ~255 tokens per image) at GPT-4o's input rate of $3.75/1M tokens.

7. A breakdown of core methodology is provided in response to Reviewer KTkS

8. Regarding patient data privacy, MedRAX's modular architecture supports locally deployed LLMs to prevent sensitive medical data transmission to external servers. Acknowledging reviewer guidelines, it also supports Azure OpenAI service with appropriate opt-out configurations. This flexibility allows institutions to implement MedRAX compliantly via on-premises deployment or properly configured cloud services, meeting their specific privacy requirements.

We thank the reviewer for their insightful feedback and appreciate their recognition of the importance of this problem in the healthcare domain.

Algorithm 1 Core Methodology. Algorithm 1 outlines the iterative reasoning process of the MedRAX agent in the ReAct loop, as follows:

1. Observe(Q, I, M) / State Preparation: This step initializes the reasoning cycle by gathering the necessary context. It aggregates the user's query (Q), any associated input images (I), and the entire history maintained in the agent's memory (M), which includes previous interactions, tool outputs, and reasoning steps. This consolidated state is fed into the LLM.

2. Reason(state, M) -> thoughts (LLM Reasoning): The LLM analyzes the current state (query, images, memory) and available tools (T) to generate internal 'thoughts'. These thoughts form a plan deciding the next action: generate a final response, ask the user for input, or select one or more tools.

3. RequiresUserInput(thoughts) (Implicit Decision): This condition is implicitly evaluated by the LLM during the Reason step. If the LLM's 'thoughts' indicate ambiguity or insufficient information that cannot be resolved by tools, it will opt to generate a clarifying question for the user instead of proceeding with a tool call or final answer.

4. GenerateUserPrompt(thoughts, M) (Implicit Action): If RequiresUserInput is effectively true, the LLM's generated output is the prompt for the user. It formulates a natural language question based on its 'thoughts' and the context in memory (M) to elicit the needed information.

5. SelectTool(thoughts, T, M) -> tool(s) (LLM Tool Selection): When the LLM's 'thoughts' indicate a need for specific capabilities (e.g., classification, segmentation), it selects the most appropriate tool or multiple tools from the available set (T). This selection is based on the LLM matching its reasoning to the predefined descriptions and capabilities of each tool. For each selected tool, the LLM also formulates the necessary input arguments in a structured format (e.g., JSON).

6. Execute(tool(s)) -> result(s) (Tool Execution): The agent executes the selected tool function(s). This involves invoking each tool with the structured arguments prepared by the LLM in the SelectTool step. The output(s) of these execution(s) are captured as result(s).

7. M ← M ∪ (thoughts, tool(s), result(s)) (Memory Update): Following successful tool execution(s), the agent's memory (M) is updated. This crucial step logs the LLM's 'thoughts' that led to the tool call(s), the identity of the tool(s) used, and the result(s) obtained. This ensures that outputs from tools become part of the context for subsequent reasoning cycles.

8. CanGenerateResponse(thoughts) / GenerateResponse(thoughts, M) (Response Generation): Following Reason, this checks if the LLM's 'thoughts' plan a tool call. If not (CanGenerateResponse effectively true), the LLM synthesizes the final natural language answer (GenerateResponse), drawing upon its concluding 'thoughts' and information in memory (M). If 'thoughts' indicate a tool is needed, this step is bypassed, and the agent proceeds to SelectTool and Execute.

ChestAgentBench Statistics. Thank you for the thoughtful comment. The distribution of common findings and department origins of cases are provided in response to Reviewer DGwp. We have revised Figure 3d to incorporate these detailed statistics.

ICML Relevance. We submitted MedRAX under the "Application-Driven Machine Learning" track highlighting areas like healthcare. Our work directly addresses significant challenges within this critical domain, and we chose ICML because we believe high-quality, impactful healthcare AI research, while perhaps historically under-represented, is important for the ML community.

CheXbench Image-Text Reasoning. This benchmark assesses fine-grained visual reasoning, requiring models to differentiate between options with subtle but critical distinctions (e.g., 'left' vs. 'right' pleural effusion). We observed poor performance across all evaluated models on this task. This suggests the challenge lies in the inherent difficulty of fine-grained radiological interpretation rather than the benchmark dataset itself (derived from the widely-used OpenI).

Clinical Relevance. Thank you for raising this important question. MedRAX integrates into clinical workflows as an interactive radiologist copilot. Clinicians can ask questions about CXRs (e.g., for disease classification, localization, VQA) via the user-friendly interface to assist in diagnosis. Its flexible deployment options (local or cloud) and modular design address practical IT and privacy barriers to adoption.

Figure 2. We have replaced Figure 2 with a clearer case showing distinct unilateral findings.