Aligning Multimodal Models for Clinical Reasoning using Rule-based Rewards

1. GPT-4 is used to generate the conversations, and a rigorous evaluation is done to ensure no hallucinations are introduced. However, the authors provide no details on the specific evaluation methods used to ensure this, which limits the reproducibility of the work. Additionally, this lack of transparency undermines the objective of reducing hallucinations with reinforcement learning, as the GPT-4-generated dataset could still contain incorrect statements.
2. The essential details on clinical decision making pathways, dataset generation, and hallucination evaluations are either missing or located in the appendix rather than the main text, making it challenging to fully understand the methodology without the appendix. Since these are important in the work, the absence from the main text disrupts the flow and limits the study’s context and reproducibility. Including these details in the main body, or at least providing a summary with clear references to the appendix for further depth, would make the paper more accessible and cohesive.
3. The presentation of the work needs improvement, especially in organization and clarity to be publishable quality. Currently, the layout requires readers to frequently go back and forth, as text descriptions are located far from the tables and figures that they describe. This disrupts the flow and makes it challenging to follow the paper. For instance, Figure 3 is introduced in Section 2 before Figure 2, which creates confusion in the narrative structure. Additionally, abbreviations are often used without prior definition; for example, Table 2 and Table 3 feature abbreviated column headings like "CQ-R" and "Hcc," but their meanings are not clarified until much later in the text or, in some cases, are only defined in the Appendix. Including these definitions upfront or alongside the tables would enhance readability. Also, typos, such as "ruke" instead of "rule" on line 144 needs to be corrected.
4. The paper lacks experiments specifically designed to address the issue of misalignment between image and text and to demonstrate how Dr-LLaVA mitigates this problem, despite this being a primary motivation for using RL over SFT. However, without empirical evidence or ablation study showing that RL successfully reduces misalignment, the paper's contribution is weakened.
5. The paper lacks a comprehensive qualitative evaluation, presenting only a single qualitative example set in Figure 4. This limited approach does not provide enough insight into the model's practical application and performance across varied cases. Conducting a broader qualitative evaluation, ideally involving human assessments by clinicians, would allow for a more in-depth assessment of the model's reliability and relevance in real-world clinical settings.
6. The paper lacks a section on limitations, which is essential for providing a balanced view of the work's contributions and areas for future improvement.
7. The paper is missing an ethics statement, which is essential in such a research involving patient information. An ethics statement provides transparency about how ethical considerations were addressed, including data handling, privacy safeguards, and compliance with relevant regulations.

1. Limited Scalability: The application area explored in this paper is narrow in scope (bone marrow pathology classification), and it seems that in order to expand to other medical applications, a lot of manual effort is necessary. In particular, (1) a physician will have to manually generate a set clinical reasoning rules in order to extend this approach to other medical application areas and (2) the rule-based reward model will need a new set of manually-constructed keywords (as in Table B.5). This weakness limits the widespread usability of this approach.

2. Additional Methodology Details: Some important details with respect to the construction of the dataset and the rule-based rewards are missing, as detailed below:

   a. Multi-Turn Conversations: Each conversation in the synthesized conversational dataset consists of five question-answer pairs (Line 135). Why do all conversations include exactly five QA pairs, when it’s possible that the conversation could end in fewer turns if a leaf node of the decision tree is reached early (e.g. low quality image)?

   b. Dataset details: No details with respect to the composition of the synthesized dataset are provided. How large is the dataset? What is the class distribution with respect to the diagnoses? Are there sufficient samples for each of the possible decision pathways in the clinical reasoning decision tree?

   c. Keyword Matching: The authors use a manually pre-defined list of keywords in order to compute rewards and accuracy metrics. How accurate is the process of keyword matching, and how often do false positives / negatives arise?

3. Need for finer-grained evaluations: The performance metrics in Table 1 are aggregate metrics. How well does the proposed approach perform in comparison to baselines across each diagnosis and each possible decision pathway?

Small things:

• Figure 1 needs to be bigger. The conversation text cannot be read unless I zoom in alot.
• Typo "rule": line 144 our conversational diagnostic system leverages ruke-based representations (Fig. 3) - should be "rule-based"

• One of my biggest concerns about this paper is that it didn't show the potential of such an approach in medical generative VLMs. To be specific, the modeling can be replaced with a discriminative classification model since the responses from the MVLM are "pseudo" free-text responses, and they can be replaced with text templates + classification labels. So is a large generative VLM (7B parameters) really needed to solve the proposed task?