CoVT-CXR: Building Chain of Visual Thought for Interpretable Chest X-Ray Diagnosis

• The authors reference an Appendix many times, but there is no Appendix.
• The baselines in the evaluation are not medical-specific models. There have been many models proposed for radiology report generation which have been trained on medical images/text, however the authors opt to use natural image/text baselines, which is an unfair comparison. For example, instead of comparing against Llava-med or llava-rad, they compare against llava; instead of comparing against med-gemini, they compare against gemini. Similarly, they exclude models specifically trained for radiology report generation, such as maira-1 or any of the numerous other proposed radiology report generators. As a result, I do not find the empirical results convincing.
• The paper is difficult to read and follow. In particular, I have a difficult time following how the dataset was created and the contents of the dataset. What are the 112 semantic classes? How were these chosen? What is a “intermediate text description”? Is there any standardization on what an “intermediate text description” comprises? The murkiness of this dataset development description makes it difficult to understand how valuable the dataset will be to others or how one would go about extending the dataset.
• Additionally, the total size of the dataset is relatively small (6k cases), and the test dataset is very small (400 cases). While I understand this is because of the manual effort required to create a chain-of-thought dataset, it does limit its upper bound on performance. I am doubtful of how well this dataset size could be increased in the future, as the chain-of-thought labelling process is not very well described. Further, if the authors could discuss how this dataset could be used in tandem with other non-chain-of-thought datasets, that would be useful.
• In the Data Curation: Motivation section and later in Section 3.1, the authors authoritatively describe a process that doctors (the authors imply all doctors) use to read chest x-rays. There are no references in this section, aside from a footnote pointing to “geekymedics” and “radiologymasterclass” websites. I find this support weak; either the language in this section should be made less strong, or additional support is needed as to how the authors determined this CXR reading process.
• Limitations and Future Work should be discussed.

In summary, I find the dataset and approach interesting and useful for the community. While the baseline comparisons leave me unsure of how well the proposed method actually works and the writing needs revision for clarity, I still find this dataset and general approach to be a valuable contribution and lean toward acceptance.

1. More model evaluation: The use of LLaVA-1.5 is noted, but LLaVA-1.6 offers significant improvements that could enhance model performance. I recommend upgrading to this latest version for fine-tuning. Additionally, exploring models like Flamingo may further improve results.
2. Evaluation Metrics: The evaluation metrics used—BLEU, ROUGE, METEOR, and CIDEr—are standard for many natural language processing tasks; however, they are not ideally suited for evaluating medical Visual Question Answering (VQA) or the factual accuracy in medical reports. These metrics often fail to capture the semantic accuracy required in medical contexts. It is critical to implement more specialized metrics that can assess the medical factual accuracy and the clinical relevance of the answers provided by the system.

The dataset appears through, designed by clinical experts, and cast as well studied ML tasks such as segmentation and question answering.

The evaluation metrics used are generally for machine translation and don't capture how useful the reports are, only that they contain similar words. A solution would be a human evaluation for some set of tasks with quantitative feedback.

I have a number of questions and comments that relate to the quality of this work.

• Whilst the CoVT-CXR dataset does sound useful, it is not possible for me to review any example instances of it. The authors explain all code and the dataset will be released upon acceptance. However, as the dataset is framed as a core contribution, it would be more useful If reviewers could review both an anonymous codebase of their experiments AND at least some instance of the proposed CoVT-CXR dataset.
• This approach appears to be inference-heavy. Visual and textual information may need to be passed to the model upwards of 12 times to perform inference for a statement $\mathcal{S}$ (as per the abstract - Line 26). Could the authors comment on this? If this is a limitation, it should probably be acknowledged in the main script.
• If the dataset required 32 medical professionals and a bespoke labelling platform, it is unlikely to be a scalable data labelling approach. What it I wanted to do this for brain MRIs, for instance? Unless the authors have a different view, this should also probably be acknowledged as a limitation.
• How did you deal with incomplete or incorrect ground-truth radiology reports in MIMIC-CXR?
• How were the 112 semantic classes defined?
• Segmentation strategy is quite unclear from lines 271-275. Are you using pretrained segmentation models with prompt engineering to ‘enable zero-short capability for medical images’, or are you fine-tuning models with CXRs, or something else? → if using a generic segmentation model, how do you account for poor segmentation results?
• How exactly does the framework autonomously identify the report all relevant targets within the images (line 325)? If inference is performed multiple times, can we not end up with the same statement being produced more than once? An explanation is not clear in the manuscript.
• The authors do not compare their framework to Llava-Med, MAIRA-1/2, CheXAgent, etc. I.e., there are no comparisons with VLMs specialized for medical tasks/CXRs specifically.
• Only lexical metrics are used for eval. These do not appropriately capture meaning as they treat all terms equally (please see https://arxiv.org/pdf/2406.04449 for more detail) — what about RGER, Chexbert, RadFact, etc? These are well known clinical evaluation metrics that are reported in most radiology report generation research.
• Unfortunately, for both ROUGE and meteor metrics (reported lexical metrics), results are somewhat underwhelming. For instance Phi-3V outperforms the proposed CoVT when finetuned. LLaVA-1.5 (not LLaVa-med) matches or outperforms CoVT for both VQA and report generation when fine-tuned.
• Line 490: Authors claim that incorporating more intermediate steps significantly improves the average sampling probability of ground truth tokens, which enhances the likelihood of generating accurate reports. Could it not simply be the case that incorporating more intermediate steps simply exposes the model to more “ground-truth” tokens, and is thus not a function of improved reasoning per se, but rather something more like oversampling these tokens and therefore generating them more frequently? It would be useful to have clinical metrics rather than purely lexical ones to help make this claim.