LLM-CXR: Instruction-Finetuned LLM for CXR Image Understanding and Generation

Thank you for your thorough reading and insightful comments. We address your concerns below.

W1. It is overclaimed to introduce bidirectional reconstruction tasks in pre-training as a novel approach.
We would like to more clearly delineate the differences between Figure 1’s (a) and (c). The image encoder and decoder in (a) have already been pre-trained with the paired vision-language dataset; and the encoder outputs pseudo-text, which the decoder in turn receives as input. In this process, the language model does not directly engage with visual information as it takes pseudo-text as input and outputs pseudo-text (this is why the LLM can remain frozen in methods like BLIP-2).

In our approach, visual information is passed directly to the LLM without passing through a vision-language pretrained network. The encoder simply performs tokenization of the image. LLM thus receives direct visual information, and vision-language alignment occurs within its own parameters on top of its pretrained language features. We believe that, as shown by our results, this is a promising avenue of multimodal LLM research that deserves attention separate from the currently dominating method of using vision adapter networks to implement multimodal LLMs.

W2. Motivation not clear. Explanation of downstream tasks?
Direction to AGI development will likely involve integration of a central LLM with various expert models operating as modules; and we envision that communication between them will heavily utilize natural language. From this perspective, expert models should have language-based communication skills. As for a specific downstream example task, models like LLM-CXR can be instructed to find patients with specific findings described by text. For example, if the inclusion criteria for a clinical trial calls for ‘patients with severe bilateral pleural effusions on CXR’, the model can be used as an agent to assist this process.

W3. Metrics used are uncommon.
We respectfully disagree on this point. We use AUROC/F1-scores based on label extraction to assess the accuracy of report generation because it is most commonly used in this area of research as shown in [1, 2 3]. Jaccard similarity index was also employed because AUROC/F1-scores favor detection of positive classes whereas the Jaccard similarity index treats both positive and negative classes equally; this is important as hallucinations are a well-known problem in multimodal language models, and negative classifications are of equal importance to positive classifications in analyzing medical images. Although we find them much less informative for medical reports, we have also added NLG metrics (e.g. BLEU, ROUGE, CIDEr).

For the image generation task, FID assesses quality of generated images, and AUROC/F1 using a pretrained CXR image classifier assesses whether generated images are in line with the given text inputs used to generate them. The latter evaluation is more important in our case as it assesses image-text alignment. To our best knowledge, there is no common or agreed upon way to evaluate the alignment between generated images from a model and the given input prompts used to generate them. We believe that this set of assessments, combined with the provided qualitative examples, showcases the capabilities of the models under investigation.

W4-1. Only LLM-based models are compared for report generation
Med-PaLM1/2 would also be a great target for comparison, but they are not publicly available yet. As per your suggestion, we include radiology report generation models that have the best reported performance as baselines for comparison. namely IFCC [3] & R2Gen [4]. These models showed weak performance compared to LLM-based models.

Table 1. CXR-to-report (part)

W4-2. IU X-ray dataset evaluation
Please refer to general response.

W5. Upsampling in section 3.1
We wanted to simulate a situation where UniXGen, which can only receive 512px images, receives 256px images (the resolution for input to LLM-CXR and XrayGPT). But we also report the outcomes for UniXGen using its native resolution 512x512 images in Table 1.

W6. How big should the image token space be?
$K_img$ is a value that is predefined as a design configuration for the VQ-GAN model architecture. We used the $K_img=1024$ as used in the original VQ-GAN paper for 256px images.

[1] Lee et al. "UniXGen: A Unified Vision-Language Model for Multi-View Chest X-ray Generation and Report Generation". arXiv 2023.
[2] Li et al. "Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation". CVPR 2023.
[3] Delbrouck et al. "Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards". EMNLP 2022.
[4] Chen et al. "Generating Radiology reports via Memory-driven Transformer". EMNLP 2020.

Dear reviewer MXmf,

We believe that we have addressed the concerns you have raised. Specifically,

1. We clarify the distinction from previous approaches in that encoders in previous works are vision-language pretrained and output pseudo-text an LLM can already understand, whereas in our approach the encoder serves to simply tokenize the image using a VQGAN encoder trained only on images and output image features that the LLM itself must learn.
2. We explain our envisioned downstream use cases.
3. We outline the detailed context of the employed metrics used to evaluate models in the paper and explain with citations that they are commonly used by other similar works as well.
4. We provide results to further experiments you suggested (i.e., comparison to non-LLM-based radiology report generation baselines and evaluation on the IU chest X-ray dataset).
5. We answer your questions about about image upsampling in section 3.1 and size of the image token space

As the deadline for the reviewer-author discussion phase is fast approaching (there is only a day left), we would like to extend a warm reminder and ask whether we have addressed your questions and concerns adequately. We would be happy to clear up any additional questions.

Best regards, Authors.

W1. Potential for catastrophic forgetting.

A. Our main goal was to leverage the pre-existing language capabilities of a large language model to tune it into an (instructionable in natural language) expert model that can describe images using text, generate images given an input text, and perform visual question-answering tasks. Because all three tasks and natural language instruction following ability require an understanding of language, we build on top of an LLM; as we simply sought to leverage the language capabilities of the LLM, it was not our goal to maintain all of the previous text-based knowledge contained in the model.

Our model is intended to be a form of expert module for future AGI that communicates in natural language, and therefore we believe that natural language skills, excluding task-related language skills, only require a basic ability to follow natural language instructions. While it would of course be beneficial to mitigate forgetting as much as possible in a model with a given number of parameters, we view this as future work that requires its own set of innovations.

W2. Lack of side-by-side comparison with models that use adapter networks that justifies the claim that our approach is better.

A. We do not mean to dismiss approaches using adapter networks for vision-language integration. Most concurrent vision-language research works use these adapter networks (e.g. XrayGPT and ELIXR) for integration of visual information to LLMs. On the other hand, there has been a surprising lack of investigation into other novel avenues (such as our method of utilizing instruction-finetuning for vision-language alignment). Moreover, the reviewer is kindly reminded that the adapter networks cannot generate images without connecting to an image generation model such as Stable Diffusion.

In the medical domain especially, we believe that our method of instruction-tuning to achieve alignment of image-text information may hold more promise as pixel-level detailed visual information must be adequately described in the text. This is in contrast to natural images where, as shown in works such as BLIP-2, the use of adapter networks to pass information to an LLM may be enough to yield sufficiently performant models as broader features confer the semantic meaning. This was an important motivation behind our work, and we wanted to convey this need for investigation of alternative approaches to image-text alignment rather than dismissing the adapter network approaches as a whole.

Our evaluation experiments comparing results for LLM-CXR to those of XrayGPT and ELIXR showcase the viability of our approach against these adapter network approaches, as they both utilize larger models but show similar or lower performance in image understanding.

W3. VQ-GAN may produce images that are not faithful to the original image.

A. As pointed out, VQ-GAN has limitations in reconstruction due to its characteristics (e.g. vector quantization). There are limitations in terms of information preservation, so VQ-GAN uses LPIPS-like loss to improve the preservation of perceptual information. An important informational aspect of CXRs is the presence or absence of certain lesions. Therefore, we introduced a clinical information-preserving loss into VQ-GAN to preserve lesion information as much as possible in the VQ-GAN structure.

Considering that our ultimate goal is not to ensure that LLM accurately restores a given image but to understand images and generate them based on text, it was more in line with our objective to aim for latent tokens that contain semantic information rather than details for exact reconstruction. In other words, in our work, we used VQ-GAN's reconstruction loss as a means of encoding and decoding an image for textual understanding rather than for complete reconstruction.

As a side note, we would like to point out that an adapter network with Stable Diffusion image generator cannot reconstruct original input images as closely as our VQ-based method.

W4. More diverse metrics e.g. human evaluations of quality and diagnostic performance evaluations against human radiologists.

A. The comprehensive evaluation by the end-users (ie, clinicians) would be the eventual target, but owing to time constraints, we cannot conduct these experiments yet. While we will try to conduct such experiments, we believe that the utilized metrics can guide further model development.

W5. Discrepancies in AUC/FID with that reported in the RoentGen paper.

A. The Difference is most likely because we used a different test set. To compare the different models from different works, we used a single test set (that is presumably different from the test set used in the RoentGen paper) across all experiments. However, we did use the model weights and inference code provided by the authors.

W6. Minor errors in writing.

A. We have fixed and improved the writing to be clearer to the reader.

Thank you for your valuable review and positive feedback. We address your concerns below.

Q1. How scalable is the tokenization and fine-tuning process for larger medical images (e.g. 3D/MR scans)?
Although we have not conducted experiments for 3D data, we presume that 3D scans will probably need a similar tokenization strategy adapted to be more appropriate for higher-dimensional data. We think that certain ideas from our approach can be extended to 3D images as well; for example, the TXV-loss we employed to preserve clinical information during tokenization would also be applicable for 3D medical images.

Q2. How interpretable are the model's decisions?
We agree that interpretability is a crucial consideration in clinical models. We believe that the natural language-based outputs of LLM-CXR intrinsically offer a higher level of interpretability than traditional computer vision models that output classification results. LLM-CXR is also able to generate images given some input text like ‘severe left-sided pneumonia’; this offers a window into the inner workings of the model as we can ask it to show images that correspond to a certain concept described using text.

As an additional note, compared to approaches in which the LLM receives images in the form of pseudo-text produced by an adapter network (e.g. image encoding networks), we believe that our method of directly processing images and treating them equal to natural language is more promising for interpretability as if the semantics of the image are delivered as pseudo-text, interpretation of the LLM’s understanding of images themselves is bound to be limited.

Q3. Comparison with RadFM
Thank you for bringing RadFM model to our attention. This work shares many of our motivations, and we were excited to see that they have open-sourced their work. We include RadFM in our assessments for CXR-to-report generation and CXR-VQA:

Table 1. CXR-to-report generation (part)

Table 3. CXR-VQA (part)

Note that RadFM is based on the LLaMa-13B model and LLM-CXR is based on the dolly-3B model, so there is about a 4-fold difference in parameter count; but LLM-CXR's performance is comparable if not better.

Overall, I think the contribution is enough as a conference paper. I still suggest acceptance.

Thank you for the recognition of our contributions and for the optimistic suggestions based on domain expertise.

After reading your review and the RadFM paper, we have also begun work on 3D applications of this methodology. Our past experience informs us that methods using 3D volumes directly is vulnerable to the 'curse of dimensionality' (data requirements increase exponentially with increase in dimensionality); and so we are considering a "2.5D" approach in which a CT or MRI volume is treated as a sequence of 2D slices, each of which can be tokenized using methodologies developed for 2D images. While results still remain to be seen, we will, as per your suggestion, add discussions on this topic in the manuscript as it may be of interest to more readers.

As for interpretability, we are also developing methods to offer more insight into model outputs such as visualizing the parts of the image that are most influential in generating certain natural language phrases (mimicking how radiologists often mark the important parts of a scan with arrows or circles). Since LLM-CXR effectively translates information between images and text within a single neural network (the LLM) and not in disparate adapter modules, we believe that making this translation process explicit will be most feasible for our model particularly. As you mentioned, because evidence-based care and clear rationales for decision-making are crucial in healthcare, we see this as important follow-up work.

Thank you again for your encouraging assessment. We were genuinely excited with this discussion. Please let us know if you have any further suggestions or ideas.

First and foremost, we would like to thank you for catching an error in our data processing process (use of the OpenAI API with MIMIC-CXR data) before publication. We hope to address your other concerns below.

W1/MethodologyConcerns-Q1. The approach of multiple instruction-tuning tasks is not particularly novel. How does our methodology differentiate itself from other existing large-scale models that employ a similar strategy?

A. The main distinction of our work from other work using instruction-tuning to build medical LLMs is that we use instruction-tuning to teach an LLM to both understand and generate images. Most instruction-tuning methodologies are geared towards changing a next word-predicting model to an instruction-following model using instruction-answer pairs (e.g. GPT3.5, PaLM) rather than promoting the integration of visual information to language-based models. There is previous work on instruction-tuning with visual information (e.g. LLaVA, InstructBLIP) which are most similar to our work; but these works employ adapter networks CLIP and BLIP respectively for image-text alignment and employ instruction-tuning techniques simply as a means of passing visual information to an LLM. We on the other hand aimed to use instruction-tuning itself as a method of aligning images and text. You may have noticed that no separate image-text aligning techniques (e.g. CLIP, BLIP, ALIGN) are used at all. In more detail, image encoders in approaches such as LLaVA and InstructBLIP have already been pre-trained with the paired vision-language dataset, and the encoder already outputs pseudo-text; the decoder in turn receives pseudo-text as input. In this process, the language model does not directly deal with images as it takes pseudo-text as input and outputs pseudo-text. However, in our approach visual information is passed directly to the LLM, and the LLM itself learns visual features and vision-language alignment on top of its previous knowledge of the language.

Another distinction is that our instruction-tuning method also enables the generation of images. This confers a few benefits: 1) we can ask the model to directly visualize a certain concept encapsulated in text (e.g., pneumonia); 2) this bidirectionality in generation turns out to be vital for aligning images and text without any adapter networks; 3) the visual instruction tuning method of LLaVA can teach the model image features but only the features that are close to the semantic meaning of the texts and not the fine-grain visual information in images that are not completely described by text.

We hope that the distinction of our work from previous ones is now clearer.

W2/ComparativeAnalysis-Q1. Comparison with XrayGPT and UniXGen is not fair because they are not trained on the same instruction-tuning dataset.

A. XrayGPT and UniXGen are trained on the same dataset as ours (MIMIC-CXR). However, we have used the dataset to train in an instruction-tuning manner.

Regarding your comment on fair comparison, we respectfully disagree with the reviewer since we started our research from the viewpoint that using an enriched instruction-tuning set itself is a distinction from previous works such as XrayGPT and UniXGen.

W3/Comparative-AnalysisQ2/3. Evaluation is not comprehensive. Compare with contemporary radiology report generation methodologies or Med-PaLM1/2.

A. The motivation for our work was to teach a large language model to understand and generate chest X-ray images so that it can serve as a more general-purpose interface for medical images than report generation models. This is why we felt most meaningful comparisons were against similar large language models that can understand CXR images (XrayGPT, ELIXR).

As per your suggestion, we have included radiology report generation models with best reported performance as additional baselines for comparison. The results confirm that our models are superior or comparable to the dedicated contemporary report generation models (Table 1, 2 in the revised manuscript).

Table 1. CXR-to-report (part)

As you mentioned, Med-PaLM1/2 would also be a great target for comparison, but they are not publicly available yet. Also, Mingjie Li et al. did not release pretrained checkpoints, and the training code does not run despite our efforts to debug the authors’ code.

W4. Evaluation metrics should also be more comprehensive e.g. results for all 14 lesions.

A. Upon reviewing the comments, we agree that a more comprehensive evaluation with all 14 lesions is appropriate and provides updated results using the 14 lesions above in the revised manuscript. This method of evaluation does not affect the overall results.

W5. Writing needs to be improved.

A. We have fixed and improved the writing to be clearer to the reader.

Thank you for helping us improve our work. We appreciate the positive feedback and are glad that our revisions have led to tangible improvements.

In our effort to address the data usage license concerns, we have re-done from scratch the parts of our work that were affected by the MIMIC data usage issue; namely, synthetic question-and-answer generation was re-done using a model that can be run locally (LLaMa2-13B) and therefore without any usage license issues. All evaluation experiments were also re-conducted using the newly trained model. While we anticipate that OpenAI API has already deleted the data (as OpenAI policy stipulates data retention for 30 days only and that the data is not used to train their models nor shared with outside parties [https://openai.com/enterprise-privacy]), we will actively continue to address the data license issues to the best of our ability.

Thank you again for catching this error in the review process.

Thank you for your insightful review. We are excited that you agree on the potential of some parts of our approach. Below, we would like to address your concerns and, where applicable, augment our study with further experiments per your comments.

W1. Experiments are done on just one dataset.

A. Please refer to the general response for added evaluation results on an out-of-domain dataset (IU chest X-ray).

W2. Visualization of alignment in feature space.

A. While it is true that a direct visualization image and text features in the embedding space would provide the most obvious evidence for image-text alignment, we found a dearth of research on the creation and interpretation of embedding spaces in GPT-based multimodal models. In fact, a recent paper [1] showed that the integration of visual and textual information in text-pretrained multimodal transformer models occurs in the neurons themselves rather than in the feature space. We think that more work on the interpretability of feature spaces of multimodal LLMs is necessary before we are able to apply such techniques to gauge image-text understanding in models. Thus instead, we demonstrate achievement of image-text alignment in the evaluation experiments and ablation studies: training with the VQA task leads to improvements in both the image-based text generation task and the text-based image generation task even though the VQA task does not directly employ these tasks, serving as evidence of improved image-text understanding in the model.

Q1. Clarification of specific application scenarios.

A. We believe that a model such as ours can be a versatile AI assistant for interacting with large volumes of medical data. For instance, it can be used to find CXR images with specific findings described using natural language from a large dataset of CXRs. This is different from using a traditional image classification network as those networks can only distinguish between predefined labels. This is also different from radiology report generation models that simply take in an image and produce a text output as models that can understand language can provide answers to more contextualized queries, which is more similar to the human radiologist’s workflow (radiologists do not look at the images alone; they understand the clinical context of the patient and the study to produce reports based on them). With confirmation of sufficient accuracy, a model like this can potentially be used to select patients for a clinical trial based on a set inclusion/exclusion criteria or be used for routine, relatively simple clinical tasks such as clearing the absence of certain lesions prior to a procedure.

Q2. L2 reconstruction loss may not be enough to reduce the loss of clinical information.

A. The preservation of information in the process of tokenizing images into a token space shared by both text and image is definitely an important, major challenge that has to be overcome to develop an accurate model using our methodology. As you have hinted, our employment of the L2 reconstruction loss, while helpful, does not address this problem entirely and there is room for improvement; however, we would like to delve further into why our VQ-GAN training method is able to take a solid step towards that direction:

The original VQ-GAN improves upon the VQ-VAE’s bottleneck architecture and a quantized latent space with LPIPS and adversarial losses. These enhancements improved reconstruction quality, especially with regards to reducing blurry outputs characteristic of models trained with pixel space L1 loss. However, because LPIPS is a loss based on a network trained on natural images (namely Inception-V3), we hypothesized that it is not adequate for extracting features of CXR images and could be improved with a loss tailored for CXR images. Hence, the TXV-loss we use is a variant of the LPIPS loss but calculated using the features of a neural network trained on CXR images (torchxrayvision library). As our ablation study shows that the use of the TXV-loss does increase the performance of the final model, we believe that our hypothesis has at least some merit.

References

[1] Schwettmann, Sarah, et al. "Multimodal Neurons in Pretrained Text-Only Transformers." ICCV. 2023.

We provide newly available CXR-to-report task results to another evaluation experiment suggested by reviewers: evaluation on the IU Chest X-ray dataset to gauge performance on out-of-distribution images from an external dataset.

The results are similar to that for in-domain data (MIMIC-CXR images): LLM-CXR shows strongest performance, followed by RadFM and then XrayGPT.