Large-scale and Fine-grained Vision-language Pre-training for Enhanced CT Image Understanding

Q4 Model's performance across different conditions and data distributions. More analysis on long-tailed disease.

A4 Thank you for this insightful question regarding the long-tail distribution of diseases and its impact on model performance. This is an excellent suggestion, and we appreciate the opportunity to expand on this important aspect. In fact, we did not observe a clear relationship between the model's performance for a disease and its frequency of occurrence. More specifically, we calculate the Pearson correlation coefficients between the diagnostic performance (i.e., AUC) and the positive case ratio across all abnormalities on the MedVL-CT69K, CT-RATE, and RadChest-CT datasets. Our method yields results of 0.08, -0.40, and -0.09 and CT-CLIP shows values of 0.06, -0.46, and -0.12. Actually, we observe that the models typically excel at detecting diffuse abnormalities, such as liver cirrhosis and pericardial effusion, while struggling to identify subtle lesions, such as liver cyst and lung nodule. For example, on the CT-RATE benchmark, the positive sample proportions for pericardial effusion and lung nodule are 2.7% and 45.0%, respectively. Despite the much higher positive sample ratio for lung nodule, our method achieves a higher AUC of 0.85 for pericardial effusion diagnosis compared to 0.63 for lung nodule. Similarly, CT-CLIP achieves AUC values of 0.75 and 0.57 for pericardial effusion and lung nodule diagnosis, respectively. We speculate that identifying these subtle lesions relies more on low-level visual features, whereas our current method only leverages the features from the last ViT block. Leveraging hierarchical visual features during training may improve the model's ability in diagnosing these localized lesions.

Q5.1 Was data selection applied during dataset collection to include more abnormal patients?

A5.1 Thank you for your insightful comment. The MedVL-CT69K dataset used in our study was collected continuously over four years, without any specific selection to include more abnormal patients.

Q5.2 Will the scaling law related to data distribution?

A5.2 Yes, the scaling law is relevant to data distribution. Specifically, we analysis the scaling law of our method on the CT-RATE dataset under the same experimental settings as MedVL-CT69K. On these two datasets, we observe different scaling laws: $AUC=0.4557D^{0.0542}$ for CT-RATE and $AUC=0.427D^{0.0598}$ for MedVL-CT69K. The scaling law for CT-RATE exhibits a larger coefficient but a smaller exponent. This indicates that in the low-data regime, our model tends to perform better on the CT-RATE dataset, while as data volume increases, the performance improvement on CT-RATE becomes less pronounced, reaching saturation more quickly. This can be attributed to the relatively lower complexity of the chest CT images in the CT-RATE dataset, which involve fewer anatomical structures and less pathological diversity compared to the abdominal CT images in the MedVL-CT69K dataset.

We would like to sincerely thank you for your thoughtful and constructive feedback during the initial review process. Your comments have been instrumental in improving the quality and clarity of our work. We have carefully addressed all of your concerns to the best of our ability and hope that our responses and revisions adequately resolve the issues you raised.

Dear 5y7k,

Thank you for your thoughtful feedback and constructive suggestions. We appreciate the opportunity to address your questions and comments, and we provide detailed responses below.

Q1 The evaluation metrics for the report generation task are limited to basic metrics (BLEU-4) and diagnostic accuracy.

A1 Thank you for your constructive comment. As suggested, in the revised paper, we have included more metrics (GREEN, BLEU 1-3, METEOR, ROUGE-L, CIDEr) for performance comparison on the downstream report generation task. We present the revised Table 2 as follows:

Q2.1 Will the MedVL-CT69K dataset or the pre-trained model checkpoint be open-sourced?

A2.1 We sincerely apologize for not clearly claiming the dataset release issue. The MedVL-CT69K dataset is subject to legal matters, and individual participant data will be made available upon request to leading physicians. We are actively working through our internal approval process and will make the code, and model checkpoints public upon approval of our institution, which will take 2 to 3 months.

Q2.2 Different numbers of diseases annotated in the validation (36) and test (54) sets.

A2.2 Our dataset follows real-world patient distribution, and the validation and test sets are divided from the entire dataset with 1,151 and 3,459 patients, respectively. Both validation and test sets were annotated by radiologists for 54 disease types. Due to the limited number of patients, the validation set only covers 36 disease types.

Q3 Limited separation between positive and negative cases in Fig. 7.

A3 Thank you for raising this important point. Our anatomical embeddings are not specific to any one disease but are designed for general diagnostic scenarios. Consequently, the embeddings encompass a rich array of semantic attributes, including the size, shape, texture, density, position, intensity of the organs, as well as characteristics related to the presence of lesions. Even if a sample does not exhibit a specific disease, it may still share certain attributes with some positive samples of that disease, leading to limited separation in the embedding space. For example, a positive sample for liver steatosis and a negative sample may both present liver cysts.

Dear wu1u,

Thank you for your detailed and insightful feedback. We appreciate your recognition of the strengths of our approach, and your constructive suggestions highlight important areas for further improvement. Below, we address each of your comments and questions:

Q1 How to process 3D volumes with a 2D ViT model.

A1 Thank you for the thoughtful feedback. After patchifying and flattening the input, both 2D and 3D images are converted into 1D token sequences, allowing us to process them using the same ViT model. The pre-trained ViT model includes four main components: a 2D convolution layer for patch embedding, a [cls] token, a learnable positional embedding layer, and a stack of Transformer blocks. To adapt the pre-trained ViT model for volume input, we replace the 2D convolution layer with a randomly initialized 3D convolution layer to perform patch embedding. For the positional embeddings, we first interpolate them across the spatial dimensions (i.e., coronal and sagittal axes) and then replicate them along the axial axis to match the size of patchified volumes. All other parameters are directly reused from the pretrained model.

Q2 It's unclear how the 2D CLIP methods are adapted to a 3D context and how the training was conducted.

A2 To adapt the 2D CLIP methods for 3D context, we replace their original 2D image encoders with the same image encoder used in our method. This modification enables these methods to process 3D volumetric inputs without altering their core innovations. Moreover, the text encoder is also replaced with the same one used in our method to ensure a fair comparison. All methods are then trained and evaluated under the same experimental setup.

Q3 In Table 3, the comparison with CT-CLIP models is ambiguous. Did the authors fine-tune a pre-trained model (based on their in-house data) on CT-RATE?

A3 To ensure a fair comparison with CT-CLIP models, we do not initialize our model with the pre-trained weights on the large-scale in-house data. Following the official settings used in CT-CLIP, we train our model on the CT-RATE training set from scratch and use the CT-RATE test set for internal validation and the whole Rad-ChestCT dataset for external evaluation.

Q4 To enhance the consistency of the embedding spaces shown in Fig. 7, the authors could consider setting a random seed before calculating the t-SNE embeddings.

A4 Thanks for your constructive comment. In fact, we have set a fixed seed before calculating the t-SNE embeddings. We would like to clarify that each feature point in Fig. 7 represents an anatomy-level feature, with each row of four figures corresponding to the features of the portal vein, liver, small bowel, and spleen, respectively. As a result, the embedding distributions vary naturally across these anatomical structures.

Q5 The authors clipped HU values to -300 to 400, raising questions about whether this range covers all abnormalities. Was this preprocessing applied across both non-contrast and contrast-enhanced scans? Additionally, was this range used for CT-RATE training, differing from the original CT-RATE paper?

A5 Thank you for your thoughtful and detailed feedback. The MedVL-CT69K dataset is derived from abdominal CT scans, and the primary goal is to diagnose abdominal diseases. In this context, we apply the soft tissue window for image normalization. In fact, we experimented with different HU clipping ranges and found that the range of [-300, 400] yielded better results compared to the broader range of [-1000, 1000]. This is likely because most of the 54 diseases we evaluated can be effectively diagnosed within the soft tissue window in both non-contrast and contrast-enhanced CT scans, and the narrower range improves image contrast, enabling the model to better capture features relevant to these diseases. In future work when training our model for whole-body diagnosis tasks, we will consider wider HU ranges to account for the lung-related abnormalities with HU values closer to -1000 and bone-related abnormalities that reach values close to 1000. For the chest CT-RATE dataset, we apply the same image pre-processing steps as CT-CLIP to ensure a fair comparison with the competitors. Specifically, we clip the HU values to a range of -1000 to 1000 and normalize them to range [-1, 1] during model training. We have clarified this point in the revised paper, as detailed in the Implementation Details section in Appendix A.2. Thanks again for your constructive comment.

We express our sincere gratitude for your valuable feedback during the initial review, especially for your careful attention to detail. Thank you for providing us an opportunity to clarify the points you raised. The insights you provide are indispensable to us, enabling significant improvements in the quality of our work. We are hopeful that our responses adequately address your concerns.

Dear Area Chair iRJK,

Thank you for your inquiry regarding the release of the code. The approval process within our institute takes 2 to 3 months. We are closely following the approval procedure, and the code will be released immediately upon approval.

Dear Reviewer wu1u,

We have provided the code through an anonymous link: https://anonymous.4open.science/r/fVLP-DF95 to facilitate the reviewing process. Besides, we are actively working through our internal approval process, which typically requires 2 to 3 months, and will make the code and model checkpoints public upon approval of our institution.

Dear Reviewer oBhB,

Thank you for your feedback and constructive suggestions. We appreciate the opportunity to clarify and expand on several aspects of our work. Below, we address each of your points in detail:

Q1 The evaluation metrics for the report generation task are limited to basic metrics (BLEU-4) and diagnostic accuracy in Table 2.

A1 Thank you for your constructive comment. As suggested, in the revised paper, we have included more metrics (GREEN, BLEU 1-3, METEOR, ROUGE-L, CIDEr) on the downstream report generation task. We present the revised Table 2 as follows:

Q2 Attributing sentences of reports mentioning a structure to the image feature of that structure may sometimes be misleading especially when manual verification/correction is not accessible.

A2 We fully appreciate the reviewer's concern. In this work, we use the Vision Transformer as the image enocder, where the self-attention mechanism enables each anatomical structure to dynamically interact with other structures and extract diagnostic-related cues from them. This results in a context-aware representation of each anatomy, facilitating the diagnosis of diseases that may involve more extensive anatomical areas.

Q3 The rapid advancement of open-source segmentation tools and LLMs raises the question of whether vision-language pretraining for CT willgradually phase out, as semantic labels could be directly constructed from text reports for supervised learning.

A3 We appreciate the insightful question and agree that advancements in open-source image segmentation tools and LLMs have expanded the possibilities for constructing semantic labels from text reports, potentially facilitating large-scale supervised learning. However, we believe that supervised learning is not an alternative to vision-language pretraining (VLP). While LLMs can be used to extract semantic labels from reports, this process typically discards many detailed descriptions that are crucial for understanding complex medical conditions. Specifically, these semantic labels are typically encoded as one-hot, which are effective for capturing discrete attributes such as the presence or absence of a disease but fall short in representing more detailed, continuous disease attributes such as size, location and density. These finer details provide essential insights into disease severity, progression, and response to treatment. Moreover, the semantic labels extracted by LLMs are not guaranteed to be 100% accurate, and manual calibration requires significant time and effort, which is not conducive to scaling datasets. In contrast, using diagnostic reports directly as supervision signals in VLP requires no additional annotations, providing a clear advantage in scaling data. Furthermore, reports are condensed records of the entire clinical diagnostic process, capturing detailed diagnostic descriptions and offering strong interpretability with comprehensive information. This enables VLP models to achieve a more holistic and nuanced understanding of complex diseases and their underlying patterns.

We express our sincere gratitude for your constructive feedback in the initial review. Your expert insights are invaluable to us in our pursuit of elevating the quality of our work.

Refs:

[1] Hamamci, Ibrahim Ethem, et al. "A foundation model utilizing chest CT volumes and radiology reports for supervised-level zero-shot detection of abnormalities." arXiv preprint arXiv:2403.17834 (2024).

[2] Ostmeier, S., Xu, J., Chen, Z., Varma, M., Blankemeier, L., Bluethgen, C., ... & Delbrouck, J. B. (2024). GREEN: Generative Radiology Report Evaluation and Error Notation. arXiv preprint arXiv:2405.03595.

[3] Zhao, Ziheng, et al. "One model to rule them all: Towards universal segmentation for medical images with text prompts." arXiv preprint arXiv:2312.17183 (2023).

We greatly appreciate your consideration of these clarifications. Your positive comments in the initial review were very encouraging, and we are truly grateful for your thoughtful feedback. We would be extremely appreciative if you could consider further raising the score.

If you still need any clarification or have any other concerns, please do not hesitate to reach out to us for any reason. Your guidance is greatly appreciated, and we are fully committed to a constructive and productive discussion.

Q2 Missing comparisons with other CT-CLIP approaches in Table 2. It's not clear how the 2D models were adapted to 3D. Report generation uses only a single NLU metric.

A2 Thank you for the constructive comment. Except IN, all other models in Table 2 are pre-trained on the MedVL-CT69K dataset. As we have adapted these 2D CLIP approaches for 3D inputs in vision-language pre-training, we can easily integrate their pre-trained image encoders with a BERT-base text decoder for CT report generation. In the revised paper, we have included more vision-language pre-training approaches (BIUD, Merlin) and metrics (GREEN [2], BLEU 1-3, METEOR, ROUGE-L, CIDEr) for performance comparison on the downstream report generation task. We present the revised Table 2 as follows:

• The revised Table 2

Q3 Comparison to the embeddings of CT-CLIP.

A3 Thank you for the insightful comment. CLIP and CT-CLIP indeed refer to the same method. As such, in our t-SNE visualizations, the embedding distributions from CLIP indeed represent the results of CT-CLIP.

Q5 The proposed approach only works for CT scans and the classes available in TotalSegmentator due to its reliance on anatomy segmentations from this tool.

A5 Thanks for your thoughtful feedback. While our experiments primarily focus on CT data, the proposed method is inherently general and can be extended to other imaging modalities and anatomical classes, provided that the anatomy masks are available. Indeed, as Reviewer oBhB mentioned, recent years have witnessed significant advancements in universal medical segmentation models, which are now capable of generating a diverse range of high-quality anatomical masks across multiple imaging modalities. For instance, SAT [3] can segment 497 anatomical classes across three modalities (CT, MRI, and PET) and 8 regions of the human body, such as brain, thorax, abdomen, and pelvis. These models greatly facilitate the deployment of our method.

Q6 Will the dataset, code and weights be released?

A6 We sincerely apologize for not clearly claiming the dataset release issue. The MedVL-CT69K dataset is subject to legal matters, and individual participant data will be made available upon request to leading physicians. We have provided the code through an anonymous link: https://anonymous.4open.science/r/fVLP-DF95 to facilitate the reviewing process. Besides, we are actively working through our internal approval process, which typically requires 2 to 3 months, and will make the code and model checkpoints public upon approval of our institution.

Q4 Ablation study on masking missing

We would like to kindly ask the reviewer to clarify the question regarding the "masking approach," as our method does not incorporate this module. Furthermore, we also wish to note that the terms "masking" and "missing" do not appear throughout our paper.

Dear Reviewer PfQv,

Thank you for your detailed and constructive feedback on our work. We appreciate your insights into both the strengths and weaknesses of our approach, and we address each of your points below:

Q1.1 Incosistency of compartive methods and metrics between Table. 1 and Table. 3

A1.1 Thanks for your valuable comment. First, we would like to clarify a misunderstanding: CT-CLIP is a renaming of CLIP in the paper by Hamamci et al. (2024) [1], representing a variant of CLIP applied to the CT scenario, and it denotes the same method as CLIP in our paper. We have clarified this point in the revised paper. Moreover, as suggested, we have included CT-VocabFine and CT-LiPro as supervised fine-tuning methods in Table 1. In Table 3, we have added comparisons with two methods, including BIUD and Merlin, to compare with more state-of-the-art (SOTA) methods. Regarding evaluation metrics, we have included weighted F1 and Precision in Table 1, and we have added Sensitivity (Sens) and Specificity (Spec) in Table 3.

• The revised Table 1

• The revised Table 3

Q1.2 How to adpat 2D approaches for 3D and how their evaluation was done.

A1.2 To adapt the 2D approaches for 3D context, we replace their original 2D image encoders with the same image encoder used in our method. This modification enables these competitors to process 3D volumetric inputs without altering their core innovations. Then, we train and evaluate all methods using the same experimental setup.

Dear Authors,

Your explanation of the internal approval process is quite clear. However, we would like to understand if there is any considerable possibility that the code release might not be approved. This information will help us better assess the situation during the review process.

Thank you for your clarification.

It is likely to be approved as we have done before, at least for non-commercial use. Previously we published our models and code on Github and Hugging Face sites.