Can Medical Vision-Language Pre-training Succeed with Purely Synthetic Data?

Q1: Results on Cleaned MIMIC-CXR Excluding 7,000+ Samples

• First, we mention these 7,000+ samples in our work to demonstrate that we successfully identified data issues in the real dataset, not to suggest that the dataset can be perfectly cleaned by removing these samples. Achieving a completely clean dataset by eliminating every problematic sample is impractical.

• Therefore, we do not expect the filtering procedure to significantly enhance VLP performance. While we removed bad samples based on image quality, another critical issue we identified—the long-tailed distribution of entities in the text—cannot be resolved by simply removing samples. This is because each report may contain entities from both head and tail groups. For example, one report could include entities that are both frequent and rare, so removing the report would not effectively address the long-tailed distribution. Consequently, even after filtering, the real dataset still suffers from this issue, which negatively affects VLP performance.

• We re-pretrained ConVIRT on the cleaned version of MIMIC-CXR, excluding these 7,000+ samples, and the results are shown below. As demonstrated in the table, the VLP method pre-trained on the filtered MIMIC-CXR dataset slightly outperforms the version pre-trained on the uncleaned dataset. However, it still underperforms compared to the method pre-trained on our synthetic data.

This can be attributed to two key factors:

1. Even after removing many imperfect samples, it remains impractical to eliminate all poor or unpaired samples from the real dataset. Consequently, some problematic samples persist, negatively impacting the VLP process.

2. The cleaning process primarily focused on filtering samples based on image quality. However, the long-tailed distribution of entities within the dataset was not addressed. Since image-text pairs cannot be easily downsampled or oversampled, this issue limits the representational balance and continues to affect the cleaned dataset.

W2: Moving RadGraph and RaTE to the Related Work

• Thank you for your detailed advice. We will move this content to the Related Work section (Section 2) in the camera-ready version.

W3: Without Theoretical Explanation or Clinician Verification

• First, we kindly disagree with the claim that our work is solely an engineering approach for a specific task. Our work systematically explores and addresses key issues in existing MedVLP datasets, such as "unpaired image-text samples," "low-quality images," and "long-tailed distributions in text." Using our proposed pipeline, we identify these challenges and design a generation framework to mitigate them, rather than brute-forcing the generation of synthetic samples. Additionally, we evaluate models pre-trained on our synthetic data across four downstream tasks and nine datasets, demonstrating the effectiveness of our approach.

• Our study investigates whether VLP can succeed using purely synthetic data and whether VLP models trained on such data can learn general representations that benefit a wide range of downstream tasks. While we acknowledge that synthetic images may contain inaccuracies, they are utilized solely during the vision-language pretraining stage to develop general representations. At this stage, the presence of noisy data is not unusual. For instance, the CLIP-LAION2B dataset includes noisy samples, and the MIMIC-CXR dataset, as demonstrated in our work and noted in BioViL-T [1], contains numerous unpaired image-text samples. Despite these imperfections, VLP methods pre-trained on these datasets have consistently demonstrated strong performance on downstream tasks across both natural and medical imaging domains [2,3]. This indicates that Vision-Language Pretraining models are capable of learning robust representative features even in the presence of noisy or imperfectly paired data.

• Therefore, while clinician verification is essential for clinical deployment, thoroughly validating synthetic samples for the VLP stage is not strictly necessary.

[1] Bannur, Shruthi, et al. "Learning to exploit temporal structure for biomedical vision-language processing." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2] Wang, Fuying, et al. "Multi-granularity cross-modal alignment for generalized medical visual representation learning." Advances in Neural Information Processing Systems 35 (2022): 33536-33549.

[3] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

W4: Qualitative Results of the Segmentation Task

• Thank you for your feedback. We will update the visualization results of the segmentation task in the camera-ready version.

W5: Experiment on Other Modalities

• Our work focuses on evaluating the effectiveness of purely synthetic data for VLP, which relies heavily on a clinician-verified image generative model. Without such verification, the quality of synthetic data would be inadequate. Currently, there is no publicly accessible generative model, validated by clinicians, that can generate 3D CT or MRI volumes from text conditions.

  We are aware of recent work [1] that explores text-conditional CT generation. However, the quality of the generated images remains limited and has not been validated by clinicians, making it unsuitable for training a MedVLP model using purely synthetic data.

  In contrast, RoentGen, designed for chest X-ray image generation, has been validated by clinicians and has demonstrated satisfactory performance in studies evaluated by human experts [2]. Hence, this work focuses on synthetic chest X-ray image-text pairs. Nevertheless, our method can be directly extended to other modalities as soon as high-quality generative models become available for those modalities, since our approach does not rely on any modality-specific priors.

[1] Hamamci, Ibrahim Ethem, et al. "GenerateCT: text-conditional generation of 3D chest CT volumes." European Conference on Computer Vision. Springer, Cham, 2025.
[2] Bluethgen, Christian, et al. "A vision–language foundation model for the generation of realistic chest x-ray images." Nature Biomedical Engineering (2024): 1-13.

W6: Typos in Line 233

• Thank you for pointing this out. We will correct top 50 frequent entiites to top 50 frequent entities in the camera-ready version.

W1, Q2: Experiments on Other Modalities

• Currently, there is no publicly accessible generative model, verified by clinicians, that can generate 3D CT or MRI volumes from text conditions. We are aware of recent work [1] that explores text-conditional CT generation. However, the quality of the generated images is still limited and has not been validated by clinicians. As a result, this model is not suitable for training a MedVLP model using purely synthetic data.

  In contrast, RoentGen has been validated by clinicians and has demonstrated satisfactory performance in studies evaluated by human experts [2].

[1] Hamamci, Ibrahim Ethem, et al. "GenerateCT: text-conditional generation of 3D chest CT volumes." European Conference on Computer Vision. Springer, Cham, 2025.
[2] Bluethgen, Christian, et al. "A vision–language foundation model for the generation of realistic chest x-ray images." Nature Biomedical Engineering (2024): 1-13.

W1, W4-1: Evaluation Details on Table 4 and Figure 3

• In the experiments presented in Table 4 and Figure 3, the "average zero-shot classification AUC" represents the average performance across all zero-shot classification results, as detailed in Section 4.1. The datasets and implementation details for the zero-shot evaluation are provided in Appendix A.1 and A.2.

W2, W3, W4-2: Content Modification and Expression in Section 3.2

• Thank you for your feedback. We will revise the content and improve the expression in Section 3.2 in the camera-ready version after acceptance.

W2: Lower F1 Scores in Zero-Shot Evaluation

• First, we directly refer to the zero-shot performance of the baseline methods as reported in [1]. Furthermore, we strictly re-pretrained all baseline methods on our SynCXR dataset following their official codebases. This ensures that we did not alter their training frameworks but simply replaced the real data with our synthetic dataset to evaluate the effectiveness of the synthetic data. Therefore, the lower F1 scores are attributable to the limitations of the baseline methods themselves, not to our synthetic dataset.

[1] Phan, Vu Minh Hieu, et al. "Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.

W4-3: Using External Datasets to Boost VLP

• In our related work, we note that "some methods also leverage external datasets to boost performance, raising concerns about generalizability." We elaborate on this in Section 3.3, where we explain that methods such as MedKLIP, KAD, and MAVL rely on external datasets like RadGraph, which are annotated by humans. Additionally, MAVL requires human re-annotation of reports, making these approaches neither scalable nor generalizable due to the high cost of human annotation. Moreover, there is no metric to define or standardize which external datasets are used.

[1] Wu, Chaoyi, et al. "Medklip: Medical knowledge enhanced language-image pre-training for x-ray diagnosis." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Zhang, Xiaoman, et al. "Knowledge-enhanced visual-language pre-training on chest radiology images." Nature Communications 14.1 (2023): 4542.

[3] Phan, Vu Minh Hieu, et al. "Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

W4-4: Metric for Identified Data Issues

• We successfully identified over 7,000 problematic images, accounting for more than 3% of the real dataset. However, thoroughly detecting all problematic images is impractical as it would require manual verification of each image at significant cost. Our goal is to highlight and address issues in the real dataset, not to create a perfectly clean dataset.

W4-5: Comparison with Rad-DINO

• Rad-DINO was pre-trained on 880k images with a resolution of 518 × 518 from multiple CXR datasets, as reported on the official model page. In contrast, the baseline methods we implemented strictly use the MIMIC-CXR dataset, consisting of 210k images with a resolution of 256 × 256. This disparity in dataset size (four times larger) and resolution (double) makes a fair comparison between our Vision-Language Model (VLM) and Rad-DINO challenging.

• Additionally, Rad-DINO does not provide fine-tuning code or downstream task data splits, which prevents us from directly comparing our VLM under Rad-DINO’s settings.

• Since Rad-DINO is pre-trained with image-only data, it cannot perform zero-shot classification as it lacks alignment with textual information. To evaluate Rad-DINO, we implemented fine-tuning classification using MGCA’s [1] ViT fine-tuning strategy and applied the same data split as our work.

• The results, shown in the table below, indicate that even though Rad-DINO was pre-trained on a significantly larger dataset than MedVLP, the MedVLP method with our synthetic data still outperforms Rad-DINO. This is because VLP incorporates semantic information from textual reports, enabling richer contextual understanding, whereas Rad-DINO, relying solely on visual SSL, focuses only on visual invariances without capturing clinical relevatn semantic information.

[1] Wang, Fuying, et al. "Multi-granularity cross-modal alignment for generalized medical visual representation learning." Advances in Neural Information Processing Systems 35 (2022): 33536-33549.

W4-11: Comparison to Other Concept Generation or Filtering Techniques

• First, there are no existing methods that use purely synthetic data to train a MedVLP model. We are the first to explore this approach.

• There is only one method, SynthCLIP [1], that uses purely synthetic image-text pairs to train a CLIP model based on natural image-text pairs. However, their approach has limitations, as it generates text with only one entity per sample. In the medical domain, real-world data typically contains multiple entities in a single sample, as explained in Section 3 of our main article.

• Therefore, a direct comparison is impractical due to the differences in methodology and the domain-specific requirements of medical data.

[1] Hammoud, Hasan Abed Al Kader, et al. "SynthCLIP: Are We Ready for a Fully Synthetic CLIP Training?." arXiv preprint arXiv:2402.01832 (2024).

W4-12: Clarifying Claims in Section 5

• For the claim, "likely due to the persistent long-tailed distribution, as the synthetic reports are generated based on real images": This is straightforward. As detailed in the last part of Section 3.1, we successfully detected the long-tailed distribution in the real dataset. Since the reports are paired with the images, the long-tailed distribution in the reports reflects the same distribution in the images. When using a report generation model to generate reports based on these long-tailed images, the resulting reports also inherit this distribution, as they aim to describe the images.

• For the claim, "as synthetic images can be curated using our image filtering procedure to ensure high quality, avoiding issues commonly found in real datasets": This is because synthetic data is generated by models, allowing us to regenerate as many samples as needed. This provides an opportunity to replace low-quality synthetic images with regenerated ones, ensuring the total number of samples remains unchanged. In contrast, for real datasets, removing low-quality images directly reduces the number of available samples, which cannot be replaced, limiting the dataset's size and balance.

W4-6: Potential Issues in Generating Reports from Sampled Entities

• First, we identified five groups of entities: 55,047 Abnormality, 36,365 Non-Abnormality, 23,017 Disease, 22,103 Non-Disease, and 40,517 Anatomy entities, totaling over 150k. We sample entities independently from each group rather than from the entire pool of 150k entities at once.

• For illogical combinations, such as "pneumothorax in rib," the pre-trained LLM, having been trained on a large corpus that includes medical data, can address these inconsistencies. For example, querying the LLM with "Can pneumothorax occur in the rib?" would result in a negative answer. This demonstrates the LLM's ability to understand relationships between entities and avoid generating unreasonable combinations.

W4-7: Time Consumption in Synthetic Data Generation

• We use the vLLM [1] package to accelerate text generation, filtering, QA, and impression generation stages. For Text-to-Image (TTI) generation, we use the diffusers package and the official code provided by RoentGen [2]. Below, we list the time required for each step in processing a single sample.

• To benchmark the generation process, we used Llama3.1-8B-Instruct for report and impression generation and InternVL2-26B for image filtering and QA. Note that the times listed below are for single-sample inference. In practice, vLLM optimizes batch processing by selecting the optimal batch size, so the total time for processing the dataset can be less than the product of the number of samples and the per-sample time. All evaluations were performed using a single A100 GPU.

[1] https://github.com/vllm-project/vllm
[2] Bluethgen, Christian, et al. "A vision–language foundation model for the generation of realistic chest x-ray images." Nature Biomedical Engineering (2024): 1-13.

W4-8: Potential Collapse Risk with Synthetic Data Training

• The paper [1], which claims that recursively training AI models with synthetic data can lead to model collapse. This process involves generating synthetic data $D_{A_0}$ with model $A_0$, training $A_0$ on $D_{A_0}$ to produce an updated model $A_1$, then using $A_1$ to generate new synthetic data $D_{A_1}$, and repeating the process iteratively.

• However, our pipeline fundamentally differs from this recursive approach. We use publicly available models, specifically InternVL and Llama, to generate the synthetic data, which is then used to train our MedVLP model. Importantly, we do not use MedVLP—or any model trained on synthetic data—to generate additional synthetic data. Since we do not recursively use synthetic data for both generation and training, the concerns described in [1] do not apply to our methodology.

[1] Shumailov, Ilia, et al. "AI models collapse when trained on recursively generated data." Nature 631.8022 (2024): 755-759.

W4-9: Explanation for Low F1 and Dice/IoU Scores

• As noted in our response to W2, we directly reference the zero-shot performance of baseline methods as reported in [1]. We also strictly re-pretrained all baseline methods on our SynCXR dataset using their official codebases. This ensures that we did not modify their training frameworks but only replaced the real data with synthetic data to evaluate its effectiveness. Therefore, lower F1, Dice, or IoU scores are due to the limitations of the baseline methods, not our synthetic dataset.

• Furthermore, as shown in Table 2(b), our results demonstrate improvements in IoU scores when using synthetic data: +1.89 for ConVIRT and +4.66 for mixed data, compared to models pre-trained on the real dataset alone. This indicates that MedVLP benefits significantly from our synthetic dataset.

[1] Phan, Vu Minh Hieu, et al. "Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.

W4-10: Pretraining and Fine-tuning Datasets in Section 4.2

• For pretraining, we used both the MIMIC-CXR dataset (referred to as the "real dataset") and the SynCXR dataset (generated by our pipeline). The "mixed dataset" combines MIMIC-CXR and SynCXR, without additional curation.
• For the fine-tuning dataset, we provide detailed information in Appendix A.1 and A.2.

Thank you for all your valuable review comments. We kindly ask you to review our rebuttal, where we have addressed your concerns in detail. We believe it clarifies and strengthens the points raised, and we would be happy to further discuss or address any remaining issues you may have.

Thank you again for your time and consideration.

Q3: Results on Cleaned MIMIC-CXR

• We re-pretrained ConVIRT on the cleaned version of MIMIC-CXR, and the results are shown below. As the table demonstrates, the VLP method pre-trained on the cleaned MIMIC-CXR dataset slightly outperforms the version pre-trained on the uncleaned dataset. However, it still underperforms compared to the method pre-trained on our synthetic data. This can be attributed to two main reasons:

  1. Even after filtering out many imperfect samples from the real dataset, it is impractical to thoroughly remove all poor or unpaired samples. As a result, some problematic samples remain in the real dataset, negatively impacting the VLP process.

  2. The cleaning process for MIMIC-CXR primarily focused on filtering samples based on image quality. However, the long-tailed distribution of entities in the dataset was not addressed, as simply downsampling or oversampling image-text pairs is not feasible. This issue limits the representational balance in the cleaned dataset.

W4: Evaluation on MGCA and Naive CLIP

• Thank you for your suggestion. We re-implemented MGCA [1] and Naive CLIP [2] using their official codebases on both our SynCXR dataset and its de-duplicated version. The results are presented below. As shown in the table, both VLP methods pre-trained on our SynCXR dataset outperform those pre-trained on the real dataset. This highlights the effectiveness of our data generation framework.

W5: Typos Correction

• Thank you for your detailed revision. We have corrected the typos and updated the figures, and these changes will be reflected in the camera-ready version.

[1] Wang, Fuying, et al. "Multi-granularity cross-modal alignment for generalized medical visual representation learning." Advances in Neural Information Processing Systems 35 (2022): 33536-33549.

[2] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

Q1: Data Quality Check on Synthetic Data

• To assess the quality of the synthetic data, we used BiomedCLIP [1], a foundational vision-language model, to compute the CLIP score for all samples in the synthetic dataset and then averaged the results. The CLIP score measures the alignment between synthetic reports and images. Similarly, we computed the averaged CLIP score for the real dataset as a reference.
  
  The results show that the averaged CLIP score for the real dataset is 0.74, while our SynCXR dataset achieves a score of 0.72. This close alignment demonstrates that our synthetic dataset exhibits a level of coherence and quality comparable to the real dataset, underscoring the robustness of our data generation framework.

[1] Zhang, Sheng, et al. "BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs." arXiv preprint arXiv:2303.00915 (2023).

Q2: Synthetic Samples

• Thank you for your suggestion. Yes, we will update the synthetic reports and images in the camera-ready version.

W3: Implementation Details of Generation and VLP

• We set k=9, m=3, and $τ_{max}=15$ in our work.

• Specific Description of tVLP with Mixed Data:
  We implemented the baseline methods on mixed data following their official codebases [1][2]. The only modification was increasing the dataset size, which resulted in doubling the training time.

[1] Zhang, Yuhao, et al. "Contrastive learning of medical visual representations from paired images and text." Machine Learning for Healthcare Conference. PMLR, 2022.

[2] Huang, Shih-Cheng, et al. "Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

W2: Concerns About the Quality of Synthetic Images

• Imperfect Synthetic Images:
  We acknowledge that synthetic images may contain inaccuracies. However, these images are used solely during the vision-language pretraining stage, where the presence of noisy data is not uncommon. For instance, the CLIP-LAION2B dataset includes noisy samples, and the MIMIC-CXR dataset, as demonstrated in our work and noted in BioViL-T [1], contains many unpaired image-text samples. Despite these imperfections, VLP methods pre-trained on both datasets have demonstrated strong performance on downstream tasks involving natural and medical images [2,3]. This suggests that Vision-Language Pretraining models can effectively learn representative features even when the data contains noise or when images and texts are not perfectly aligned. Therefore, while some synthetic images in our dataset may be imperfect or even contradict their associated reports, they are unlikely to significantly affect the VLP process.

• Anatomical Region Control:
  We agree that fine-grained regional control, such as employing models like ControlNet, could improve the correctness of synthetic images by ensuring better alignment with anatomical structures. However, as of this project's completion, implementing such control faces two key challenges: (1) the absence of fine-grained segmentation tools capable of generating accurate masks for individual anatomical regions, and (2) the lack of paired chest X-ray mask-text datasets required to train ControlNet or similar models for fine-grained regional control. Addressing these gaps remains a critical area for future research to advance synthetic medical image generation.

[1] Bannur, Shruthi, et al. "Learning to exploit temporal structure for biomedical vision-language processing." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2] Wang, Fuying, et al. "Multi-granularity cross-modal alignment for generalized medical visual representation learning." Advances in Neural Information Processing Systems 35 (2022): 33536-33549.

[3] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

Thank you for your valuable time and comments. The main concerns are addressed below.

W1: Concerns about entity sampling and constrain on report generation

• Potential Contradictory Entities:
  Leveraging the rich knowledge embedded in LLMs, these models inherently understand contradictory situations. For example, if we query an LLM with "Can one anatomical region of the lung have increased size and decreased size at the same region at the same time?", the LLM is capable of reasoning that this is impossible because these are opposing processes. The LLM effectively distinguishes between conflicting terms like "increased size" and "decreased size" by associating them with different entities. This ensures that the generated synthetic report remains logical and avoids contradictory entities.

• Repeated Entities:
  We observed repeated entities in the extracted entity set. To address this, we utilized Med-CPT [1], a text embedding model, to compute text embeddings for all entities. Next, we calculated the embedding similarity for entities within each group: ABNORMALITY, NON-ABNORMALITY, DISEASE, NON-DISEASE, and ANATOMY. For entities with an embedding similarity greater than 0.95, we identified one entity as redundant and replaced it with the other. After this process, each group was refined, resulting in a total of 98,758 unique entities. Approximately 36% of entities were removed from the initial pool. We then re-implemented our synthetic data generation procedure based on this de-duplicated set of entities.
  
  Subsequently, we re-pretrained ConVIRT and GLoRIA on the synthetic dataset generated from the de-duplicated entities. As shown in the zero-shot classification results below, the synthetic dataset consistently improves both MedVLP methods on downstream tasks. We appreciate the reviewers' valuable suggestions, which have inspired us to further enhance the quality of our synthetic data generation pipeline.

• Enforcing the Generation of the Impression Section: We focus on generating the "Impression" section of reports because the generative model, RoentGen [1], is trained exclusively on text conditioned by "Impression" data. As a result, this model is designed to accept only "Impression"-style text as input. Additionally, in the MIMIC-CXR dataset, which serves as the reference real-world dataset in this work, there are 213,384 image-text pairs that include an "Impression" section, compared to only 137,485 samples that include a "Findings" section. This discrepancy indicates that the "Impression" section is more commonly present in radiology reports than the "Findings" section, making it the logical focus for our generation pipeline.

[1] Jin, Qiao, et al. "MedCPT: Contrastive Pre-trained Transformers with large-scale PubMed search logs for zero-shot biomedical information retrieval." Bioinformatics 39.11 (2023): btad651. [2] Bluethgen, Christian, et al. "A vision–language foundation model for the generation of realistic chest x-ray images." Nature Biomedical Engineering (2024): 1-13.

I sincerely appreciate the author's feedback and it is very exciting to see all those extra experimental results with promising outcomes. Overall, the author's feedback addressed most of my concerns and questions.

Still, there are 3 minor questions.

• First of all, the fact that LLM can understand the relationship between each contradictive entity does not necessarily mean it can avoid this issue when contradictive entities are provided in the input. LLM can still generate problematic captions simply using all the given entities rather than figure out if they are reasonable or not. The reviewer believes it is still necessary to double-check this problem, and maybe take a closer look into the generated captions.
• Meanwhile, the new results after filtering out the repeated entities improved the performance of the pipeline and this may indicate a major revision in terms of the paper's content is needed to reflect this and further clarify the whole image synthesis pipeline.
• Lastly, I largely agree with the author's comments on the synthesis image quality issue. However, it concerns me without qualitative or quantitative evidence.

These issues put me in a hard position, I would like to recommend acceptance for the paper given its contribution and the inspiring results, but it seems that there are still things that can be further optimized in the image synthesis pipeline, which makes it slightly below the bar of 8: Directly accept. I would really like to raise my score to 7: weakly accept, but there is no such a number here. So I choose to maintain my current score of 6 for now and wait to see what other reviewers think.