Medical Vision-Language Pretraining through Contrastive Learning of Positive and Negative Mention

1. The claim of PNC evaluation as a novel contribution is questionable, as similar evaluation methods have been previously published [1]. "We define the procedure as follows. First, we compute logits with positive prompts (such as atelectasis) and negative prompts (that is, no atelectasis). Then, we compute the softmax between the positive and negative logits. Lastly, we keep the softmax probabilities of the positive logits as the probability that the disease is present in the chest X-ray." This is similar to the definition of PNC in the paper "in PNC evaluation method, both positive and negative mentions of different disease categories are used as text inputs, designated as “There is {disease}” and “There is no {disease}”. After calculating the similarity between all texts and images, the similarities of positive and negative mentions within the same category are normalized by softmax, and then the normalized."
2. Evaluating models trained only with POS using PNC metrics is unfair. During training, POS models are only exposed to positive mention patterns (e.g., "disease"), so the models may fail when processing negative mentions (e.g., "no disease") during evaluation. The model's text encoder, having never learned to distinguish the semantic significance of negation words like "no," may primarily focus on the disease term while failing to capture the negation context. This could result in similar embeddings for both "disease" and "no disease" prompts, undermining the validity of the PNC evaluation metrics. (Minor, Figure 2 PC should be POS?).
3. Figure 7 shows faster performance degradation in PNC compared to POS as categories decrease, but lacks theoretical explanation.
4. Some recent work[2] reports significantly higher performance on similar tasks.
5. The visualization analysis effectively demonstrates the model's advantages, however, it only compares with CARZero - including more baseline models would strengthen the empirical validation.

[1] Tiu, E., Talius, E., Patel, P. et al. Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning. Nat. Biomed. Eng (2022). https://doi.org/10.1038/s41551-022-00936-9. [2] Dai, Tianjie, et al. "UniChest: Conquer-and-Divide Pre-training for Multi-Source Chest X-Ray Classification." IEEE Transactions on Medical Imaging (2024).

• Constructing visual entailment relationships depends on LLMs for disease category extraction, making quality control challenging, especially when using models like Meta-Llama-3-8B-Instruct for tagging.
• The PNC evaluation method has already been utilized in previous works, including one of this study's baselines, CheXzero. The primary difference is in the prompt format (“{Pathology}”, “No {Pathology}” → “There is {disease}” and “There is no {disease}”).
  [1] Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning, Tiu et al., Nature Biomedical Engineering, 6(12):1399–1406, 2022.
  [2] Xplainer: From X-Ray Observations to Explainable Zero-Shot Diagnosis, Pellegrini et al., MICCAI 2023.
  [3] Significantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoders, Jang et al., Sci Rep 14, 23199 (2024).
• This model uses 25 labels extracted from the test reports for training, covering all labels in their test set. Such a setup makes zero-shot classification comparisons unfair. A more fair comparison would involve an out-domain dataset and labels, such as PadChest.

The paper lacks sufficient clarity and justification in key areas. Definitions and grouping of labels, particularly the neutral classification and the merging of distinct categories like ‘other diseases’ and ‘normal,’ are not well-explained, raising questions about label consistency. The evaluation relies too heavily on NLG metrics, omitting a clinically relevant, label-based comparison to verify accurate report matching. Claims about the model’s capability in handling rare diseases lack substantiating performance metrics. Details regarding data cleaning methods and the accuracy of LLM-based label extraction remain unverified, and certain experimental aspects, such as selection criteria in ablation studies, are insufficiently described. For further details, please refer to the Questions section.

1. This work first claim that negative mentioning is a distinctive difference between medical and general image domain, however, I didn't see any evidence to support this claim. Though I partially approved this claim with some emperical bias and the experiments results also reflects this pattern to some extend. I still want to see a clear statistic comparison between medical and general image domain about your claim.
2. I really want to suggest the authors to reformat or reporduce their figures, especially for Figure 2, I can barely see the '-' negative sign in Fig2(b). I have to enlarge the figures to 150% on my 27 inch monitor to see that. Other reviewers or readers might neglect such important signs with a glimpse. As your major improvement here is all about negative samples, why not make it obvious and highlighted?
3. This work relies on LLMs to extract positive and negative labels but not showing any evaluation on the LLMs performance on this task. Is the LLM employed and the labels it generate really reliable, at what degree of certainity? This is crucial for your further exploration.
4. For equation (2), as you described, you only sample one sentence from a report j, and you use $y_j$ as the text input. This confuse me in following ways: a. would you re-sample a sentence from the same report later in your training process? b. since you are randomly sampling a sentence $k^m$ from a report j on epoch m, I would assume that for next epoch, the sampled sentence $k^{m+1}$ is different sentence from last epoch. Then, the input $y_j$ is actually different now. Are you supposed to use $y_{jk}$ to represent the randomly selected sentence k from report j in your equation?
5. Image-Text retrieval is one of the most important use of CLIP model, but you only report the retrieval-based generation task. I'd like to see some experiments results on image-text retrieval tasks on the MIMIC-CXR dataset. And I think this might be one of the most important contribution of the this work to the community, as we don't have a very efficient tool for medical image-text retrieval models on the market.

My main concerns fall in the nolvelty.

1. The proposed Positive-Negative Contrastive (PNC) Evaluation Method lacks significance for medical vision-language models. In the case of CheXZero (https://www.nature.com/articles/s41551-022-00936-9), the authors have employed positive-negative pair prompts, a method established in 2022.

Furthermore, I believe this distinction is not crucial for medical VLMs. The choice between using positive-only prompts or positive-negative pairs is contingent upon the training pipeline and recommendations outlined in the original papers. It is entirely reasonable to compare these methods directly, as they all provide a measure of similarity to assess the likelihood of a specific disease and no difference between them in practical use. The introduction of PNC evaluation settings does not offer any additional advantages. Therefore, I fail to see the necessity of introducing the PNC framework.

2. The VECL is also not interesting enough. The textual pre-process methods are similar to MedKLIP (CVPR 2023 https://openaccess.thecvf.com/content/ICCV2023/papers/Wu_MedKLIP_Medical_Knowledge_Enhanced_Language-Image_Pre-Training_for_X-ray_Diagnosis_ICCV_2023_paper.pdf) which used triplets supervision signals to maintain and emphasize entailment, contradiction, and neutral relationships.

Based on the above concerns I don't feel that this paper is attractive enough to ICLR attendees to reach the publication requirements.

• Motivation: The motivation "previous studies only evaluate the distance between both negative and positive mentions" is not correct. Studies like [1], they evaluate both of them in the RSNA dataset. Instead, the evaluation method in this paper is a way to normalize the scores between different classes by leveraging the negative mentions.

• Evaluation: VECL is designed specifically for the PNC evaluation. Therefore, the "zero-shot classification" is not truly "zero-shot" and the comparison is somehow unfair for other methods. A better way to evaluate the performance is to evaluate the quality of the backbone model by finetuning it in the downstream tasks, like the setting in [1]

[1] Huang, Shih-Cheng, et al. "Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.