SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understanding

Lack of quality control for caption and QA dataset. The only quality control is for SlideBench which includes 5 experts to review the QA dataset. There is no discussion like, what were the criteria for resolving disagreements; how many questions did each expert go through; what are their specialties; are they all familiar with these 10 tumor? Since it's the key foundation of this paper. There must be a rigorous human-in-the-loop system to handle the creations.

Over-reliance on ONE language model (GPT4) from ONE company. The entire dataset is generated by GPT4 which may have the same bias and distribution shift. There is a lack of justification for solely using one model and a lack of diversity options on the other model. How do other language models like Claude do if used for generating captions and questions?

The benchmark claims SlideChat's zero-shot performance on VQA tasks. However, the training set and testing set basically have the same or very similar distribution (BCNB). Compared with the other baseline, SlideChat can never be treated as zero-shot. The real zero-shot results are the performances of baselines.

It's known that the TCGA report is very noisy. They have some reports with only one or two lines of description. They could also contain redundant information about radiology and other modalities. There is no quality control system for the SlideInstruction/Caption dataset which may lead to some hallucinations for the language models.

The handling of one report linked to multiple WSIs is not appropriate. The report will indicate which image they referring to in the corresponding descriptions. Even in the case where they don't specify or only provide a general impression. These are not essentially noisy data. And these three WSI image tokens can be grouped to pair with the report.

Evaluation metrics for medical use. should have less penalty for I don't know. It's also not proper to only report accuracy since some diagnosis questions favor metrics like false positive rate, false negative rate, and F1. It will be better to propose a specific way to handle "I don't know" responses in the evaluation.

1. Experiments are NOT adequate in the following aspects:

   a. To validate the incremental contribution of “slide-level” MLLM, patch-level MLLMs, such as Quilt-LLaVA [3], should be compared. I understand patch-level MLLMs cannot handle the long-sequence input of a slide. However, a global feature (e.g. mean/max pooling of patch features, or the slide-level features extracted by whole-slide pathology foundation models, such as Prov-GigaPath [4] or mSTAR [5]) can be used as the visual feature and fed into patch-level MLLMs.

   b. Performance on task-specific applications should be presented. In clinical practice, accuracy is paramount. Although the authors claimed that pathology foundation models rely on finetuning, their predictions are much more precise than MLLMs from results in appendix B.3, especially on BCNB datasets (which is originally a classification dataset) where **SlideChat is even worse than random guess in grading, subtyping and HER2. Please demonstrate classification performance on these tasks to see how much of the gap is between them.

   c. WSI-VQA, a SOTA slide-level VQA model, should be compared in the experiments.

   d. There is no out-of-domain captioning slide dataset for evaluation, which makes me worry if the model is trying to memorize something because the template is quite similar which will make the metrics caring about text-matching become high while the predicted results are wrong.

2. From the perspective of clinical significance, I can't imagine how pathologists would use this tool, as they wouldn't be asking the model these questions. While the captioning task may have some clinical significance, aiding in report writing, its current performance demonstrated in this paper falls short of supporting this requirement.

3. Methodology-wise, it is a bit incremental by simply combining an existing MIL model and LLaVA.

4. Data-wise, captioning data is much less than others, such as WSICaption. Both are curated from TCGA where over 10K cases of WSI-Report pairs are available. Why are there only 4K cases of captioning data?

5. Train/test split ratios should be given.

[3] Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos, CVPR, 2024

[4] A whole-slide foundation model for digital pathology from real-world data, Nature, 2024

[5] A Multimodal Knowledge-enhanced Whole-slide Pathology Foundation Model, arxiv, 2024

The main weakness of the study lies in the lack of comparison with pathology specific baselines in evaluation. In particular, while authors chose Llava-Med and MedDr as biomedical specialized MLLMs, neither of which are specifically trained for pathology. On the other hand, domain-specific MLLMs such as PA-LLava (https://github.com/ddw2AIGROUP2CQUPT/PA-LLaVA) and Quilt-llava (https://openaccess.thecvf.com/content/CVPR2024/papers/Seyfioglu_Quilt-LLaVA_Visual_Instruction_Tuning_by_Extracting_Localized_Narratives_from_Open-Source_CVPR_2024_paper.pdf) were not evaluated. This may in turn, over-inflate SlideChat's on pathology-specific VQA relative to other methods. The methodology of evaluating general purpose MLLMs like GPT-4o on just a thumbnail or randomly sampled, small crops completely independently before majority vote is also somewhat questionable, and would benefit from further experimentation to make the comparison more convincing.

Additionally, while the release of the 176k VQA question answer pairs is likely still potentially useful for the community (especially the release of SlideBench-VQA, which has been reviewed by expert pathologists to ensure quality), it is difficult to assess its impact relative to other datasets already available in the community, given it stems from just 3,294 patients from TCGA. The previous MI-Gen work, which the authors cite, have already OCRed / used GPT to clean-up ~10k TCGA WSI captions and have made them publicly available - a significantly larger dataset than the 4.2k captions to be released by the authors.

1. Although the paper shows better performance than other baseline MLLMs, none of the baselines are pathology-specific. In WSI-captioning, SlideChat is worse than the only pathology-specific model in the baseline - MI-Gen. It is not clear if SlideChat is better than other pathology MLLMs such as Quilt-Llava in the proposed tasks (including WSI captioning and VQA).
2. The evaluation scheme for patch-level baselines (e.g. GPT 4o) is concerned. While a 1024x1024 thumbnail destroys the aspect ratio of the slide and is too small to make the diagnosis, a random sampling of 30 patches creates randomness between the run and may not well represent the diversity of the patches within the slide, given a slide may have 10^4 224x224 patches (e.g. the diagnostic patch may not be included in the sampled patches). Instead, prototypical-based patch selection methods (which can be as simple as feature-based clustering methods) can be considered for a fair comparison.
3. The small scale of the training (only on 4181 WSIs) limits the diversity of the slides seen during training. How does the model do on out-of-distribution data (such as non-cancer cases or rare diseases e.g. EBRAINS [1]) in terms of diagnosis and IDH status prediction (which can either be in the form of VQA or WSI captioning)?
4. There is no failure case / hallucination analysis, which is especially critical for medical MLLMs. People can gain a better understanding of the model's limitations if the authors can show and analyze when and why model fails, and how does the model hallucinate.
5. There is lack of comparison with supervised baseline on BCNB given the prediction of ER, PR, and HER2 status are standard WSI classification tasks which are commonly included in existing patch-level foundation models [2]. How does the model compare to ABMIL trained on CONCH features?

[1] The digital brain tumour atlas, an open histopathology resource. Scientific Data. [2] Phikon-v2 A large and public feature extractor for biomarker prediction. arXiv:2409.09173

1. Generating 175,754 visual question-answer pairs from 4,181 WSI-caption pairs results in nearly 40 QAs per caption. However, the reports for these WSIs often contain only a few key points (around 3-4 important points per report), which can lead to many ineffective or redundant QAs.
2. There may be some overlap between SlideBench-VQA and WSI-VQA, so the authors should clarify this. If any WSIs from the WSI-VQA test set appear in the SlideBench-VQA training set, it could result in an unfair comparison.
3. Comparing other models that use WSI-thumbnails or randomly select 30 patches to represent the WSI is not a fair approach, as both WSI-thumbnails and the limited selection of patches typically lack critical diagnostic information. I’m also curious about the effects of selecting more patches, say 300, instead of just 30.
4. A comparison with the previous open-source slide-level MLLM, PRISM [1], is necessary.

[1] Shaikovski G, Casson A, Severson K, et al. PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathology[J]. arXiv preprint arXiv:2405.10254, 2024.