EndoAssistant: A Large-scale Vision-Language Dataset for Endoscopic Surgery Understanding from Open-Source Videos

limited analysis of potential biases within the data, impacting the generalizability of the model.

• Curated training data is unknown about the source hospitals and patient demographics, as such information is not available from video. They are all endoscopy-surgery: 91% Gastrointestinal Endoscopy, 6% ENT Endoscopy, 2% Ophthalmic Endoscopy, 1% Neuroendscosopy.
• To evaluate the generalizability, we evaluate on various testing data from different data origin and surgical types:

The dataset relies on relatively straightforward image-text pairing and may not fully capture deeper semantic alignment between the visual and language modalities

Our dataset generates question-answer pairs based on video captions, which contain deep semantic alignment between visual and language information. Compared to existing benchmarks that only contain simple question-answer pairs, our dataset introduces complex questions that require deep understanding of deep semantic alignment between visual and language information.

include in the limitations a discussion on the challenging conditions

Thank you for highlighting these challenges, which are critical for improving our work. During data collection, we focused on curating high-quality samples, filtering out cases with blurriness or poor lighting, which are indeed prevalent in real-world scenarios. Moving forward, we aim to incorporate more diverse and challenging cases into our dataset to enhance scalability, specifically targeting such challenging conditions. We believe that including these challenging scenarios will significantly improve the model's robustness in real-world applications. Regarding data bias and generalizability, we have addressed these aspects from both training and testing perspectives in our discussion. Additionally, we have included a detailed discussion of these common challenges in endoscopic surgery within the limitations section.

Fine-tuning on similar medical datasets could make the evaluation more aligned with the dataset's intended use. [1] Hecvl: Hierarchical video-language pretraining for zero-shot surgical phase recognition [2] Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation

Thanks for suggesting these related works. As the data collected in [1,2] are not released, we tested their released checkpoint (noted as surgVLP) under our settings and added comparisons in table1, table2, figure5.

• Zero-shot cross-modal retrieval: text-to-image (T2I) and image-to-text (I2T)

• Zero-shot classification on four anatomical detection datasets

• Linear-probing on four anatomical detection datasets

How does EndoAssistant perform on surgical tasks beyond classification and VQA, such as surgical phase recognition or anomaly detection?

• Surgical phase recognition: EndoVis18-VQA and Cholec80-VQA datasets (table3) we used for visual question answering tasks contain phase/tool recognition questions.
• Kvasir, Hyper-Kvasir, NBI-InFrames, and GastroVision datasets (figure5 and table1,2) are all for anatomical detection. The categories of Kvasir include polyps, two classes related to polyp removal and three anatomical landmarks in the GI tract. The anatomical-landmarks in Hyper-Kvasir dataset include cecum, ileum, pylorus, retroflex-rectum, retroflex-stomach and z-line. The classes in the NBI-InFrames dataset include tissue with intraepithelial papillary capillary loop-like vessels, leukoplakia, hypertrophic vessels, and healthy tissue. GastroVision also exhibits a diverse range of classes, including anatomical landmarks, pathological findings, polyp removal cases and beyond.

SurgVLP is a work on surgical phase and tool recognition. I do not understand why the authors compare it with zero/few-shot methods on the pathological and endoscopic diagnosis datasets. Comparing it with surgical understanding tasks is more reasonable.

What is the average / std for the length of captions?

The average/std of the caption lengths is 15.7/11.7.

How are the 120 image / caption pairs used in "Cross-modal retrieval" selected? Will this data be made available for future works to allow comparisons?

Expert surgeons manually selected and verified an endoscopy subset from PathCap (Sun et al., 2023) and PMC-CLIP datasets (Lin et al., 2023). We will release the code, checkpoints, and the curated data (training and testing) to the public upon acceptance.

Can the authors offer any additional insights about the quality of the dataset?

• Visual-Contextual Alignment: We aligned images and captions using video timestamps, providing a highly stringent and consistent alignment guarantee.
• Contextual Richness: The current image-caption pairs have achieved state-of-the-art (SOTA) performance across four anatomical detection datasets, excelling in all standard vision-language pre-training evaluation protocols: cross-modal retrieval (Table 1), linear probing (Table 2), and zero-shot/few-shot classification (Figure 5). This highlights their richness and effectiveness.
• QA Conciseness: We observed some hallucinations when generating explanations alongside answers using GPT.
  • Gathering clinicians' feedback for prompt design and professional evaluation/correction adds significant cost.
  • The primary aim of this paper is to demonstrate the effectiveness of the collected data from video, minimizing additional input from GPT to better validate its utility.

As a result, we chose to use only the data without explanations.

• Expert Evaluation: On randomly sampled 5k data, one surgeon checked (1) cleaned text all containing medical relevant and high-informative information; (2) texts were matched with the associated time-aligned images to ensure alignment with surgical terminology and context; (3) automatically curated QA dataset achieves 100% surgical relevancy, 95% accuracy, and includes 62% of questions that require image reference to be answered.

What is the motivation for using both a CLIP model and a custom pretrained CNN classifier to classify endoscopic vs. irrelevant content?

The motivation for using a two-step filtering process involving both a general CLIP model and a custom specific classifier lies in their complementary strengths in handling diverse and domain-specific content respectively.

Step 1: CLIP for Broad Filtering

• Generalization Capability: CLIP is pre-trained on tons of data, effectively distinguishing broad categories like “surgical endoscopy image” vs. “radiology image” vs. “presentation slide or lecturer talking" vs. “Doctors or patients”.
• Necessity to re-classify the images around boundary: given the aforementioned prompts, we noticed that the classification accuracy of pretrained CLIP is almost 100% when it gives a high confidence (probability > 0.25 for one certain class), otherwise it requires an additional classifier.

Step 2: Custom classifier for Fine-grained Classification

• Endoscopy and non-endoscopy training data: we collect positive samples from existing endoscopy datasets but the non-endoscopy images can be quite diverse and thus require self annotation. We manually labeled 56k images (enough to cover the diversity of the negative samples).
• High Precision: We train a binary classifier specifically on the aforementioned data, which is able to refine identified subtle features unique to endoscopy, like organ textures and surgical lighting, removing residual non-endoscopic images missed by general CLIP.

Overall, the introduction of CLIP alleviates our labor to annotate the training data for custom classifier. CLIP gives broad generalization capabilities for initial filtering and in-house annotated data gives domain-specific expertise for fine-grained classification. This ensures both scalability and accuracy in distinguishing endoscopic content from non-endoscopic content.

Implementation details

• Contrastive language-image pre-training: we have updated the paper including all the training hyper-parameters for pre-training, few-shot, and linear probing settings.
• LLaVA: we used the default finetuning hyper-parameters provided by https://github.com/haotian-liu/LLaVA/blob/main/scripts/v1_5/finetune_task_lora.sh on llava-v1.5-7b with 10 epochs.

Evaluate on internal datasets.

Thank you for the suggestion! It seems valuable to turn the surgeon-evaluated subset into a testing benchmark. However, for now, we have used all of it for training. We are considering setting aside some for testing once we collect new data and have it reviewed by the surgeon.

Open release.

We will release the code, checkpoints, and curated data (including training data from open-source videos and testing image-caption pairs from PubMed) to the public upon acceptance.

How are the Question-Answer Pairs constructed? Some QA not related to the images.

As described in Section 3.2, we collaborated with surgeons to design prompts that guide GPT in generating QA pairs from each caption text. On a randomly sampled subset of 5,000 data points, a surgeon verified that the automatically curated QA dataset achieved 100% surgical relevance, 95% accuracy, and included 62% of questions requiring image references to be answered. We note QA pairs not directly related to the images also contribute to downstream tasks by embedding essential surgical knowledge, which is crucial for addressing related topics.

The proposed Visual Question Answering (VQA) models seem more akin to a VLM model than a traditional VQA model.

The model is built on a large multimodal foundation model (LMM) that supports multi-round conversations. The VQA task, involving a single round of conversation, represents a subset of LMM's capabilities. In our case, we fine-tuned and tested the model using a single round of QA, rather than the multi-round visual instruction tuning typically utilized in generative LMMs.

All symbols used in the tables should be clearly defined, such as shading and underlining.

Thanks for the suggestion! We have updated the paper accordingly.

The sampling method lacks temporal association

It is indeed insightful to correlate caption text with temporal frame sequences rather than individual frames. During preprocessing, even after keyframe extraction (images with significant differences), each caption corresponds to an average of 5.25 frames. Although adopting a sequence-to-text mapping would enable more precise learning of visual and textual embeddings, we simply map each frame to the caption and still achieve the SOTA performance across four anatomical detection datasets, excelling in all standard vision-language pre-training evaluation protocols: cross-modal retrieval (Table 1), linear probing (Table 2), and zero-shot/few-shot classification (Figure 5). We plan to explore sequence-text data in future work as existing benchmarks primarily evaluate single image-text pairs. Additionally, we have included a discussion of this limitation in the paper's discussion section. Thanks again for the insightful suggestion!

Multimodal associations between images and texts.

Apologies for the confusion; we did not perform any sampling. We would like to clarify that the image-text correspondence has been carefully maintained throughout the entire process. While the image and text data are initially processed through independent cleaning branches, both retain their original timestamps, which provide a strong foundation for dual-modality alignment. In the final step, these precise timestamps are used to align the cleaned image and text data, ensuring both topic relevance and accurate correspondence. Overall, we did not diminish any multimodal associations present in the raw video. On the contrary, the keyframe extraction process removes redundant and endoscopy-irrelevant frames, which enhances the multimodal association by eliminating noise.

Add SurgGPT in comparison

We have updated Table 3 to include SurgGPT (advanced version of surgicalVQA) in the comparison. Notably, specialized models like SurgGPT tailor different models for each specific testing task, whereas our model is general-purpose: a single model for all tasks. Despite the inherent disadvantage in such a comparison, our model still achieves superior result on Cholec-VQA and comparable result on EndoVis-18-VQA dataset.

Surgery-specific tasks

• EndoVis18-VQA and Cholec80-VQA datasets (table3) we used for visual question answering tasks contain phase/tool recognition questions.
• Kvasir, Hyper-Kvasir, NBI-InfFrames, and GastroVision datasets (figure5 and table2) are all for anatomical detection. The categories of Kvasir include polyps, two classes related to polyp removal and three anatomical landmarks in the GI tract. The anatomical-landmarks in Hyper-Kvasir dataset include cecum, ileum, pylorus, retroflex-rectum, retroflex-stomach and z-line. The classes in the NBI-InFrames dataset include tissue with intraepithelial papillary capillary loop-like vessels, leukoplakia, hypertrophic vessels, and healthy tissue. GastroVision also exhibits a diverse range of classes, including anatomical landmarks, pathological findings, polyp removal cases and beyond.

The zero/few-shot performance of a VLM trained on the proposed dataset may be expected.

Table1 and Figure5 shows performance comparison on various datasets in zero-shot and few-shot settings respectively. Our models outperform all the counterpart models on four anatomical detection datasets under both settings.

Fine-tuning LLaVA on different datasets but evaluating on the same benchmark surgical datasets

Table3 shows comparisons on LLaVA fine tuned with different datasets and tested on the same benchmark surgical datasets: ‘LLaVA’ represents fine tuning with the corresponding training set, while ‘LLaVA + ours’ represents fine tuning with our data plus training sets.

1. Although models like MedFuse or SurgicalGPT are only trained and tested on one single test, they are also much more lightweight compared with MLLMs, with better deployment possibilities in narrow & accurate scenarios.
2. Zero/few-shot studies on surgical phase recognition are lacking (e.g., the authors may use AutoLaparo & MultiBypass140). The current surgical understanding evaluation has involved the related training set in the training process, which cannot demonstrate the power of MLLMs.
3. SurgVLP is a work on surgical phase and tool recognition. I do not understand why the authors compare it with the pathological and endoscopic diagnosis datasets. Comparing it with surgical understanding tasks is more reasonable.

Comparison with specialized models

Thank you for your suggestion! For a fair comparison, we use one single model to test on two phase/tool recognition datasets for overall performance. While validation on external testing datasets is a critical and necessary practice in real-world medical applications, specialized models often struggle with cross-domain testing (as shown below). Our model demonstrates superior performance on in-domain testing for Cholec80-VQA and comparable results for EndoVis18-VQA. Moreover, we achieve a significant improvement on overall performance (generalizability), calculated as a weighted average based on the dataset sizes (#2769 for EndoVis18-VQA, #9096 for Cholec80-VQA).

Zero/few-shot studies on surgical phase recognition

As you suggested, we test accuracy on AutoLaparo and MultiBypass140 under zero-shot and few-shot (1/4/8/16) settings. Overall performance is calculated as a weighted average of the three metrics, using the number of images in each dataset as weights: AutoLaparo contains 28,060 images, while the Bern and Strasbourg subsets of MultiBypass140 include 92,842 and 139,324 images, respectively. Due to time constraints, we compare ours with the most competitive counterpart, SurgVLP (Yuan et al., NeurIPS 2024) for the few shot settings. Notably, our model achieves the highest overall performance across all settings.

• Zero-shot (%)

• One-shot

• 4-shot

• 8-shot

• 16-shot

Anatomical detection tasks

Anatomical detection is a critical subtask in surgical procedures: it serves as a foundation of computer-assisted surgery. Under one-shot and few-shot settings, our model consistently demonstrates state-of-the-art performance across two surgical phase recognition datasets and four anatomical detection datasets, highlighting its remarkable generalizability for real-world surgical applications.