Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation

[Q1. Plan to Expand Dataset] Scaling and diversifying the surgical vision-language pretraining dataset is challenging due to privacy concerns and the cost of expert annotations. Even though the SVL pretraining dataset covers diverse laparoscopic surgeries, it lacks surgeries in different organs, such as the brain and heart. We fully agree with the reviewer that expanding the dataset would be crucial for developing more generalizable models. To address this, we plan to expand the pretraining dataset using diverse media such as textbooks, instructional videos, and intra-operative video recordings from diverse sources. We also aim to diversify the pretraining dataset by considering laparoscopic, endoscopic, and microscopic surgeries on different organs.

[Q2. Transcription Errors and Solution] ASR errors can be divided into two categories: misspelling errors and incorrect punctuation. In this work, we address these errors using specific methods to mitigate their negative impacts:

• Misspelling Errors: We first follow [1] and use multiple text preprocessing to correct the ASR errors first, including USMLE-based spelling checking. Then, we use the LLM to generate large-scale clean descriptive texts that detail the steps for diverse surgeries. Noisy transcriptions are then assigned to semantically similar step texts based on their semantic similarity, as shown below:

  < Table S2>. Original noisy transcription and augmented text based on LLM.

  Raw Transcript	Assigned Step Text
  but you think about two richard deliver from with amoxicillin truck and to work with your left hand with the right four unit	Placement of five trocars: typically one for the camera, two for the surgeon's instruments, one for the liver retractor, and one for the assistant.
  so I be try to prepare my posterior dissection on the joel	Dissection of the Esophageal Hiatus: Dissect the phrenoesophageal ligament to expose the esophageal hiatus.

  This ensures that the noisy texts are aligned with accurate descriptions. The use of the above LLM-based augmentation significantly improves the model's performance by providing cleaner and more accurate training data. As shown in the table below, SurgVLP shows improvement in accuracy when pretrained with augmented transcriptions.

  < Table S2>. Zero-shot phase recognition performance on Cholec80 dataset.

  	SurgVLP	SurgVLP + LLM Augmentation
  Accuracy	34.7	36.1
  F1-Score	24.4	26.8

• Incorrect Punctuation Errors: We follow SurgVLP and use Whisper, an ASR system based on a language decoder, which generates complete sentences with fewer punctuation errors. Whisper-transcribed sentences help define video correspondence boundaries more accurately.

We anticipate that the modeling-based approach can also be applied in the future. For example, developing ASR systems specifically trained on surgical data can further reduce errors and improve transcription accuracy. Also, pretraining text encoders on a surgical corpus [2] can make the model more robust to ASR errors.

[1] Ikezogwo, Wisdom, et al. "Quilt-1m: One million image-text pairs for histopathology." Advances in neural information processing systems 36 (2024).

[2] Bombieri, Marco, et al. "Surgicberta: a pre-trained language model for procedural surgical language." International Journal of Data Science and Analytics 18.1 (2024): 69-81.

[Q3. Computational Overhead] We manage the computational overhead associated with hierarchical pretraining and dynamic time-warping (DTW) processes by using efficient resource allocation and high-speed computational infrastructure. Below is a summary of the time and computational cost when applying hierarchical pretraining and DTW processes:
< Table S3>. Training configurations and time for SurgVLP.

In this work, the speed bottleneck is at the online video loading and preprocessing. In future, we aim to apply the asynchronous processing pipelines to further parallelize the workload and reduce bottlenecks.

[Q4. Acknowledge the Plans to Mitigate Data Size and ASR Errors ] Thank you for the insightful suggestion. In the revised manuscript, we have added a discussion section to elaborate the strategies about expanding dataset and improving transcription accuracy, as we discussed in prior responses.

[Q5. Potential Negative Impacts] Thank you for the insightful suggestion. As discussed in the paper, our dataset is compiled from open educational platforms accessible to all learners, minimizing any potential societal impact. However, in the rapidly advancing field of surgical data science, there are still risks of privacy concerns and data security associated with the collection and use of surgical video data. Ensuring robust data protection measures and maintaining patient confidentiality is crucial to mitigate these risks.

Moreover, the development of automated intra-operative surgical procedural understanding systems should aim to provide intuitive cognitive aids for surgeons, enhancing their decision-making during complex procedures. These systems should support surgeons by delivering relevant information at critical moments, rather than overwhelming them with extraneous details. We have added a discussion on the potential negative societal impact in the revised manuscript.

[Q1. Transcription Errors and Solution. ] In the following Table 1, we quantitatively demonstrate the impact of LLM-based augmentation and transcription processing strategies on transcription accuracy for the zero-shot phase recognition on Cholec80 dataset. We train the SurgVLP model using Clip-level text Augmentation which uses ASR transcripts enhanced by LLM-based augmentation and before mentioned transcription processing strategies. The results are shown in the first two rows on the table. The SurgVLP model when trained with the Clip-level text Augmentation, including transcription processing strategy and LLM-based augmentation, brings – 2.2% improvement in the F1 score.

Similarly, we also train the HecVL model using processed transcripts without LLM-based augmentation, resulting in improved performance compared to the baseline HecVL. Notably, the improvements from transcription processing can be further enhanced when combined with LLM-based augmentation.

[Q2. Computational Overhead] Thank you for your follow-up comment. In this work, we train directly on the surgical videos without converting them into frames. As a result, a key bottleneck lies in the online video decoding process, where videos must be decoded and processed in real-time to provide training batches. Currently, we are using ffmpeg for the online video decoding process followed by Pytorch to process the decoded frames. In future, we plan to implement a more efficient multi-threaded video decoding pipeline where multiple videos can be decoded in parallel on GPUs using advanced libraries such as NVIDIA Data Loading Library (DALI) [1] or Decord [2]. We expect to minimize the idle GPU time.

[Potential Negative Impacts] Thank you for a follow-up comment. To address patient consent and data privacy in the collection of intra-operative surgical videos, we will ensure that all recordings are fully anonymized by removing any identifiable patient information and out-of-body frames using open-source tools [3]. Specifically, since our focus is on laparoscopic videos, anonymization will involve removing any captions or metadata that could identify the patient, as well as ensuring that any external out-of-body views or patient-identifiable features captured in the video are thoroughly anonymized.

[1] https://developer.nvidia.com/dali

[2] https://github.com/dmlc/decord

[3] Lavanchy, Joël L., et al. "Preserving privacy in surgical video analysis using a deep learning classifier to identify out-of-body scenes in endoscopic videos." Scientific reports 13.1 (2023): 9235.

[Q1. Dataset Limitations] Thank you for the insightful suggestion. In the rebuttal letter pdf, we have added a table to summarize the top 42 types of surgical videos and their amounts in the pretraining dataset. As shown in Table 1 of the rebuttal letter PDF, the SVL dataset predominantly consists of laparoscopic surgeries and covers a diverse range of surgical types, including stomach, duodenum, hernia, colon, gallbladder, and tumor surgeries. This diversity ensures that the SVL dataset provides broad and generalizable language supervision for surgical multi-modal representation learning.
In this paper, the downstream datasets are Cholec80 (Laparoscopic cholecystectomy), AutoLaparo (Laparoscopic hysterectomy), and MultiBypass (Laparoscopic gastric bypass). Our SVL pretraining dataset contains sufficient videos that cover the visual concepts required for these downstream tasks. We agree with the reviewer that expanding the pretraining dataset to cover more types of surgeries can improve diversity and improve generalizability. We leave this exploration to future research endeavors.

[Q2. Generalizability] Thank you for the insightful question. Evaluating the generalizability of our system across different types of surgeries is crucial for understanding its broader applicability. Our system has been tested extensively on laparoscopic surgeries, including Cholec80 (Laparoscopic cholecystectomy), AutoLaparo (Laparoscopic hysterectomy), and MultiBypass (Laparoscopic gastric bypass). These evaluations demonstrate the model's ability to effectively recognize phases and understand the workflow in these contexts.
Since we do not have ophthalmic surgical videos in our pretraining dataset, we do not test the PeskaVLP’s performance in this domain. Future work would involve incorporating and testing diverse surgical datasets, including ophthalmic surgery, to ensure the model's effectiveness across various surgical domains. This will help us assess the system's capability to generalize and perform effectively in a broader range of surgical contexts.

[Q1. Augmentation Removes Variation] Thank you for pointing out one of the key insights of this work, i.e., using LLM to build a large, versatile, and accurate surgical knowledge base to enrich and correct narrations of different types of videos during the pretraining. Since we enrich the narration based on the built knowledge base, the question becomes if the LLM-generated surgical knowledge base is diverse enough.

In this work, we aim to bring the diversity and versatility of the built knowledge base by curating 917 lecture titles, covering diverse surgical procedures such as colorectal, transanal and proctological, cholecystectomy, hernia, and sigmoidectomy surgery. We also manually design input-output examples to instruct the LLM to generate diverse steps. Additionally, during the pretraining, we randomly select either the pseudo step or the original narration text to maintain textual semantics.

Our approach might face risks when LLM generates an over-standardized surgical step knowledge base. However, our experiments show that the downstream zero-shot performance clearly improves when the LLM-generated knowledge base is applied. This implies that the advantages of correcting noisy narration texts outweigh the potential variation and diversity risks.

< Table S1>. Zero-shot phase recognition performance on Cholec80 dataset.

[Q2. Other Components] In the following table, we show that the combination of visual self-supervision and language supervision at the clip-level vision-language pretraining are effective even without the knowledge augmentation.

< Table S2>. Zero-shot phase recognition on Cholec80 and Autolaparo datasets.

The results indicate incorporating knowledge augmentation and visual self-supervision can individually benefit the pretraining performance.

[Q3. Different Levels of Knowledge Augmentation] Knowledge augmentation from different hierarchies improves the final performance in different ways. We have summarized results in the table below: < Table S3>. Performance on Cholec80 dataset with different augmentations.

We observe that applying knowledge-augmented text data in SurgVLP and HecVL achieves a clear improvement, though they still underperform the PeskaVLP. These results demonstrate the effectiveness of the other components of PeskaVLP, i.e., combination of visual self-supervision and language supervision and procedure-aware pretraining objective.

[Q4. Overly Standardized Texts] We thank the reviewer for this interesting question. We apply the LLM augmentation strategy for pretraining by considering the goal of representation learning, downstream surgical workflow analysis applications, and the nature of pretraining surgical lecture videos. The goal of PeskaVLP is to learn the correspondence between vision and language modality with the aim of improving the performance on the surgical downstream tasks. Therefore, the jargons and interjection can introduce the noisy alignment, which is filtered out in our pretraining dataset.

Our pretraining videos are primarily lecture videos where narrations are scripted and more formal. The GPT-augmented strategy is well-suited for this type of data, as it enhances the clarity and completeness of the text.

We acknowledge the potential limitations in real-world applications. For future work, incorporating more diverse and real-time surgical audio [1] into the pretraining process could help improve the performance.

[1] Jia, Shuyue, et al. "MedPodGPT: A multilingual audio-augmented large language model for medical research and education." medRxiv (2024): 2024-07.

[Q5. Modality Gap HecVL] In the rebuttal letter PDF, Figure 1, shows HecVL's embeddings, showing a smaller modality gap for video-abstract pairs compared to SurgVLP. This shows the benefit of hierarchical vision-language pretraining for aligning long-form videos with high-level summaries. PeskaVLP consistently outperforms prior baselines, demonstrating its effectiveness in closing the modality gap and enhancing pretraining for vision-and-language tasks.

[Q6. Table 3] Thank you for pointing this out. We have corrected the table to highlight Moco (third row) as providing the best results on the Cholec80 dataset and Moco (second row) on the StrasBypass70 dataset. In the revised manuscript, we have discussed that Moco pretrained on Cholec80 outperforms the others because it is specifically trained on cholecystectomy procedures, thereby losing versatility. This specialization allows Moco to achieve superior performance on Cholec80 but limits its generalizability to other datasets, as summarized below:

< Table S4>. Correct version of table 3 in manuscript.

[Q1 SVL Dataset]

[Q1.1. Types of surgeries in SVL dataset] In the rebuttal letter PDF, we have added a table summarizing the top 42 types of surgical videos in the pretraining dataset. As shown in Table 1 in the rebuttal letter PDF, the SVL dataset predominantly contains laparoscopic surgeries, focusing on the stomach, duodenum, hernia, colon, gallbladder, and tumors. Also, the diverse content within each surgical type ensures that the SVL dataset provides generalizable language supervision for representation learning.

[Q1.2. Suitable for Pretraining] This paper uses Cholec80, Autolaparo, and MultiBypass datasets for downstream tasks, with Table 1 showing that the SVL pretraining datasets sufficiently cover the required visual concepts. The SVL videos focus on laparoscopic surgeries, covering common concepts like instruments and bleeding, while their descriptive texts offer valuable language supervision for tasks like instrument recognition and adverse event detection.

[Q1.3. Affect Downstream Dataset] We hypothesize that the composition of different types of surgical videos in the pretraining dataset will affect performance. While research [1] suggests that altering the pretraining dataset to the downstream dataset improves zero-shot adaptation, we aim to build a generalizable pretrained model and thus do not filter videos based on the downstream datasets.

[1] Datacomp: In search of the next generation of multimodal datasets

[Q2. LLM-Generated Text for Clip-level Alignment] In this paper, we control the LLM-generated textual quality based on the in-context learning ability of LLMs, which enables faithful predictions with a limited context. By thoughtfully designing contextual input-output examples, LLM can generate diverse and rich texts that enrich the clip-level narration texts, as shown below:

< Table S1>. Original noisy transcription and assigned step text based on LLM.

The generated surgical steps from LLM are not always perfectly aligned with the clip-level frames. To address this, we randomly pick either the original or the augmented texts during video-language pretraining.

Noisy image-text alignment is a challenge for all CLIP-based methods. Scaling up the dataset could help mitigate this issue by providing a broader context and improving alignment, and we leave this exploration to future research.

[Q3. Textual Similarity] Textual similarity is calculated using cosine similarity between feature vectors from pseudo steps and narrations. We use BioClinicalBert to extract 768-dimensional embeddings for each sentence, normalize them, and compute the dot product to get similarity. This approach is consistent with methods used in previous literature.

[Q4. Effectiveness of Three Levels Augmentation] Knowledge augmentation from different levels improves performance in different ways, as summarized in the table below:

< Table S2>. Performance on Cholec80.

We found that phase-level knowledge augmentation significantly improves surgical vision-language pretraining due to its concise and less noisy keystep texts, whereas LLM-generated surgical knowledge base is less effective due to noisy image-text alignment in pseudo-step assignment.

[Q5. Augmentations Affect Corresponding Text's Semantics] For visual self-supervision, we use spatial augmentations like random cropping and horizontal flipping, which might introduce vision-language misalignment. We find that around 9% of sentences contain spatial words. Since one video maps to multiple sentences, the actual impact from augmentation is less than 9%, which is a less significant source of noise than misspellings and incomplete transcripts. Also, our experiments show that these augmentations consistently boost performance, with their benefits outweighing potential misalignments.

[Q6. Procedural Information] We clarify that the surgical procedural information learned from pretraining is not just a linear sequence of phases but includes complex dependencies among surgical key steps. PeskaVLP learns from various surgical lecture videos, capturing possible paths to complete a surgery. For instance, if one pretraining video includes [.., gallbladder dissection, cleaning and coagulation, ..] and another [.., gallbladder dissection, gallbladder packing, ..], our procedure-aware pretraining can identify multiple possible steps following gallbladder dissection. This procedural information is useful for temporal modeling in surgical phase recognition when applying methods like temporal convolutional networks or task graphs to perform online phase prediction.

[Q7. Table 3] We have updated the manuscript and added discussion. The revised manuscript explains that Moco's strong performance on Cholec80 is due to its specialization in cholecystectomy, which limits its generalizability to other surgeries.

< Table S4>. Correct version of table 3 in manuscript.

Thank you for the detailed responses. The rebuttal has addressed most of my concerns, and I would like to raise my score accordingly.