Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Data

Thank you for your thoughtful review and insightful comments. We provide detailed responses to your questions and concerns below.

I'd like to ask how the frames were extracted. Were they keyframes, sampled at regular intervals, or something else?

R: We extract frames by sampling them at regular intervals from the videos. Specifically, for video recognition tasks, we uniformly sample a fixed number of frames (e.g., 15) at equal time intervals to ensure consistent temporal coverage across the video.

The model heavily depends on a weakly-supervised dataset generated from surgical lecture videos, which, despite its scale, may introduce considerable noise due to the lack of domain-specific annotation quality control. If we could analyze the approximate distribution of entity keywords(tools or actions, etc.) in the captions, it would allow for a preliminary observation.

R: Thank you for pointing this out. To provide a preliminary observation of the entity keyword distribution, we have added a word cloud in Figure 3 (in the revised version) that gives an overview of the tag frequency in our dataset. In addition, to reduce noise in our weakly-supervised dataset, our method employs a domain-specific named entity recognition (NER) model to filter medical-related terms from the captions. Additionally, clinician experts were consulted to refine and validate high-frequency tags to ensure their relevance to surgical procedures. This multi-layered quality control approach helps maintain a high level of accuracy and domain-specific relevance in the dataset.

While the temporal-attention fusion layer is a key innovation intended to enhance video-based recognition, the paper lacks a rigorous justification for why this layer outperforms simpler alternatives, such as average pooling or recurrent networks.

R: The temporal-attention fusion layer is designed to capture temporal dependencies between frames, which are critical for recognizing dynamic surgical actions. We also experimentally validated the effectiveness of the temporal fusion layer, as demonstrated in Table 4. For instance, temporal attention achieves a significant improvement of 6.0 mAP in action recognition tasks over the average pooling baseline, highlighting its effectiveness for capturing temporal information.

This architecture includes a label decoder and a text decoder, where the text decoder aligns the visual content with the transcript during training. However, is there potential redundancy between the label decoder and text decoder modules? Would an ablation study be necessary to verify this?

R: The text decoder is implemented for text reconstruction following the setup in the original RAM paper. Its primary purpose is to provide additional textual semantic information during training, rather than serving as a core contribution of RASO. Given its auxiliary role, we did not conduct an ablation study specifically on this module.

We greatly appreciate your thorough review and constructive feedback. Your comments and suggestions were instrumental in improving both the clarity and quality of our paper. We have carefully read through them and made the following changes in our revised paper:

• We have changed Endovis18 in the experiment section to RSS (L319).
• We have added a reference to Fig. 1 (L93).
• We have updated Fig. 2 to improve the text decoder part.
• We changed the paragraph starter from discussion to implications (L235).
• Updated the references if applicapable (L608, L623, L630, L539, L535).
• We have removed replicated sub-tables in Table 4 and added an ablation study for supervised video recognition.
• We have updated the description to the image-based method for video recognition. (L431).

We address the other questions below:

Line 184ff: N and T both represent a number of features, the final embedding size is NxTxD, its unclear what the difference between N&T is and why there are two numbers of frames.

R: N refers to the number of frames sampled from the video, while T represents the number of image tokens extracted from each frame. The embeddings of one frame are TxD, and the embeddings for N frames are NxTxD. We have clarified this in the revised version.

Line 260: How are the generated tags split into the pretraining and fin-tuning set?

R: Response The pretraining dataset consists of labels generated from our label engine, as described in Section 3, while the fine-tuning dataset includes VLM-annotated tags. For supervised experiments, the fine-tuning dataset also incorporates the training data from CholecT50.

Table1: EndoVis18 (RSS) is listed with 997 images. The dataset consists of 19 snippets with 300 frames each, why/how was it down sampled?

R: EndoVis18 (RSS) has a public test split. We use the test split without any changes, which is publicly available here. The test split includes four sequences with 249, 248, 248, and 248 labeled frames, totaling 977 images.

Table2: why aren't the values of P & R highlighted? Esp. for P the text references the models superiority, this should be visualized in the table.

R: Highlighting precision (P) or recall (R) individually can be misleading without considering the balance between the two. For example, on the CLIP dataset, CLIP achieves a recall rate of 64.7%, but its precision is only 16.6%. In contrast, RASO achieves a recall rate of 43.3% with a significantly higher precision of 27.0%. Furthermore, by adjusting the threshold, RASO can achieve a recall rate of 64.8% with a precision of 21.8%. The inclusion of P and R in Table 2 aims to provide a more comprehensive understanding of model performance, rather than emphasizing one metric over the other. For more direct comparisons, metrics like F-beta and mAP, which aggregate both precision and recall, are more appropriate. We believe these metrics better reflect the overall performance and effectiveness of the models in question.

Why is the Fbeta with beta=0.5 instead of the more common F1 score used?

R: The choice of F-beta with β=0.5 instead of the more common F1 score (β=1) reflects our effort to balance precision (P) and recall (R) more effectively. The β parameter adjusts the relative importance of P and R. While F1 gives equal weight to both metrics, we found that using β=1 often results in excessively high recall at the expense of precision for most methods. To address this imbalance, we chose β=0.5 to give slightly more weight to precision, which is critical for tasks requiring high specificity. For example, the table below shows the performance on the GraSP dataset, comparing results for β=0.5 and β=1.0 (F1 score):

Line 417ff: Figure2 introduces the RASO architecture to include a temporal-attention fusion, but the video recognition paragraph compares the video aggregation against a frame-wise approach? How does the frame-wise method work? Which features are aggregated?

R: For the frame-wise method, we run inference for each frame independently, and the predicted labels from all frames are aggregated using a union operation. Thus, we don't aggregate the features here.

Thank you for taking the time to review our paper and for providing valuable feedback. Your comments have helped us identify areas where further clarification and improvements were needed. We have addressed your concerns in detail below.

The work presents a modification to a previous RAM model and a pipeline for data generation. Even though leveraging web-based data from unstructured medical sources can be relevant for medical machine-learning purposes, I am unsure about the significance of the contribution from the general machine-learning perspective.

R: Our work makes significant contributions to general machine learning by addressing both a fundamental data challenge and a structural limitation in current recognition models. First, our approach tackles a key ML challenge—constructing datasets in domains where manual annotation is costly or requires expertise. By leveraging domain-specific cues, such as transcripts and voiceovers, our pipeline generates accurate image-label pairs with minimal manual intervention. This general problem exists across many fields, making our pipeline broadly applicable. For instance, beyond surgery, it can be adapted to instructional videos in general medicine, chemistry, or mechanical engineering, enabling tasks like automated knowledge extraction or content summarization. Second, RASO is a novel video recognition structure that extends existing image-based models to effectively process videos. We introduce a temporal-fusion layer to capture temporal dependencies between frames, enabling efficient video recognition.

The work introduces a methodology to generate weak annotations, which can also make it possible to fine-tune existing foundation models with extensive surgical data. What are the additional benefits of the introduced model compared with standard foundation models fine-tuned on the weakly annotated data?

R: Compared to traditional foundation models like CLIP and surgical domain-specific models such as SurgVLP, RASO addresses a key limitation in the reliance on image-text pairs for training. These models often suffer from noise introduced by irrelevant textual content when using weakly supervised data, which hampers their ability to learn surgical-specific visual concepts. RASO mitigates this issue through a joint training approach that integrates both image-tag and image-text pairs, preserving the richness of image-text information while leveraging the precision of image-tag pairs to improve recognition of visual concepts. As shown in Table 2, RASO outperforms SurgVLP with mAP improvements of 2.9, 4.5, 10.6, and 7.2 across four benchmarks, demonstrating its superiority in surgical object recognition tasks.

It is interesting to see that in Table 4, average pooling performs better than temporal attention in the instrument recognition tasks. What would be the main insights regarding this result?

R: In tasks involving the recognition of static tags, such as surgical instruments or anatomical structures, the goal is to identify the union of static tags appearing across frames. Since the order of video frames does not affect the result for these static tags, methods like average pooling that disregard temporal information can still perform effectively. However, for recognizing surgical actions, temporal information is crucial, as these actions depend on the sequential dynamics between frames. While Table 4 shows that average pooling outperforms temporal fusion by 1.4 mAP in instrument recognition tasks, the temporal attention mechanism brings a significant 6.0 mAP improvement in action recognition tasks, highlighting its importance for tasks that require capturing temporal dependencies.

Figure 3 is written WshiperX, probably instead of WhisperX.

R: Thank you for pointing this out. We have corrected the spelling in Figure 3.

We sincerely thank you for your thoughtful feedback and detailed comments. We appreciate your recognition of our efforts in presenting a well-structured paper and conducting comprehensive evaluations. Below, we address your concerns and clarify the points you raised.

It is not clear in Figure 2 how the embeddings from I and Vs are processed in the Tag Decoder. How does the network balance the importance of information coming from images or videos?

R: We appreciate your observation. For image inputs, embeddings are generated using the Image Encoder and passed along with tag embeddings through the Tag Decoder to produce tag predictions. For video inputs, we uniformly sample 15 frames from the video and process each frame through the same Image Encoder to generate frame-level embeddings. These embeddings are then combined using a temporal-fusion layer, which captures temporal dependencies across frames, to form a unified video embedding. This aggregated video embedding, together with tag embeddings, is then passed through the Tag Decoder to generate predictions. We have updated the Figure 2 by adding a concat operation to explain how frame tokens are aggregated more clearly. Our network does not explicitly "balance" the importance of images and videos; instead, it relies on the shared Image Encoder for consistent frame-wise feature extraction and the temporal-fusion layer for capturing video-specific temporal information. This design ensures that both image and video inputs are projected into a unified visual representation space.

It was not clear why pre-training and fine-tuning were performed for a small number of epochs. Does the loss saturate when fine-tuning for 4 epochs only?

R: Our choice of a small number of pretraining and fine-tuning epochs stems from the scale of our weakly-supervised dataset and the pre-initialization from RAM weights. The pretraining dataset includes approximately 900K images, allowing RASO to learn effectively within 10 epochs. For fine-tuning, we find 4 epochs to be a sweet spot, balancing performance and training time. The fine-tuning dataset is much smaller than the pretraining dataset, which helps mitigate the risk of overfitting during fine-tuning.

VLM automated annotation: it is not clear the effect of the ability of GPT-4o in generating accurate annotation. An assessment of this step is needed.

R: We recognize the importance of evaluating the quality of GPT-4o-generated annotations. The intuition behind using GPT-4o for annotation is that we observed GPT-4o tends to use common tags with reasonable accuracy, which complements the broader tag coverage of the pretraining dataset. This synergy enhances the model's performance on frequently occurring tags. To this end, we conducted an assessment by randomly sampling 1,000 annotated images and assigning them to seven clinicians for expert tag annotations. The agreement rate, defined as the percentage of tags matched between GPT-4o annotations and clinician annotations, was calculated for 20 frequent tags. The results are as follows:

These results indicate that GPT-4o provides annotations for common surgical objects with moderate accuracy, offering a reliable foundation for weak supervision.

Explain or confirm the main technical differences with RAM

R: The primary technical difference is that RASO extends RAM's capabilities from image-only tasks to video recognition with improved accuracy and effiency. This is achieved through the introduction of a temporal-fusion layer that captures temporal information across video frames, enabling more robust video-based recognition. Additionally, RASO introduces a general and scalable data generation pipeline to construct a large-scale weakly-supervised surgical dataset from surgical lecture videos, significantly enhancing its domain-specific applicability.