Meta-weighted Diffusion Model for Reliable Online Surgical Phase Recognition

W1: (1) We propose a classifier-guided diffusion model to obtian reliable surgical phase prediction by regularizing uncertainties including imbalance phase distribution and low image quality in surgical videos. As for low image quality, in this work, it refers to frame image with high semantic similarity across different phase categories that may raise ambiguity for precise recognition, which is exactly reviewer YAUo concerned. at training time, given an input surgical video frame, our model can output multiple prediction trajectories for the input frame. By calculating the expectation in Eq. 9, we aggregate information from these trajectories and ensure that the distribution of denoised predictions for a given training video frame is more compact around its true label, thus it helps us to regulate uncertainty to some extent. At test time, we can also utilize these multiple trajectories for each test video frame, using PIW and t-tests to evaluate the uncertainty of the denoised predictions, as described in Section 2.3. Intuitively, if the variance of the denoised predictions for a given test video frame is high, the uncertainty for that frame is also likely high, which can be used to remind users to get noticed. (2) As for evaluation, we also employ relaxed metric when unclear boundaries occur. The relaxed metric allows frames within a certain time range at the intersection of two phases to be classified as belonging to either phase and still be considered correct.

W2: In order to assign appropriate weights for different video frames in the case of imbalanced phase distribution, we design a meta learning strategy to optimize ELBO in Eq. 9 based on a balanced meta dataset. Namely, different phases dominate the meta dataset equally. Hence, there should be no concern about the biases for meta-weight net caused by imbalanced meta dataset.

W3: (1) As discussed in W1 (1) for reviewer YAUo, uncertainty regularization for model parameters is already been implemented at training time. That is, given some ambiguous video frames, the model can try it best to give predictions with low variance. (2) As for evaluation, PIW ensures that the model’s predictions cover the range of the true values by providing a reasonable prediction interval, preventing excessive contraction and thereby reducing the probability of false negatives. Meanwhile, the PTST metric measures the effectiveness of the prediction interval in containing the true value, ensuring that the prediction interval correctly covers the target in most cases, thus reducing the probability of false positives. These two metrics work together to balance the width of the prediction interval and the accuracy of covering the true value, statistically lowering the occurrence of false negative and false positive events.

W4: We conduct experiments on the OphNet dataset and replenish results in Table 13 of the appendix A6. As we can see that our model outperforms other competitive competitors. For convienence, we show Table 13 as follows:

Table 13 : Results on the OphNet dataset using metrics following OphNet. The best results are marked in bold.

Q1: (1) In this work, we focus on the task of surgical videos, hence, we employ a simple backbone consists of ConvNext and LSTM to merge temporal features for constructing conditions of MetaDiff; (2) MetaDiff is flexible and could also be generalize to other medical image tasks involving uncertainty issues with only a few modification, that is, we only need to replace backbones for videos with those for images.

W: We explain the writing confusions as follows and will make them clear in the final version of manuscript.

(1) L, C, I in line 38 and line 39 refer to video frame length, category of phase number, and channel number per frame, respectively.

(2) We amend Fig.1 of the manuscript to Fig. 7 of the appendix A6 and try to make it clearer.

(3) We have used the ICLR format citation in our template and the local PDF in our computer is normal with green box citation, however, once we update it online, the green box disappears. Fortunately, we check that the hyperlink feature is still there. We will fix that in the final version of manuscript.

(4, 5) We modify the notation inconsistencies and grammer errors and make more careful proofreading.

(6) The meaning of "modestly" in the context: In order to obtain reliable predictions by awaring the uncertainties conveyed in surgical video frames, rather than calculate one prediction per frame, our model outputs multiple prediction trajectories per frame. At training and test time, we should collect countable amount of predictions per frame to carry out model learning and inference, respectively. And we refer this behavior as "modest collection".

(7) We modify the expression ""we use model the diffusion process endpoint through the incorporation of the conditional embedding.. "" to "we model the diffusion process endpoint ..." by deleting "use".

(8) We amend Fig.2 of the manuscript to Fig. 8 of the appendix A6 and try to make it clearer. Meanwhile, CDM is our proposed parametric diffusion model while MOA is the developed algorithm to opimize parameters of CDM. Combining CDM and MOA forms our MetaDiff model.

(9) We replace all "clip" in Sec. 3.2 with "frame" for clarity.

(10) We bold LoViT (Pr) and modify the corresponding description as: "MetaDiff significantly outperforms most of competitors, such as SKiT and LAST, across various metrics including accuracy (Acc), precision (Pr), recall (Re), and Jaccard (Ja)."

(11) We change "DM" to "CDM" in Table 6 of the manuscript.

Q1: (1) As illustrated in Fig. 7 of the appendix A6 (a modified version of Fig. 1), image quality in this work mainly refers to frame images with high semantic similarity across different phase categories that may raise ambiguity for precise recognition. (2) We propose a classifier-guided diffusion model to deal with uncertainty in surgical phase recognition mainly caused by image quality by leveraging the model’s generative capability. Specifically, at training time, given an input surgical video frame, our model can output multiple prediction trajectories for the input frame. By calculating the expectation in Eq. 9, we aggregate information from these trajectories and ensure that the distribution of denoised predictions for a given training video frame is more compact around its true label, thus it helps us to regulate uncertainty to some extent. At test time, we can also utilize these multiple trajectories for each test video frame, using PIW and t-tests to evaluate the uncertainty of the denoised predictions, as described in Section 2.3. Intuitively, if the variance of the denoised predictions for a given test video frame is high, the uncertainty for that frame is also likely high, which can be used to remind users to get noticed.

Q2: We replenish details on the training time and memory usage in Table 16 of the appendix A6. As for inference, results have been reported in Table 5 of the manuscript. For convienence, we report Table 16 as follows:

Table 16. Ablation studies on training time, CPU and GPU memories during training.

W1: (1) Performance of most of the downstream tasks depend on abilities of pre-trained backbones, and our model in the scope of surgical phase recognition is also no exception. For tasks that ConvNext do not generalize well, if we still want to use our model, other domain adaptation techniques for ConvNext such as fine-tuning need to be considered. (2) We replenish other widely used pre-trained backbones including ResNet and ViT to validate robustness of our model, results are reported in Table 14 of the appendix A6. Again, we claim that such kind of robustness comes from our well-designed conditional diffusion model for modeling uncertainties in surgical videos. For convenience, we also show Table 14 as follows:

Table 14: Results on the Cholec80 dataset using different backbones. The best results are in bold.

W2: We validate our model on an nursing procedure video dataset NurViD [1] and report results in Table 17 of the appendix A6. The results demonstrate that our model also achieves promising performance compared with other competitors thanks to our well-designed uncertainty-awared conditional diffusion model.

[1] NurViD: A Large Expert-Level Video Database for Nursing Procedure Activity Understanding

W3: (1) We have conducted experiments on some practical and challenge situations including semi-supervised and few-shot surgical phase recognition tasks, results have been reported in Fig. 5(a) of the manuscript. As for semi-supervised setting, we downsample the whole video with ratio ranging from 0.2 to 1, and use both the sampled labeled frames and non-sampled unlabeled frames (without labels) to train our model. As for few-shot setting, we only use the sampled labeled frames to train our model. (2) Annotations for surgical phase recognition are inherently coarse-grained and manually labeled. For cases where boundary ambiguity arises due to manual labeling, the paper addresses this issue using the relaxed metric. The relaxed metric allows frames within a certain time range at the intersection of two phases to be classified as belonging to either phase and still be considered correct. (3) When it comes to background frames for non-surgical phases like camera adjustments and so on. We implement experiments on the OphNet dataset as follows: i) ignoring the background frames (ignore bg); ii) ignoring the background labels while using their frames (ignore bgl), and iii) treating the background frames as a single class (use 1 bgl). Results are shown in Table 15 of the appendix A6. For convienence, we report Table 15 as follows:

Table 15: Results on the OphNet dataset regarding the effect of background frames.

W4: We replenish comparisons between our model and reviewer suggested methods including SurgPLAN++, SR-Mamba, and SPR-Mamba in Table 12 of the appendix A6. Our model achieves competitive or even better results compared with the competitors. For convienence, we show Table 14 as follows:

Table 14: Results on the Cholec80 dataset. R denotes the relaxed metric. The best results are marked in bold.

W1: We propose a classifier-guided diffusion model to deal with uncertainty in surgical phase recognition mainly caused by image quality by leveraging the model’s generative capability. Specifically, at training time, given an input surgical video frame, our model can output multiple prediction trajectories for the input frame. By calculating the expectation in Eq. 9, we aggregate information from these trajectories and ensure that the distribution of denoised predictions for a given training video frame is more compact around its true label, thus it helps us to regulate uncertainty to some extent. At test time, we can also utilize these multiple trajectories for each test video frame, using PIW and t-tests to evaluate the uncertainty of the denoised predictions, as described in Section 2.3. Intuitively, if the variance of the denoised predictions for a given test video frame is high, the uncertainty for that frame is also likely high, which can be used to remind users to get noticed.

W2: We further validate the broader applicability of MetaDiff on ophthalmic surgical videos from OphNet [1] and nursing procedure videos from NurViD[2] and report results in Table 13 and Table 17 of the appendix A6. As we can see that our model achieves promising performance compared with the most recently developed competitors. For convenience, we report Table 13 and Table 17 as follows:

Table 13: Results on the OphNet dataset using metrics following OphNet.The best results are marked in bold.

Table 17: Results on the NurViD dataset using the metrics following NurViD. The best results are marked in bold.

[1] OphNet: A Large-Scale Video Benchmark for Ophthalmic Surgical Workflow Understanding

[2] NurViD: A Large Expert-Level Video Database for Nursing Procedure Activity Understanding

Q1: (1) The tasks of PDPP, MS-PCD and ours are different, where the former two are video procedure planning and the latter is surgical phase recognition. As for PDPP and MS-PCD, the initial and end frames are concatenated along with Gaussian noise as condition for diffusion decoder to generate the intermediate frames with one trajectory per frame. On the contrary, our model takes features of the current surgical video frame as condition for decoding multiple prediction trajectories for that frame, which could be collected for reducing and evaluating uncertainty at training and test time, respectively. (2) Rather than intuitively concatenated conditions with Gaussian noise as in PDPP and MS-PCD do, we first model the condition as a Gaussian distrbution at the endpoint of forward process and derive a tractable backward process posterior corresponding to the forward process. Then we introduce an ELBO as in Eq. 9 of the manuscript to optimize parameters of our model.

Q2: (1) As reviewer pointed out that the visual information across different sets of frames may vary considerably, in this work, we mainly use cascaded ConvNext and LSTM to fuse temporal information into the current frame feature so as to set the model aware of such frame variance. (2) Other temporal modeling techniques could also be considered, such as Transformer and its complicated variants. However, in this work, we mainly want to illustrate that modeling uncertainty with our well-designed condtional diffusion model could indeed bring some significant benefits, thus, we only employ a very basic backbone consisting of ConvNext and LSTM for computing conditions.

Dear AC,

Given that the rebuttal phase is about to end, however, it seems that reviewer EE1r hasn't read our responses. Could you please remind the reviewers to check out our responses and we wonder that whether our responses have addressed reviewer EE1r's concerns.

Best wishes.

Paper 8858 Authors