Zero-Shot Video Semantic Segmentation based on Pre-Trained Diffusion Models

1. The goal of VSS (Video Semantic Segmentation) is to predict the semantic category for each pixel in each frame based on predefined classes. EmerDiff has shown good performance in image segmentation tasks. Would its approach, with the addition of a certain level of temporal modeling, serve as a more suitable baseline or comparison method to validate the effectiveness of the proposed approach?

2. VSS is a highly complex task, and balancing the effectiveness of capturing temporal and spatial information with computational efficiency is key. An important concern is whether the proposed method might increase the computational burden, leading to excessively long training times.

3. Lack comparison and discussion with recent important work on video scene segmentation (VSS) task, General and Task-Oriented Video Segmentation, in ECCV 2024.

4. How was the KNN classifier chosen, and is it the best clustering model?

W1: The comparisons in experiments are weak and can’t show the superiority of these methods. The baselines that are surpassed are all focused on image segmentation, so it does not effectively demonstrate the method’s strengths. Additionally, although the paper compares its approach with other supervised video segmentation methods, it does not achieve state-of-the-art performance. Furthermore, there are no comparisons with other zero-shot video segmentation methods. Therefore, I suggest adding baselines of other zero-shot video segmentation methods to highlight the advantages of the proposed approach.

W2: I understand one of the main innovations of this paper lies in the CBR strategy, but the ablation study does not provide quantitative results for the CBR ablation and doesn’t have enough analysis for it.

1. Technical contributions. The proposed paper seems to heavily rely on existing works, i.e. Emerdiff [Namekata et al., 2024] to extract per-frame semantic features and generate coarse segmentation maps, with additional correspondence-based refinement (CBR) and mask modulation to refine and upsample the coarse segmentation maps. On the other hand, the proposed segmentation map refinement strategy, CBR which employ pixel-wise correspondence has been widely studied in previous work, especially video segmentation related vision tasks, e.g. Tang et al. (2023) as the authors pointed out in the paper and [a-c] etc. While the mask modulation operation is essentially a weighted summation operation to mask out latents and difference map, the overall technical contributions c.f. video semantic segmentation seems incremental and lacking significance.
2. Sufficiency of Experiments. While the focus of the proposed work is zero-shot VSS, several aspects are missing to convincingly support authors' claims. a) It is important to show the performance gain of the proposed pipeline versus naive frame feature aggregation to prove the effectiveness of the proposed CBR and mask modulation. I.e., what is the performance like with SD spatial features + average/max pooling of the features for the final video-level segmentation? Does row 1 and 2 in Table 2 refer to a similar case? If so, it would be better to be clarified in the paper. b) While "zero-shot" being one of the major contributions claimed in the paper, it would be important to compare the proposed pipeline with other foundation models like SAM/SAM2. One additional minor point is that, the proposed pipeline requires GT labels to assign class labels to the segmentation map for the first frame, whether the proposed pipeline is fully "zero-shot" may be argueable.
3. Computation Efficiency. The proposed pipeline for processing videos requires to process per-frame features, while also computing pixel-level correspondence (in the proposed CBR module), which is quite computationally intensive. This has also validated by the authors in the supplementary material. There has been works explored object-level correspondence for more efficient computation, e.g. [b], I wonder if the authors can provide more insights from this aspect.
4. Minor: many references are missing the conference information.

[a] Cheng et al., Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation, NeurIPS 2021. [b] Cheng et al., Putting the Object Back into Video Object Segmentation, CVPR 2024. [c] Sun et al., Alignment before aggregation: trajectory memory retrieval network for video object segmentation, ICCV 2023.

• There is an inconsistent behaviour in the results that needs to be explained. Their results show better performance on VSPW than Cityscapes, when it is widely acknowledged that VSPW is much harder dataset with more categories. For example refer to OCRNet[1] results on VSPW at 36.68% while on Cityscapes at 81.8%. While these are fully supervised models it is still not explained why unsupervised ones will act differently? It seems something is wrong in the setup. Even if the claim this has to do with the dataset resolution, that wouldn't be the case on CamVid and still why fully supervised methods face more challenges in VSPW than Camvid/Cityscapes.

• Is there a way to setup their baseline for EmerDiff to just be evaluated per image? is there a reason why they are only reporting the adaptation to the video segmentation with their proposed context model. Currently, it seems their baseline is still dependant on their own adaptation so a simple comparison to image based baselines that rely on diffusion models will serve the purpose of confirming the benefit from their video based approach.

• It would be interesting to see its performance w.r.t DiffSeg[2] another image based segmentation mining diffusion models with focus on attention maps.

• While not sure but seems their ablation is conducted on part of the VSPW val set, why 30 videos only? os is this the full number of videos in the val set?

[1] Yuan, Yuhui, et al. "Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation. 2021." 1909.

[2] Frick, Raphael Antonius, and Martin Steinebach. "DiffSeg: Towards Detecting Diffusion-Based Inpainting Attacks Using Multi-Feature Segmentation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

1. The label setting and motivation are not clear. In Section 5.1, Evaluation Protocol, it says that the labels from the ground truth of the first frame are used to assign labels to the method's clusters (l. 359-360). However, the paper claims that the method is zero-shot [1], leading to confusion on the task the method is designed for. If the method is using the first frame of the video to assign classes then it should be categorized as Video Object Segmentation [2] and compared to the baselines for this task, rather than Video Semantic Segmentation.
2. The method is not well-validated and deserves further analysis. In Table 2, it is not clear based on the difference in mIOU and mVC whether all components are necessary to achieve the best performance. The analysis does not delve into the contributions of different components (e.g. when the feature aggregation can fail: in l. 472 it says "some occasions" but what occasions?) Other aspects of the model were not validated, such as choosing the features from blocks 6-8.
3. The computational efficiency (training time, inference time and memory use on the GPU) of the non-diffusion-based baseline methods are missing. This is important in assessing the ease-of-use of the method and contributes to one's decision to apply the method.

[1] "Zero-Shot Semantic Segmentation". Maxime Bucher, Tuan-Hung Vu, Matthieu Cord and Patrick Pérez. NeurIPS, 2019. [2] "The 2017 DAVIS Challenge on Video Object Segmentation". Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung and Luc Van Gool. Arxiv:1704.00675, 2018.