DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut

We thank the reviewer for the helpful comments and appreciate the positive feedback.

Weaknesses

1. The main contribution explicitly states the improvements over the TokenCut and MaskCut models, but does not compare to these models in the experiments.

Unlike TokenCut or MaskCut, our method can provide dense segmentation maps and adapt the number of detected segments based on the visual content of an image. MaskCut and TokenCut cannot do this because they can only detect a fixed number of segments. (l.101-104). Therefore, these methods are not well-suited for an image segmentation task. Technically, MaskCut uses an iterative graph partitioning method and masks the graph nodes associated with detected objects. As a result, each segment is treated as a single object that cannot be refined once detected, which in practice severely limits its ability to identify a large number of objects. To support this claim and following the reviewer’s suggestion, we provide below a comparison between the performance of MaskCut and DiffCut. Overall, the improvement of DiffCut highlights our two main contributions: the quality of our visual features for semantic segmentation, and the capacity of the recursive Ncut algorithm to adjust the number of segments to the visual content of each image. We would be glad to include these results in the final paper.

2. In my view, the comparison of the respective strengths and weaknesses of different base models is not detailed enough. [...]

In addition to the experiments conducted in Section 4.2 (Fig. 3 and 4), we evaluated in Supplementary D the potential of different base models for zero-shot segmentation using a simple KMeans clustering on their features for various datasets. This comparison reveals that among the different vision encoders assessed, the diffusion model shows the greatest potential for zero-shot segmentation, as the quality of clustering on its features consistently outperforms other backbones. Given this, we believe that comparing segmentation results after clustering obtained from different base models would yield similar observations.

Questions

1. The paper should state more clearly that it targets semantic segmentation to avoid confusion with other types of image segmentation. In my view this would have a major impact on the clarity of the paper.

We thank the reviewer for this helpful comment. We will make sure to clearly state and emphasize that this paper targets semantic segmentation to avoid confusion with other types of segmentation.

2. How large is the computational improvement over previous methods? The differences are mentioned in terms of parameter count, but a comparison in terms of runtime would be interesting as well.

The primary bottleneck in graph-cut methods is the eigensystem solving which requires $O(n^3)$ operations. In MaskCut, the graph size remains constant, whereas in DiffCut, it has its maximum size only during the first iteration of the recursive NCut. For each subsequent partition, only the relevant nodes from the original graph are selected, reducing the eigensystem's cost. The concept assignment consists in multiplying the matrix of concepts with the feature map and retrieving the indices of the concepts with the highest similarity.

Following your remark, we provide in the general response a table comparing the runtime execution of MaskCut, DiffCut and DiffSeg. We will be glad to include these results in the final manuscript.

3. In comparison to standard semantic segmentation, the method proposed here predicts unlabelled classes. Are the features consistent enough to allow for an easy (e.g. linear) mapping to the labelled classes across images?

We have not explored this path yet. However, based on the results in Fig. 3 and 4, we believe that such a mapping would work well, as features from different images belonging to the same semantic class show high similarity. Consequently, we estimate that mapping these features to a class label should be consistent.

Limitations

1. I don't fully agree with the statement that the method does not have potential societal impacts. [...]

Thank you for pointing this out. You are correct that our method which reuses pre-trained models in a zero-shot fashion can indeed save computational ressources as well as reduce energy consumption and human labor, contributing to more sustainable AI practices. We appreciate your remark and will ensure to highlight this in our paper.

2. One limitation in my view stems from the fact that the diffusion model used as a basis probably does not work well on all types of images. [...]

Since the diffusion backbone is not specifically trained to generate images in specialized domains like biomedical imaging for example, the method may struggle to transfer to out-of-distribution images. However, this issue could potentially be mitigated by fine-tuning the pre-trained diffusion model on the target data domain. As this limitation was not addressed in the main paper, we will discuss it in a Limitations section added to the Supplementary.

We thank the reviewer for the helpful comments and appreciate the positive feedback.

Weaknesses

1. [...] The proposed method is not applied with SD1.4, which I think would be mandatory. [...] to allow a direct comparison with the original DiffSeg method. [...].

As mentioned in the general response, we emphasize that the ablation study carried out in Tab. 2 uses the same diffusion backbone (SSD-1B) for DiffCut and Diffseg, making the comparison between both approaches fair. Therefore, the large gain of DiffCut over DiffSeg in Tab. 2 directly validates our two main contributions: the choice of the diffusion features and the clustering algorithm. Following the reviewer’s suggestion, we present in the general response a quantitative comparison between DiffCut with SD1.4 and the original DiffSeg method.

2. The DiffSeg results in Tab. 2 seem to be worse than in Tab. 1. I would appreciate a clarification of the effect of using SD-XL features.

The discrepancy between the results of DiffSeg in Tab. 1 and Tab. 2 can be explained by the different backbones used as well as the absence of the refinement module in Tab. 2 as mentioned at lines 293-294. As DiffSeg relies on an attention aggregation process and iterative attention merging, its performance depends on the architecture of the backbone, especially the number and placement of attention modules which significantly differ between SSD-1B and SD1.4.

3. [...] I assume, DiffSeg would again underperform DiffCut. However, having DiffSeg in this table would be a more complete comparison [...].

We agree that having DiffSeg in this table would be a more complete comparison with this particular method. Nevertheless, having demonstrated in previous sections the superiority of DiffCut for generating high-quality masks, the aim here is rather to demonstrate that a straightforward extension to an open-vocabulary setting is competitive with existing approaches targeting this task.

4. When considering Fig. 9, the masks [...] do not always align with the object boundaries [...]

The masks not always aligning perfectly with object boundaries can indeed be attributed to using low-dimensional resolution and relying solely on bi-linear upsampling. The low-dimensional resolution does not capture the fine boundaries of objects, and bi-linear upsampling does not refine the edges to match the true boundaries. Exploring more sophisticated feature upsampling techniques could mitigate this misalignment.

5. [...] I am unsure if the application of the Hungarian matching [...] is fair when comparing with methods that do not apply this strategy [...]

All baselines but ACSeg, ReCO and MaskCLIP are evaluated with Hungarian matching, which is the standard evaluation protocol for methods that provide unlabeled clusters. While ACSeg also evaluates with Hungarian matching, we reported scores with labeled clusters, which are significantly higher (47.1 vs. 53.9 on VOC and 16.4 vs. 28.1 on Stuff-27). Finally, while ReCO and MaskCLIP may not seem directly comparable, we believe that assigning labels to predicted masks can offer more advantageous performance. Hungarian matching performs a one-to-one assignment between each ground-truth label and the single predicted cluster with the greatest overlap, meaning some predicted clusters may not be matched with any ground-truth label. While predicting a class may seem more challenging, it allows assigning a label to every predicted cluster, giving each a chance to be correctly classified, which can ultimately lead to better performance.

6. I like the additional baseline AutoSC [...] which model is faster, AutoSC or DiffCut?

AutoSC is a variant of DiffCut where we replace the recursive clustering by the proposed automated spectral clustering that automatically estimates the number of clusters. Despite its simplicity, it is highly effective and demonstrates competitive performance. Differences mainly appear in datasets containing small objects (e.g., COCO-Object, Cityscapes, ADE20K). DiffCut's recursive partitioning offers flexibility in detecting small objects by allowing large segments to be further divided into smaller ones. In contrast, AutoSC is effective at uncovering the most salient clusters but may overlook smaller objects.

For the set of exponents, we chose the same set we explored for DiffCut. We also examined AutoSC's behavior with higher alpha values, but this did not lead to performance improvements as the index of the largest relative eigen-gap generally corresponds to what is found for alpha in $\\{1, 5, 10, 15\\}$. Lastly, since AutoSC does not require a recursive partitioning of the graph, it benefits from lower runtime execution.

Questions

1. ReCO is first listed training-free [...]?

We thank the reviewer for pointing this out. This is an oversight on our part; ReCO and MaskCLIP should be listed as training-free methods since the reported scores do not involve any training. We will update the table accordingly.

2. [...] I would maybe consider replacing the term robustness with hyperparameter sensitivity (also in G). [...]

We appreciate the reviewer's suggestion and agree that this terminology is more appropriate for these sections. We will update the paper accordingly.

3. [...] Did you also study the results if only 64x64 resolution were used?

We have not analyzed the results using only $64 \times 64$ resolution. Since the image semantics are concentrated in the layers near the UNet’s bottleneck, with the outer layers focusing more on details, we would expect the performance to be less favorable with only $64 \times 64$ resolution.

4. l.339: Could you please discuss this strategy more comprehensively in the final manuscript [...].

We thank the reviewer for this valuable feedback. We will discuss this strategy more comprehensively in the final manuscript with complementary details in Supplementary.

We thank the reviewer for the helpful comments and appreciate the positive feedback.

Weaknesses

1. Writing is a bit too vague [...]. For example, can you demonstrate the "patch-level alignment" with visual examples?

For vision tasks, we expect the vision encoder to be semantically coherent i.e to produce similar patch representations for regions belonging to the same semantic concept. We refer to it as the “patch-level alignment”. We show in Fig 4. and Fig 10. some qualitative results that illustrate the inner patch-level alignment for different vision encoders. We visualize the patch level alignment by encoding an image and computing the cosine similarity between a reference patch and all other patches. For SSD-1B, in Fig 4. the dog is uniformly and strongly highlighted, indicating a strong patch-level alignment. Following the reviewer’s remark, we will refer to Fig.4 at l.168 for better clarity on this point.

2. Are there any visual verification and description for what kind of concepts you get?

In section 3.3, each concept is obtained via a reduction of the spatial dimension of the corresponding segment (Masked SMM l.195) . This way, a single concept has the form of a feature vector that globally describes its corresponding segment. Providing a visual description of a single concept is not straightforward and would imply a form of “decoding” of this concept.

3. The experiments [...] execute with SSD-1B. What about using other SD models? [...]

Following the reviewer’s recommendation, we conduct a new evaluation to demonstrate that DiffCut provides competitive performance with other smaller diffusion backbones than SSD-1B. Specifically, we use SD1.4 and SSD-vega (another distilled version of SDXL). For SD1.4 the UNet encoder has 260M parameters (~ 30% of the UNet), for SSD-vega, the UNet encoder has 240M parameters (~32% of the UNet).

The results obtained with these two backbones are consistent with those achieved using SSD-1B. Although there is a slight performance drop with the SD1.4 encoder backbone compared to SSD-1B, the method still significantly outperforms DiffSeg. Remarkably, DiffCut with SSD-vega UNet encoder achieves performance comparable to SSD-1B, despite being only half its size. We will be glad to add these results in the final paper.

Questions

1. What do you mean by "perform a many-to-one matching"?

In datasets that include a background class, this label implicitly encompasses a variety of concepts related to "things" or "stuff," which vary depending on the dataset. Since our method generates a segment for every detected object in an image, a one-to-one matching between a single cluster ID and the entire background's ID does not accurately represent the model's true categorization capabilities. Therefore, in such cases, we use a many-to-one matching approach by associating the clusters that primarily overlap with the background to its ID. We will be glad to include this clarification in the final paper.

2. Are the hyperparameters the same for DiffCut on different datasets?

As highlighted in the general response and noted in the Implementation Details section (l.230), we used a fixed set of hyperparameters, $\tau$ and $\alpha$, across all datasets for our evaluations. This approach ensures a fair comparison against the baselines and is particularly important for our evaluation of DiffSeg, where we also maintain a single hyperparameter value across different datasets.

Limitations

1. How is the computation efficiency of the proposed method comparing to other methods? [...]

The primary bottleneck in graph-cut methods is the eigensystem solving which requires $O(n^3)$ operations with $n$ being the number of nodes in the graph. In MaskCut, the graph size remains constant, whereas in DiffCut, it has its maximum size only during the first iteration of the recursive NCut. For each subsequent partition, only the relevant nodes from the original graph are selected, reducing the eigensystem's cost. The concept assignment essentially consists in multiplying the matrix of concepts with the feature map.

Following your remark, we provide in the general response a table comparing the runtime execution of MaskCut, DiffCut and DiffSeg. We will be glad to include these results in the final manuscript.

2. The semantic clustering knowledge in pretrained diffusion models might not be able to transfer to data domains like biomedical images.

Since the diffusion backbone is not specifically trained to generate images in specialized domains like biomedical imaging, the method may struggle to transfer to out-of-distribution images. However, this issue could potentially be mitigated by fine-tuning the pre-trained diffusion models on the target data domain. As this limitation was not addressed in the main paper, we will discuss it in a Limitations section added to the Supplementary.

3. If LD, AX, or UA is given, would the proposed method still outperform the corresponding SOTAs? [...]

Since our method is entirely training-free and does not rely on LD, AX, or UA, we believe that incorporating any additional information could further enhance its performance. Specifically, given the superior quality of our masks compared to MaskCut, we believe that integrating our DiffCut method to generate high-quality pseudo-masks within an unsupervised training scheme, would lead to even better performance.

We thank the reviewer for the helpful comments and appreciate the positive feedback.

Weaknesses

1. [Unsupervised Instance or Panoptic Segmentation Performance]

We have not yet evaluated DiffCut’s performance in instance or panoptic segmentation. The goal here is to demonstrate that the features of a diffusion UNet surpass those of other vision backbones in precise semantic segmentation. Given that our method can generate high-quality semantic masks under the most restrictive conditions (training-free and independent of LD, AX, and UA), future work could involve relaxing these constraints and incorporating DiffCut as a high-quality pseudo-label generator within an unsupervised training pipeline—such as proposed in U2Seg [1]. This could significantly enhance performance in unsupervised semantic, instance, and panoptic segmentation.

2. [Technical Contribution and Comparison with MaskCut]

Unlike TokenCut or MaskCut, our method can provide dense segmentation maps and adapt the number of detected segments based on the visual content of an image. MaskCut and TokenCut cannot do this because they can only detect a fixed number of segments. (l.101-104). Therefore, these methods are not well-suited for an image segmentation task. Technically, MaskCut uses an iterative graph partitioning method and masks the graph nodes associated with detected objects. As a result, each segment is treated as a single object that cannot be refined once detected, which in practice severely limits its ability to identify a large number of objects. To support this claim and following the reviewer’s suggestion, we provide below a comparison between the performance of MaskCut and DiffCut. Overall, the improvement of DiffCut highlights our two main contributions: the quality of our visual features for semantic segmentation, and the capacity of the recursive Ncut algorithm to adjust the number of segments to the visual content of each image. We would be glad to include these results in the final paper.

3. [Lack of Comparisons with Previous Works]

We thank the reviewers for these recent references. In comparison with EAGLE [2], we propose a much simpler and effective training-free pipeline that reaches higher results on COCO-Stuff and Cityscapes due to the quality of the extracted features from the diffusion backbone. Their pipeline is based on DINOv1 which we proved in Fig. 3 has worse feature correspondence than our backbone. U2Seg [1] proposes a framework to unify unsupervised semantic, instance and panoptic segmentation. This method only evaluates on COCO-Stuff-27 for unsupervised semantic segmentation. We will be glad to add these baselines to our main Tab. 1.

4. [Adequacy of Low-Resolution Features]

We empirically observe in Fig. 9, 11 and 12 that the excellent quality of the features provided by the diffusion model at a reasonably low spatial resolution $(32 \times 32)$ effectively enables the detection of medium to small-sized objects. Increasing the value of the hyperparameter $\tau$ can even further help to detect objects at a finer granularity. This spatial resolution of feature maps does not present a significant drawback for small object localization.

5. [Recursive Normalized Cuts or Multi-class Spectral Clustering]

In Tab. 2, we explore the use of spectral clustering for multi-object segmentation by introducing a variant of DiffCut called AutoSC. This variant adapts the number of segments based on the visual content using a heuristic called “relative-eigen-gap” which estimates the number of connected components in a graph. This estimated number of connected components, $k$, is then used to determine the number of clusters in $k$-way spectral clustering. We observe that AutoSC also achieves excellent results, often comparable to those obtained with DiffCut.

We note that the main differences between AutoSC and DiffCut appear in datasets containing small objects (e.g., COCO-Object, Cityscapes, ADE20K). DiffCut’s recursive partitioning offers flexibility in detecting small objects by allowing large segments to be further divided into smaller ones, while the “relative eigen-gap” heuristic in AutoSC is effective at uncovering the most salient clusters, potentially overlooking smaller objects.

[1] Niu, Dantong, Xudong Wang, Xinyang Han, Long Lian, Roei Herzig, and Trevor Darrell. "Unsupervised universal image segmentation." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22744-22754. 2024.

[2] Kim, Chanyoung, Woojung Han, Dayun Ju, and Seong Jae Hwang. "EAGLE: Eigen Aggregation Learning for Object-Centric Unsupervised Semantic Segmentation." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3523-3533. 2024.