G4Seg: Generation for Online Segmentation Refinement with Diffusion Models

W1 & Q1

The contribution of this paper is limited, and the idea of reducing discrepancies is relatively incremental. The contribution and novelty could be improved

Limited Contribution We respectfully disagree with this. We'd like to argue that the distinctive aspect of our G4Seg, is the first work to solve the discriminative segmentation task with dynamic generative content mining, which is based on an interesting finding regarding the effectiveness of the generated-original visual bias on the segmentation mask. In this way, we make contributions on performing a plug-and-play training-free refinement framework with the pre-trained diffusion models leveraging the generation pior. We have made a comparison with other segmentation approaches utilizing the diffusion model in Table 1 in the revised manuscript. And a more detailed table compares training-free diffusion-based semantic segmentation methods in terms of the generative content, the strategy mimic generation and segmentation gap, and external discriminative assistance.

In addition to re-claiming our contribution to the methodology, G4Seg also consistently improves the performance both on in-exact segmentation and fully/semi-supervised segmentation via leveraging the generation of prior knowledge.

For fully/semi-supervised open-vocabulary segmentation:

For fully supervised closed set sematic segmentation/panoptic segmentation:

We have added these to the Appendix of the revised paper.

Incremental contribution for reducing discrepancies

Here, we want to emphasize that there is an inherent gap between the generative prior and the segmentation results. For a diffusion model, its training objective is to progressively denoise an image step-by-step from noise. The aforementioned concurrent works are also trying to address this gap, but they overlook the capabilities of generating specific images of the diffusion model. Therefore, we aim to leverage a generative nature of the diffusion model with dynamic generative content mining to tackle a downstream pixel-level discrimination task, without relying heavily upon external discriminative assistance.

W2 & W3 & Q2

The proposed method focuses on discrepancies. For irregular objects, it is difficult to give an accurate generation. Does the proposed method have negative results in these situations? The proposed method has limited improvement by performing a forward diffusion denoising process. More discussions about the discrepancies, and more experimental comparison.

limited improvement and irregular condition We agree with the generation limitation. In some extreme cases, when the object is in irregular shape and initial mask is of poor quality, then the improvements become limited. We stratified the samples based on different quality levels of coarse segmentation and calculated the mean IoU improvement for each group according to their initial mask quality, as measured by the per-sample IoU metric on the VOC dataset:

ControlNet means Explicit Mask Injection using a pre-trained Controlnet As seen from the Table, we make the following comments:

• Irregular shape and bad initial mask. For initial coarse mask for relatively low performance, all three methods(including one fully supervised method) have limited improvements compared to higher initial segmentation.
• Limited Improvement Since we have introduced a mask conditioning method which is based on cross-attention and self-attention, is the performance bottleneck. These attention-based generation methods do not perform well in mask-conditioned generation. If we adopt a stronger mask conditioning method, such as Contronet[7], the performance would significantly improve, as shown in the Table.

We do not adopt such a stronger mask injection strategy because it would violate the "inexact segmentation" setting in our approach.

Discussion of the discrepancies In addition to the theoretical explanations in the main text, we have included more intuitive explanations, as shown in the Appendix F . Furthermore, we found that the primary bottleneck for G4Seg in leveraging discrepancies lies in the method of mask injection. As demonstrated in the table above, if we adopt a more precise mask injection approach, our method's utilization of discrepancies becomes more effective and reasonable.

We have added these discussions to the revised paper with Appendix I.

W4

The proposed method is a refinement processing, which lacks of comparison with existing mask refinement method, such as CRF.

Sorry for the missing comparison. We have conduct a comparison between other mask refinement method on VOC and Context datasets with SCLIP and MaskCLIP.

According to the Table, we make the following comments:

• Although CascadePSP and SegRefiner use many pixel-level labels for training, the performance improvement in exact semantic segmentation is still quite limited.
• DenseCRF, as a method that refines coarse predictions by leveraging the information of image formation, improves the initial segmentation with a significant margin when the number of classes is limited. However, as the number of classes increases, the improvements achieved by DenseCRF become less significant. The improvements of our method are roughly comparable to those of DenseCRF on Context dataset.
• Since the source of segmentation knowledge in our method differs from that of other approaches (CascadePSP relies on annotations, and DenseCRF leverages image-based priors), G4Seg can be further combined with these methods to achieve additional improvements.

W5

It lacks the analysis about the hyper-parameters like injection weight and mixing coefficient. For implementation details, the settings on different tasks are different.

Missing ablation study Sorry for the missing experiments. We add these experiemnts following with different injection values and different mixing coefficient.

Performance on VOC with different injection values

So we finally choose injection value 1.0. Then the mixing coefficient ablation for weakly supervised semantic segmentation is listed as follows: Performance on WSSS VOC with different mixing coefficients

The ablation for text-supervised semantic segmentation on VOC is listed in the following:

Different settings We'd like to emphasize that our experiment settings remain the same for both weakly supervised semantic segmentation and text-supervised semantic segmentation. Our approach utilizes existing base methods to generate a coarse mask, which is then refined using a pre-trained diffusion model. For different experiments, we use different hyperparameters. The widely used DenseCRF follows a similar procedure[9].

The $\beta$ coefficient represents the reliance of initial segmentation confidence. A lower mixing weight is chosen for text-supervised semantic segmentation since the base segmentation performance of TSSS is relatively poor, requiring us to rely more heavily on the corresponding component for further refinement.

W6

The proposed method seems increasing the computational cost during inference according to Table 5.

We appreciate the authors' constructive criticism. As a method that leverages the generative capabilities of diffusion models for post-processing without any annotation, it is natural for our approach to incur additional computational overhead. However, after comparing the computational latency of several mask refinement methods, we found that our approach operates within a reasonable range of computational cost. Finally, the computational cost of diffusion models could be improved with pruning, quantization, or using lighter versions of Stable Diffusion models.

SegRefiner is a fully supervised mask refinement method with a diffusion process

[1] Unleashing text-to-image diffusion models for visual perception. ICCV2023.

[2] Open-vocabulary panoptic segmentation with text-to-image diffusion models. CVPR2023.

[3] Diffusion models for zero-shot open-vocabulary segmentation. arXiv:2306.09316.

[4] Diffusion model is secretly a training-free open vocabulary semantic segmenter. arXiv:2309.02773.

[5] Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation. CVPR2024.

[6] Dataset diffusion: Diffusion-based synthetic data generation for pixel-level semantic segmentation. Neurips2023.

[7] https://huggingface.co/lllyasviel/sd-controlnet-seg

[8] CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement. CVPR2020.

[9] https://github.com/lucasb-eyer/pydensecrf/issues/76

W1

G4Seg is based on Stable Diffusion and suffers from heavy time costs, resulting in poor practicality especially when inference at scale. It would be better to report the specific inference cost for each dataset, not be confined to VOC but the complex COCO.

inference cost on CoCo

Since our method updates one segment with one pass, and the COCO dataset contains an average of multiple categories per image, the overall processing time per image is relatively longer.

Inference cost comparison with other refinement methods After comparing the average inference speed on three datasets of several post-hoc mask refinement methods, we found that our approach operates within a reasonable range of computational costs.

W3

The feature maps in Stable Diffusion are heavily downsampled (e.g., 64x64). Does G4Seg work for small objects?

Since the feature maps are downsampled, as for extremely small objects, we can use a diffusion model with a larger input resolution such as SDXL(128x128 feature map size). Furthermore, resizing the image to a larger resolution and then applying sliding inference also help to detect small targets. We conduct these two strategies on the VOC dataset and observe the performance on small object, which is defined with total occupied object pixels is smaller than 10% of the total pixels in the image.

W5

Detailed parameter sensitivity analyses for the injection weight and mixing coefficient are lost, which is important to assess the robustness of G4Seg.

Sorry for this. Based on the following table, as the increase of injection value (in equations (5) and (6)), the performance meets a initial increase followed by a subsequent decrease. Therefore, we choose 1.0 as our hyperparameter to achieve the best numerical value. We will add this.

Performance on VOC with different injection values

Then the mixing coefficient ablation is following: Performance on VOC with different mixing coefficient

Q1

In Table 6, how to inject boxes, points and scribbles into Stable Diffusion?

Sorry for this. Actually, in the test phase, a coarse mask is obatined firstly based on these visual prompt via the specific segmentation method. Then, G4Seg is applied on top of it. We have modified paper to clarify it.

I am very curious about your pipeline. Since in the CascadePSP paper, the highest improvement in IoU is only 5% IoU among all the datasets, which is in a simpler segmentation task (with only one foreground per image). In more complex multi-class segmentation tasks, it seems that the 10% IoU improvements on SCLIP and the 7% IoU improvements on CLIP-ES are unclear or not well explained. Could you please clarify how these improvements were obtained? Did your reproduction align with the original paper? Could you share your code with an anonymous link?

As you mentioned, the original CascadePSP is designed to process the simple foreground-background case. To apply it to semantic segmentation with multi-class segmentation tasks, we need to first convert the semantic mask to several binary masks for each class (as the step 1 listed above). For example, if the coarse mask contains two classes person and dog, we need to first split it into two binary masks (valued 0 and 255) for each class and perform refinement based on them.

Additionally, the coarse masks used in CascadePSP are generated by fully supervised models like Deeplab V3+, which are already of high quality, making the observed improvements less significant. In contrast, for WSSS, the coarse masks have more room for improvement, so achieving higher gains is reasonable. Furthermore, the metric used in CascadePSP is image-wise IoU, differing from the class-wise mIoU typically used in semantic segmentation tasks. To ensure consistency with baseline methods such as SCLIP and CLIP-ES, mIoU should be adopted instead.

I am also very curious about the difference between our reproductions. I would appreciate it if the authors could correct me and share their reproduction details of CascadePSP on VOC.

W1

The reliance on Stable Diffusion may potentially limit the model's efficiency compared to baseline methods, raising concerns about computational requirements and processing speed.

While there may currently be computational challenges, compared with the other segmentation refinement method using diffusion generation with multi-step inference [1], our method only needs one step of model inference for a specific class.

Furthermore, the model computational cost of diffusion models could be improved with such as pruning, quantization, or using lighter versions of Stable Diffusion. We conduct our experiments with a slighter version of stable diffusion achieving comparable results.

SegRefiner is a fully supervised mask refinement method with diffusion process

W2 & Q3

The improvements are most pronounced when applied to less accurate segmentation models, such as GroupVit, which is trained solely with text labels. When used with stronger base segmentation models (e.g., SCLIP and DiffSegmenter), the performance gains appear to be less substantial, potentially limiting the method's applicability across a broader range of models. Could this approach be extended or adapted to those fully-supervised methods that can already achieve accurate segmentation results, such as SegFormer and Mask2Former?

To show the versability of our method, we have conduct experients under a fully supervised setting on ADE20k with three fully supervised segmentation approaches(SegFormer[4],Mask2Former[5]) with semantic segmentaion and panoptic segmentation(ODISE[6]).

Fully supervised results on ADE20k datasets

We have demonstrated the applicability of our method in fully supervised settings. Note that our method is applied without training or finetuning with any pixel-level annotation.

The reason To further clarify the limited performance on SCLIP and DiffSegmenter comparing GroupViT, we make following explanation. Specifically, the knowledge for mask quality improvements comes from two components: Explicit Mask Injection, which relies on the attention of the diffusion backbone and the CLIP embedding space for correspondence mapping; DiffSegmentor, which relies solely on the attention of the diffusion backbone, while SCLIP depends on CLIP's discriminative capability. Therefore, both methods share some overlap in terms of knowledge sources with ours, which limits the extent of improvement. If we apply another mask injection(Seg Controlnet[3]) and embedding space(Diffusion Backbone Feature), the accuracy improvements of our pipeline would be more significant.

Q1

How does the proposed method handle cases where the initial segmentation is significantly inaccurate or contains multiple errors? Is there a threshold of initial accuracy below which the refinement process becomes less effective or unreliable?

Thank you for the constructive comments. To show how sensitive the proposed method relying on the initial segmentation quality, since our approach is sample-wise, we performed stratification based on different quality levels of coarse segmentation and then calculated the mean IoU improvement for samples with different levels for the VOC dataset:

Table Improvement for Different Initial Performance

When the initial mask quality is poor, the improvement of our method is also limited. The improvement from our method is most significant for initial IoU values between 40 and 80. This indicates that our approach is particularly effective when the initial segmentation is already of reasonable quality. When the initial segmentation is already nearly perfect(80-100), the improvement from our method becomes limited due to the bottleneck caused by errors inherent in the mask injection process.

Q2

Given that the method leverages generative priors from Stable Diffusion, how sensitive is it to domain shifts or out-of-distribution images? Have the authors explored its performance on datasets or image types that differ significantly from those used in training Stable Diffusion?

Thank you for your advice. To evaluate the performance of our method in cross-domain settings, we adopted a baseline[7] for nighttime semantic segmentation on NightCity-fine[8].

Our method remains effective under out-of-domain conditions.

[1] SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Process.NeurIPs2023

[2] https://huggingface.co/OFA-Sys/small-stable-diffusion-v0

[3] https://huggingface.co/lllyasviel/sd-controlnet-seg

[4] SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers. NeurIPs2021

[5] Mask2Former: Masked-attention Mask Transformer for Universal Image Segmentation. CVPR2022

[6] ODISE: Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models. CVPR2023

[7] Disentangle then Parse: Night-time Semantic Segmentation with Illumination Disentanglement. ICCV2023

[8] Night-time scene parsing with a large real dataset. TIP2021

Dear Reviewer ADSn:

Thank you again for taking the time to review our work and for providing valuable and insightful feedback. We sincerely hope the responses have clarified the points raised. If you have any additional questions or concerns, please do not hesitate to reach out—we would be happy to discuss them further.

Best regards,

Authors of Submission 6302

W1

While results are promising, deeper comparisons with similar methods (e.g., training-free segmentation approaches) could be emphasized to highlight advantages and potential trade-offs more explicitly.

We have compared our method with other diffusion based segmentation approaches in Table 1 in the manuscript. Furthermore, detailed and further comparison with similar training free segmentation methods is shown in following table. We dicussed in terms of the Generative Content, the strategy mimic generation and segmentation gap(Gen->Seg) and external discriminative assitance.

Advantage As the table shows, the DiffSegmentor only relies on the attention mechanism in the diffusion backbone as the clue to the target mask, which does not fully excavate the generation prior of the diffusion model. The OVDiff and Freeda utilize many generated images with specific classes and obtain the discriminative prototype of the class, where the prototype is retrieved based on the region of interest from cross/Self-attention aggregation. Due to the strong external discriminative assistance, it is challenging to determine whether the generative capacity of the diffusion model contributes to the performance. Our method aims to fully exploit the generative prior for a discriminative task, specifically inexact segmentation, by adopting a GPT-like approach to solve the discriminative task in a generative manner without overly relying on extra assistance.

Potential trade-offs

Avoiding over-reliance on external discriminative features highlights the generative capacity of diffusion models but may sacrifice performance in challenging segmentation tasks. Furthermore, solving a discriminative task in a generative manner might struggle with tasks requiring precise boundary delineation, which are typically better handled by explicit discriminative features or accurate annotations.

We have added these discussions in the revisions of Appendix M.

W2

Although tested on popular benchmarks, it is not entirely clear how G4Seg would scale to other datasets.

We extended our method to two scenarios: one involving cross-domain datasets and another featuring datasets with a larger number of categories.

We firstly evaluate the performance of our method on a cross-domain setting and adopt a baseline[7] for nighttime semantic segmentation on NightCity-fine[8].

We further conduct our method on ADE20k with three fully supervised segmentation approaches(SegFormer[4],Mask2Former[5]) with semantic segmentation and panoptic segmentation(ODISE[6]).

Fully supervised results on ADE20k datasets

We have added these experiments to our revision with Appendix H.

W3

While computational cost is discussed, there is limited exploration of scenarios where G4Seg may not perform optimally (e.g., highly cluttered or ambiguous images).

Thank you for the constructive comments. To show how sensitive the proposed method relying on the initial segmentation quality, since our approach is sample-wise, we stratified the samples based on different quality levels of coarse segmentation and calculated the mean IoU improvement for each group according to their initial mask quality, as measured by the per-sample IoU metric on the VOC dataset:

Table Improvement for Different Initial Performance

When the initial mask quality is poor, the improvement of our method is also limited. The improvement from our method is most significant for initial IoU values between 40 and 80. This indicates that our approach is particularly effective when the initial segmentation is already of reasonable quality. When the initial segmentation is already nearly perfect(80-100), the improvement from our method becomes limited due to the bottleneck caused by errors inherent in the mask injection process.

[1] Diffusion models for zero-shot open-vocabulary segmentation. arXiv:2306.09316.

[2] Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation. CVPR2024

[3] Diffusion model is secretly a training-free open vocabulary semantic segmenter. arXiv:2309.02773

[4] SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers. NeurIPs2021

[5] Mask2Former: Masked-attention Mask Transformer for Universal Image Segmentation. CVPR2022

[6] ODISE: Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models. CVPR2023

[7] Disentangle then Parse: Night-time Semantic Segmentation with Illumination Disentanglement. ICCV2023

[8] Night-time scene parsing with a large real dataset. TIP2021

Dear Reviewer ervM:

Thanks for reviewing our work and providing insightful comments. We hope our answers have helped to clarify the points discussed. Please let us know if you have further comments or need more discussions.

Best regards,

Authors of Submission 6302

W1

The reliance on a specific class label in the text prompt limits this method to only a single category for a given image. It is not clear how this method extends multiple classes. Running the proposed method multiple times with multiple yet separate class labels would face the problem of post-consolidating, most likely handling contrasting segmentation results outside the diffusion steps, which seems non-optimal.

Multi-class Extension Sorry for the misleading. Our current approach does indeed involve running the process multiple times with separate class labels in order to obtain results for different categories. For each specific class, We treat all other segments as background and update the current segment logit independently. Then the final segmentation is refined with updated logit after normalization. We have added these illustrations in the paper in Sec 4.1.

Non-optimal post-consolidating We did not incorporate contrasting segmentations of different classes within the diffusion step for the following two reasons:

• The challenge of handling the composed injection of different classes in a single pass. We did a validation experiment for injection with multi-classes with concatenation of different class names as prompt and restrict their corresponding cross-attention and self-attention. The results are presented in the following table. Performing such multi-class injection significantly degrades the overall performance, as the interference among classes might affect the details in the generation.
• The separation of the generation and updating processes: We have modularized and decoupled the mask injection generation from the segmentation results updating process. This design enhances the flexibility of our pipeline, allowing for the substitution of stronger mask injection modules or more advanced updating strategies, thereby facilitating further exploration in future work.

W2

The main idea is that, given an imperfect segmentation mask, the generated image will have a discrepancy from the original image. This drives the intuition given in the paper that with a more accurate segmentation mask, the probability of generating the original image is more likely to be maximized. How this is implemented is described in 3.2.2 and Appendix E. Some explanation is given 317-323; however, the explanation also highlights the reliance on a good initial segmentation mask. The ablation study does not present how sensitive the proposed method is to the initial segmentation accuracy.

Thank you for the constructive comments. To evaluate the sensitivity of the proposed method to the quality of the initial segmentation, we stratified the samples into groups based on different levels of coarse segmentation quality. Since our approach is sample-wise, we can directly calculate the mean IoU improvement for each group according to their initial mask quality, as measured by the per-sample IoU metric on the VOC dataset:

Table Improvement for different initial performance

According to the above table, when the initial mask quality is poor, the improvement of our method is also limited. The improvement from our method is most significant for initial IoU values between 40 and 80. This indicates that our approach is particularly effective when the initial segmentation is already of reasonable quality. When the initial segmentation is already nearly perfect(80-100), the improvement from our method becomes limited due to the bottleneck caused by errors inherent in the mask injection process. We have added this into the revision in Sect 4.2.

W3

The segmentation mIoU improvement is marginal. Besides, the comparisons with stronger SOTA are missing OVAM (CVPR 2024) and CoDe (CVPR 2024). DeOP (ICCV 2023), which could be another interesting baseline.

We sincerely thank the comments, particularly regarding the expansion of the experimental dimensions. Marginal Improvements: We argue that G4Seg is a purely generative prior-based attempt at segmentation refinement, which only leverages a diffusion model pretrained with a generation target. This framework does not rely on any pixel-level annotations or extra fine-tuning. In Sect. 3.3.1, we introduced a mask conditioning method, which is based on cross-attention and self-attention. These attention-based generation methods do not perform well in a mask-conditioned generation. If we adopt a stronger mask conditioning method, such as ControlNet[1], the performance would significantly improve, as shown in the following Table:

We do not adopt such a stronger mask injection strategy because it would violate the "inexact segmentation" setting in our approach. We have added these discussion to Appendix F.

Other Baselines Sorry, we couldn't identify which paper "CoDe" refers to. Please help give us more details about "CoDe" and we will include it into the results and discussion. For the present, we first clarify the remaining two papers:

• OVAM[2]: OVAM uses manually annotated masks of generated images to update token embeddings, which are then used to generate more images and corrected cross-attention-based pseudo masks. Strictly speaking, this method is not an "inexact segmentation" approach. Therefore, we categorize this method as a semi-supervised/fully supervised approach.

• DeOP[3]: DeOP is inherently a fully supervised method, so we also categorize it as fully supervised approach.

Table Results on semi-supervised/fully supervised methods.

Note that the OVAM is evaluated with cross attention pseudo mask generated with text token paper offered. DeOp use the pretrained weight provided with GitHub repository.

We have added these discussions to Appendix G and H.

Q1.

How could this method extend to other UNet-based text-to-image diffusion models beyond SD2.1 (such as Wurstchen, SSD-1B, SDXL, SD1.5, etc.) and their distilled versions (such as single or reduced step variants, LCM versions)?

How to extend Our pipeline contains two step, Explicit Mask Injection and Semantic Correspondence Alignment after the generation. EMI is based on the backbone attention perturbation.

SD1.5, SD2.1, LCM, and SDXL (with SSD-1B as a lighter distilled version of SDXL) share largely similar U-Net backbone architectures, incorporating cross-attention and self-attention layers. Consequently, the EMI step is executed in a nearly identical manner across these models. Then after the generation, the SCA step remains the same.

Würstchen includes a semantic compressor, which introduces uncertainties regarding its impact on our pipeline. As a result, directly utilizing it may present certain challenges, requiring further investigation and analysis to ensure its compatibility and effectiveness.

Experimental Results We have compared the results with SD1.5, SD2.1, SDXL and LCM.

As can be seen from the Table, the version of pretrained diffusion model has no obvious impact on the segmentation results. We appreciate the reviewer's comments and have added this to the revision as in Appendix L.

Reference

[1] https://huggingface.co/lllyasviel/sd-controlnet-seg

[2] Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models.CVPR2024

[3] Open Vocabulary Semantic Segmentation with Decoupled One-Pass Network. ICCV2023

Dear Reviewer EypU:

Thanks for reviewing our work and providing insightful comments. We hope our answers have addressed your concerns. Please let us know if you have further comments or need more discussions.

Best regards,

Authors of Submission 6302