High-Precision Dichotomous Image Segmentation via Probing Diffusion Capacity

Q1: Why optimizing with $x_0$ instead of noise & UNet degeneration & other approach for efficient inference

Thank you for raising these valuable concerns, here are our clarifications:

Our one-step denoising paradigm yields the same outcome whether we supervise the noise $\epsilon\_{\theta}(x\_{T}, T, cond, d\_{lab})$ or the denoised $x_0$ derived from it. This is due to the linear relationship between the two, which allows for an easy transformation using the formula provided (See Eqn. 4).

Moreover, we believe that the UNet in our model is not degenerated to a common UNet. Similar to SD, the UNet still predicts the noise distribution. The core of one-step denoising lies in simplifying and streamlining the multi-step denoising paradigm, and thereby minimizing the generative randomness introduced by noise and achieving reliable outcomes in perceptual tasks.

Regarding other efficient inference methods based on ODE solvers, we have indeed employed a consistent objective training approach with a 10-step denoising strategy (See Tab. 4, last row) based on DDIM. However, we found that it did not outperform the direct prediction of noise from $x_T$. We believe this is because multi-step denoising introduces noise variables iteratively, which can lead to results that are more generative in nature, while one-step denoising is more appropriate for binary segmentation tasks that require high determinism.

As for other ODE solvers and inference optimizations, we will further exploring these avenues in our future work.

Q2: Sensitivity of noise during sampling

Thank you for your insightful comment. We conducted new experiments with the following setup:

Multiple Noise Predictions: For given samples, we used $5$ distinct noise for prediction. We evaluated the results of each noise-predicted mask individually and also the average result across all $5$ predictions.

Pyramid Noise Initialization: We initialized the input with pyramid noise and compared the resulting predictions to those obtained with our standard noise initialization.

As shown in the result above, DiffDIS is stable to the noise sampled during inference due to our streamlined one-step denoising paradigm.

Q3: Integration of additional text information

Thank you for your valuable suggestion. We have experimented with CLIP to encode text embeddings for images, but found it didn't lead to any improvements. We believe that this is because the semantic information in CLIP's text embeddings is too broad to provide targeted guidance for segmenting individual objects.

We will continue to explore ways to incorporate additional text cues from other label models to further enhance our model's performance.

Q4: Difference between DBIA and common cross-attention

Our Detail-Balancing Interactive Attention (DBIA) enhances the standard attention mechanisms in the SD network by introducing a mutual cross-domain attention specifically for edge and mask features, where we employ a commonly-used cross-attention design----queries from one domain interact with the keys and values from the other domain. This mechanism leads to a deep contextual understanding, while enabling better alignment and focus on subtle features. It ensures that these details are not overlooked and are effectively balanced between the two domains.

Q5: Other attempts of scale-wise injection

Thank you for the suggestion of alternative approaches for the multi-scale injection. In response to this issue, we conducted new experiments with the following setup:

We first incorporated condition signals into the network blocks using a combination of simple convolution operations and summation. Additionally, following the authors' suggestion, we enabled the scale-wise conditions to interact with the network features via extra cross-attention mechanisms. The results are shown in the table below:

Q6: Inference speed and parameters

We measure the inference speed and present a comparison of our FPS and parameters with other methods below (all tested on a single NVIDIA 3090 GPU):

Despite our model's higher parameter count and relatively slower inference speed, it represents an early attempt at using diffusion for high-precision segmentation. As more optimized architectures emerge, such as pruning techniques for SD models, we plan to exploring more efficient solutions in the future to improve our model's performance.

W1: Performance on other scenario:

Thank you for your constructive comments. We conduct extra experiments on HRSOD benchmark. Following[1], our model is trained with the combination of HRSOD-TR and UHRSD-TR, yielding the following results. We will show more applications in the final paper.

[1]Pyramid Grafting Network for One-Stage High Resolution Saliency Detection. CVPR 2022.

W2: Novelty of one-step denoising in DiffDIS:

Thank you for highlighting this interesting concern. However, we would like to clarify a few points regarding DiffDIS’s relation to DDPM-Seg and other one-step denoising works:

Firstly, DDPM-Seg conducts full-step denoising during inference, extracting semantic information from specific U-Net layers at certain reverse steps for segmentation. In contrast, our method starts from a fully noised latent at step 999 and directly accomplishes one-step denoising, skipping intermediate $x_t$ predictions, thus presenting an end-to-end model.

Regarding other works based on DDPM-Seg that utilize one-step denoising, they typically extract features from frozen SD and use these features as visual /semantic cues in additional adapter/refiner modules. This kind of design often leads to additional complexities in module design and increased computational overhead. Moreover, in complex scenes with fine details, straightforward feature extraction and fusion are insufficient for accurate recovery, thereby failing to fully leverage the inherent priors within SD by freezing the model. And, the randomly selected $t$ can also introduce generative randomness into the results.

Our one-step denoising paradigm, as a simplified version of the DDPM reverse process, minimizes the generative randomness introduced by noise, and makes more efficient use of SD's priors, making it more suitable for high-precision binary segmentation tasks that demand high determinism.

We appreciate the opportunity to clarify these points and thank you for raising this issue. It has been a valuable insight, and we will consider how to better utilize intermediate features from the U-Net in future work.

W3: Novelty of BDE & DBIA

We appreciate the feedback, and would like to clarify that our core contributions extend beyond these modules themselves and the use of BDE and DBIA serves to support the broader goals.

Specifically, our primary focus is on harnessing the power of diffusion models to address the challenges faced by traditional methods in high-precision DIS task, with task-specific designs for higher processing speed, stronger detail perception capability, and higher reliability. This includes the adoption of a one-step denoising strategy and the incorporation of an edge estimation task to provide enhanced guidance and detail balancing.

To incorporate the edge information to enhance mask generation, we design BDE and DBIA, a suite of necessary, simple yet effective mechanisms, to conduct the essential cross-domain distinguishment and communication between masks and edges, ensuring a seamless and conflict-free domain processing, while significantly enhancing cross-modality information transfer.

Q1: How can VAE decode HR image from low-dimension:

VAEs conceptualize the data distribution as a mixture of several Gaussian distributions. The encoder in a VAE learns to approximate the parameters of this distribution (typically the mean and variance), from which latent variables are sampled, enabling the compression of images into a very low-dimensional space. Additionally, by training the reconstruction capability on a vast array of natural images, VAE develops robust generalization abilities, allowing itself to effectively compress and reconstruct any image. We have demonstrated the superior performance of VAE in specialized tasks, such as binary image reconstruction (See Tab. 1).

For high-resolution segmentation tasks, this indicates that the information are fully preserved within the latent space. All we need to do is to design additional detail-enhancing algorithms to identify and capture these most subtle details, such as edge-assisted estimation, DBIA, and SWCI, aiming for the best denoising effects to refine the representations and enhance the clarity of the features.

Q2: Transferability to other architectures:

We appreciate your suggestion to explore the transferability of our methods to other architectures such as pixel-space diffusion UNets and latent DiT.

However, we believe that applying pixel-level diffusion directly to high-precision segmentation tasks is not practical for several reasons: Firstly, as input resolutions increase (e.g., 1024x1024), pixel-space diffusion becomes computationally intensive and memory-hungry compared to latent space diffusion. This can be particularly challenging for HR images where the data volume is already substantial. Secondly, pixel-space diffusion can significantly slow down processing speeds, exacerbating the already slow inference times for high-resolution images. Thirdly, the continuous Gaussian distribution in the VAE latent space is more aligned with the continuous feature space inherent in diffusion processes. Lastly, adopting pixel-space diffusion might necessitate training from scratch, bypassing the powerful priors provided by models like SD. This could lead to slower convergence and reduced accuracy, as evidenced by Tab. 4.

Regarding the use of latent DiT as our backbone, we apologize for not having had sufficient time to explore it in the rebuttal period, however, we will investigate it in the final paper.

Typo:

Thank you for pointing out these typos and grammar issues, we appreciate your detailed attention! We will address these issues in our revision and conduct a thorough self-check for any other errors throughout the manuscript. The corrections have been incorporated into the updated PDF file of our revision.

W1: Key differences from GenPercept & task-specific optimizations in DiffDIS:

Thank you for your feedback! GenPercept utilizes a deterministic framework that maps conditions (e.g. image latents), directly to the target perceptual output without undergoing a noise-corruption and denoising process, while DiffDIS maintains the ability of diffusion models to extract intricate details from noise by rooting in DDPM, and employing a simplified denoising paradigm that recovers the target from globally noised variables, which minimizes the randomness inherent in generative models while preserving the detail-extraction capabilities of diffusion models. This design aligns well with high-precision tasks like DIS, leading to superior performance against GenPercept (See Tab. 2).

W2: Key differences from EdgeConnect:

Thank you for your concern, however, we beg to differ on this issue:

EdgeConnect used a two-stage, sequential prediction strategy, generating the edges of the image to be inpainted first, followed by the image itself. In EdgeConnect, the edges act as essential intermediates/connectors between the edge estimation task and image inpainting task (hence "EdgeConnect" is named).

Different from them, we use a parallel, interactive approach that generates edges and masks simultaneously (with BDE helping manage different modalities within each batch), where the edge estimation acts as an auxiliary task, aiming at enhancing the mask generation by imposing finer constraints, and diminishing the stochastic nature of diffusion processes.

DiffDIS is distinctly tailored for the DIS task, with a clear focus on achieving higher inference speed, stronger detail extraction, and higher determinism, rather than "albeit applied in segmentation instead of inpainting".

W3: Comparision of joint predicton & stage-wise processing:

In response to your suggestion, we implemented a two-stage mask generation process following EdgeConnect's training workflow :

In the first stage, an edge generation model produces edges as intermediate outputs, which are then used as additional conditions for the mask generation task in the second stage. To ensure fairness, we also integrated the DBIA module in the second stage. The results, along with inference times, are shown below:

As shown in the result above, our joint prediction strategy enhances the network's ability to capture fine details while being more efficient and resource-friendly.

W4: Training time comparison:

Thank you for your suggestion. We compiled a comparison of training times with 11 other methods, and the results are as follows:

W2: Overkillness of VAE:

We thank the reviewer for pointing out this issue! If a single-channel VAE is used, it would require retraining to achieve reconstruction performance comparable to that of the three-channel structure. In our work, we initially aimed to leverage the prior from the SD model, which led us to adopt the classical three-channel VAE it provides. However, the suggestion of examining the potential overkill with fewer channels in VAE is very insightful, and we plan to further investigate this approach in future work.

Q1: BDE vs. learnable task embedding in EMU-edit

Thank you for your suggestion. Our BDE diverges from EMU-edit’s learnable task embedding in several key aspects. We employ one-dimensional orthogonal vectors, contrasting with the task vectors they fetch from an embedding table. Our embeddings interact solely within the ResBlock layers, whereas theirs are integrated into both residual and cross-attention layers. Given that EMU-edit's source code is not publicly available, we independently implemented learnable task embeddings in DiffDIS, yielding the following results:

Q2: Details on one-step sampling:

During the inference phase, the one-step denoising process aligns with the training phase by replacing the noised mask and edge latents with Gaussian noise. Using the image latent of the target segmentation as a condition, it guides the generation of noise $\epsilon\_{\text{pred}\_m}$. Denoising is then conducted using the process outlined in Fig. 5, where $l\_{\text{pred}\_m} = \left(m\_{T} - \sqrt{1 - \bar{\alpha}\_T} \times \epsilon\_{\text{pred}\_m} \right) / \sqrt{\bar{\alpha}\_T}$. This constitutes the one-step sampling procedure.

Q3: How the pretrained weights are handled when doubling input channels:

For the input convolution layer, we duplicated the 4-channel weights and then scaled all of them by 0.5