Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation

We deeply appreciate your acknowledgment of our motivation and approach. We will try our best to address the questions and weaknesses you have raised.

W1: Lack of proper references Thank you for your feedback. We will add appropriate references in the next version.

W2: Definition of $\beta_{1}$ and $\beta_{2}$ Yes, you are correct. We will clearly define $\beta^{1}$ and $\beta^{2}$ in Section 4.4 and explain the intuitive meanings of the two values. $\beta^{1}=(\beta_{start},\beta_{end})$, where $\beta_{start}$ and $\beta_{end}$ respectively represent the initial and final values of $\beta$ in the DDIM scheduler. We also mentioned this in our discussion with Reviewer uveh. Intuitively, $\beta^{2}$ adds more noise at smaller time steps compared to $\beta^{1}$. Experimental results show that adding more noise at smaller time steps improves the model's performance, which is why we ultimately chose OI2M. .

W3: Other typos Thank you once again. We will fix these errors in the next version.

Questions: 5-shot performance

We are grateful for your question. We have also mentioned this issue in our response to Reviewer zc26. We have tried two approaches to address the N-shot setting issue.

Inference Time Improvement

The first approach is an improvement in the inference phase. Considering the transition from 1-shot to n-shot, where additional support samples' Keys and Values are concatenated (see Appendix A.6), this leads to a significantly larger number of Keys and Values for the self-attention layer to process during inference compared to the training phase. An intuitive solution is to randomly sample the Keys and Values of the support samples at each layer during inference, ensuring the number of samples matches the number of Keys and Values during training. We found that this approach indeed improves the model's performance in 5-shot and 10-shot settings. The experimental results are shown in the table below.

Table: Performance of DiffewS with Inference Time Improvement

Another more straightforward idea is to introduce multiple support samples during the training phase. This way, the model can learn how to select among multiple support images during training. To save computational resources, we conducted experiments under the ablation study setting on the COCO fold0. The experimental results are shown in the table below.

Table: Performance of DiffewS with Training Time Improvement

(train 1-5 shot) indicates that we randomly select 1 to 5 support samples as input during a single training iteration. From the table, we can see that when multiple support samples are introduced during the training phase, the model's performance significantly improves in both 5-shot and 10-shot scenarios. Additionally, as the number of support samples increases during training, the model's performance in the 10-shot scenario gradually improves. We also found that when multiple support samples are introduced during the training phase, the 1-shot performance may decrease. This could be due to the inconsistency between training and inference.

In summary, both approaches effectively alleviate the issues associated with the N-shot setting, demonstrating the potential of DiffewS in Few-shot Semantic Segmentation. We also hope that our work can provide some inspiration for researchers in the fields of diffusion-based models and Few-shot Semantic Segmentation.

We sincerely thank you for your very careful and detailed review. We are delighted that you found our work to be comprehensive and convincing. The questions and weaknesses you've pointed out are incredibly helpful to us, and we will do our utmost to address them below.

W1: Why should the diffusion model be applied to FSS tasks? In L43-L53, we briefly introduce the motivation for applying the diffusion model to FSS tasks. We will make this clearer in the revised version. The pretrained diffusion model exhibits significant potential in fine-grained pixel prediction ability[1,2] and semantic correspondence ability [3-5]. Both of these capabilities are essential for Few-Shot Segmentation (FSS) tasks, making diffusion models particularly suitable for this application.

For example, SAM has strong fine-grained pixel prediction capabilities but lacks semantic abilities, leading to a series of works attempting to address this issue[6,7]. Therefore, it is difficult to use SAM in FSS. For example, PerSAM[8] is based solely on SAM's capabilities but does not perform well in FSS.

On the other hand, LLaVA has strong semantic abilities but lacks fine-grained pixel prediction capabilities due to its reliance on only image-text pairs for pretraining(lacks fine-grained supervision). Therefore, LISA[7] needs to rely on SAM to achieve segmentation prediction.

W2: Multi-step denoising process We believe that the supervised objective of the multi-step denoising process is not suitable for FSS, but this does not mean that single-step prediction fails to unleash the potential of the diffusion model (DM). The difference between OI2M and MI2M/MN2M lies only in their training objectives (see Appendix A.2). Compared to OI2M, MI2M and MN2M use noisy target masks as input, which could potentially lead to information leakage in segmentation tasks, especially when the noise level is low, thus hindering the model's learning. The experimental results in Figure 3 also confirm this, showing that setting $\beta_{2}$ is more effective than $\beta_{1}$. Intuitively, $\beta_{2}$ adds more noise at smaller time steps compared to $\beta_{1}$, and OI2M can be understood as an extreme case where the input contains no information about the target mask at all (or the target mask has become pure noise). We will provide more theoretical explanations in Section 4.4.

W3: Inference speed and training cost Thank you for your suggestions. We can compare the advantages of DiffewS in two aspects:

1.Compared to traditional FSS methods, DiffewS converges faster and has lower training overhead. For instance, taking DCAMA[9] as an example, both our model and DCAMA were trained on 4 V100 GPUs. The comparison of training time is as follows:

Table: Training Time Comparison

2. Compared to other generative methods based on multi-step denoising, DiffewS has a faster inference speed. For instance, considering MI2M and MN2M mentioned in this paper, the inference speed comparison is as follows:

Table: Inference Speed Comparison

In our implementation, MI2M and MN2M use DDIM sampling, requiring the model to forward 10 times, while OI2M only needs to forward once. Thus, OI2M should be 10 times faster than MI2M/MN2M."

W4:Other studies Thank you for your valuable suggestions. We will include discussions of DifFSS and Ref LDM-Seg in the Related Work section, as well as comparisons with our work. DifFSS uses generated images as auxiliary support images, which can improve the performance of existing FSS methods to some extent. This approach enhances FSS performance from the data level, making it orthogonal to our method. Ref LDM-Seg is a contemporaneous work to ours. Unlike DiffewS, which utilizes self-attention to achieve interaction between the query and support images, Ref LDM-Seg uses cross-attention for this purpose. Additionally, Ref LDM-Seg introduces multiple linear projection layers, increasing the number of parameters in the diffusion model, which might also disrupt the original priors of the diffusion model. Our work provides a more comprehensive and systematic analysis of applying diffusion models to Few-shot Semantic Segmentation tasks. Considering that Ref LDM-Seg has not been open-sourced, and the training data and experimental setups cannot be strictly aligned, we currently cannot conduct a direct and fair experimental comparison.

References

[1]Ke B, Obukhov A, Huang S, et al. Repurposing diffusion-based image generators for monocular depth estimation. CVPR2024

[2]Lee H Y, Tseng H Y, Yang M H. Exploiting Diffusion Prior for Generalizable Dense Prediction. CVPR2024

[3]Tang, Luming, et al. Emergent correspondence from image diffusion. NeurIPS2023

[4]Luo, Grace, et al. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. NeurIPS2023

[5]Zhang, Junyi, et al. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. NeurIPS2023

[6]Li F, Zhang H, Sun P, et al. Semantic-sam: Segment and recognize anything at any granularity. ECCV 2024

[7]Lai X, Tian Z, Chen Y, et al. Lisa: Reasoning segmentation via large language model. CVPR 2024

[8]Zhang R, Jiang Z, Guo Z, et al. Personalize segment anything model with one shot. ICLR2024

[9]Shi, Xinyu, et al. Dense cross-query-and-support attention weighted mask aggregation for few-shot segmentation. European Conference on Computer Vision. ECCV2022

Due to space constraints, we will respond to your valuable questions in the global response.

Thank you very much for acknowledging our motivation and approach. The questions and weaknesses you've pointed out are incredibly helpful to us, and we will do our utmost to address them below.

Weaknesses: N-shot setting

Thanks again for pointing out this issue. As we mentioned in the Experiment and Limitation sections of our paper, our current work primarily focuses on unleashing the potential of Diffusion Models in Few-shot Semantic Segmentation. Consequently, our experiments and optimizations are mainly concentrated on the One-shot setting. Since the model receives only one support image at a time during the training phase, using 5-shot or 10-shot during the inference phase can indeed cause inconsistencies. The model might not learn how to effectively utilize multiple support images during training, which could result in interference from lower-quality samples. However, we fully agree with your point that we should further explore the performance of DiffewS with additional support samples. Reviewer zc26 raised a similar request, asking, "can this be mitigated by making adjustments during training?"

Therefore, we have tried two approaches to address the N-shot setting issue.

Inference Time Improvement

The first approach is an improvement in the inference phase. Considering the transition from 1-shot to n-shot, where additional support samples' Keys and Values are concatenated (see Appendix A.6), this leads to a significantly larger number of Keys and Values for the self-attention layer to process during inference compared to the training phase. An intuitive solution is to randomly sample the Keys and Values of the support samples at each layer during inference, ensuring the number of samples matches the number of Keys and Values during training. We found that this approach indeed improves the model's performance in 5-shot and 10-shot settings. The experimental results are shown in the table below.

Table: Performance of DiffewS with Inference Time Improvement

From the table, we can see that following the original DiffewS inference method, the 10-shot performance is even lower than the 1-shot result. This might be due to the reasons we analyzed earlier. However, after adopting the random sampling strategy during inference, there is a significant improvement in the model's performance for both 5-shot and 10-shot scenarios

Training Time Improvement

Another more straightforward idea is to introduce multiple support samples during the training phase. This way, the model can learn how to select among multiple support images during training. To save computational resources, we conducted experiments under the ablation study setting on the COCO fold0. The experimental results are shown in the table below.

Table: Performance of DiffewS with Training Time Improvement

(train 1-5 shot) indicates that we randomly select 1 to 5 support samples as input during a single training iteration. From the table, we can see that when multiple support samples are introduced during the training phase, the model's performance significantly improves in both 5-shot and 10-shot scenarios. Additionally, as the number of support samples increases during training, the model's performance in the 10-shot scenario gradually improves. We also found that when multiple support samples are introduced during the training phase, the 1-shot performance may decrease. This could be due to the inconsistency between training and inference.

In summary, both approaches effectively alleviate the issues associated with the N-shot setting, demonstrating the potential of DiffewS in Few-shot Semantic Segmentation. We also hope that our work can provide more inspiration for researchers in the fields of diffusion-based models and Few-shot Semantic Segmentation.

Questions:open-vocabulary segmentation

We believe it is feasible; however, it would be more appropriate to inject text information through cross-attention, as discussed in (Sec 4.1 Tokenized Interaction Cross-Attention). A similar idea has already been validated in the Referring Segmentation task (UniGS [1]). We will include a discussion on this in the Related Work section. Our work focuses more on how to effectively achieve interaction between the features of two images, which is why we ultimately chose the KV Fusion Self-Attention approach. However, these two methods are not mutually exclusive and can coexist. Nevertheless, to achieve open-vocabulary segmentation, more segmentation datasets with richly annotated text are needed for training, as merely using a few-shot dataset is insufficient. Therefore, we have not conducted experimental verification.

[1] Qi L, Yang L, Guo W, et al. Unigs: Unified representation for image generation and segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 6305-6315.