Completing Visual Objects via Bridging Generation and Segmentation

We thank the reviewer for the time and effort to review our paper. Our answers to the questions are as follows.

1. Some symbols in the paper need to be further improved.

We would like to thank the reviewer for the detailed inspection. We have incorporated the suggested changes to the revised manuscript.

2. It's not rigorous to name the progressive completion process as 'mask denoising'. The object's mask generated from SAM is taken as a condition to diffusion instead of the denoising unit as $x_t$.

Thanks for the comments. We agree that the mask denoising is very different from the image denoising diffusion while we hope the word "mask denoising" can help the reader understand our iterative mask completion process. Given a partially visible object $I_p$ and its corresponding partial mask $M_p$, the conventional object completion task aims to find a generative model $\mathcal{G}$ such that $I_c\leftarrow\mathcal{G}(I_p)$, where $I_c$ is the complete object. Here, we additionally add the partial mask $M_p$ to the condition $I_c\leftarrow\mathcal{G}(I_p, M_p)$, where $M_p$ can be assumed as an addition of the complete mask and a noise $M_p=M_c+\Delta$. By introducing a segmentation model $\mathcal{S}$, we can find a mask denoiser $\mathcal{S}\circ\mathcal{G}$ from the mask completion process: $M_c\leftarrow\mathcal{S}\circ\mathcal{G}(I_p, M_c+\Delta)$. If we consider the occlusion $\Delta$ as a noise, $\mathcal{S}\circ\mathcal{G}$ can be assumed as conducting mask denoising for each IMD step. We will further clarify the usage of "mask denoising" in the revision.

3. It is not clear whether the other comparison methods are retrained in the evaluation data. If yes, the training details and the incomplete mask's interaction in their networks should be claimed, otherwise the authors should explain how to obtain these completion results with the off-the-shelf generation models.

We would like to clarify that the baseline methods are not retrained in the evaluation dataset. We consider the baselines can be directly utilized due to (1) the object completion can be treated as a subtask of the in/outpainting objectives that the baseline methods are trained on (2) the object categories in the evaluation set are common which should be within the large-scale training data of the baselines. In addition, to evaluate on a more general object setting, our model is also not trained on the evaluation set (DYCE) but on OpenImage (Line 231). In this way, we consider our comparison can fairly reflect the effectiveness of the proposed method.

4. It would be better if the authors can give some analysis on the sampling speed and more qualitative results of the proposed method.

Table D: Inference time for each stage of IMD on a single V100 GPU.

Thanks for your suggestion. We provide the sampling time of each component as shown in Table D. We notice that the most time-consuming part is the diffusion process. To improve the inference speed, we notice that decreasing the diffusion step number in the first several IMD steps will not severely degrade the performance. By incorporating this idea into MaskComp, the average running time was reduced to 2/3 original time. We also provide more qualitative results and results in a potential application (layered image generation) as shown in Figure A (github.com/iclr23anonymous739/more_vis.pdf) and Figure B (github.com/iclr23anonymous739/application.pdf). We believe the idea of using segmentation to boost generation and the strong capability of MaskComp can be of interest to the community. We will incorporate the suggested results in the revision.

Thanks again for the authors feedback. My concerns have been resolved. I am glad to update my rate.

We thank the reviewer for the time and effort to review our paper. Our answers to the questions are as follows.

1. The justification for the entire iterative procedure is lacking. It is possible to have noisy output (worse image/inaccurate mask) in both the generation and segmentation stages.

We would like to provide an analysis of the noise-tolerant capability of the proposed MaskComp. We agree that worse images and inaccurate masks can appear in the generation and segmentation stages respectively. Since all the generation processes are conditioned on the original partial image (as shown in Figure 1), both worse images and inaccurate masks will just influence the conditioned mask for the next iteration. We have two mechanisms to mitigate the influence of the noisy conditioned masks:

• Error Suppression through Mask Voting: We vote among multiple masks to form the input of the next iteration. In this way, if only a few masks are degraded, the impact of them can be recovered by the voting operation.

• Error Tolerance in IMD Iteration: We train the mask-denoising ControlNet to handle a wide range of occluded masks. Consequently, if the conditioned mask undergoes minimal improvement or degradation due to the noises in a given iteration, it can still be improved in the subsequent iteration. While this may slightly extend the convergence time, it is not anticipated to have a significant impact on the ultimate image quality.

In this way, we consider the IMD process can work in a robust manner. We will show a quantitative analysis in the later response to question 7.

2. The proposed method lacks mode diversity for object completion due to the averaging operator in the IMD process.

We agree that MaskComp lacks shape diversity due to the averaging operator. However, we consider that object completion tasks typically only prioritize the estimation closest to the ground truth object. For example, in the closely related amodal segmentation task, the evaluation is conducted by calculating the IoU between the predicted mask and the GT mask. We consider retaining only the most plausible estimation to be reasonable since the rough shape of objects can be deduced by the visible parts in most cases.

In addition, we consider the lack of shape diversity will not hinder the usage of MaskComp in practice. We discuss the potential application of MaskComp in Figure 10. Since the current image editing models struggle to modify partial images, MaskComp can serve as the first step to complete objects and then feed to an image editing model to further process the image. We show another cool application, layered image generation, of MaskComp as shown in Figure A (github.com/iclr23anonymous739/application.pdf).

3. In the generation stage, how does the interpolation is implemented? How does the method generate the interpolated masks and its time embedding?

We designed several occlusion strategies to create diverse occlusion (the details can be found in Line 417-424). During training, we conduct the random occlusion process twice for each complete mask $M_c$. The partial mask $M_p$ is achieved by considering the occluded areas in both of the occlusion processes. And the interpolated mask $M$ is generated by using one of the occlusions. The time embedding used for the gating operation is shared with the time embedding for encoding the diffusion step in the stable diffusion.

4. What are the notations M, E in the paragraph of "Diffusion model"? The details of the adopted diffusion model are missing in the main text.

The $M$ denotes the interpolated mask (Line 146) with an occlusion rate between the complete mask $M_c$ and partial mask $M_p$ and $E$ denotes the text prompt of the object (please note that the text prompt is necessary to fine-tune stable diffusion while optional during inference). We leverage frozen stable diffusion (Line 242) as our underlining diffusion model. We further clarified the notations in the revised version.

5. It is unclear how the baselines and this method are compared. Are the background removed before comparison, for the FID computation and user study?

Yes, we aim to ignore the background during all the evaluations. For the quantitative comparison, the computing of FID only considers the ground-truth object area to eliminate the influence of the generated background (Caption of Table 1). For the user study, we give detailed instructions to participants to focus only on the foreground object region and ignore the difference presented in the background (Line 457).

6. Comparison with a combination of amodal segmentation and condition image generation as a baseline.

Table A: Performance comparison with amodal segmentation baseline.

Thanks for the suggestion. We conducted an additional experiment with the masks from the SOTA open-sourced amodal segmentation method ([A] AISFormer) and leveraged ControlNet to generate images based on the amodal masks. We report the performance as shown in Table A. We notice that our method shows obvious superior performance compared to the baseline setting.

[A] AISFormer: Amodal Instance Segmentation with Transformer

7. What if there are different degrees of segmentation error?

Table B: Ablation with noisy conditioned mask. We leverage FID to evaluate the performance. The error is added based on the size of the foreground object.

We conducted an experiment to manually add random errors to the masks. As shown in Table B, we ablate on the iteration number and the degree of segmentation error. We notice that the segmentation error will just increase the converge iteration number while the final performance will not be severely influenced. In addition, since MaskComp predicts a black background, it is easy for segmentation models to segment the foreground objects. Therefore, a large error in segmentation is not expected.

8. It would be more convincing if the method is evaluated on the cases with diverse modes of object shape in the occluded regions.

Table 2 (b): Performance with different occlusion rates.

Table C: Performance with different occlusion types. Object denotes the occlusion is created with a random object shape.

We reported the performance with different occlusion rates as shown in Table 2 (b). We additionally report the performance with different occlusion types as shown in Table C. Since the object occlusion has a more complex boundary, the results with rectangle and oval occlusions show a lower FID than that with more complex object occlusions.

Thanks for your time and effort in reviewing our work. We hope our earlier response can resolve your concern. As the deadline for the discussion period is approaching, if any aspect of our response or the method itself remains unclear, please do not hesitate to reach out. We are more than willing to provide further clarification.

Here is a summary about the suggested changes to the manuscript:

• Further discussion about the errors and robustness of the IMD process (Line 317-337).
• Further explanation about the interpolated mask and time embedding (Line 253-257).
• Clarification of the $M$, $E$ and the underlined diffusion model (Line 153-158).
• Comparison with a combination of amodal segmentation and condition image generation (Table 4 (c) & Line 354-359).
• Performance with different occlusion types (Table 4 (d) & Line 359-360).

We look forward to your continued feedback and sincerely hope that you can improve the rating based on the responses.

Thanks for the detailed response.

• 1&7. The argument on the robustness does not fully convince me. The voting does help but it would hurt diversity, which is a dilemma. While enlarging the range of masks may help, there may also be cases which the training does not cover. Thanks for the empirical study, which does show the method is stable on this benchmark.

• 2. The groundtruth may not be unique, especially for articulated objects.

• 3-5 Thanks for the clarification. It would be better to apply a GT mask before sending the results to any evaluation process.

• 6. Thanks for the additional results, which addressed my previous concern.
• 8. Thanks for the results, but I would like to clarify that "diverse modes of object shape" refer to potential multiple GT masks due to different poses of complex objects, such as human/animal classes.

We thank the reviewer for the feedback. However, we do not agree with the reviewer due to the lack of concrete reasons to support his point. We consider we have explained the unnecessity of shape diversity in the completion task from the (1) conventional (amodal evaluation), (2) practical (application) and (3) empirical (analysis of human annotator) perspectives and have provided convincing evidence to support our claim that sacrificing shape diversity to gain reliability and image quality is deserving.

We would encourage the reviewer to discuss the specific concern with other reviewers and AC during the discussion phase. Thank you.

We thank the reviewer for the time and effort to review our paper. Our answers to the questions are as follows.

1 & 2. Using SAM to generate masks is straightforward and the mask voting process gives no surprises. If generating a mask is the key to generating high-quality images, why not directly use an encoder-decoder model like U-Net or an SD to predict the target complete mask in one step?

Table A: IoU of predicted masks on AHP dataset.

In this paper, we investigate the relation between object generation and segmentation. We found that even if the segmentation model is not trained, the segmentation can still benefit the generation. It is a good idea to conduct experiments to investigate other segmentation models. We report the results of using a ControlNet (frozen SD) and UNet with the partial object as the condition to predict the complete mask in Table A. We notice that the complete mask predicted from the IMD process shows obviously higher IoU to the ground truth which indicates the IMD process is an effective mask prediction approach.

In addition, we would like to clarify that the mask prediction is not simply conducted by the SAM but by the entire IMD process with both generation and segmentation stages. We treat the segmentation as the shape property of the generated object. Therefore, with an object generation model, we can directly obtain the shape using SAM without training another mask-specific model.

We also want to further clarify the usage of the voting operation. Given a partial object $I_p$, the generation model samples from a distribution that contains both realistic and unrealistic images. As shown in Figure A (github.com/iclr23anonymous739/dist.pdf), since the image distribution is complex, the expectation $E[I_c]$ cannot represent a realistic image. However, when we consider the shape of the generated images, we are excited to find that the expectation $E[\mathcal{S}(I_c)]$ leads to a more realistic shape. We consider this observation interesting as, for most of the other generation tasks (non-conditioned), averaging the object shape will just yield an unrealistic random shape. This observation (Figure 4) serves as one of our core observations to build the IMD process. Here, SAM serves as the tool to extract the object shape and voting is a way to binarize the expectation of object shapes to a binary mask.

3. More ablation studies of the proposed method should be given. E.g., different segmentation models, and different mask voting strategies.

Table B: Performance with different segmentation models.

Table C: Performance with different voting strategies. Logits: mask logits before binarzing.

We conducted ablation experiments to determine the design choice in the segmentation stage. We report the ablation studies about segmentation models and voting strategies in Table B and Table C. The current design choice of using SAM and voting with logits is based on the ablation results.

4. Comparisons and discussions of the running time.

Table D: Inference time for each stage of IMD on a single V100 GPU.

Thank you for your suggestion. We demonstrate the running time of each component in IMD as shown in Table D. We acknowledge that the inference speed of MaskComp is slow. To improve the inference speed, we notice that decreasing the diffusion step number in the first several IMD steps will not severely degrade the performance. By incorporating this idea into MaskComp, the average running time was reduced to 2/3 original time. We will add the discussion to our limitation section.

5. The FID only considers the ground-truth object area. Thus, this metric will yield a higher FID score if the generated object has a large discrepancy with the ground truth even if it is realistic.

Table E: Performance comparison with FID scores calculated with SAM masks.

Thanks for the comment. We report the FID scores evaluated by cropping the object with SAM in Table E. We notice that, for the FID scores calculated with both GT mask and SAM mask, our method outperforms baseline methods with an obvious marginal which indicates the effectiveness of the proposed method.

6. Qualitative comparisons with ControlNet and Kandinsky.

Thanks for pointing this out. We add additional comparisons with ControlNet and Kandinsky in Figure 6 (github.com/iclr23anonymous739/comparison.pdf). We notice that MaskComp achieves superior performance compared to baseline methods.

7. The results of using the ground-truth mask as input to see how different methods perform with such an oracle (This tells the generation upper bound of mask-guided methods).

Table 2 (a): Performance with different conditioned masks. Occlusion rate: partial mask > noisy mask > complete mask.

We reported the results of using the ground-truth mask as the condition in Table 2 (a) and there is also a visual comparison among different conditioned masks in Figure 2. With the ground-truth mask, the generation model shows a promising performance to complete the objects. This also serves as the core motivation for our iterative mask denoising process (IMD) which starts from a partial mask to gradually generate a complete mask to boost the generation process.

We deeply appreciate the time and effort invested in reviewing our work and providing valuable feedback. Since the deadline for the discussion period is approaching and we haven't heard back from you, here's a summary of our earlier response to the concerns raised:

• Weakness 1 & 2: We discuss the motivation for using of SAM and voting with our observations and conduct additional experiments to compare the IMD process with baselines. Moreover, we clarify that the mask prediction is not simply conducted by the SAM but by the entire IMD process with both generation and segmentation stages.

• Weakness 3: We provide the suggested ablation results and incorporate the discussion in Table 4 & Line 349-353 in the revised version.

• Weakness 4: We discuss the inference speed and incorporate the analysis in Table 3 (c) and Line 295-300 in the revised version.

• Weakness 5: We report the FID score calculated with masks predicted from the generated images and update the Table 1 to better illustrate the performance comparison.

• Weakness 6: We update Figure 6 to additionally compare with ControlNet and Kandinsky.

• Weakness 7: We further discuss the results in Table 3 (a) about the performance with GT mask as condition.

We are open to further discussions if there are any aspects that remain unclear. We look forward to your continued feedback and sincerely hope that you can improve the rating based on the responses.

We thank the reviewer for the time and effort to review our paper. Our answers to the questions are as follows.

1. Readers of ICLR may expect more theoretical ideas or fundamentals that can be transferrable to stimulate other research. This paper is heavily dependent on Zhang 2023 paper.

Thanks for the important comment. We would like to further clarify that the idea behind MaskComp is to investigate the relation between image generation and segmentation. Previous work PaintSeg [A] has proved that image generation can benefit the segmentation and lead to training-free object segmentation capability. In this work, our results indicate that segmentation can also be beneficial to image generation with the proposed IMD process. To better understand the process, we also provide a theoretical analysis of the process in Section 3.3. We consider that the idea of leveraging segmentation to boost generation can be of interest to the community.

[A] PaintSeg: Training-free Segmentation via Painting, NeurIPS 2023

2. If more details of the user study are provided, it will be more convincing. How many people participate? How to select the subjects? What exactly were the sentences for questions? How was the user interface or the testing environment?

We provide information about the user study in the supplementary materials (Line 450-464). There are 16 people participated in the user study. All participants were selected from graduate students who have relevant knowledge to understand the task. We illustrate the instructions for the participants in Line 456-564. We demonstrate the user interface in Figure A (github.com/iclr23anonymous739/platform.png).