CLIPAway: Harmonizing focused embeddings for removing objects via diffusion models

We would like to thank the reviewer for their detailed feedback which help us improve the paper further.

Applicability to Higher Resolution Images
We appreciate the reviewer's insight regarding the applicability of our method to higher resolution images. As noted, the current limitation is inherent to existing diffusion-based image inpainting methods. Our primary goal was to enhance object-removal performance within these existing constraints. Future research could indeed focus on adapting our method for higher resolutions, potentially bridging the gap between diffusion-based techniques and LaMa-like methods. This would involve significant advancements in diffusion model architectures to handle high-resolution images effectively.

Comparison with “LaMa + SD-Inpaint” Baseline
Thank you for suggesting the comparison with the “LaMa + SD-Inpaint” baseline. This is a very interesting work. We conducted this comparison using the same setup as in Table 1 of our manuscript. The results are as follows.

Incorporating SD-Inpaint as a post-process significantly improved sample quality, as indicated by the improvement in the FID score. However, the object removal metrics reveal a decline in the model’s ability to effectively remove objects. We attribute this decline to SD-Inpaint’s tendency to insert and hallucinate objects, as noted in our paper. We also would like to mention that in KID, which is the other visual quality metric, CLIPAway model achieves better scores in addition to the significantly better scores on the object removal CLIP based metrics. We appreciate the reviewer’s suggestion, which helped us highlight the trade-offs involved in using SD-Inpaint in combination with LaMa. We will include these results in our manuscript to provide a more comprehensive evaluation of our pipeline.

We also add visual results of this comparison in the attached Rebuttal Fig. 6. It can be seen that LaMa fills the erased area in a blurred way, and the SD-Inpaint, followed by LaMa, adds objects again. For example, in the first row, a horse is added to the image.

Analysis of Failure Cases
We appreciate the reviewer's suggestion to present and analyze more failure cases to highlight the limitations of our method.

One significant limitation of our method is the constraint on applicable image resolution as noted by the Reviewer, which is inherent to diffusion-based inpainting methods. Specifically, the model in a Stable Diffusion (SD) pipeline expects input images and masks to be in the 512x512 pixel resolution, while SD-XL pipelines expect 1024x1024 resolution. Deviating from these resolutions degrades the performance of the UNet in the diffusion pipeline, resulting in less effective inpainting, regardless of the quality of the embeddings provided by the CLIPAway pipeline. We will include a detailed discussion of these limitations in our manuscript to provide a comprehensive evaluation of our method.

Potential Extensions to Other Frameworks
We appreciate the reviewer's suggestion to apply our method to SDXL. After examining the SDXL-Inpaint model, we observed that the issue of object addition is even more pronounced. SDXL recommends setting the strength parameter to 0.99, using the unerased image as input, and introducing noise at this strength (for reference: https://huggingface.co/diffusers/stable-diffusion-xl-1.0-inpainting-0.1). The strength parameter determines how much the masked portion of the reference image is transformed, ranging from 0 to 1. A higher strength adds more noise to the image, requiring more denoising steps. When the strength is set to 1, maximum noise is added, leading to degradation in image quality as noted on the webpage's limitations, with ongoing work to address this issue.

Fig. 7 of the Rebuttal PDF illustrates that using the original input image with a strength of 1.0 results in lower output quality. Conversely, a strength of 0.99 often causes the model to regenerate the erased foreground object, as it retains some information about it. This setting yields a CLIP-1 score of 67.22, the lowest among those reported in Table 1 of the main paper. When combined with our CLIPAway method, the SDXL-Inpaint model achieves a CLIP-1 score of 85.91.

We will include comprehensive experiments with SDXL in the revised manuscript. We believe that eliminating the reliance on unerased images will improve results for both SDXL-Inpaint and our approach.

Lastly, we thank the reviewer for recommending that we open-source our code and for finding our work beneficial. We are pleased to announce that the code and model are ready and will be released soon.

I have carefully reviewed the authors' rebuttal and the other reviewers' comments. The authors have adequately addressed my initial concerns. I would recommend adding the additional baseline comparisons and SDXL experiments in the final version.

We would like to thank the reviewer for their thorough review and constructive feedback. We will certainly incorporate the additional baseline comparisons and SDXL experiments into the final version of the paper as recommended.

We would like to thank the reviewer for their detailed feedback.

Eliminating Shadows
We appreciate the reviewer’s concern about shadows in the inpainted images. Since our method uses SD-Inpaint as its backbone, inpainting is performed only on the masked regions, which ensures that only the areas the user wants to remove are altered, while the unmasked areas are preserved. LaMa, MAT, and ZITS++ follow a similar approach. To address the issue of shadows, we conducted additional experiments where both the objects and their shadows were included in the masks. As shown in the Rebuttal PDF (Fig. 3), this adjustment significantly improves the results by effectively removing both the objects and their shadows, leading to cleaner and more accurate inpainted images. This way, only the areas the user wants to remove are altered, the user can choose to keep the shadow for some artistic designs or choose to remove it.

Additional Visual Evidence
We thank the reviewer for their suggestion to provide more visual evidence. We agree that additional examples would better demonstrate the efficacy of our method. To address this, we have included more visual results in the Rebuttal PDF (Fig. 2) that show the behavior of foreground-background, and projected embeddings. Specifically, we observe that the foreground embedding is dominated by the foreground object, while the foreground object appears at a smaller scale in the conditional generation results of the background embedding. These new examples highlight the effectiveness of our projection block in removing objects of interest. We will update our submission to include these new results, strengthening our observations and making them more convincing.

Improving Visual Quality Assessment
To provide a more comprehensive assessment of visual quality, we have included Kernel Inception Distance (KID) as an additional photorealism metric as given below (lower is better):

As shown in the table, KID results align with the photorealism metrics reported previously. We did not use LPIPS because it necessitates reference images with the objects removed, which are not available. With the addition of KID, we now have three metrics to evaluate visual quality, complementing FID and CMMD, which were previously reported in the quantitative comparisons in the main paper.

Prompt Use in Baseline Comparisons
We understand that using a specific prompt, such as "an empty table" when removing a laptop, can sometimes improve results. However, we found that this approach does not consistently improve the baselines. Sample results demonstrating this are included in the Rebuttal PDF (Fig. 4). For example, "empty table top" prompt results in inpainting the region with adding an additional component on top of the table rather than completing it directly in the first row, "an empty runway" still adds planes to the image, or "an empty bed" adds a pillow to the bed. Additionally, generating specific prompts for every situation in a large dataset is impractical and time-consuming. Our method circumvents this issue by using vector arithmetic in the CLIP space, eliminating the need for custom prompts. We believe this makes our method more robust and user-friendly. We hope the reviewer finds this perspective agreeable.

Thank you for providing extensive additional results in the rebuttal, they do improve my understanding of the proposed method and fully resolve my concerns. I raised my score, and hope the additional results can be included in the paper.

We would like to thank the reviewer for their detailed feedback.

We appreciate the reviewer's recognition of our method's ability to leverage existing work effectively and its superior performance in removing objects smoothly compared to competitors. To address the concern regarding the qualitative judgment based on a few examples, we will extend our supplementary material with additional qualitative examples to better demonstrate the effectiveness of our method. Additionally, we would like to emphasize that the quantitative analysis provided in Table 1 shows we achieve consistent improvements over many baselines.

We appreciate the reviewer's comments on our ablation studies showing the impact of background and foreground embeddings. While background and foreground embeddings may sometimes contain overlapping information, our method aims to orthogonalize these embeddings to effectively isolate relevant background information. The orthogonal embeddings are designed to include only information related to the background. Although these embeddings are not expected to exactly match the original background of the image, they are constructed to exclude the foreground object and provide a contextually coherent background representation. To further clarify and validate the effectiveness of our approach, we provide additional examples in the Rebuttal PDF (Fig. 2) which show the behavior of foreground-background, and projected embeddings. We will add more examples in the revised paper. We would like to thank the reviewer for bringing this to our attention.

Regarding Contextual Understanding
Since we use SDInpaint as our backbone, inpainting is carried out only on the regions identified by the provided masks as happens in other competing methods as well, LAMA, MAT, ZITS++. To address the reviewer's concern about the method not removing shadows, we conducted new experiments where both objects of interest and their shadows were included in the corresponding masks. We share these results in the Rebuttal PDF (Fig. 3). Our method produces highly satisfactory results, effectively eliminating the objects and their shadows if the user prefers to.

Clarification on "Plug and Play" Capability
We apologize if the use of the term "plug and play" caused any confusion. Our method bridges the AlphaCLIP embedding space to the CLIP embedding space through projections, which requires some architectural decisions during the training process. However, once trained, our CLIPAway module can be integrated into any framework that employs the CLIP embedding space or other CLIP embedding spaces with similar properties — without requiring additional training. As demonstrated in Table 1, adding CLIPAway to SD-Inpaint, Blended Diffusion, and Unipaint frameworks significantly improves their performance on the object removal task.

Computational Overhead and Efficiency
While diffusion models do have higher computational overhead compared to GAN models, they are currently favored due to the significant improvements they offer. Additionally, there is extensive research focused on optimizing diffusion models and reducing the number of required denoising steps to improve their efficiency. Therefore, we believe it is important to continue improving these models for various tasks. . We are not aware of any straightforward post-processing technique that can elevate simpler inpainting methods to a state-of-the-art level of quality. Reviewer GcHN brought the LaMa + SD-Inpaint baseline to our attention, where the faster LaMa model generates the initial inpainting results and SDInpaint, which is a diffusion-based inpainting method, serves as post-processing. However, this combination is more computationally intensive than our approach and yields worse results.

Expanding Beyond Object Removal
While our main focus in the paper is on object removal, our framework, can be applied to other tasks. Based on the Reviewer's comment, we investigated the capabilities of our framework on reference-based background inpainting as given in the Rebuttal PDF (Fig. 5). By computing the projected embedding of a reference image representing the target background, we inpaint the background of the input image while preserving its foreground. We will revise our submission to include these new results, demonstrating that our method can open directions for image manipulation and general image inpainting problems. Additionally, we would like to emphasize that object removal is a significant task in its own right, and previous research has proposed generating datasets specifically for this purpose.

many thanks to the authors for the detailed response. In my view, this is still a limited task that can be dealt with in different ways with varying success. However, I am happy with the answers that the reviewers have provided and have raised score accordingly

We would like to thank the reviewer for their detailed review.

We appreciate the reviewer's feedback regarding the simplicity of our method. We share the same preference for simplicity over complex solutions. While our solution might seem obvious now, previous research attempted to address object removal tasks by generating synthetic datasets. For instance, InstInpaint [30], MagicBrush [33], InstructPix2Pix [2], and concurrently ObjectDrop [26] explored different approaches to create training datasets for this purpose. Our method did not require a dataset generation and achieve better results than competing methods which makes it more elegant.

The reviewer asks about the flexibility of CLIP embedding spaces. This is an excellent point. Our method is not confined to the CLIP embedding spaces used in our initial experiments. To demonstrate this flexibility, we trained the MLP for AlphaCLIP with model keys VIT-L/14@336px, and VIT-B/16 in addition to the VIT-L/14 that was presented in the paper. After training, we evaluated each model using the same setup as in Table 1 of the manuscript. The qualitative scores for each model are provided in the table below:

The results consistently improve the SD-Inpaint model with different CLIP embeddings.

The reviewer asks if the projection could be done on the AlphaCLIP space rather than OpenCLIP embedding space. Our projection method is not restricted to the OpenCLIP embedding space. Since AlphaCLIP’s vision transformer is trained with similar objectives as the CLIP vision transformer, their feature spaces are conceptually similar. Therefore, projections can be performed in the AlphaCLIP embedding space or other CLIP embedding spaces with similar properties. To support this, we evaluated the projection method on the AlphaCLIP feature space (projection on AlphaCLIP space followed by MLP) using the same setup as described in Table 1 of the manuscript. The results, shown below, confirm that our projection approach is applicable beyond the OpenCLIP embedding space.

We will include these results in the revised paper. We believe they will make the paper more comprehensive. We would like to thank the reviewer for these valuable suggestions.

We also provided qualitative example object removal results in the Rebuttal PDF (Fig. 1), further confirming the effectiveness of our projection approach across different CLIP embedding spaces. These results illustrate our method’s flexibility and robustness, supporting its broader applicability.

Training the MLP layer for AlphaCLIP with the model key VIT-B/16 using a single Nvidia A40 GPU takes approximately 7 hours which is a negligible GPU time compared to training the diffusion models from scratch.