InstantSwap: Fast Customized Concept Swapping across Sharp Shape Differences

We sincerely thank the reviewer for their insightful comments and recognition of our work. We greatly appreciate the acknowledgment of our comprehensive and promising experimental results, fluent writing, and the benchmark we constructed. We have refined the paper, incorporated additional experiments, and clarified the following points in the revised version. The changes relevant to your concerns are marked in brown and we will remove these colors in the final version.

W1: Detailed introduction on customization methods

Thanks for pointing this out. We have added a detailed introduction of the customization methods we used in Section 3.2.3 and Appendix L.

• Specifically, in the target branch of our method, we first convert the target concept into semantic space with DreamBooth. We use a specific rare token (e.g., sks) to represent the concept.

• During concept swapping, we can use this rare token in the target prompt to semantically represent our target customized concept and perform the customized concept swapping.

W2: Apply our method to DiT-based architectures

• Thank you for your illuminating suggestions. Our SECR enhances cross-attention maps of concepts using semantic information. This technique can be applied to any generative model with a cross-attention layer, regardless of architecture, to enrich semantic information in regions of interest.

• Since there is no explicit cross-attention layer in SD3 [4] and it only contains a self-attention layer, we have included a detailed analysis in Appendix M to explore the architecture of SD3 and investigate how our SECR could be applied to it.

• We focus on the interaction between images and text within SD3. The results indicate that the attention layer in SD3 can be decomposed into a cross-attention operation between image and text and a self-attention operation of the image or text. Our SECR can be applied to the cross-attention part of the attention layer in SD3 to enhance semantic information in regions of interest.

• However, since SD3 is trained based on Rectified Flow, existing score distillation methods (e.g., SDS [5], DDS [6]) cannot be directly integrated with it. In future work, we will continue to explore this potential direction.

[4] Scaling Rectified Flow Transformers for High-Resolution Image Synthesis. In ICML, 2024.

[5] Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2023.

[6] Delta denoising score. In ICCV, 2023.

W3: Impact of semantic-enhanced operation on the naturalness of images

We employ the semantic-enhanced operation on foreground concepts within the cross-attention layers to derive an enhanced representation of the source and target objects. This enhanced representation effectively interacts with the features of surrounding objects and the background in the self-attention layers, mitigating inconsistencies and unnaturalness in the target image.

We have provided detailed results in Appendix N. The qualitative results in Fig. 23 show that:

• Our method seamlessly inpaints objects in the background that the foreground concept occludes (upper left examples).
• It generates reflections of the target concept in water (upper right examples).
• It accurately places the target concept on background objects (lower left examples).
• It also generates natural shadows of the target concept that are consistent with the environmental lighting angles (upper left and lower right examples).

Moreover, considering that image naturalness is a subjective human assessment, we use the human preference metric [7] to quantify the naturalness of images. A higher score indicates a better alignment with human preferences. The results show that our method is the only one that receives a positive human evaluation score, demonstrating a significant advantage over other methods.

[7] Imagereward: Learning and evaluating human preferences for text-to-image generation. In NIPS, 2023.

Dear Reviewer VFJh:

Thank you once again for dedicating your valuable time to reviewing our paper and providing constructive comments! As the end of the discussion period approaches, we kindly ask if our responses have satisfactorily addressed your concerns. If you have any further inquiries about our method, or if you still have any concerns about our paper, please do not hesitate to inform us. We are more than willing to engage in timely discussions with you.

Sincerely,

The Authors

Dear Reviewer VFJh:

We would like to thank you again for your valuable feedback on our paper.

As the period for the Author-Reviewer discussion is closing very soon, we would like to use this opportunity to kindly ask if our responses sufficiently clarify your concerns. We sincerely appreciate your time and consideration.

Best Regards,

The Authors

Dear Reviewer VFJh:

We notice that all other reviewers have posted their post-rebuttal comments to our response but we still have not yet received any further information from you. We greatly appreciate your initial comments and fully understand that you may be extremely busy at this time.

As the deadline for the Reviewer-Author discussion phase is fast approaching (less than two days left), we respectfully ask whether we have addressed your questions and concerns adequately. Your feedback would be greatly appreciated, and we would be delighted to engage in further discussions if needed.

Best Regards,

The Authors

We sincerely thank the reviewer for their insightful comments and recognition of our work, especially for acknowledging our comprehensive experiments, solid results, clear motivation, and smooth writing. We have refined the paper, incorporated additional experiments, and clarified the following points in the revised version. The changes relevant to your concerns are marked in blue and we will remove these colors in the final version.

W1: Customization is time-intensive

• The overall time added is very mild. As shown in the table below, DreamBooth requires about 8.5 minutes to learn a single concept. Meanwhile, our method can be combined with other more efficient customization methods such as Custom Diffusion [2] which only requires about 3.5 minutes for each concept.

We have added the above discussion in Appendix J. The visual results of our method combined with Custom Diffusion can be found in Fig.22.

[2] Multi-concept customization of text-to-image diffusion. In CVPR, 2023.

W2: Combine our method with Textual Inversion

Thanks for your suggestion. Our method can indeed be combined with other customization methods such as Textual Inversion (TI) [3]. We have included results in Appendix K and present our quantitative results here. Please refer to our modified manuscript for more qualitative results and analysis.

• The results show that our method integrates well with Textual Inversion, achieving similar performance in background consistency and inference time.
• Due to the limited customization capabilities of Textual Inversion, its foreground consistency is not as high as with our original method.

[3] An image is worth one word: Personalizing text-to-image generation using textual inversion. In ICLR, 2023.

W3: More failure cases

We have presented failure cases in the limitations part of Appendix P. We have added more failure cases to the modified manuscript and conducted a detailed analysis. Specifically, due to the limited capability of the base diffusion model (e.g., Stable Diffusion), it is unable to perceive the relative sizes between specific concepts. We visualize this phenomenon in Fig. 25:

• First row: an extremely large cup.

• Second row: a toy duck the size of a little girl.

• Third row: a pair of sunglasses nearly as large as the table.

Similar issues occur in all compared methods. Nevertheless, our method still achieves the best concept swapping results.

W4: How to expand the foreground region

• The goal of Customized Concept Swapping is to smoothly replace the foreground object with the customized object. We care about (1) whether the foreground region is re-occupied with the customized object and (2) whether the transition between the foreground and background remains natural and smooth. As a 2D image editing task, whether the physical size of an object is consistent with the surrounding environment (highly dependent on the perspective angle) is not a focus in this field.

• As we mentioned in the limitation part of Appendix P, perceiving the relative size of specific concepts is very challenging for the current base diffusion model. Previous image editing methods also struggle to effectively manage this issue.

• If we really want to handle this scenario, we can expand our obtained bounding by adjusting the threshold $\beta$ adaptively during the swapping process to make the bounding box larger, thereby expanding the foreground region.

Dear Reviewer iqew:

Thank you once again for dedicating your valuable time to reviewing our paper and providing constructive comments! As the end of the discussion period approaches, we kindly ask if our responses have satisfactorily addressed your concerns. If you have any further inquiries about our method, or if you still have any concerns about our paper, please do not hesitate to inform us. We are more than willing to engage in timely discussions with you.

Sincerely,

The Authors

Dear Reviewer iqew:

We would like to thank you again for your valuable feedback on our paper.

As the period for the Author-Reviewer discussion is closing very soon, we would like to use this opportunity to kindly ask if our responses sufficiently clarify your concerns. We sincerely appreciate your time and consideration.

Best Regards,

The Authors

We sincerely thank the reviewer for their insightful comments and recognition of our work, particularly for acknowledging the integrity, effectiveness, and efficiency of our method. We have refined the paper, incorporated additional experiments, and clarified the following points in the revised version. The changes relevant to your concerns are marked in red and we will remove these colors in the final version.

W1: Theoretical analysis of background gradient masking (BGM)

Thanks for your suggestion.

• As shown in Fig. 3 of our paper, a complete optimization step of our method consists of two stages: forward pass and backward propagation. During the forward pass, we input the source and target images and compute gradient $\nabla_{z} \mathcal{L}$ as $\nabla_z \mathcal{L}=w(t)\left(\epsilon_\phi\left(z_t, t, \tau\left(P_t\right)\right)-\hat{\epsilon}_\phi\left(\hat{z_t}, t, \tau\left(P_s\right)\right)\right.$ (Eq. 10 of the main paper). No mask is applied during gradient computing, allowing sufficient interaction between the foreground and background within the self-attention layer. This ensures that no distribution shift occurs between the foreground and background in the gradient $\nabla _{z} \mathcal{L}$.

• Furthermore, the gradient $\nabla_{z} \mathcal{L}$ shares the same dimensions as the image latent $z$ (see more gradient visual results in Fig. 18 of the Appendix). The backward propagation of the gradient $\nabla_{z} \mathcal{L}$ is a pixel-wise update on $z$. Based on this, we directly apply the mask on the gradient of the background pixels in $\nabla_{z} \mathcal{L}$ before backward propagation, resulting in our $\nabla_{z} \mathcal{L}_{BGM}$.

• The update of our gradient on the foreground is the same as that of $\nabla_{z} \mathcal{L}$ on the foreground. Since there is no distribution shift in $\nabla_{z} \mathcal{L}$, our $\nabla_{z} \mathcal{L}_{BGM}$ similarly avoids introducing distribution shifts in the foreground update.

• Besides, as shown in Fig. 23 in the Appendix, our method consistently generates results that are natural and coherent in both the foreground and background, further proving that there is no distribution shift between the foreground and background in our $\nabla_{z} \mathcal{L}_{BGM}$.

• If any explanation remains unclear, please do not hesitate to reach out. We are very happy to clarify and address any misunderstandings.

Dear Reviewer nPYy:

Thank you once again for dedicating your valuable time to reviewing our paper and providing constructive comments! As the end of the discussion period approaches, we kindly ask if our responses have satisfactorily addressed your concerns. If you have any further inquiries about our method, or if you still have any concerns about our paper, please do not hesitate to inform us. We are more than willing to engage in timely discussions with you.

Sincerely,

The Authors

We appreciate your invaluable feedback! We shall exert our utmost efforts to address your inquiries:

Q1 Adapt InstantSwap to other Stable Diffusion models

• We combine our method with Stable Diffusion 1.5 and conduct comprehensive experiments. We present the quantitative results in the table below. We also provide qualitative results in: https://anonymous.4open.science/r/ICLR_InstantSwap-3030/SD1.5.pdf

• The results show that our method can integrate well with Stable Diffusion 1.5.

Q2 Handle concepts with complicated textures

• Generating high-fidelity concepts with complicated textures is indeed a challenging problem. We analyze that it is primarily caused by customization methods like DreamBooth, which struggles to consistently generate complex concepts in different images.

• We provide an example in the anonymous link below, where we directly use DreamBooth with the target prompt to generate images. It is evident that concepts generated by DreamBooth are different for different images. https://anonymous.4open.science/r/ICLR_InstantSwap-3030/more_failure_case.pdf

• However, even if DreamBooth cannot effectively customize these complex concepts, our method still completes the concept swapping precisely (see qualitative results in Q3 below). Besides, our method can be integrated with other customization methods despite DreamBooth, which is discussed in W1 and W2 of Reviewer iqew.

Q3 More challenging concepts

• During the experiment, we pair each concept from ConceptBench with each source image from SwapBench, resulting in a total of 1,600 images for evaluation. This is larger than the 700 images in PnPInv [11] and the 55 images in PnP [12].

• Additionally, we include more uncommon challenging concepts from DreamBench++ [13]. Due to time constraints, we only conduct experiments on the first 50 images in SwapBench. We present the quantitative results in the table below. We also provide qualitative results in: https://anonymous.4open.science/r/ICLR_InstantSwap-3030/challenging_concept.pdf

• The results show that these challenging concepts indeed reduce the foreground consistency of our method. Nevertheless, our approach still faithfully completes the concept swapping and surpasses all compared methods on all metrics. We will include more challenging concepts, conduct experiments on the whole SwapBench, and report the results in the final version.

[11] Pnp inversion: Boosting diffusion-based editing with 3 lines of code. In ICLR, 2024.

[12] Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation. In CVPR, 2023.

[13] Dreambench++: A human-aligned benchmark for personalized image generation. In Arxiv, 2024.

Lastly, we thank you once again for your response. If you have any further questions, we would be delighted to continue the discussion with you.