Add-it: Training-Free Object Insertion in Images With Pretrained Diffusion Models

We thank the reviewer for providing the feedback. In the following, we respond to each of the concerns in the review. Additionally, the modifications in the revision are marked in blue, for your convenience.

1. The proposed techniques are incremental, lacking sufficient novelty to make a significant contribution to the field

   We appreciate the reviewer's feedback and would like to address the concerns in two parts: first, by clarifying the contributions of our proposed techniques, and second, by highlighting additional core contributions of our work.

   Techniques Contribution: The reviewer is correct that some of our method’s components build on prior work. However, even though some techniques have been available for more than a year, object insertion remains a challenging, unsolved problem (as also noted by reviewer 5Sc6). This indicates that our application and refinement of these techniques are non-trivial and advance the state of the art.

   We also highlight that the Weighted-Extended Self-Attention mechanism is based on our novel findings regarding attention distribution within MM-DiT blocks, which the reviewer acknowledged as interesting. While previous methods have explored weighting components like text embeddings [1], our approach uniquely leverages MM-DiT attention sources—a component that has not been explored before. Unlike [1], our weighting parameters are derived from an analytical study of attention dynamics in multi-modal diffusion transformers, contributing to a deeper understanding of models as SD3 and FLUX.

   Further Contributions: Our work makes additional substantial contributions to the field. First, we demonstrate significant improvements in the task of object insertion to images, supported by quantitative metrics, human evaluations, and qualitative examples. Second, we address a critical gap in previous object addition methods: ensuring that objects are placed in plausible locations (affordance). By constructing the Additing Affordance Benchmark, we revealed that existing methods often fail to insert objects appropriately. This benchmark underscores the field's current limitations, while our method, Add-it, achieves an affordance score of 82.8, significantly outperforming the best previous method at 47.4. Finally, we present the first analysis of attention sources in the MM-DiT block and leverage this understanding to balance information from both the source and target images.

   [1] https://huggingface.co/docs/diffusers/using-diffusers/weighted_prompts#prompt-weighting

2. Include a comparison with additional tuning-free methods such as Masactrl [3], Proximal Guidance [4], and InfEdit [5]

   Per the reviewer's suggestion, we have included comparisons to MasaCtrl and InfEdit in the updated PDF. The authors of the Proximal Guidance work have not released their code, so we were unable to compare with their method. In Section A.3 of the updated PDF, we present an evaluation of these baselines along with another baseline requested by Reviewer 5Sc6. Add-it significantly outperforms these baselines, with the most notable improvement in the affordance metric, as shown in Tables 1 and 3.

3. Incorrect usage of quotation marks on line 237: The word 'choose' is improperly quoted

   We thank the reviewer for catching this error. It has been corrected in the updated PDF.

4. The findings in lines 210-212 (Attention sources distribution in the MM-DiT block) are interesting. Could the authors provide insights into the underlying reasons for this observation?

   We appreciate the reviewer's positive feedback on our findings. The newly introduced MM-DiT block integrates information from two sources: the text prompt and the generated image patches, simultaneously. By extending this block to also incorporate information from a source image, our method uniquely combines information from three sources within the same layer. In contrast, prior Unet-based diffusion models handle these sources separately—using a cross-attention layer for text prompts and a self-attention layer for image patches. The need to balance these information sources arises specifically from the architecture of the extended MM-DiT block, where all three sources are blended together within a single layer. Finally, in Section A.4 of the appendix, titled “The Role of Positional Encoding,” we further analyze how positional encoding impacts the extended attention between the source and generated images.

We are happy to address any other questions.

We thank the reviewer for providing the feedback. In the following, we respond to each of the concerns in the review. Additionally, the modifications in the revision are marked in blue, for your convenience.

1. What are Additing Affordance Benchmark advantages over other benchmarks? Provide more details

   The core strength of our proposed benchmark, and its advantage over others, is its ability to quantitatively and automatically assess whether an object was added correctly to an existing scene.

   In Lines 88-93, we highlight a significant gap in current object insertion evaluation protocols: the lack of a mechanism to assess whether an object was added to a plausible location. Our experiments reveal that existing methods often struggle to find the “right” spot for object placement, which we refer to as affordance. To address this, we introduced the “Additing Affordance Benchmark,” which includes carefully labeled regions indicating plausible insertion areas for each image sample in the dataset.

   We provide information about the benchmark in Section 4, with additional details on its construction and evaluation protocol in Section A.6, titled “Additing Affordance Benchmark.” Sample examples are shown in Figure 19.

   Following the reviewer's suggestion, we included additional discussion in section A.6 of the updated PDF, highlighting the benchmark’s advantages and necessity in the field. We believe this benchmark is a major contribution, as determining plausible insertion locations is crucial for future object insertion methods. Following Reviewer rVMn suggestion, we have expanded the benchmark from 200 to 500 samples.

2. Comparison with the MagicBrush benchmark

   In response to the reviewer's suggestion, we now added comparisons of Add-it with other baselines on the MagicBrush benchmark, as discussed in Section A.3 of the updated PDF. We include both qualitative and quantitative comparisons, demonstrating that Add-it outperforms all prior methods, including the MagicBrush model itself.

3. The paper lacks a comparison with traditional image quality metrics such as FID and IS, which are common in measuring generated images quality.

   We agree with the reviewer that incorporating additional quality metrics can be valuable. We first explain why standard FID and IS metrics are not the best choice for image editing, and then propose a modification, which we added to the paper.

   Metrics like FID and IS are commonly used in image generation but they tend to be less meaningful when evaluating image editing methods, and as a result are generally not employed in popular image editing methods and benchmarks [1,2,3,4,5,6]. This is due to two main reasons: First, image editing keeps most of the original image unchanged, making it difficult to quantify the edit influence on the score. For instance, a model that simply refrains from adding any new objects could achieve a near-perfect FID score. Second, FID performs best with datasets containing at least 5,000 samples [7], whereas image editing benchmarks are typically much smaller, often with only a few hundred samples

   With that said, we agree with the reviewer that incorporating additional quality metrics is valuable for the field. To adapt these metrics for the image addition task, we employ an off-the-shelf open vocabulary detection model to identify regions containing the added objects. We crop these regions to create a dataset of object crops, then evaluate KID and IS by comparing the cropped dataset against the original source images. This method allows us to directly assess the quality and diversity of the inserted objects. We opted for KID over FID due to its reliability on smaller datasets [7]. As discussed in Section A.3 of the updated PDF and shown in Table 6, Add-it achieves the best performance in both KID and IS metrics.

   [1] InstructPix2Pix: Learning to Follow Image Editing Instructions

   [2] EraseDraw: Learning to Insert Objects by Erasing Them from Images

   [3] MagicBrush : A Manually Annotated Dataset for Instruction-Guided Image Editing

   [4] Paint by Inpaint: Learning to Add Image Objects by Removing Them First

   [5] Emu Edit: Precise Image Editing via Recognition and Generation Tasks

   [6] Prompt-to-Prompt Image Editing with Cross Attention Control

   [7] Training Generative Adversarial Networks with Limited Data

We thank the reviewer for the positive and comprehensive feedback. In the following, we provide a response to each of the concerns in the review. Additionally, the modifications in the revision are marked in blue, for your convenience.

1. Lack of comparison with the existing methods for adding objects [1,2]. [1] Paint by Inpaint: Learning to Add Image Objects by Removing Them First [2] ObjectAdd: Adding Objects into Image via a Training-Free Diffusion Modification Fashion

   We appreciate the reviewer for highlighting these additional baselines. Regarding baseline [1] (PaintByInpaint), In Section A.3 of the updated PDF we present an evaluation of [1] along with two other baselines requested by Reviewer 9iYS. Our Add-it significantly outperforms these three baselines, with the most notable improvement in the affordance metric, as shown in Tables 1 and 3. Add-it achieves a score of 0.828, whereas the highest score among these baselines is 0.366.

   Regarding baseline [2] (ObjectAdd). This paper addresses a different task than ours. There, the user specifies the region for object addition as input, making it more akin to an inpainting task. In our paper, the location of the added object is found by the model, not given. This difference prevents a direct comparison with the other baselines. Additionally, the official code released for [2] is missing crucial files, which prevents us from running it.

2. Potential limitations of relying on SAM-2 for refining the object mask. Previous work uses self- and cross-attention maps to relieve this requirement

   Following this comment we added discussion in In Section A.5 of the updated PDF. We discuss various methods for localizing the added object during the Latent Blending step. We provide both quantitative (Table 7) and qualitative (Figure 18) comparisons of these alternatives, specifically coarse attention masks and masked-based SAM-2 prompt. Our proposed point-based SAM-2 prompting approach outperforms both alternatives.

   While it is true that SAM-2 may not always perfectly capture object boundaries, in our experiments it consistently outperforms coarse attention-based masks. Note that we do leverage self- and cross-attention maps to generate point-based prompts for SAM-2.

3. Potential biases inherited from the pretrained diffusion models

   We acknowledge the risk of potential biases and have addressed this in the Limitations section (Section 6). However, these biases are not unique to our method; they are common across all current editing approaches that rely on pretrained diffusion models. We included qualitative examples illustrating these biases in Figure 15 of the updated PDF, and discuss it further in section A.2.

We are happy to address any other questions.

We thank the reviewer for the positive and comprehensive feedback. In the following, we respond to each of the concerns in the review. Additionally, the modifications in the revision are marked in blue, for your convenience.

1. The evaluation affordance dataset is relatively small, containing only 200 images

   Following this feedback, and to strengthen our benchmark, we have added 300 additional examples, increasing the dataset to a total of 500 images. The newly added examples have been included in the updated supplementary material.

   With this increase, our benchmark now has 2.5x more object insertion samples than in other popular benchmarks such as EmuEdit and MagicBrush (only 200 for the adding task).

2. Computation Speed, running SAM2 could impact processing speed

   SAM2 is a real-time video segmentation model, and its computational time is minimal compared to the runtime of the diffusion model. In our experiments, SAM2 inference takes 0.04 seconds on average, whereas FLUX requires 6.4 ± 0.13 seconds per image. Overall, Add-it inference takes about 7.225 ± 0.05 seconds per image—only one second longer than the original FLUX model (both statistics were derived from 200 generations). This slight increase is due to caching attention maps and the additional computation for the extended attention mechanism, with SAM2 contributing only a negligible portion to this difference. We now included statistics on computation time in Section A.1 of the updated PDF.

3. Add-it text prompts may introduce additional complexity over instruction based methods

   The text prompts used in Add-it are generally quite simple, as demonstrated in Figures 1, 5, 6, and 11. These prompts are similar in structure to instruction-based prompts, making the transition between the two straightforward. For instance, given an input image with the caption “A skateboarder with no shirt doing tricks on a rail” and instruction “Add a football jersey to the torso of the skateboarder”, the Add-it style prompt would be “A skateboarder with a football jersey doing tricks on a rail”. We find that an LLM can easily translate between these formats, as we demonstrated by using ChatGPT to convert instructions into Add-it style prompts for the EmuEdit dataset. We included further details on this process in Section A.7 of our updated PDF.

4. Did the authors experiment with using a coarse mask as an input for SAM2?

   Yes, we initially experimented with using the coarse attention mask as input for SAM2. However, we found that it generally performs worse than our proposed point-based algorithm. Following the reviewer's question, we have included a discussion in Section A.5 of the updated PDF concerning alternative methods for localizing the added object during the Latent Blending step. We provide both quantitative (Table 7) and qualitative (Figure 18) comparisons of these alternatives, specifically (i) coarse attention masks, and (ii) masked-based SAM-2 prompts. Our proposed point-based SAM-2 prompting approach outperforms both alternatives, with an Affordance score of 0.828 compared to scores of 0.77 and 0.809 for the alternatives.

5. Figure 7(a) shows peak at gamma=1.07, surpassing the gama=auto setting

   You are correct. The scale value we used in our experiments is gamma = 1.05. While using gamma=auto will usually yield better results. We opted for a fixed value for simplicity and efficiency. The fixed value was determined by averaging the auto-gamma values that balanced the attention sources on a separate validation set. To avoid test leakage, we did not determine a new optimal gamma for each evaluation set individually. However, even with the fixed gamma optimized for our validation set, Add-it still significantly outperforms all previous methods. While selecting gamma=1.07 could improve results slightly, it would risk introducing test leakage.

We are happy to address any other questions.