Q5: Why not use Diffree to address mask-free removal? This should also be addressed by OABench.

A5: We appreciate your interest in extending Diffree to mask-free text-guided object removal. While OABench could indeed facilitate training for such tasks, our focus in this paper is on object addition, which presents unique and significant challenges:

1. Complexity of Object Addition: Adding new objects requires the model to determine appropriate placement, scale, and integration with the existing scene, without any prior information about the object’s presence.
2. Existing Object Removal: In object removal, the target object is already present, and its location and size are known, making the task more straightforward as the model can directly manipulate the existing content.

We chose to concentrate on object addition to address these challenges comprehensively. Developing Diffree for mask-free object removal is an interesting direction for future work, and we believe our approach and dataset could be adapted to support this extension.

Q6: What did "with/without mix with previous image" mean in Figure 8?

A6: Thank you for pointing out the need for clarification. In Figure 8, “with/without mix with previous image” refers to whether we perform a mix operation using the mask generated by the OMP module during iterative object additions. (1) With Mix: We use the OMP generated mask to mix the newly generated object into the original input image, ensuring that the background remains consistent across iterations. (2) Without Mix: We directly use the output image from the previous iteration without mix, which can lead to background degradation over multiple additions.

We have updated the caption of Figure 8 in the revised manuscript to make this clearer. The ablation study demonstrates that using the OMP module and performing the mix operation significantly improves background consistency and quality during iterative additions. This capability was not possible in previous text-guided methods due to the lack of available object masks.

For more details about the OMP module and its role, please refer General Response.

Q3: The generalization of the Diffree should be concerned. As shown in Figure 9 (a), the generation failed while the mask was very small.

A3: Thank you for raising the concern about Diffree’s generalization, specifically regarding Figure 9(a). We would like to clarify that:

Figure 9(a) Clarification: The image on the right side of Figure 9(a) is generated by AnyDoor [1] using Diffree's mask, not generated by Diffree. The dog generated by Diffree (left side) is depicted facing away, which we consider a plausible variation and maintains consistency with the background.

The following is the evidence we believe supports generalization:

1. Use of Real Images: Diffree is trained using real images as outputs, avoiding artifacts commonly introduced by synthetic datasets or inpainting models. This ensures that the generated objects are naturally integrated with the background.

2. Overfitting Analysis: To assess potential overfitting in input, we further conduct experiments where we removed using an inpainting model (e.g., PowerPaint [2]) from an image containing multiple identical objects and then added back using Diffree to see if it would be added onto the same position and size. As shown in Figure A13 of the Appendix, the added objects appeared at different positions and sizes, indicating no overfitting to artifacts and demonstrating generalization.

3. Diverse generation conditions: We have successfully applied Diffree to: 1) Real images without artifacts. 2) Anime images (Figures A2 and A3 in the Appendix). 3) Adding objects not present in our dataset, demonstrating robustness to out-of-distribution scenarios.

We acknowledge certain limitations of Diffree in Section A4 of the Appendix. However, the evidence suggests that Diffree generalizes well across different contexts, maintaining high-quality results without significant failure due to mask size or overfitting.

[1] Chen, Xi, et al. "Anydoor: Zero-shot object-level image customization." CVPR 2024.

[2] Zhuang, Junhao, et al. "A task is worth one word: Learning with task prompts for high-quality versatile image inpainting." ECCV 2024.

Q4: For the object addition guided completely by texts, the authors should provide more results to verify whether the proposed method could handle different orientational descriptions.

A4: Thank you for this important suggestion. To enhance precise control in object addition, we extended our model by re-labeling our dataset with accurate location descriptions using GPT-4o-mini and retrained Diffree with these detailed annotations and original annotations.

To address your concerns, we conducted experiments using contextually detailed prompts specifying different locations within the same scene or involving multiple similar objects. As shown in Figure A14, our model can accurately add objects based on context-related descriptions (e.g., “Add a red knitted wool hat on the head of the doll on the left”). This demonstrates that Diffree can be extended to handle more precise control. We plan to further enhance this capability by constructing larger datasets with precise annotations.

We sincerely thank Reviewer js6X for their valuable time and insightful comments on our work. We have summarized the updating of our revision in Summary of Paper Updating and provided an in-depth discussion about our Object Mask Predictor (OMP) module in Genera Response. Below, we address each of your concerns individually.

Q1.1: Diffree is trained to predict precise object masks, most objects contain shadows or reflections, which are not included in object masks. Without proper shadow and reflection generation, the inserted objects suffer from "copy and paste" artifacts.

A1.1: Thank you for highlighting the importance of shadows and reflections in object insertion. We acknowledge that including these elements is crucial for realistic results. However, Diffree effectively handles them for the following reasons:

1. Diffree is trained to add objects in images through text and output the masks of the objects, instead of predict object masks. The model learns to render objects within the scene context, naturally incorporating appropriate shadows and reflections to generated images.

2. Use of Real Images in Training: Our data collection process utilizes real images with objects as training targets, unlike previous methods that may rely on synthetic data. This ensures that the generated objects in Diffree inherit the inherent shadows and reflections present in real-world scenarios, leading to seamless integration with the background (e.g., avocado in Figure A14).

3. Objects Casting Long Shadows: We recognize that existing mask-guided and text-guided inpainting methods pose challenges for shadow generation, especially for objects casting long shadows. These challenges arise for two reasons: first, there is a misalignment between the data masks and the objects and actual shadows (e.g., long shadows); second, for mask-guided method, it is challenging for users to draw the estimated masks of long shadow area with objects , which is also challenging for the model understands the respective parts of shadows and objects in the mask. Therefore, combining inpainting works with shadow generation works can lead to better results for objects casting long shadows.

   To address this, Diffree’s OMP module generates object masks that can be integrated with shadow generation methods [1], as shown in Figure A11 of the Appendix. These methods are designed to produce plausible shadows for composite objects using the object mask and the composite image. By combining Diffree with such techniques, we can generate coherent objects with realistic long shadows, enhancing the overall visual quality.

   This integration reflects the good scalability of Diffree and its ability to produce realistic results without “copy and paste” artifacts.

[1] Liu, Qingyang, et al. "Shadow Generation for Composite Image Using Diffusion Model." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Q1.2: The metric of background consistency is also problematic.

A1.2: Thank you for expressing concerns about the metric used for evaluating background consistency. We agree that an appropriate metric is essential for accurate assessment.

We use the Learned Perceptual Image Patch Similarity (LPIPS) metric [1], which is widely recognized for measuring perceptual similarity between images. To specifically evaluate background consistency, we focus on the regions outside the generated object:

1. We replace the area of the generated object in the output image with the corresponding area from the input image. This isolates the background for comparison.
2. We compute the LPIPS score between the input image and this modified output image, effectively measuring differences only in the background regions.

This approach allows us to quantify how closely the background in the generated image matches the original background. An ideal score would indicate no perceptual difference, signifying perfect background consistency.

We believe this method provides a robust and accurate evaluation of background consistency, ensuring that any changes in the background introduced during object insertion are appropriately measured.

[1] Zhang, Richard, et al. "The unreasonable effectiveness of deep features as a perceptual metric." CVPR 2018.

Q2: Missing discussion and ablation about the necessity of the joint training of OMP and diffusion model.

A2: We appreciate your suggestion to elaborate on the necessity of ablation study and joint training for the OMP module and the diffusion model. In Figure 8, we present an ablation study demonstrating the impact of omitting the OMP module. The results show that without the OMP module, background consistency deteriorates rapidly in iterative additions. OMP is a lightweight module that outputs the mask of added objects. Please refer General response (Point 3) for more details about the joint training of OMP and diffusion model.

Dear Reviewer js6X,

Thanks again for your valuable time and insightful comments. We hope that our response can address your concerns. As the deadline for Author/Reviewer discussion period is approaching, we really appreciate if you can let us know whether there still exists any further question about the paper or the response. We are looking forward to further discussion.

Best regards,

Authors of Paper 2788

Q8: Ambiguity in Object Placement in Complex Scenes: The method struggles with complex images involving multiple entities (e.g., two people). When required to add an object to one specific entity, the model may face ambiguity, as there is limited functionality for users to adjust the mask and specify the target placement.

A8: Thank you for your question. As discussed in A2 (Point 2), we retrained Diffree to improve its ability to add new objects through context-related descriptions. For images with multiple entities, Diffree achieves precise object addition by utilizing detailed prompts that specify attributes or positions (e.g., “Add a red knitted wool hat on the head of the doll on the left” in the Figure A14). We believe this effectively addresses your concerns.

Q9: How does Diffree handle ambiguities in scenes with multiple entities (e.g., if there are two people in an image, how does it decide which one to add the object to)? Could user input be incorporated to clarify target placement in such cases

A9: Please refer to our response in A8.

Q10: Could more fine-grained prompts be incorporated or generated to enhance the model's ability to handle specific requests for object details and interactions within a scene?

A10: Yes. As mentioned in A2 (Point 2) and A8, Diffree benefits from detailed textual descriptions (e.g., “Add a red knitted wool hat on the head of the doll on the left”) that include specific attributes, locations, and interactions. By incorporating precise language in the prompts, users can guide the model to produce more accurate and contextually appropriate additions.

Q5: Inconsistency in Visuals (Figure 1): In Figure 1, example 2 appears visually inconsistent, with an unrealistic rendering of the added object. This raises questions about the model's object realism under certain conditions.

A5: We appreciate your observation regarding the visual consistency in Figure 1. In this example, we performed 16 iterations of object additions to the image, with many newly added objects interacting with previously added ones (e.g., [fabric sofa, table, carpet, teddy bear, ice cream, birthday cake]). Achieving such a high number of additions with complex interactions is challenging and was nearly impossible with previous text-guided methods.

We recognize that multiple overlapping additions can degrade realism due to compounded inpainting effects on image quality in these overlapping areas, challenging all methods. Conversely, this scenario highlights the importance of our OMP module in maintaining background consistency and overall image quality. As shown in Figure 8, without the OMP module, the image quality deteriorates significantly after multiple additions.

Despite these challenges, the positions and sizes of the new objects remain reasonable and the perspective relationship and reflection of objects are consistent with the environment, demonstrating Diffree’s capability in rational object addition even in complex scenarios.

Q6: Lack of User Study, particularly on user satisfaction and perceived realism

A6: Thank you for pointing out the importance of user studies in evaluating satisfaction and perceived realism. The user study results has provided in Figure A6 of the Appendix on the reasonableness of added object locations.

For better address your concerns, we conduct a further user study to assess overall satisfaction. We compare the results of Diffree and InstructPix2Pix across 1,000 cases from the COCO and OpenImages datasets. Participants were asked to evaluate which method produced more satisfactory results. The results are as follows:

• User study of overall satisfaction in 1000 cases of COCO and OpenImages dataset, with the win rate defined as the ratio of our Diffree wins to the total wins by either Diffree or InstructPix2pix.

  Dataset	InstructPix2Pix Win	Tie	Diffree Win	Win Rate
  COCO	13	110	877	98.5%
  OpenImages	23	187	790	97.2%

These results demonstrate that Diffree significantly outperforms InstructPix2Pix in terms of user satisfaction and perceived realism. We believe this comprehensive user study effectively showcases the advantages of our method.

Q7: The model is only compared against PowerPaint and InstructPix2Pix. Including additional comparison baselines would provide a more comprehensive assessment.

A7: Thank you for your question. We have included qualitative comparisons with additional methods [1,2,3,4] in Figure A10 of the Appendix. We would like to emphasize two key points:

1.Different Data Collection Processes: Our data collection process differs significantly from other text-guided methods. Previous methods often struggle with object addition, yielding low success rates (e.g., 18% for InstructPix2Pix, as shown in Figure 7). Our approach uses real images as training targets and images with removed objects as inputs, along with a complex filtering process to obtain high-quality data. This results in improved consistency where the background remains unchanged.

2.Comparisons with Mask-Guided Methods: Although mask-guided methods like PowerPaint [1] require additional input conditions (e.g., masks), we have demonstrated that Diffree can yield competitive results without such inputs. In Appendix A7, we discuss the challenges of input masks and provide a fair comparison by supplying complete masks for methods that require them. As shown in Figure A10, methods that lack mask information fail to produce satisfactory results.

We provide a comprehensive assessment of Diffree’s performance relative to existing methods through these comparisons.

[1] Zhuang, Junhao, et al. "A task is worth one word: Learning with task prompts for high-quality versatile image inpainting." ECCV, 2024.

[2] Nichol, Alex, et al. "Glide: Towards photorealistic image generation and editing with text-guided diffusion models." ICML 2022.

[3] Li, Yuheng, et al. "Gligen: Open-set grounded text-to-image generation." CVPR 2023.

[4] Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." CVPR 2022.

We would like to express our sincere gratitude to Reviewer zVNG for their valuable time and effort in reviewing our work. We have summarized the updating of our revision in Summary of Paper Updating and provided an in-depth discussion about our Object Mask Predictor (OMP) module in Genera Response. Below, we address each identified weakness.

Q1: Minor Typographical Issue: a missing period at the end of line 101.

A1: Thank you for pointing this out. We have corrected the typo in the updated manuscript.

Q2: Since the dataset primarily relies on the COCO dataset, prompts are often generic object labels rather than detailed, fine-grained descriptions. This limitation can hinder the model's ability to respond to specific or interactive prompts.

A2: We appreciate your concern regarding the use of generic object labels in training. We address your points on specific object attributes and context-based interactions separately:

1. specific object attributes

Despite the generic labels during training, Diffree demonstrates strong generalization and effectively responds to detailed, fine-grained prompts. As shown in Figure A3 of the Appendix, Diffree successfully follows instructions such as “add shiny golden crown” and “add reflective sunglasses,” producing appropriate additions that match the detailed descriptions. This capability stems from our fine-tuning approach based on the pre-trained SD1.5, which inherently generates objects with specific attributes from text descriptions. This reflects Diffree’s robustness and adaptability, making it applicable to a wide range of scenarios.

2. context-based interactions

To enhance precise control in object addition, we extended our model by re-labeling our dataset with accurate location descriptions using GPT-4o-mini and retrained Diffree with these detailed annotations and original annotations.

To address your concerns, we conducted experiments using contextually detailed prompts specifying different locations within the same scene or involving multiple similar objects. As shown in Figure A14, our model can accurately add objects based on context-related descriptions (e.g., “Add a red knitted wool hat on the head of the doll on the left”). This demonstrates that Diffree can be extended to handle more precise control. We plan to further enhance this capability by constructing larger datasets with precise annotations.

Q3: Further clarification about the description of Object Mask Predictor (OMP) module. why the image latent and latent noise are concatenated as inputs for the OMP module during both training and inference phases?

A3: Thank you for seeking clarification on the OMP module. We further discusse the OMP module in detail in Appendix A7 and the General Response (Point 3). The concatenation of the image latent and noise-free latent allows the OMP module to access both the input image information and the predicted latent representation. This combination enables the OMP to predict the object mask during the initial decoding steps, as the noise-free latent contains essential information about the generated object. By utilizing both inputs, the OMP module can produce accurate masks corresponding to the objects being added, facilitating efficient integration with various applications. We also updated the caption of Figure 6 to provide a clearer explanation of our Diffree's training process.

Q4: Figure 6 Presentation: Figure 6 needs a more detailed caption to allow readers to follow the workflow easily by reading only the figure caption.

A4: Thank you for the suggestion. We have updated the caption of Figure 6 to provide a clearer explanation of our Diffree's training process.

Thank you for your response and effort in addressing my concerns. While I appreciate your attempt to clarify the points raised, there are several critical areas where the work still falls short.

A2.1

The attributes presented by the authors fail to convince me of the robustness of the proposed method. The attributes of the objects are overly specific. For example, prompts such as “glasses with a corgi on the left side and a duck on the right side” require a level of detail that your method seems ill-equipped to handle. More intricate and realistic prompts of this nature would likely challenge the effectiveness and scalability of your approach.

A2.2

You mentioned re-labeling the dataset using GPT-4. However, the process of re-labeling lacks transparency. How exactly was the re-labeling conducted? How did you ensure objectivity and detail in this process? Additionally, how was this evaluated to confirm its efficacy?

The example provided demonstrates functionality primarily for location attributes, but what happens when users require specific modifications such as:

• Adjusting the size (“make it slightly bigger”): What if the adjustment becomes exaggerated?
• Changing the shape (“curve it more elegantly”): How would your method handle such structural transformations?
• Altering partial color attributes (e.g., “change the color of the mirror’s border”): How is this managed?

The lack of discussion around these nuanced cases is a significant oversight.

A5

I did not specifically mention "complex interactions" between objects in my earlier comments. My concern here is that the objects depicted in Figure 1 appear highly unnatural and unrealistic. They look as though they have been hard-pasted into the scene, lacking any visual cohesion or natural blending. This issue undermines the claims of your method's ability to generate realistic outputs.

A6

The user study appears to be rushed, poorly planned, and subjective. Several key details are missing:

• Participant demographics: Who were the participants? What were their ages, genders, or professions?
• Evaluation process: How was the evaluation conducted? Was the process designed to ensure objectivity?
• Setup and methodology: What specific setup was used for the study? Were participants given clear and standardized tasks?
• Evaluation metrics: What does a "win" or "loss" mean? Is "win" based on realism, retention of background integrity, or some other metric? Without clear definitions, the reported results seem arbitrary and subjective.

Drawing meaningful conclusions from such a study is highly problematic without addressing these issues.

Summary

While I acknowledge the effort made in responding to my concerns during the rebuttal period, the submission still lacks the necessary rigor and detail required to meet the standards of ICLR. The paper shows potential but needs significant refinement. The evaluations, methodologies, and overall presentation require a more meticulous and objective approach.

I am still deliberating on whether to lower the score. My final decision will depend on the perspectives of other reviewers and their comments.

We appreciate the reviewers’ thorough feedback on our manuscript. Below are our responses to the specific concerns raised:

1. Response to Reviewer‘s A2.1: ill-equipped to handle detail prompts such as “glasses with a corgi on the left side and a duck on the right side”

   As highlighted in A2: context-based interactions, our model is capable of accurately adding objects based on detailed, context-specific descriptions. For instance, the example “Add a red knitted wool hat on the head of the doll on the left” in Figure A14 demonstrates the model’s ability to handle similar detailed prompts effectively.

2. Response to Reviewer‘s A2.2: users require specific modifications (e.g., "curve it more elegantly")

   As noted in the opening sentence of our Abstract: "this paper addresses an important problem of object addition". Not for specific modifications.

3. Response to Reviewer‘s A2.2: The process of re-labeling

   Regarding re-labeling the dataset‘s details using GPT-4o-mini:*

   We provide the following inputs to GPT-4o-mini for precise re-labeling:

   1. Cropped area image of the object to help GPT-4o-mini understand the object’s attributes.
   2. Original image containing the object to establish context and correspondence.
   3. object label description.
   4. A Prompt of the object to guide the task: You will be provided with the following: 1.A real image of a scene. 2.A cropped image of a specific object from the scene. 3.The category text of the object (e.g., ‘cup’, ‘chair’). Based on this information, generate a concise description of the object’s appearance and its spatial position in the scene. The description should be no longer than 20 words and focus solely on the object and its immediate spatial relationship. Example: ‘A transparent cup on the table.’ Avoid adding unnecessary context or details beyond the object’s appearance and position.

   Using these inputs, GPT-4o-mini generates descriptions detailing the specific attributes and relative positional relationships of the objects through structured prompts.

4. Response to Reviewer‘s A5: Figure 1 appear highly unnatural and unrealistic and not mention "complex interactions" between objects

   We argue that Figure 1 demonstrates a good level of perspective alignment and consistent reflections between objects and their environment, especially considering the cumulative 16 iterations of text-guided object additions.

   Regarding potential reasons ("complex interactions" you not mention) for highly unnatural and unrealistic results you might think of, we analyze that the added interaction areas of multiple objects may lead to a decline in quality, which is an issue faced by all methods.

5. Response to Reviewer‘s A6: The user study appears to be rushed, poorly planned, and subjective, which asked for supplemented during the rebuttal period. Lack details of participants's ages, genders, or professions, etc.

   1. The original paper already includes a user study in the appendix, contrary to your initial comment suggesting its absence.

   2. In response to your reviews of needs for a user study on overall satisfaction, we further conducted additional userstudy as per your feedback. We provide the following details:

      • Participant demographics: Ages: between 20 and 30 years old. Genders: both men and women. Professions: master’s or doctoral students engaged in Computer Vision related research.

      • Evaluation process: Participants were randomly presented with two images (one from Diffree and one from InstructPix2Pix) in a randomized order. They were asked to evaluate which performed better across various metrics used in our quantitative experiments.

      • Setup and methodology: Evaluations were based on the Unified Metric in our quantitative experiments, which includes criteria such as “Background Consistency,” “Location Reasonableness,” “Object Correlation,” “Object Quality,” and “Diversity.”

      • Evaluation metrics: As indicated in the table caption, we assessed overall satisfaction (i.e., Unified Metric in the experiments section). Only “Win” was marked instead of both “Win” and “Loss”. “Diffree Win” or “InstructPix2Pix Win” means that the user found the results of Diffree or InstructPix2Pix superior. Statistical analysis was performed after participants’ blind evaluations.

Q4: The “add dragon” prompt results in “a dragon painting in the frame.” Is this behavior as expected?

A4: We appreciate your observation regarding the “add dragon” prompt resulting in “a dragon painting in the frame”. This behavior is indeed expected and showcases the model’s contextual understanding. Since “dragon” does not exist in our training dataset, Diffree generalizes by integrating the dragon in a contextually appropriate manner. In the setting of a room, adding a dragon as a painting within a frame is a plausible and aesthetically coherent choice. This outcome highlights Diffree’s ability to handle out-of-distribution (OOD) objects by creatively incorporating them into the environment. It demonstrates the model’s generalization capabilities and sensitivity to scene context, ensuring that even unfamiliar objects are added in a way that maintains the coherence and realism of the image.

Q5: How does Diffree perform with out-of-distribution (OOD) objects not included in OABench?

A5: Diffree demonstrates strong performance when handling out-of-distribution (OOD) objects not included in our OABench dataset. As detailed in the Appendix (Lines 921–922), our model is capable of adding various objects absent from the training data. For example, Figures A1 to A3 illustrate that Diffree can successfully add objects like “dragon”, "necklace" or other OOD items, even in anime-style images, despite being trained on real-world images.

This generalization capability stems from our fine-tuning approach based on the pre-trained SD1.5, which inherently can generate various objects from text descriptions. Therefore, even objects not present in our fine-tuning dataset can be added by Diffree. This reflects Diffree’s robustness and adaptability to different image styles and unseen objects, making it applicable to a wide range of scenarios.

We thank Reviewer ethC for their valuable time and effort in reviewing our paper. We appreciate your thoughtful feedback and have addressed your comments in our revised manuscript. We have summarized the updating of our revision in Summary of Paper Updating and provided an in-depth discussion about our Object Mask Predictor (OMP) module in Genera Response. Below, we respond to each of your concerns in detail.

Q1: An in-depth analysis and comparison with SmartMask [1] would be beneficial.

A1: Thank you for your suggestion. We agree that a comparison with SmartMask [1] is valuable. SmartMask differs from Diffree in several key aspects, particularly in its reliance on additional models and input requirements during inference:

1. Dependency on Additional Models:

   SmartMask requires two extra models during inference: a panoptic segmentation model [2] and a mask-guided inpainting model [3]. The process involves:

   ​ (1) Converting the input image into a panoptic segmentation map using the segmentation model.

   ​ (2) Using this segmentation map as input to SmartMask to generate the object mask.

   ​ (3) Feeding the mask into the inpainting model to perform object addition.

   Diffree, in contrast, accomplishes object addition using a single model without relying on additional models. This makes Diffree more efficient and less resource-intensive. Moreover, the performance of SmartMask heavily depends on the accuracy of the segmentation and inpainting models, which may introduce limitations due to potential inaccuracies or limited label outputs.

2. Input Requirements:

   SmartMask requires a converted semantic layout map, an object description, and a context description of the final scene to generate the object mask. This reliance on detailed scene context can limit its applicability, especially when such detailed descriptions are unavailable or insufficient.

   Diffree only requires the object description for addition, simplifying the inference process and reducing dependency on additional inputs.

Unfortunately, since SmartMask is not currently open-source, conducting quantitative experimental comparisons is challenging. Nevertheless, we believe Diffree offers significant advantages in terms of efficiency, simplicity, and reduced dependency on external models and detailed context inputs. We have also updated the related work section to include a discussion of SmartMask.

[1] Singh, Jaskirat, et al. "SmartMask: Context Aware High-Fidelity Mask Generation for Fine-grained Object Insertion and Layout Control." CVPR 2024.

[2] Kirillov, Alexander, et al. "Panoptic segmentation." CVPR 2019.

[3] Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. "Adding conditional control to text-to-image diffusion models." ICCV 2023.

Q2: How Diffree generalizes to more precise control.

A2: Thank you for this important question. To enhance precise control in object addition, we extended our model by re-labeling our dataset with accurate location descriptions using GPT-4o-mini and retrained Diffree with these detailed annotations and original annotations.

To address your concerns, we conducted experiments using contextually detailed prompts specifying different locations within the same scene or involving multiple similar objects. As shown in Figure A14, our model can accurately add objects based on context-related descriptions (e.g., “Add a red knitted wool hat on the head of the doll on the left”). This demonstrates that Diffree can be extended to handle more precise control. We plan to further enhance this capability by constructing larger datasets with precise annotations.

Q3: While user-friendly for object insertion, it restricts users from adjusting the mask. In standard image processing, users or designers often need to make adjustments to achieve their desired results.

A3: We appreciate your concern regarding the ability to adjust masks. Diffree addresses this by generating both the edited image and the object mask through our OMP module. This mask serves as a strong starting point for users and designers who may wish to refine the results. Users can adjust the mask using standard image editing tools or incorporate it into existing mask-guided methods for further refinement.

By providing an initial high-quality mask, Diffree reduces the effort required to obtain an ideal mask, which can be a challenging task when starting from scratch. This feature enhances user control and facilitates a more efficient and flexible editing workflow, aligning with the needs of designers and standard image processing practices.

Dear Reviewer ethC,

Thanks again for your valuable time and insightful comments. We hope that our response can address your concerns. As the deadline for Author/Reviewer discussion period is approaching, we really appreciate if you can let us know whether there still exists any further question about the paper or the response. We are looking forward to further discussion.

Best regards,

Authors of Paper 2788

Q5: The Local CLIP Score results when combining both successful and failure cases.

A5: As detailed in Lines 358–366, the Local CLIP Score requires calculating the CLIP Score for the added object area. For most of InstructPix2Pix’s results, additions fail or alter the entire image, making calculation impossible. Compared to InstructPix2Pix, Diffree has a significantly higher success rate (18% vs. 98%). To ensure fairness, we set the Local CLIP Score of failure cases to zero for both methods. The results highlight Diffree’s advantages:

• The Local CLIP Score results of InstructPix2Pix and Diffree, combining both successful and failure cases. We set the Local CLIP Score of failure cases to zero for both methods.

  Dataset	InstructPix2Pix	Diffree
  COCO	5.10	28.53
  OpenImages	5.52	28.23

In order to better address your concerns, we further conduct a user study on the overall satisfaction with all results between InstructPix2Pix and our Diffree in 1000 cases of COCO and OpenImages. The following comparison demonstrates our significant advantage:

• User study of overall satisfaction in 1000 cases of COCO and OpenImages dataset, with the win rate defined as the ratio of our Diffree wins to the total wins by either Diffree or InstructPix2pix.

  Dataset	InstructPix2Pix Win	Tie	Diffree Win	Win Rate
  COCO	13	110	877	98.5%
  OpenImages	23	187	790	97.2%

These results emphasize Diffree’s significant advantage over InstructPix2Pix.

Q6: Generalization problem, assuming that a real image contains two dogs, and I first use PowerPaint to delete the two dogs, and then let the proposed Diffree generate one dog, what will the output image show? Will the position of the new dog duplicate the position of the previously removed one?

A6: Thank you for this insightful question. We further conduct experiments to address this concern (Figure A13 in the Appendix). After removing one object from an image containing multiple identical objects and using Diffree to add it back, we observed that the new object’s position and size generally differ from the removed one. This indicates that Diffree does not duplicate previous positions and demonstrates good generalization ability.

Q7: Were the compared methods (e.g., InstructPix2Pix) trained on the proposed dataset as well? A7: No, InstructPix2Pix and Diffree were trained on their respective datasets based on pre-trained weights of SD 1.5. For fairness concerns, please refer to our detailed discussion in A2.

Thank you for your thoughtful review, which has helped us strengthen the manuscript. We have summarized the updating of our revision in Summary of Paper Updating and provided an in-depth discussion about our Object Mask Predictor (OMP) module in Genera Response. Below, we address each of your identified weaknesses:

Q1: Lack of ablation of the validation model design, e.g., the output if the OMP is removed.

A1: We appreciate your suggestion. In Figure 8, we provided an ablation study on the impact of omitting OMP module in Diffree’s iterative results. Without OMP module, the background deteriorates rapidly after multiple additions, as the model cannot ensure background consistency. This demonstrates the crucial role of the OMP module. Please refer to our General Response about the OMP module for more details.

Q2: The comparison in Figure A10 in the Appendix, which does not retrain or fine-tune other methods on our dataset, is unfair.

A2: We respectfully disagree for the following reasons:

1. Consistency with Prior text-guided methods: In related text-guided methods [2,3], models fine-tuned with their respective training data are compared without the need for retraining on new datasets.

2. Fundamental Differences in Data Requirements: Mask-guided methods differ from text-guided methods in data requirements. Mask-guided methods can be trained directly on real datasets without synthetic data, as they utilize additional mask conditions. Specifically, the input image condition can be a real image with the object area removed using the mask, while the same real image (without modifications) serves as the training target. Fine-tuning them on our high-quality, relatively small-scale synthetic dataset, while removing the object area from the inputs, would not improve their performance or alter their input requirements.

3. Our Data Generation and usage process are Key Contributions

   In previous text-guided methods [1,2,3], the input and output both are synthesized images for object addition, which leads to poor consistency and makes it difficult to obtain high-quality data pairs. Creating a high-quality object addition dataset is considered challenging [2].

   Our method uses real images as training targets and images with removed objects as inputs, with a complex filtering process for high-quality data. This approach enhances background consistency in data pairs. Comparing with models trained on their own datasets, like InstructPix2Pix, is fair in this context.

[1] Brooks, Tim, Aleksander Holynski, and Alexei A. Efros. "Instructpix2pix: Learning to follow image editing instructions." CVPR 2023.

[2] Sheynin, Shelly, et al. "Emu edit: Precise image editing via recognition and generation tasks." CVPR 2024.

[3] Zhang, Kai, et al. "Magicbrush: A manually annotated dataset for instruction-guided image editing." NeurIPS 2024.

Q3: The explanation of the charts lack sufficient detail.

A3: Thank you for pointing this out. We have updated the paper to enhance clarity, providing more detailed explanations of the charts. Please refer to the updated manuscript for specifics.

Q4: Concerns about the generalization performance of the Diffree model due to potential overfitting to artifacts introduced by the inpainting model.

A4: We understand your concern but believe Diffree demonstrates strong generalization:

1. Output independence from artifacts: Diffree uses real images as outputs during training, preventing artifacts from affecting the generated results. The added objects naturally integrate with the background, enhancing realism.

2. Input robustness: To address your concerns, we further conduct additional experiments where objects were removed using an inpainting model (e.g., PowerPaint [1]) from an image containing multiple identical objects and then added back using Diffree to see if it would be added onto the same position and size. As shown in Figure A13 of the Appendix, the added objects appeared at different positions and sizes, indicating no overfitting to artifacts and demonstrating generalization.

In addition, our visualizations include real images without artifacts and anime images (Figures A2 and A3 in the Appendix), showcasing Diffree’s adaptability and robustness. Furthermore, we discussed in Lines 921-922 of data collection, and for objects that are not in our dataset, the object addition can also be achieved.

[1] Zhuang, Junhao, et al. "A task is worth one word: Learning with task prompts for high-quality versatile image inpainting." ECCV 2024.

Dear Reviewer t9nW,

Could you kindly review the rebuttal thoroughly and let us know whether the authors have adequately addressed the issues raised or if you have any further questions.

Best,

AC of Submission2788