Object-Aware Inversion and Reassembly for Image Editing

This paper achieves high-quality object-level image editing through a simple yet effective method. Authors find that the editing for different parts requires different noising levels used to inverse the given image to latent space. As such, they search for optimal inversion steps for different parts based on a search metric for both the edited and non-edited regions respectively. In order to produce a natural final result, the results of different parts are seamlessly blended, which yields a harmonic and globally coherent editing output. Albeit its simplicity, the proposed method outperforms competitive works when editing real images and shows promise in real usage.

This paper discovers that various editing pairs exhibit differing levels of editing complexity. Additionally, it is observed that neglecting the varying difficulty levels of different editing pairs in multi-object editing tasks results in the problems of concept mismatch and poor editing. To address these issues, the paper introduces a novel training-free image editing approach called OIR. This method follows the approach of assembly first and then reassembly. 1) In the assembly strategy, the paper introduces a novel search metric. This metric automatically identifies the optimal inversion step for different editing pairs, enabling automatic control of editing difficulty. This approach allows different editing pairs to undergo separate denoising, preventing concept mismatch issues. Moreover, employing the search metric to find the optimal result represents a new paradigm in single-object editing. 2) In the reassembly strategy, the article suggests merging the editing regions and non-editing region during the reassembly step. This operation takes place in the denoise latent space. The reassembly process incorporates a re-inversion strategy, enhancing the image editing's edge smoothness, improving image editability, and enabling interaction across regions. 3) To assess OIR's capabilities, the authors collect two datasets, which are employed to evaluate both the single-object editing proficiency of the search metric and the multi-object editing capability of OIR. Numerous experimental results indicate that the search metric performs on par with existing state-of-the-art editing methods in single-object editing tasks. Moreover, in multi-object editing tasks, OIR demonstrates strong performance, outperforming the previous SOTA methods

This paper introduces advancements in text-driven image editing using diffusion-based methods. Existing techniques involve a fixed number of inversion steps to edit images aligned with a target prompt, but the optimal number of steps varies for different editing pairs. The paper introduces a new approach called Object-aware Inversion and Reassembly (OIR) to enable fine-grained, object-level editing. The object-level fine-grained editing is achieved by segmentation from SAM. It uses a search metric to determine the optimal inversion step for each editing pair and combines them for the final edited image, demonstrating superior performance, especially in multi-object editing scenarios.

The paper proposes object-aware inversion and reassembly for image editing. The motivation is that the inversion steps vary from the editing of different objects. Therefore, we need to choose different inversion steps for different objects. Also, for different objects in one image, we need to merge the editing results. Then the reassembly strategy is introduced. The proposed method achieves state-of-the-art performance.

We sincerely thank all reviewers for their valuable comments.

Novelty

Reviewer pgMn, Reviewer nVsr and Reviewer r1AF well recognize the novelty of our method:

• "This paper identifies a fundamental challenge in multi-object image editing tasks. The new search metric introduced in this article is simple yet effective. The OIR introduced in this article is novel and effective, presenting a new solution for the multi-object editing task." (Reviewer pgMn)
• "This paper identifies an interesting and useful phenomenon that the structure of different objects corresponds to a different level of latent as modeled in the diffusion process. Since the denoising process of different image parts is disentangled, the proposed method supports fine-grained multiple object editing, which is not featured in prior works." (Reviewer nVsr)
• "The idea of working on each object separately and reassemble is a interesting way to do multi-object editing." (Reviewer r1AF)

Promising Results

All reviewers agree that:

• "The proposed method achieves state-of-the-art performance. They do bring a lot of performance gain." (Reviewer xVir)
• "Employing the search metric yields promising results, comparable to other image editing methods. The experimental comparisons with other methods are extensive and thorough." (Reviewer pgMn)
• "Results are competitive compared to other inversion based editing techniques." (Reviewer r1AF)
• "The image editing results are indeed impressive, as shown in the main text and the appendix. Both the quantitative and qualitative results demonstrate the advantage over strong prior works." (Reviewer nVsr)

Summary of changes

We have revised our submission and summarized our updates as follows:

1. We have provided image editing speeds for different methods in Appendix A.6. (Reviewer xVir, r1AF and nVsr)
2. We have presented the image editing results of our method under different mask generators in Appendix A.7. (Reviewer r1AF)
3. We have supplemented the comparison results between our method and the inversion-based method with a segment model in Appendix A.10. (Reviewer r1AF)
4. We have provided the experiment to visualize the distribution of optimal inversion steps in Appendix A.9. (Reviewer pgMn)
5. We have presented the combined approach and the experimental results of our method with the inversion-based image editing method in Appendix A.8. (Reviewer nVsr)
6. We have revised the paper to enhance clarity in expression. (Reviewer nVsr)

Q1: The method is time-consuming (Reviewers xVir, r1AF and nVsr).

A1:

• We have introduced a parallel acceleration solution for our method in Appendix A.4 of the initial submission, which is recognized by Reviewer nVsr. In the new version of the paper, we move the details of the parallel acceleration method into "Acceleration for generating candidate images" in Section 3.1, and these details are highlighted in blue.
• We observe that the sequential optimal inversion step searching process in OIR is time-consuming. To address this issue, we propose a splicing strategy over the denoising process. Firstly, we pair different denoising processes with varying inversion steps to achieve equal total steps. For example, the denoising processes with 1 and 50 inversion steps are paired. Similarly, the 2 and 49-step inversions are paired. In this way, denoising processes of the same total length can proceed simultaneously for parallel acceleration, as there is no dependency between denoising processes. In addition, the optimal inversion step search of different editing pairs can be parallelized on multiple GPUs to achieve further acceleration.
• The table above presents a comparison results of editing speeds. It reveals that our method can be significantly accelerated in multi-GPU scenarios, whereas other methods lack clear parallel acceleration solutions due to the temporal dependency between the denoising steps. Detailed content is presented in blue font in Appendix A.6.

Q2: The contribution and significance of the method is limited (Reviewer r1AF).

A2: We identified and addressed a fundamental challenge in multi-object image editing tasks, where existing methods typically apply the same inversion steps for all editing pairs, resulting in suboptimal editing quality. To this end, we propose a new editing paradigm to enable object-aware fine-grained editing, which integrates two tightly coupled components:

• A simple yet effective new search metric that automatically and dynamically selects the optimal inversion step for each editing pair, unlike previous inversion-based methods that set the inversion step as a fixed value. This approach effectively balances the trade-off between editability and fidelity. Moreover, we can easily accelerate this process by splicing denoising processes of varying lengths into equal lengths and running denoising processes of the same length in parallel.

• A novel disassembly and reassembly paradigm that is specially designed to disentangle the denoising process for each editing pair, which is not featured in prior works. The significance and contributions of our paper have been well recognized by Reviewer pgMn and Reviewer nVsr.

Dear reviewers,

As the Author-Reviewer discussion period is going to end soon, please take a moment to review the response from the authors and discuss any further questions or concerns you may have.

Even if you have no concerns, it would be helpful if you could acknowledge that you have read the response and provide feedback on it.

Thanks, AC

This paper proposes object-aware inversion and reassembly for image editing. The main idea is that different editing pairs (i.e., source and target) requires different inversion steps. The proposed method searches for the optimal inversion step for each subject pair, and reassemble the output. Reviewers think that the proposed method leads to performance gain, but introduces computational overhead. There are no consensus reached in the discussion period. After going through the paper, reviews, and responses, the AC thinks that the strengths outweighs the weaknesses. Therefore, an acceptance is recommended. The authors are advised to include the discussion of the computational overhead in the main paper to maximize the impacts brought to the community.

Accept (poster)