PnP Inversion: Boosting Diffusion-based Editing with 3 Lines of Code

We would like to sincerely thank the reviewer for his valuable insights and acknowledgment of our study. Following the feedback, we have revised our manuscript, specifically clarifying previously ambiguous areas. We believe these enhancements significantly contribute to the clarity and impact of our research.

Question1: Why the proposed inversion method can not correct PnP’s background.

Answer1:

Since Plug-and-Play (PnP) does not have a background protection mechanism but performs editing globally, it often fails to preserve the background effectively. Although our method can improve essential content preservation, it is highly related to the editing model (The $DDIM$ _ $Forward_{Editing}$ _ $_{Model}$) base. Our proposed method aims to maximize the efficacy of existing editing rather than instituting fundamental changes. As evidenced in Figure 1, our approach successfully amends specific background areas, marked in red, in line with our objectives. The further advancement of the performance of these editing methods is designated as an area for future research, as detailed in Section G of the main paper.

Question2: What does the deviation from $z_0$ to $z_0^{''}$ looks like.

Answer2:

The deviation of pure DDIM Inversion and DDIM Forward is shown in Figure 4 (b) and Figure 8 (b). The deviation would completely distort the original image content, making it hard to perform essential content preservation.

We express our sincere gratitude to the reviewer for the insightful comments and the recognition of our work. We especially appreciate the acknowledgment of our approach's simplicity, effectiveness, and superior performance. In response to the feedback, we have meticulously refined our paper and clarified unclear parts in the revised version. We believe these enhancements significantly contribute to the clarity of our research.

Question1: Use meaningful subscripts to denote source latent, source prompt forward latent, and other patents.

Answer1:

We sincerely thank the reviewer for pointing out a better writing approach for articulating and explaining our methods. We have revised the denotations as follows:

• We use $z_{0}^{src}$ to denote the source latent.
• We use $z_{t}^{I}$ to denote DDIM inversion latent. Specifically, we use $z_{t}^{I}$ to denote the nonexistent ideal DDIM inversion latent, and use $z_{t}^{I_{p}}$ to denote the perturbed DDIM inversion latent.
• We use $z_{t}^{F}$ to denote DDIM forward latent. Specifically, we use $z_{t}^{F_s}$ to denote the DDIM forward latent with source prompt without guidance scale, and use $z_{t}^{F_t}$ to denote the DDIM forward latent with target prompt without guidance scale. To further distinguish the DDIM forward process with guidance scale, we use $z_{t}^{F_{sg}}$ to denote the DDIM forward latent with source prompt with guidance scale, and use $z_{t}^{F_{tg}}$ to denote the DDIM forward latent with target prompt with guidance scale.

We also revised some other denotations:

• We use $C^{src}$ to denote source prompt embedding (or embedding of null for some editing methods), and use $C^{tgt}$ to denote target prompt embedding.
• We use $d_{t}^{rec}$ to denote the difference between the perturbed DDIM inversion latent $z_{t}^{I_{p}}$ and the DDIM forward latent of the source prompt $z_{t}^{F_{sg}}$.

Question2: The meaning of step 1,2 in Figure2.

Answer2:

The step denotes the order of steps in the editing process.

Specifically, for techniques such as Null-Text Inversion, StyleDiffusion, and Negative-Prompt Inversion, the process involves initially inverting $z_{0}^{src}$ to $z_{1}^{I_p}$ (step 1) followed by inverting it to $z_{2}^{I_p}$ (step 2). Subsequently, editing on $z_{2}^{I_p}$ is executed using a two-branch diffusion model to compute the denoised latent $z_{1}^{F_{sg}}$ and $z_{1}^{F_{tg}}$(step 3), and eventually $z_{0}^{F_{sg}}$ and $z_{0}^{F_{tg}}$(step 4).

For our method, steps 1 and 2 mirror those in Null-Text Inversion, StyleDiffusion, and Negative-Prompt Inversion. Post these steps, editing on $z_{2}^{I_p}$ is conducted using the two-branch diffusion model to determine the denoised latent $z_{1}^{F_{sg}}$ and $z_{1}^{F_{tg}}$(step 3), followed by step 4, which involves adding differences to rectify deviations. Steps 5 and 6 are repetitions of steps 3 and 4. Notably, steps 4 and 6, introduced by our method, are denoted by a solid circle. A comprehensive explanation of these steps is provided in the main paper.

Question6: The use of CLIP Similarity.

Answer6:

• Include more metrics to make the evaluation more comprehensive: Since the editing is expected to only perform in a part of the image and not make huge changes in the image structure. It is true the CLIP score varies modestly among different editing methods. However, we hold the view that CLIP score is still possible to distinguish the editing quality since it is aligned with visualization results, shown in the visualization of the main paper. To make the evaluation more comprehensive, we added two human evaluation metrics to rate the editing quality. Due to the limited time, we randomly selected 50 images and let two volunteers rate 1-10 source-editing image pairs on two dimensions: editing image quality and alignment with the target prompt. Both two volunteers rated all 50 images, and we averaged their scores to formulate the table below. The Table shows the same compared methods as in Table 1 of the main paper. For the editing method Prompt-to-Prompt (P2P), we compare four different inversion methods: DDIM Inversion (DDIM), Null-Text Inversion (NT), Negative-Prompt Inversion (NP), and StyleDiffusion. For editing methods MasaCtrl, Pix2Pix-Zero (P2P-Zero), and Plug-and-Play (PnP), we compare with DDIM Inversion (DDIM). Results show that the human evaluation metrics show the same tendency with CLIP Similarity scores.

  Method	Whole Image CLIP Similarity	Editing Image Quality	Alignment with Target Prompt	Average of Human Evaluation Metrics
  DDIM+P2P	25.01	1.74	5.68	3.71
  NT+P2P	24.75	6.32	5.19	5.76
  NP+P2P	24.61	5.19	4.65	4.92
  StyleD+P2P	24.78	6.21	5.15	5.68
  Ours+P2P	25.02	6.50	5.52	6.01
  DDIM+MasaCtrl	23.96	4.54	3.88	4.21
  Ours+MasaCtrl	24.38	4.61	4.01	4.31
  DDIM+P2P-Zero	22.80	2.86	3.01	2.94
  Ours+P2P-Zero	23.31	4.33	4.20	4.27
  DDIM+Plug-and-Play	25.41	6.39	5.60	6.00
  Ours+Plug-and-Play	25.41	6.56	5.61	6.09

  Table 2. Comparing our methods with other inversion techniques across various editing methods on Whole Image CLIP Similarity and Human Evaluation Metrics.

  We will further evaluate the human evaluation metrics on the whole benchmark with 700 images and report the value in the final version.

• The input text of CLIP Similarity evaluation: We directly use the target prompt to calculate both CLIP Similarity in Whole and Edit. It is possible that the masked image is not completely aligned with the target prompt for Edit Region CLIP Similarity. However, since most mask regions in PIE-Bench take up around half of the image size, our visualization results show alignment between the Edit Region CLIP Similarity and masked image. Moreover, the same variation trend of Edit Region CLIP Similarity, Whole Image CLIP Similarity, and Human Evaluation Metrics may also demonstrate the reasonableness of using this metric. While acknowledging the logical basis of this metric, we concur with the identified potential risks associated with its use. In response, we have decided to switch it to human evaluation metrics in the final version of our work.

Question3: Better explanation of the algorithm.

Answer3:

We apologize for the lack of clarity and unclear denotations of our proposed algorithm in the historical version. We have revised the algorithm and polished section 4.2 according to your constructive advice:

• The bracket notations: The bracket notations denote the batch concatenation for the inputs and outputs involved in DDIM Inversion and DDIM Forward processes instead of calling a function twice. Since existing editing methods simultaneously process the source prompt $C^{src}$ and target prompt $C^{tgt}$, with a batch size of 2, to ensure essential content preservation and editing by integrating information between the source and target batches. In adherence to these established methods, we write the algorithm with an input concatenated in batch dimension. We apologize for not specifying it clearly in the previous version of our manuscript and have added the corresponding clarification in the revised version.
• The misalignment of batch size in input and output: We thank the reviewer for pointing out the misalignment. We have revised our manuscript and ensured the batch size is maintained consistently in input and output.

Additionally, we have updated the algorithm in response to constructive feedback from other reviewers. For detailed information, please refer to the revised version of the paper.

Question4: How source branch interacts with the target branch.

Answer4:

The interaction is related to the implementation of $DDIM\underline{}Forward_{Editing\underline{}Model}$. We have provided a detailed review of how previous editing methods perform the source and target branches interaction in the related work. For more information, please refer to Section A in the revised version.

To illustrate this interaction, we utilize the Prompt-to-Prompt method as an example. Prompt-to-Prompt uses the cross-attention map in the source branch to enhance or replace the ones in the target branch. The selection of the cross-attention map is based on the correspondence between the cross-attention map and the text prompt. For instance, in transforming an image of a cat to a dog, the method involves using the source prompt 'a cat' and the target prompt 'a dog'. Here, the cross-attention map associated with 'dog' is exchanged with that of 'cat' in the source branch, thereby infusing the edit with relevant information. Concurrently, other attention maps in the source branch are maintained as is, ensuring the preservation of essential content.

This also explains why an accurate source branch latent would be important for essential content preservation during the editing. This is because the feature (such as attention map) in the source branch will be further used in target branch editing.

Question5: The distribution of images for each editing type in the PIE-Bench.

Answer5:

We have clarified the number of images in each editing type in the revised Section B, listed as follows:

0. random editing written by volunteers (140 images)

1. change object (80 images)

2. add object (80 images)

3. delete object (80 images)

4. change object content (40 images)

5. change object pose (40 images)

6. change object color (40 images)

7. change object material (40 images)

8. change background (80 images)

9. change image style (80 images)

We distribute the editing into several classes: random (type 0, 140 images), object-related (type 1-3, 240 images), object attribute change (type 4-7, 160 images), and large region edit (type 8-9, 160 images). Thus, PIE-Bench covers a wider range of editing categories instead of only local editing.

Thanks to the authors for the clarification. I've increased my rating.

I agree that the target prompts in the PIE bench may not effectively demonstrate the local editing quality when assessing CLIP similarity for certain types of edits. I believe the Editbenchmark dataset is a better fit for evaluating the local editing quality, as it primarily includes prompts concentrating on local editing.

I'm not asking for additional experiments during the rebuttal period, but I believe these results could be more convincing for the EDIT metric. Additionally, I think it's possible to identify a similar subset within the PIE bench with target prompts focused on local regions. Measuring EDIT metric using this subset might be better.

Question2: The explanation of the method is unclear.

Answer2:

We apologize for the lack of clarity in the explanation of our proposed algorithm and polish section 4.2 according to your constructive advice:

• The bracket notations: The bracket notations denote the batch concatenation for the inputs and outputs involved in DDIM Inversion and DDIM Forward processes. Since existing editing methods simultaneously process the source prompt $C^{src}$ and target prompt $C^{tgt}$, with a batch size of 2, to ensure essential content preservation and editing by integrating information between the source and target batches. In adherence to these established methods, we write the algorithm with an input concatenated in batch dimension. We apologize for not specifying it clearly in the previous version of our manuscript and have added the corresponding clarification in the revised version.
• The $o_t$ : The $o_t$ represents the difference between DDIM inversion latent and DDIM forward latent in each time step $t$ . To clarify the notation, we have revised it to $d_t$ , representing the difference in time step t. Moreover, we add a detailed explanation of the calculation of the difference in the main text.
• Whether $z_t$ encompasses both src and tgt: We apologize for the unclear notation in previous manuscripts. In the revised version, we have explicitly distinguished different branches in different batch channels with [] notation. Specifically, the process of generating DDIM Inversion latent is only performed in the source branch. During editing, the source and target embeddings concatenate in the batch dim to simplify the interaction between the two branches.
• The comparison to CycleDiffusion: CycleDiffusion directly calculates an offset between the latent generated by the source prompt and the target prompt of the current time step t in the DDIM Forward process, then adds the offset to the latent embedding of time step t-1. This is quite similar to the methods used in [3][4][5], which use the difference between the source and target prompts to perform image editing. However, our methods are distinguished from their approaches in several aspects: (1) Our method targets editing methods that involve detailed editing operations such as attention map fusion and latent fusion. These editing methods usually perform more satisfying editing results than [3][4][5] and CycleDiffusion. (2) Our method is a plug-and-play improvement for diffusion inversion targeted for two-branch prompt-based editing methods, instead of an editing method. (3) Our method largely improves the essential content preservation and editability by two-branch disentangling, while [3][4][5] and CycleDiffusion directly use a one-branch approach to perform editing. Nevertheless, we are thankful to the reviewer for pointing out the missing citation for CycleDiffusion. We add citations for [6][7] to our related work.

[3] General Image-to-Image Translation with One-Shot Image Guidance

[4] SINE: SINgle Image Editing with Text-to-Image Diffusion Models

[5] SEGA: Instructing Diffusion using Semantic Dimensions

[6] Unifying Diffusion Models' Latent Space, with Applications to CycleDiffusion and Guidance

[7] A Latent Space of Stochastic Diffusion Models for Zero-Shot Image Editing and Guidance

Question3: Scores for each category.

Answer3:

In fact, we have already provided the results and analysis of our method added to Prompt-to-Prompt on different editing categories in supplementary file E.6. We agree that a discussion and reporting of scores on various editing types would be helpful for a better understanding of the paper. Thus, we additionally add each category's result of MasaCtrl in supplementary file E.6.

Different editing types perform differently in each category. For instance, Prompt-to-Prompt excels in material alterations, resulting in an overall superior performance in category 7. Conversely, MasaCtrl demonstrates proficiency in modifying object poses using the mutual self-attention mechanism, leading to relatively better results in the respective category 5. The detailed results and analysis can be found in the supplementary file.

Question4: Only the result of Ours+P2P is shown in Figure 4.

Answer4:

The target of Figure 4 is to provide a comparison of different inversion methods and our inversion method. For a fair comparison, all inversion methods are combined with Prompt-to-Prompt. Additional visualization results of our method combined with different editing techniques can be found in Figure 1 and supplementary files from Figure 8 to Figure 13. Due to the page limit, most visualization results are in the supplementary files.

We sincerely thank the reviewer for your insightful comments and recognition of this work, especially for acknowledging the convincing and comprehensive experimental results as well as the contribution of benchmark construction and well-organized evaluation criteria. We have polished the paper, added the experiments, and made the clarifications in the revised version.

Question1: Figure 5 is more relevant to the content of the paper than Figure 3. A more detailed explanation of the benchmarks and evaluation metrics in the main body of the text would be beneficial.

Answer1:

We would like to express our sincere thanks to the reviewer for the constructive feedback on the structure of our paper. We fully agree that Figure 5 is more relevant to the content of the paper in comparison with Figure 3 and have accordingly updated the revised manuscript to reflect this change. But we still want to explain why we put substantial effort into introducing previous works. This is because our key insight is the disentangling of two branches, which is ignored by previous works. Based on this insight, we review previous research and dig into the approaches they employed to preserve essential content and facilitate editing. Then, we analyze how to disentangle the two branches in previous works to combine them with our proposed inversion method better. In the revised version, we have put the main part of the related work and Figure 3 into the supplementary file. Moreover, we added Figure 5 to the main text with more explanation.

However, we apologize that due to the page limit, it is hard for us to put a detailed explanation of evaluation metrics into the main text. In this paper, we do not introduce new evaluation metrics but explore existing metrics suitable for the task and incorporate them into our benchmark. We posit that a comprehensive explanation of these metrics in the supplementary files suffices and should not detract significantly from the reader's comprehension.

We sincerely thank the reviewer for the insightful comments and recognition of this work, especially for acknowledging that our approach is simple and effective with superior performance. We have polished the paper, added the experiments, and clarified the below points in the revised version.

Question1: More explanation of the method section.

Answer1:

We apologize for the lack of clarity in the explanation of our proposed algorithm in the historical versions and polish section 4.2 according to your constructive advice:

• The bracket notations: The bracket notations denote the batch concatenation for the inputs and outputs involved in DDIM Inversion and DDIM Forward processes. Since existing editing methods simultaneously process the source prompt $C^{src}$ and target prompt $C^{tgt}$, with a batch size of 2, to ensure essential content preservation and editing by integrating information between the source and target batches. In adherence to these established methods, we write the algorithm with an input concatenated in batch dimension. We apologize for not specifying it clearly in the previous version of our manuscript and have added the corresponding clarification in the revised version.
• Explain lines 3, 7-9 in more detail: We have reformulated the algorithm to enhance its readability, adopting a new style of presentation. The revised algorithm and added explanation can be found in the edited rebuttal revision. Specifically, our method achieves editing performance enhancement in 3 steps: (1) we first compute the perturbed latent in DDIM forward; (2) we then store the difference between the original latent in the inversion process and the perturbed latent in the forward process with input condition $C^{src}$; (3) we finally add the difference to the source/reconstruction branch in existing editing model.

We express our gratitude for the reviewer's suggestions, which have significantly improved the quality and readability of the paper.

Question2: It worth adding discussion and comparison of work [1]. [1] Pan, Zhihong, et al. "Effective Real Image Editing with Accelerated Iterative Diffusion Inversion." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

Answer2:

We regret the initial oversight of not providing discussion and comparison with the AIDI method introduced in the current literature. The method under consideration, known as Accelerated Iterative Diffusion Inversion (AIDI), employs an iterative procedure to find a fixed-point solution for the ideal diffusion latent, as depicted by the dashed circle in Figure 2.

It is pertinent to note that AIDI is distinct from and complementary to our approach; while AIDI concentrates on refining DDIM Inversion, our methodology is aimed at the DDIM Forward correction. We have conducted additional experiments to both compare and combine our method with AIDI. The results have been incorporated into the supplementary material in Section E.2 since AIDI reports proficiency in reconstruction. Due to the absence of an official AIDI implementation, we adapted the Prompt-to-Prompt (P2P) codebase to include AIDI with the iteration parameters set to 5 and 20, which is denoted as AIDI(5) and AIDI(20).

Table 1. Results of comparing and combining our method with AIDI.

The table above illustrates that our method outperforms AIDI in terms of structural distance, background preservation, and editability. Additionally, the integration of AIDI into our method results in further enhancements to performance. This improvement substantiates the orthogonality and potential synergistic relationship between our approach and AIDI.

Moreover, given that AIDI necessitates an iterative process to locate the fixed-point solution, it is inherently more time-consuming than our proposed method, even when the iteration parameter is limited to 5. Our method achieves an average inversion time of 16.15 seconds, compared to 21.39 seconds for AIDI. These results have been updated in the main body of the paper.

Question3: The name "direct inversion" clashes with another existing work [2]. [2] Elarabawy, Adham, Harish Kamath, and Samuel Denton. "Direct inversion: Optimization-free text-driven real image editing with diffusion models." arXiv preprint arXiv:2211.07825 (2022).

Answer3:

We thank the reviewer for highlighting the potential for ambiguity with the name ‘Direct Inversion’ used in existing work. To avoid confusion and maintain our methodology's distinctiveness, we have renamed our approach to 'PnP Inversion.' We have updated our manuscript accordingly to reflect this change.

Question4: Typo errors.

Answer4:

We appreciate the reviewer's careful reading of our manuscript and the assistance in identifying typographical errors. We have thoroughly reviewed the document and corrected the typos that were pointed out.

Question5: The paper shows promising empirical results but is still not theoretically motivated.

Answer5:

While providing mathematical proof for our method is challenging, we offer some theoretical considerations. For more details, please refer to Sections 4.1 and 4.2 of our paper.

We establish our theoretical foundation on the observation that optimization-based inversion techniques attain satisfactory performance by correcting the perturbations in DDIM forward latent, reverting them to their corresponding DDIM inversion counterparts. Despite their efficacy, optimization-based inversion methods are impeded by several limitations: (1) undesirable latent space distances affecting essential content preservation, (2) misalignment in the generation model’s distribution, and (3) extended processing times.

In response to these challenges, we find a simple and effective solution neglected by previous works. This solution adopts the principles of optimization-based methods—namely, the adjustment of DDIM forward latent—while innovating through two pivotal adjustments: (1) disentangling the source and target branches, and (2) empowering each branch to excel in its designated role: preservation or editing.

Using Prompt-to-Prompt as an illustrative example, we further explain why the solution is theoretically reasonable. The key idea of Prompt-to-Prompt is using the cross-attention map in the source branch to enhance or replace the ones in the target branch. As a result, an accurate source branch latent would be important for essential content preservation during the cross-attention map integration. Put simply, a source branch that fails to reconstruct the original image will invariably find it impossible to conserve pertinent information within it. Therefore, correcting the DDIM forward latent back to the DDIM inversion latent is necessary for achieving better structure and background preservation. Simultaneously, the editing branch, targeted for the incorporation of novel concepts, benefits from remaining unmodified and achieving better editability. This motivation is also proved in our ablation study in Section 5.4.