Invertible Consistency Distillation for Text-Guided Image Editing in Around 7 Steps

Thanks for your careful reading and valuable feedback! We respond to your questions below.

1. it feels that the editing method is limited on Prompt2Prompt (P2P). Is it still possible to apply the model on non-rigid editing such as MasaCTRL?

Our model is not limited to P2P. Alternative editing methods, such as MasaCTRL, are also applicable in the same manner as to original diffusion models. The attached file (Figure 3) provides a few examples of our approach combined with MasaCTRL for non-rigid editing. The results show performance improvement due to more advanced editing methods. We will add these results in the revision. Please note that our work introduces a fast inversion method that is quite orthogonal to editing approaches. Therefore, we evaluate P2P as one of the most well-known approaches to compare various inversion methods fairly.

2. Compare the inversion&sampling time with other baselines

Thanks for the valuable suggestion! We present the time required to invert a single image in the table below. We will add these results to the revision.

3. Compare the model with other training based methods

Below, we compare our approach with Intruct-Pix2Pix and observe that it outperforms Intruct-Pix2Pix in terms of both content preservation and editing strength while being training-free. We also present a few visual examples in the attached file (Figure 2, Bottom). We will add this comparison to the revision.

4. Does the performance degrade if the distillation step decreases?

The scaling performance of our approach is similar to that of consistency distillation in text-to-image generation. Based on our experiments, the optimal number of steps is 6-8 (3-4 encoding + 3-4 decoding steps) and performance noticeably degrades for 2-4 steps (please see the tables below). We believe that future work on consistency models will greatly contribute to improving performance in fewer steps, which can then be transferred to our method.

Inversion:

Editing:

5. Figure 1 seems very redundant for me

Thanks for the suggestion! We will update Figure 1 in the revision.

Thanks for your careful reading and valuable feedback! We respond to your questions below.

1. The description of the method is unclear... we need to have a coupling of the image and noise.

An interesting aspect is that neither additional data nor teacher inversion is required. Compared to the original CD, the only modification needed is the boundary condition. This is possible because the forward CD operates on the same ODE trajectories as the original CD (please see Figure 3 in the paper). The primary difference lies in the direction of the consistency function. Thus, image-noise couplings are "implicitly" created during training using one step of the ODE solver, much like in the original CD. We will provide more details in the revision to ensure clarity.

2. Given a distilled diffusion model, can we use this model to do fast inversion?

It is an interesting question for future research. Since existing distilled models do not support reversibility like diffusion models, we believe that fast inversion of the already distilled models necessiates an extra model to learn image-noise connections. Our intuition is supported by extensive literature on GAN inversion, where an encoder is trained to perform inversion.

3. The method basically needs two models...I feel like it is a bit ad-hoc...Two models introduce significantly more parameter overhead....

We believe that the distilled models need to be redesigned to handle bidirectional sampling (data to noise and noise to data): it can be either an extra input argument, LoRA adapters, or an additional model. As described in Appendix A, our method employs LoRA adapters as a highly convenient and compact way to enable invertibility. Specifically, we use a single diffusion model and simply activate the corresponding LoRA weights depending on which model is used (forward or reverse). Note that LoRA adapters take up a small portion of the total number of parameters (less than 10%). Thus, the parameter overhead is rather negligible compared to storing an extra model.

4. inversion-based image editing only handles a small subset of image editing tasks

Besides image editing, fast inversion methods can be useful for other problems, such as anomaly detection [1], text-to-3D generation [2,3], image restoration [4].

[1] Hend et al. Out-of-Distribution Detection with a Single Unconditional Diffusion Model

[2] Lukoianov et al. Score Distillation via Reparametrized DDIM

[3] Liang et al. LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching

[4] Chihaoui et al. Blind Image Restoration via Fast Diffusion Inversion

Thanks authors for the response.

Now I am clear about the data you are using. But I am still not convinced that if it worths to have two models just for fast inversion. I will increase my score to 5 to reflect your clarification.

Thanks for your careful reading and valuable feedback! We respond to your questions below.

1. Discussion of Failure Cases.

We present some inversion failure cases in the attached file (Figure 1). Our method sometimes oversaturates images for high guidance scales and struggles to reconstruct complex details like human faces and hands. We will add the discussion and more examples in the revision.

2. More SDXL Demos Needed.

The paper has SDXL illustrations in figures 8 and 16. To strengthen the evaluation, we will provide more SDXL examples in the revision.

3. Comparison with SDXL Turbo using SDEdit.

Thanks for the important suggestion! Please see the comparisons in the table below and the qualitative results in the attached file (Figure 2, Top). SDXL Turbo with SDEdit significantly hurts reference image preservation due to stochasticity, as confirmed by the DinoV2, image CLIP score and qualitative comparisons. This highlights the importance of accurate image inversion for editing. Moreover, our approach outperforms SDXL Turbo in terms of editing strength (ImageReward).

Thanks for your careful reading and valuable feedback! We respond to your questions below.

1. almost every technique comes from another earlier work...contribution less impactful.

Our techniques present a natural continuation of previous works without compromising novelty, in our opinion. Invertible Сonsistency Distillation and its corresponding training pipeline have not been formulated before, and dynamic guidance was not explored from the inversion perspective. We believe our approach makes an important contribution to the field, as it can be useful for future works on inversion-based problems.

2. The proposed framework is highly customized toward CD, not be able to generalize SD-turbo, DMD, and UFOGen.

Unlike GAN- or DMD-based distillation methods, CD exploits the ODE perspective of diffusion models. In our ablation study in Table 2E, we show that using ODE perspective via CD objective largely enhances our inversion method. The inversion of other distillation methods remains an open and interesting question.

3. Can the proposed method match the performance with more steps and what the marginal performance would be if we scale up the number of steps?

In the table below, we present the results of the improved version of our model after more careful tuning. 8 steps of our inversion closely approaches the performance of the teacher inversion. Therefore, further scaling does not seem necessary. We will provide more details in the revision.

4. The overall quality of this paper can be greatly improved by replacing Figure 1.

Thanks for the suggestion! We will update Figure 1 in the revision.