The Blessing of Randomness: SDE Beats ODE in General Diffusion-based Image Editing

We thank reviewer mBr1 for the acknowledgment of our contributions and the valuable comments. We respond below to your questions and concerns.

Weakness1: Comparison with the latest methods.

Thank you for your valuable comment. The preliminary goal of the inpainting and image-to-image translation experiments is to show the versatility of SDE in general image editing, instead of improving the latest methods. Our dragging experiment against the latest baselines is able to show the effectiveness of the SDE formulation. Besides, based on the results, we believe that the SDE formulation is orthogonal to other improvements for a specific task and applies to the latest ODE-based methods.

Moreover, in our submission, we conduct experiments on recent work including SD (CVPR 2022), DiffEdit (ICLR 2023), DDIB (ICLR 2023), and DragDiffusion (arXiv 2023.06). Nevertheless, if you would like to point out specific papers, we are happy to discuss the relation and will try to do the experiments.

Weakness2: Costing time for inpainting and image-to-image translation tasks.

Thank you for your valuable question. In the last paragraph (on page 9) of Sec. 5.4 in our submission, we introduce that the SDE counterpart takes nearly the same time as the direct ODE baseline in all experiments. To more clearly demonstrate the time efficiency, we list the cost time per image for all inpainting and image-to-image translation experiments here and we will add it to the revision as soon as possible.

From the table, we can see that the time efficiency of both SDE and ODE mainly depends on the number of function evaluations, which are the same using the same sampling strategy. Therefore, when applied to the latest ODE baselines, SDE-based methods would have a similar time cost. Also see the discussion on comparison with the latest methods in our response to your weakness1.

We thank reviewer UXKo for the acknowledgment of our contributions and the valuable comments. We respond below to your questions and concerns.

Weakness1: The relationship between our work and CycleDiffusion.

Thank you for your valuable question. we emphasize our contributions and explain the differences and links between our work and CycleDiffusion in detail:

1. We introduce a unified perspective for multiple diffusion-based image editing tasks (I2I, inpainting, dragging) that encompasses various editing techniques. In contrast, CycleDiffusion focuses on I2I. Given the fact that existing research in such tasks is mainly independent, we believe the unified perspective is a nontrivial and valuable contribution.
2. To our knowledge, we present the FIRST theoretical analysis (which is one of our core contributions) on the superiority of SDE over ODE in image editing, while all existing methods including Cycle-Diffusion are purely empirical.
3. We present a new algorithm (SDE-Drag) as well as a new benchmark for open-set image dragging. Besides the SDE formulation, the core contribution of SDE-Drag is a simple copy&paste manipulation strategy for the latent variables, which significantly outperforms existing optimization-based methods. Such contributions are unique in our paper compared to existing work including Cycle-Diffusion.
4. The core idea of Cycle-Diffusion is to construct an invertible SDE algorithm, inspired by the ODE methods. It is mainly compared with non-invertible SDE baselines. However, our core idea is that the general SDE formulation (including Cycle-Diffusion and others) is better than ODE in editing. Our theory applies to other SDE formulations and we apply other SDE-based methods in the inpainting task other than Cycle-Diffusion.

We believe the above contributions make our paper distinct from Cycle-Diffusion.

Weakness2: More image editing methods are not mentioned.

Thank you for your insightful suggestion and for pointing out the related works, as briefly discussed below.

Our framework analyzes the difference between SDE and ODE when the prior distribution mismatch is due to manipulation or domain transformation during inference. Therefore, it also applies to RePaint, DDNM, and DPS. However, training-based methods like T2I-Adapter and ControlNet without mismatch, are excluded from our framework.

We will add the above discussion in our revision.

We thank reviewer NG3Y for the acknowledgment of our contributions and the valuable comments. We respond below to your questions and concerns.

Weakness1: The reason to minimize $KL(\tilde p_0 || p_0)$ .

Thank you for your meticulous and insightful comment. We agree that the final objective of image editing is to analyze $\tilde p_0$ and $q_0$ under certain divergence. Below, we explain why making $\tilde p_0$ and $p_0$ close (i.e., the contribution of this paper) is meaningful both theoretically and empirically.

Theoretically, we indeed assume that the diffusion model characterizes the true score functions, namely $KL(p_0 || q_0)=0$ $(p_0=q_0)$ in the submission (See Line 3, Paragraph 1 in Sec. 3.2). Then our theory on $KL(\tilde p_0|| p_0)$ holds for $KL(\tilde p_0||q_0)$ as desired. Intuitively, the assumption can be relaxed to a bounded approximation error of the score function, i.e., $\mathbb E_{q_t}[ \|s_{\mathbf \theta}(\mathbf x_t, t) - \nabla_{\mathbf x_t} \ln q_t \|^2] < \epsilon$ based on the latest theoretical work [a, b, c]. In particular, if $\epsilon \rightarrow 0$, then the total variation distance (TV) for $q_0$ and $p_0$ tends to zero, namely, $TV(q_0, p_0) \rightarrow 0$, making $p_0$ an accurate approximation for $q_0$.

Empirically, experimental results suggest that $p_0$ is a good surrogate for $q_0$, especially compared to $\tilde p_0$. For instance, Stable Diffusion [d] can generate high fidelity images in human perception. Besides, Tab. 5（on page 26）in our submission shows that the FID for sampling (namely $p_0$) is significantly lower than the FID for editing (ODE baseline, namely $\tilde p_0$), suggesting that $p_0$ is much closer to $q_0$ than $\tilde p_0$, making it meaningful to minimize $KL(\tilde p_0|| p_0)$.

We sincerely appreciate the valuable comments. We will revise the paper to clarify that the final objective of image editing is to analyze $\tilde p_0$ and $q_0$ and present the above discussion on why minimizing $KL(\tilde p_0 || p_0)$ is meaningful. If you have any further concerns, please feel free to post more comments.

[a] Chen, Sitan, et al. "Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions." arXiv preprint arXiv:2209.11215 (2022).

[b] Lee, Holden, Jianfeng Lu, and Yixin Tan. "Convergence of score-based generative modeling for general data distributions." International Conference on Algorithmic Learning Theory. PMLR, 2023.

[c] Chen, Sitan, Giannis Daras, and Alex Dimakis. "Restoration-degradation beyond linear diffusions: A non-asymptotic analysis for ddim-type samplers." International Conference on Machine Learning. PMLR, 2023.

[d] Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

Weakness2: Pretrained models and datasets in the experimental section.

Thank you for thoroughly reading our paper and highlighting the issues in detail. For fairness, we use the same pretrained models (including the training dataset) as the corresponding baselines for all experiments. In particular, we use Stable Diffusion 1.5 [d] trained on the LAION dataset [e] for the inpainting, DiffEdit and dragging tasks. We use ADM [f] trained on the ImageNet dataset [g] for the DDIB experiment.

We will add the information in Sec. 5 in the revision.

[e] Schuhmann, Christoph, et al. "Laion-5b: An open large-scale dataset for training next generation image-text models." Advances in Neural Information Processing Systems 35 (2022): 25278-25294.

[f] Dhariwal, Prafulla, and Alexander Nichol. "Diffusion models beat gans on image synthesis." Advances in neural information processing systems 34 (2021): 8780-8794.

[g] Deng, Jia, et al. "Imagenet: A large-scale hierarchical image database." 2009 IEEE conference on computer vision and pattern recognition. Ieee, 2009.

Weakness3: Expression in the paragraph "Samplers" from Section 2.

Thank you very much for pointing out our mistakes. We will change our expression: Samples can be generated from the diffusion model by discretizing the reverse SDE (Eq.(4)) or ODE (Eq.(5)) from $T$ to $0$.

Weakness4: Details about ODE inversion.

Thank you for your insightful suggestion. We are not sure about the "random Gaussian noise" in your comment. We guess you mean the $\bar{w}_s$ in Eq. (6) in the submission. Note that when using ODE inversion, we set $\eta=0$, which implies that $\sigma_s = 0$. Namely, the coefficient of the random Gaussian noise in Eq. (6) is zero and it is a deterministic process. If we misunderstand your comment, please feel free to post further questions.

In case you are not familiar with ODE inversion, it generally aims to obtain $\mathbf x_T$ given $\mathbf x_0$ based on the invertibility of the ODE. According to Eq. (5) in the submission, the general formulation of ODE inversion is defined as $\mathbf x_T = \mathbf x_0 + \int_0^T \left[f(t) \mathbf x_t + \frac{g^2(t)}{2 \sqrt{1 - \alpha_t}} \mathbf \epsilon_{\mathbf \theta}(\mathbf x_t, t)\right] dt$. This implies that for each given data point $\mathbf x_0$, there exists a uniquely determined $\mathbf x_T$ associated with it through ODE inversion.

As for the specific DDIM sampler, one step is Eq. (6) in our submission. We rewrite it here for clarity (only ODE formulation, i.e., $\eta=0$): $\mathbf x_t = \sqrt{\alpha_{t}} \left( \frac{\mathbf x_s - \sqrt{1 - \alpha_s} \mathbf \epsilon_{\mathbf \theta}(\mathbf x_s, s)}{\sqrt{\alpha_s}} \right) + \sqrt{1 - \alpha_{t}} \mathbf \epsilon_{\mathbf \theta}(\mathbf x_s, s)$. When $t>s$, it is a step of DDIM inversion.

We will provide the details of ODE inversion in the Appendix in the revision.

Weakness5：Assumption in our theoretical analyses.

Thank you for your insightful suggestion. For completeness, we will list the assumption in Appendix D, which is the same as Assumption A.1. of [h].

[h] Lu, Cheng, et al. "Maximum likelihood training for score-based diffusion odes by high order denoising score matching." International Conference on Machine Learning. PMLR, 2022.

Question1: Is CycleSDE still an SDE?

Thank you for your question. We interpret "noise is fixed" to mean that the noises in the backward process of Cycle-SDE are solved by the forward process. Actually, given $\mathbf x_{t_0}$, the noises in the backward process are not iid standard Gaussian noise like the origin SDE, but instead a combination of noises in the forward process. Consequently, the reverse process is stochastic, and thus Cycle-SDE is still an SDE.

Question2: Is the log-Sobolev inequality hold for the data distribution in practice?

Thank you for your question. The log-Sobolev inequality is a classical and standard assumption in theoretical analyses of Langevin-type algorithms [i, j], which cannot be verified in large-scale and high-dimensional real data like LAION [e]. Note that without the assumption, although it is nontrivial to obtain the strong linear convergence result, our main results (i.e., Theorem 1,2&3) still hold.

[i] Chewi, Sinho, et al. "Analysis of Langevin Monte Carlo from Poincar'e to Log-Sobolev." arXiv preprint arXiv:2112.12662 (2021).

[j] Vempala, Santosh, and Andre Wibisono. "Rapid convergence of the unadjusted Langevin algorithm: Isoperimetry suffices." Advances in neural information processing systems 32 (2019).

Finally, we believe the quality of the paper has been improved following your insightful suggestions. If you have any more questions, we are happy to discuss them and will do our best to address them!

Dear Reviewer NG3Y,

Thank you again for your great efforts and valuable comments. We have carefully addressed the main concerns in detail. We hope you will find the response satisfactory. As the discussion phase is about to close, we are very much looking forward to hearing from you about any further feedback. We will be very happy to clarify further concerns (if any).

Best, Authors

We thank Reviewer NG3Y for the time and consideration. We are happy to see the comment "The authors answered all my interrogations" and there is no further question. We hope the Reviewer NG3Y can update the score accordingly based on the initial review "I am prepared to improve my score by taking into account the author's feedback."

We understand that Reviewer NG3Y can be very busy but we kindly remind Reviewer NG3Y that the author-reviewer discussion period ends in one day.