Response to reviewer #qmRm

This paper targets at two issues related to the image morphing, identity loss and directional trasition. However, I do not see the importance of these two issues. Why they are important and worthing efforts for this task.

For instance, (1) For the second case in Fig. 5, IMPUS exhibits identity loss, where the third generated image deviates from the original identity; (2) In Fig. 16, IMPUS shows a lack of similarity between the second and third generated images, leading to unsmooth transitions.

I understand that the smooth transition is important, is it the same as the meaning of directional transition?

We believe that these two terms are different. A smooth transition implies a gradual shift from the left image to the right image, maintaining continuity throughout. On the other hand, a directional transition signifies a progression where there is no loss of identity or regression to a previous state.

What is the successful rate, are the results cherry-picked

Our successful rate is very high and the results are not cherry-picked. To reinforce this, we conducted user studies where 30 volunteers, comprising animators, AI researchers, and gaming enthusiasts aged between 20 and 35, participated. The outcomes of the user studies can be found below, showcasing the effectiveness of our proposed method through a subjective evaluation.

|IMPUS|DiffMorpher|Slerp|Ours|

|Preference|17.16%|14.89%|7.82%|60.13%

Response to reviewer #NcSG

This paper is apparently not well prepared for publication.

Thanks for the notification from the reviewer. We have rectified the identified typos and meticulously proofread the paper to ensure it is more refined and polished.

In equation (7), since m has the same size as z, what does it mean by “m=1”?

It means that the value of each item in m is equal to 1.

The image captions of Fig.2 and Fig.3 is too simple. Reader have to move to the main text to try to understand the figures.

Thanks for the suggestion. We have carefully added explanation for Fig.2 and Fig.3 in their captions.

In the introduction, it mentions a comprehensive evaluation system, including a new dataset and specialized metrics. But in Sec 4.1, it says “following IMPUS and DiffMorpher, we ... using the following metrics”. Where are the proposed specialized metrics?

We have rectified the claim in the introduction. In this paper, we present an evaluation benchmark featuring a novel dataset comprising four unique sets of image pairs classified based on their semantic and layout similarities.

Without a human perceptual study, it’s difficult to get a conclusion when comparing the two methods.

We have carried out user studies involving 30 volunteers, encompassing a diverse group of participants such as animators, AI researchers, and gaming enthusiasts, within the age range of 20 to 35. The results are provided below, showcasing the effectiveness of our proposed method through a subjective evaluation.

|IMPUS|DiffMorpher|Slerp|Ours|

|Preference|17.16%|14.89%|7.82%|60.13%

In my understanding, the diffusion process of DDIM is to directly add noise without using the trained network.

Indeed, throughout the training process, the diffusion process will first generate random noise and then iteratively denoise the image to produce the clearer images.

However, during the reverse diffusion steps, the network operates conversely by commencing with the image itself and utilizing the pre-trained UNet to progressively obtain the noise pattern corresponding to that image.

For the experiment “Ours(Var-A)” in line 469, what does it mean to “omit the original attention mechanism”? Completely remove the attention?

In line 469, the term "Ours (Var-A)" denotes the approach that solely employs spherical interpolation (as in Eq. (3)) in both the reverse diffusion and forward denoising steps. To prevent any potential confusion, we will provide further clarification on this in the revised version.

Thank you gain for your constructive feedback.

Q1: The explanation about the inconsistency between the qualitative and quantitative comparison of "DiffMorpher" and "spherical interpolation".

a) In Table 1, LPIPS, FID, and PPL are computed by extracting features of adjacent image pairs through the deep neural networks. While the results from "spherical interpolation" appear visually clearer than those from "DiffMorpher," "spherical interpolation" often generates semantically irrelevant content. For instance, as illustrated in Fig. 5, the images generated by "spherical interpolation" lose the semantic essence of the motorcycle. Additionally, there are significant changes between adjacent images, including both foreground and background elements, which result in larger feature variations between adjacent image pairs. In contrast, despite introducing noticeable image artifacts, "DiffMorpher" effectively preserves the semantic consistency between the source and target images. Furthermore, the transitions between adjacent images exhibit smoother background changes and maintain similar semantic content. These factors contribute to "DiffMorpher" outperforming "spherical interpolation" in Table 1.

b) In the Supplementary Material, we provide additional generated results. These results show that for cases with similar semantics and layouts, "DiffMorpher" produces superior outputs compared to "spherical interpolation". MorphBench, an evaluation dataset constructed by "DiffMorpher," is composed of images with similar semantics and layouts, which aligns well with the strengths of "DiffMorpher". Consequently, "DiffMorpher" achieves better performance than "spherical interpolation" on MorphBench. On the other hand, Morph4Data, our newly proposed evaluation dataset, includes cases with differing semantics and layouts. On Morph4Data, "spherical interpolation" achieves a better FID score than "DiffMorpher". This also demonstrate that our proposed evaluation datasets are more comprehensive.

Q2: more information about the user study.

We provided a total of 50 examples to 30 volunteers for evaluation. Each example included the results of our method alongside the results generated by three other comparison methods.

Q3: The reverse diffusion steps. Diffusion inversion is a technique used in image edition and image inpainting. Its goal is to start from a given image $x_0$ and progressively "add noise" to revert it back to the initial noise state $x_T$.

Here, we use the DDIM as the example for illustration. Instead of generating $x_{t-1}$ from $x_t$ via the DDIM inference formulation:

$$x_{t-1} = \sqrt{\bar\alpha_{t-1}}(\frac{x_t-\sqrt{1-\bar\alpha_{t}}\epsilon_\theta(x_t)}{\sqrt{\bar\alpha_{t}}})+ \sqrt{1-\bar\alpha_{t-1}-\sigma_t^2}\epsilon_{\theta}(x_t) + \sigma_t\epsilon _t,$$

we generate $x_t$ from $x_{t-1}$ using the reversed version of the above formula:

$$x_t = \sqrt{\frac{\bar\alpha_t}{\bar\alpha_{t-1}}}x_{t-1}+\sqrt{\alpha_t} (\sqrt{\frac{1}{\alpha_t}-1 } - \sqrt{\frac{1}{\alpha_{t-1} } -1})\epsilon_{\theta}(x_{t-1}).$$

Hence, by inputting $x_0$ into the pre-trained U-Net, we can obtain the predicted noise $\epsilon_{\theta}(x_{0})$. Using the above formula, we can then compute $x_1$. By iteratively applying this process, we can derive the initial noise state $x_T$ from a given image using the pre-trained U-Net. For more detailed information, please refer to [1, 2].

[1] Edict: Exact diffusion inversion via coupled transformations. CVPR 2023

[2] Prompt-to-prompt image editing with cross attention control. ICLR 2023

Response to reviewer #XAhp

Attention Interpolation for Text-to-Image Diffusion (He et al., 2024) is cited in the related work section but never compared to, despite the methods seeming quite similar.

Similarly, Smooth Diffusion (Guo et al., CVPR 2024) is quite closely related and could also be compared to, although it requires additional tuning and thus seems a lot less relevant as a baseline.

Thank you for the reviewer's suggestions. We have conducted experiments to compare our approach with the two methods mentioned and find that these method can only handle images with similar layout and semantics. Instead, our method can efficiently handle this case. Additionally, it's important to note that “Attention Interpolation for Text-to-Image Diffusion” relies on the IP-Adapter for image morphing, which compromises training efficiency. Furthermore, “Smooth Diffusion” necessitates tuning, making it slower and less efficient than our method. We will add more discussion and comparisons with these methods in the revised edition

Eq. 3 looks wrong to me. Defining it with j as an integer in the range [1, 5] does not result in a slerp, as far as I know.

In our experiments, we configured the pipeline to generate five transition images. That's the reason why we compute five sampling points (with j as an integer in the range [1, 5]) in Eq. (3), each corresponding to a transition image. Moreover, our approach allows for setting j to any value. As the number j increases, Eq. (3) will increasingly resemble a more smooth slerp operation.

I think using a DCT instead of an FFT could help improve quality by getting rid of the correlation between the pixels on the left and right (and top and bottom respectively) edge of the image.

Thank you for the suggestion. We have carried out experiments to contrast the DCT with the FFT and observe that DCT perform on par with FFT. This situation might because our Gausian noise inject doesn't help improve a lot for the result.

The number of decimal digits in tables is excessive. Please reduce it to a reasonable number (more than 3 or 4 digits should not be necessary in most cases) to aid readability.

Thanks for the suggestion from the reviewer. We have made refinements to the decimal digit precision in Tab. 1 and Tab. 2.

There are a bunch of small mistakes. I'd recommend going over the paper and fixing them for a potential camera-ready version, as the current state looks a bit sloppy. Some examples: l. 122: "ur" -> "Our", l. 130: "Recently, Recently,"; l. 132: "wang2023interpolating"; l. 194: wrong citation for LDM, l. 211: I think DiffMorph should be attributed to different people.

Thanks for the suggestion from the reviewer. We have diligently proofread the paper and rectified those minor errors in the revised edition.

Response to reviewer #JPFT

There are some unverified claims at the core of this task. For example, see lines 254-255.

In lines 254-255, we aim to emphasize that the multi-step denoising procedure is notably nonlinear, potentially leading to inconsistencies in the generated image morphing.

To demonstrate the nonlinear nature of the multi-step denoising process, we would like to refer the reviewer to Eq. (9) in the DDIM paper[1]

What constitutes good image morphing is somewhat underdefined. The intended effect that this paper seeks to achieve feels too vague and abstract, making it challenging to identify a clear definition of effective image morphing.

An effective image morphing outcome should exhibit gradual transitions from the initial (left) image to the final (right) image while preserving the original identities. For instance, (1) In the second example depicted in Fig. 5, IMPUS displays an identity loss where the third generated image deviates from the original identity; (2) Within Fig. 16, the lack of similarity between the second and third generated images results in abrupt transitions, detracting from smoothness.

Even with quantitative metrics like PPL, high scores do not necessarily indicate results that align with human preferences, which can be highly subjective.

We concur with the reviewer's observation that existing quantitative metrics face challenges in accurately assessing result quality. In our research, we follow DiffMorpher and IMPUS to conduct PPL, LPIPS, and FID evaluations.

To enhance our comparative analysis and include human preferences, we further conducted user studies. Specifically, we invited 30 volunteers encompassing animators, AI experts, and gaming enthusiasts aged between 20 and 35 to choose their most favorable results. The results are presented below, demonstrate the effectiveness of our proposed approach from a subjective standpoint.

|IMPUS|DiffMorpher|Slerp|Ours|

|Preference|17.16%|14.89%|7.82%|60.13%

While the authors have introduced several modules, their use appears to have been determined empirically. The selection of hyperparameters and the final module composition could benefit from clearer justification, as they seem somewhat arbitrary at times.

In Fig. 2, our experiments involve substituting the key and value features within the attention mechanism. Through this analysis, we notice the pivotal role played by these features in facilitating seamless transitions and maintaining the inherent identity within the images. Consequently, the core enhancement in our design stems from the modifications made to the key and value features.

The design outlined in Sections 3.2 and 3.3 is crafted to handle their respective issues, aiming to achieve optimal performance. Indeed, our pipeline's final iteration is a product of iterative experimentation. Despite its simplicity, our strategy proves effective and logical. Moreover, our approach surpasses existing techniques without necessitating extensive training.

What's the intuition behind Equation 5. How does it relate to identity preservation?

In Figure 6, we observe that the absence of Eq. (5) can lead to challenges in generating smooth transitions in extreme scenarios. Instead, relying on the key and value features generated through spherical interpolation, as described in Eq. (3), leads to gradual changes (see Fig. 2). Therefore, the key lies in determining the optimal method for this replacement of key and value features.

Through our experiments, we find that by averaging the results of spherical interpolations, we are able to achieve the best performance.

How does Equation 6 connect to motion flow? What do you mean by "step-oriented"?

In our "step-oriented motion flow" process, we compute the gradual transition from the initial (left) image to the final (right) image. Initially, we opted to term it "motion flow" to signify this progression. However, upon reflection, we recognize that "motion flow" typically refers to changes in motion within 2D or 3D spaces, potentially causing confusion. As a result, we will adjust this terminology in the updated version.

The alterations in attention features evolve progressively based on the step j. This is precisely why we coined it as "step-oriented."

[1] Denoising Diffusion Implicit Models. ICLR 2021.