Motion Guidance: Diffusion-Based Image Editing with Differentiable Motion Estimators

We thank the reviewer for their positive and comprehensive feedback. Below are our responses.

 

Target Flow Construction

Generating a target flow field may seem complicated but is actually rather straightforward, as outlined in Appendix A1. All flows in our paper (except for the motion transfer results, which are extracted from real videos), are compositions of simple, elementary flows. As such, it is relatively straightforward to imagine a UI, similar to existing software such as MS Paint, Keynote, or Illustrator, which allows a user to “paint” a target flow.

Additionally, in Appendix A1 we describe how we create target flows that are pixel-aligned with the input image by using SAM [Kirillov et al. 2023], a simple, fast, and highly user-friendly, off-the-shelf segmentation network.

We also agree with the reviewer that some flows are hard to construct, which is why we include the motion transfer results. We show that these flows can be obtained by extracting them from existing video.

 

Recursive Steps in Motion Transfer Results

We agree that fewer recursive steps being needed for the motion transfer experiments may have something to do with the pixel misalignment. Our hypothesis is that fewer recursive steps helps in this case because we do not want to optimize our guidance loss function too aggressively, given that our flow is somewhat misaligned. This is because we are effectively taking fewer “gradient steps” on our guidance function. We provide a visualization of the extent of misalignment between the extracted motion and the source images in Appendix Section A6 and Figure A9. Overall, motion transfer works because our method optimizes a soft objective, which is controlled partly through the number of recursive steps.

 

Flow Legend

We have already included a flow legend in the form of a colorwheel in Figure 1, where we show how the hue and value of a color corresponds to the motion we are trying to depict. We opted not to include this legend in each figure due to space considerations, but we have updated the caption of each figure to refer back to this legend to clear up any confusion. We thank the reviewer for pointing this out.

 

Recursive Denoising Value

Information on hyperparameters can be found in section A1, “Implementation Details,” in the appendix. We use a value of K=10 for all experiments except for the motion transfer experiments, in which we use a value of 10, 5, and 2 for the cat, pug, and panda respectively. We realize we did not specify the exact parameters for the motion transfer experiments, and we have updated Appendix Section A1 accordingly. We thank the reviewer for spotting this.

 

Role of the Flow Loss

The reason why removing the flow loss results in hallucinations is because the color loss provides no signal to disoccluded regions, but the diffusion network still denoises in the disoccluded areas. There is nothing preventing the diffusion model from hallucinating what it wants in those areas, in particular, from hallucinating an extra apple or an additional waterfall. We also believe the reviewer’s hypothesis for why the flow loss helps. By adding the additional constraint we encourage the network to remove any potential ambiguity in the flow.

 

Efficiency of Flow Backpropagation

Backpropagating through the entire loss function takes up roughly 50% of the total sampling time. Using the target flow in the color loss to circumvent the backpropagation overhead is an interesting idea, but does not work because we have to backpropagate through the flow loss anyways. We do note this trick is still useful, and we use it in our ablations when we try to completely remove the flow loss from our guidance loss.

 

We hope these comments have addressed the reviewer’s questions. We would like to thank them for their time, and ask that they consider raising their score for our paper.

We thank the reviewer for their positive and comprehensive feedback. Below are our responses.

 

User Interface

Generating a target flow field may seem complicated but is actually rather straightforward, as outlined in Appendix A1. All flows in our paper (except for the motion transfer results, which are extracted from real videos), are compositions of simple, elementary flows. As such, it is relatively straightforward to imagine a UI, similar to existing software such as MS Paint, Keynote, or Illustrator, which allows a user to “paint” a target flow.

 

DragGAN

We see the relationship between our method and DragGAN as a series of tradeoffs. While DragGAN can work for sparse point displacements, they sacrifice precision and control. Our method on the other hand works with dense flows, which can be much more accurate. DragGAN is quite fast, but the tradeoff is that GANs are trained only on narrow datasets, while our method can handle a far wider breadth of images and can be controlled by text prompting.

 

Out of Domain DragGAN Results

The goal of Figure 5 was to illustrate the limitations of DragGAN and how our method can overcome them. We wanted to point out two things. Firstly, that DragGAN only works on narrow domains. And secondly, that for DragGAN to work on real images, it also requires a very good GAN inversion method. This is discussed in section 4.4.

 

Video Generation

We thank the reviewer for this idea. We have updated the appendix in the supplemental material to add a sequence of deformations by increasingly larger target flows, which can be found in Figure A10 and Section A7. While the motion edits look good, the images are sampled independently of each other. Because of this the model fills in disocclusions inconsistently, and the frames form a slightly choppy video. We believe more work must be done to address this issue, and leave this for future work.

 

We hope these comments have addressed the reviewer’s questions. We would like to thank them for their time, and ask that they consider raising their score for our paper.

We thank the reviewer for their positive and comprehensive feedback. Below are our responses.

 

Quantitative Results

We first point out that we quantify the trade-off between CLIP alignment scores and Flow Loss for two different datasets, and two different flow methods in Figure 6. The majority of our results are qualitative because there is no standard benchmark for the task that we consider, and we believe that qualitative results effectively convey both the use cases of our method and the high quality of the samples from our method.

 

Random Samples

We have already provided completely random samples using our method in the appendix, in Section A3 and Figure A4. In total, there are 60 images sampled with a set random seed, all of which are of high quality in terms of image content and adherence to the target flow.

 

Hand-Drawn Optical Flow

All experiments in this paper use “hand-drawn” optical flow, which is described in Appendix section A1. These flows are compositions of simple, elementary flows, making it relatively easy to generate by hand. The only experiments for which we use optical flow from a video is the “motion transfer” results in Figure 7.

 

Guidance Function Equation

We thank the reviewer for catching this. We have updated equation 4 to be more precise and have included a short explanation.

 

We hope these comments have addressed the reviewer’s questions, and we would like to thank them for their time.

We thank the reviewer for their constructive and comprehensive feedback. Below are our responses.

 

Comparison to Recent Works

Per ICLR reviewer guidelines, authors are not expected to compare with work published within four months of the main paper deadline (Sept. 28th, 2023). Not only are [Shi et al. 2023] and [Mou et al. 2023] not even published, their preprints were only just uploaded to arXiv within this four month period. In fact, it appears that one of these works may have been submitted to this conference as well, clearly making it concurrent work.

Our paper does, however, contain a comparison to recent related work, DragGAN, even though we were not required to since it published on July 2023. This is because we thought the reader would benefit from such a comparison, and because we thought we could make the comparison fairly since DragGAN had released a finalized version of their code. Overall, we hope this conveys to the reviewer that we have made a good faith effort to compare our method to relevant baselines.

If the reviewer has any additional questions or comments, we would love to discuss further. However, we believe we have addressed the reviewer’s main concern, and we would like to ask the reviewer to consider improving their rating.

 

Complex Flow Fields

We disagree with the assessment that our flow fields are “coarse and global,” especially when compared to prior work in this area, which handle only sparse point-wise motion edits. For example, the deformation of the topiary in Figure 1, the waterfall in Figure 3, the homography on the laptop lid in Figure 3, and the river in Figure 4 are all quite complex. Each of these flow fields consists of continuously varying, highly non-global, fine-grained displacements. This is not to mention the rotations and scaling flow fields, which while conceptually very simple, are quite mechanically complex and require fine-grain control of the content to achieve.

 

Hallucinations in Motion Transfer

First, we would like to note that our method does not hallucinate for the vast majority of our examples, as seen in Figures 1, 2, 3, 4, and 5. We believe that “motion transfer” is an extreme case, and results in small changes in lighting or subject identity because the extracted optical flow does not perfectly match the source image. We add a demonstration of this in Figure A9 of the appendix, with discussion in Section A6. Because of this mismatch, the diffusion model must do significant work in hallucinating some of the missing or unrealistic detail, and “correct for” the target flow. This is one of the benefits of our soft optimization method. In fact, to some extent objects must be hallucinated. For example, naively stretching the topiary in Figure 1 would result in artifacts in the leaves. That the diffusion model “hallucinates” details can make the edit much more realistic.

 

Edit Mask

Our method does not necessarily need an edit mask to work. For example see the motion transfer results in Figure 7. We add it for two reasons. One is because we wanted to mimic the edit masks of other methods such as [Pan et al. 2023], which seems useful. And the other is that guidance is actually a soft constraint. It is akin to doing gradient descent on two different objective functions. Therefore, even by specifying zero flow in an area, we are not guaranteed zero flow in that area. This is quite useful sometimes, as we can follow the “intention” of the target flow while not being bound to its exact form (as discussed above with respect to motion transfer). But it also means that an edit mask can help to make this constraint hard.

 

One Step Approximation and Flow Estimates

We have already provided examples of one step approximations and corresponding flow estimates in Figure A6 in the appendix. As can be seen, the one step approximations are quite good, even in earlier timesteps. This allows our guidance function to control the optical flow throughout the entire denoising trajectory.

 

Differentiating Through Classical Flow Methods

We choose to use neural network based flow methods because they are state of the art. We choose RAFT specifically because it is a very high quality, fully-differentiable optical flow method, and has been shown to work in diverse scenarios. This allows us to make motion-based edits on a diverse set of source images with a broad range of target flows.

 

We hope these comments have addressed the reviewer’s questions. We would like to thank them for their time, and again ask that they consider raising their score for our paper.