How I Warped Your Noise: a Temporally-Correlated Noise Prior for Diffusion Models

We thank the reviewer for the constructive feedback. We will address the remaining questions below.

Zero-shot and training

I guess most experiments in this paper are conducted in zero-shot way which applies the image processing model and method to video processing problems. Looking forward to seeing whether this helps when training a video model, like the PYoCo model.

This is correct, we propose our noise warping method primarily as a novel tool to improve temporal coherency of video editing tasks in a zero-shot setting. This is part of an effort to leverage pretrained image models for video tasks without requiring additional resource-intensive training of models on video datasets. However, as the reviewer mentioned, employing this temporally-correlated noise prior for training or fine-tuning video diffusion models is definitely an exciting direction for future work, and we will be happy to see adoption of our $\int$-noise in these contexts.

Comparison with video models

Considering video restoration (Zeroscope Text-to-Video) and pose-to-human (dreampose) already have good performance, I wonder the gap between image model+ integrate noise and these video models.

In our work, we raised awareness on the importance of noise priors for temporal consistency and focused on solving the mathematical problem of distribution-preserving Gaussian noise warping. We then validated our proposed solution on several zero-shot video editing tasks with diffusion models, making it an effective tool for improving temporal coherency in these applications.

While our proposed method is compatible with other zero-shot techniques like cross-frame attention (which we implemented in some of our results), it is challenging to directly compare our method alone with video models for several reasons.

First, as many previous work on zero-shot video editing with image diffusion models have shown, it is often necessary to combine different techniques together to achieve good temporal coherency. For instance, Pix2Video [1] uses a combination of DDIM inversion, loss guidance and depth conditioning in their pipeline. A fair comparison against existing video models would have to be task-specific (super-resolution, stylization, video generation etc.), as different works adopt different pipelines. Our approach is a general method to enforce noise priors for diffusion models, and could be easily incorporated into different pipelines.

Secondly, a comparison with these video models requires at least similar inputs to be fair. In the example of pose-to-person, Dreampose [2] is trained to take a very informative DensePose [3] parameterization as input, while we based our method on PIDM which only takes an Openpose poses [4]. Note that we did additionally use DensePose in our application results, but only for motion estimation (see Appendix D.3).

All in all, we believe combining our $\int$-noise with other techniques like depth conditioning can further improve the quality of the results.

On another note, we have added more visual comparisons of our methods with other noise priors such as Control-A-Video and DDIM inversion, which we believe can be helpful. Please refer to the new project webpage for additional results: https://warpyournoise.github.io/

Integration with DeepFloyd IF

I wonder if this is applicable to other pixel-level diffusion models like the open-sourced deepfloyd from stabilityAI, whose capacity is comparable or better than stable diffusion.

Our noise warping method is also applicable to other non-latent diffusion models like DeepFloyd from StabilityAI. We added some examples of using our noise warping with the open-source model for the task of video super-resolution and stylization to support our claim. The results can be visualized on our project webpage: https://warpyournoise.github.io/#deepfloyd We thank the reviewer for pointing this out, as this broadens the generalization abilities of our method beyond the category specific models we showcased in our initial manuscript.

Conclusion / References

We hope this clarified some of the interrogations. Please feel free to reach out for any further questions/comments regarding the submission and our answers above.

References:

[1] Ceylan, Duygu, Chun-Hao P. Huang, and Niloy J. Mitra. "Pix2video: Video editing using image diffusion." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Karras, Johanna, et al. "Dreampose: Fashion image-to-video synthesis via stable diffusion." arXiv preprint arXiv:2304.06025 (2023).

[3] Güler, Rıza Alp, Natalia Neverova, and Iasonas Kokkinos. "Densepose: Dense human pose estimation in the wild." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[4] Cao, Zhe, et al. "Realtime multi-person 2d pose estimation using part affinity fields." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.

We thank the reviewer for the constructive feedback. We will address the weaknesses mentioned below.

Treatment of non-diffeomorphic warping fields

From the theoretical point of view, the presented approach relies on the warping field being a diffeomorphism, while in practice to the best of my knowledge optical flows estimated with off-the-shelf methods are rarely invertible mappings.

It is true that the theoretical derivation relies on the warping field being a diffeomorphism, which is generally not the case for optical flow maps. In particular, disocclusions can reveal empty spaces that aren’t mapped to any part of the initial frame.

We solve this in practice by replacing missing values with newly sampled noise. On very long sequences, we also resample the high-resolution noise periodically, which effectively updates the anchor frame. We will add a more detailed explanation regarding how we handle warping in practical settings in the Appendix.

Image quality measure in quantitative evaluation

According to Tab. 1, while the proposed method typically increases temporal coherence, at the same time it worsens frame-wise image plausibility metrics such as LPIPS or FID.

For the appearance transfer example, we used FID as a measure of image quality, since there is no ground truth in this case. FID is computed between the LSUN Bedroom dataset and a dataset we create from frames of the video results we have. Unfortunately, FID simultaneously measures sample quality and diversity, while we are only interested in quality. Since we are computing it over a sequence of frames from the same video, the diversity is naturally very low, which explains the overall high values we have in Tab. 1 (typical FID values are <50). Additionally, less temporally coherent methods like ‘Random Noise’ will likely have more variations between frames, which improves diversity. This explains why random noise performs the best wrt. FID, and our method typically performs a bit worse. All in all, the main take-away from the FID comparison is that our method performs in the same range as other Gaussian noise priors (random, fixed, PYoCo, Control-A-Video) and much better than the other interpolation methods. This, in complement with the warp error metric, shows that we are able to take the “best of both worlds”.

Regarding the LPIPS metric for video super-resolution and restoration, we do observe a slight decrease in image quality according to the metrics, which we cannot fully explain. Similarly, the main take-away is that we are performing in the same range as the other noise priors methods, while having much better temporal coherency.

More comparisons with Control-A-Video

None of the video samples in the supplementary except for fluid dynamics provides the results of noise handling with the method proposed by Control-A-Video model. Please note that this model more close to the presented method in spirit since it also takes input video into account while PYoCo does not.

The residual noise from Control-A-Video behaves like fixed noise in regions of uniform color, and like random noise around moving edges between objects. Our experiments have shown that this generally leads to the “worst of both worlds'', as both flickering and texture sticking happens. We have updated the results with more comparisons with the residual noise prior. They can be found on our project webpage: https://warpyournoise.github.io/

We thank the reviewer for the constructive feedback. We will address the remaining questions below.

Computational Efficiency

The proposed method is computationally inefficient as compared to prior arts. Quantitative comparisons on this aspect would have been more helpful for future research works.

We estimate both wall time and CPU time for all methods on a video sequence from the surrogate rendering application (resolution at 256x256). By evaluating the run time for different sub-sequence lengths, we can linearly regress the average per frame computation time of each method. This is evaluated on a GeForce RTX 3090 GPU and an Intel i9-12900K CPU. The table below summarizes the run time of our method against previous approaches. We will add this table to the updated manuscript appendix.

Our method is indeed less efficient. This is a trade-off that we decided to make, as our intent was to solve Gaussian noise warping as accurately as possible, to understand how far that could bring us. It is worth noting that we have explored several simpler, less computationally heavy ways to warp the noise by making different simplifying assumptions (such as the root-bilinear interpolation methods shown in the table, a modification also suggested by Reviewer 3 (H5Nc) in Question 1). However, none of those led to satisfying solutions.

An analysis of the table shows that simple noise sampling methods like PYoCo, fixed or random noise can be executed efficiently on GPU. Methods that rely on information from the input sequence in a simple way such as Control-A-Video or interpolation methods are almost 3 orders of magnitude slower than the simple noise sampling methods. Finally, our warping method requires ~1s per frame to warp the noise. While being considerably less efficient than aforementioned methods, the proposed approach is comparable to DDIM inversion. With a standard setting of 50 DDIM steps, DDIM inversion takes longer than our method to produce a noise map, while being similar to ours in run time if we reduce the number of steps to 20. Please refer to our answer to Reviewer 1 (13WQ) for more qualitative comparisons with DDIM inversion.

Additionally, our method has two parameters $k$ and $s$ which control the resolution of the upsampled noise and the discretization level of the pixel polygons respectively. These parameters provide users with the possibility to trade off accuracy in the temporal correlations against efficiency. We refer to the table above to demonstrate how the parameters impact the performance. The measurements represent the wall clock time in seconds that our method requires to compute noise for a video sequence of 24 frames at resolution 256x256. Additionally, we refer to the Section 4 of the paper on how these parameters affect the quality of the results.

Lastly, our implementation was not optimized for efficiency. We believe it can still be drastically improved, which would make the computational cost of using our noise scheduler negligible.

Better visual comparison for fluid super-resolution

Figure 5 illustration is hard to follow, it’s not clear how to judge the performance based on the images shown.

Unfortunately it is hard to show video results properly on paper. We will update the visualization in the submission to reflect the feedback. Additionally, the video results of this figure can be found on our project webpage at https://warpyournoise.github.io/#fluid-sr, where the visual comparison is much clearer.

Conclusion / References

We hope this clarified some of the interrogations. We will update the paper to integrate the feedback. Please feel free to reach out for any further questions/comments regarding the submission and our answers above.

References:

[1] Ge, Songwei, et al. "Preserve your own correlation: A noise prior for video diffusion models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Chen, Weifeng, et al. "Control-A-Video: Controllable Text-to-Video Generation with Diffusion Models." arXiv preprint arXiv:2305.13840 (2023).

[3] Song, Jiaming, Chenlin Meng, and Stefano Ermon. "Denoising diffusion implicit models." arXiv preprint arXiv:2010.02502 (2020).

We thank the reviewer for the constructive feedback. We will address the remaining questions below.

Implicit assumption on temporally-coherent noises

The major weakness from the practical point of view is the implicit assumption that temporally correlated noise maps can induce temporally consistent video editing results, which is often not true.

While there is no theoretical guarantee that diffusion models produce temporally-consistent video editing results with a given temporally-correlated noise map, we observe that this is often the case in practice. Take the translating pizza example on Appendix E: if we add a noise that undergoes the same translation as the input video, the generated images are more temporally coherent. For better visualization and understanding of these results, we refer to the videos at https://warpyournoise.github.io/#warp-ldm.

Other examples in the paper also highlight that transformations that are applied to the noise will produce similar image transformations, even though that’s not explicitly enforced during training of diffusion models. With this in mind, we believe that our proposed temporally coherent noise would indeed improve the training of video diffusion models, similar to PYoCo [1].

Sensitivity with respect to flow map

I’m wondering how sensitive the method is with respect to the underlying estimated flow map? In other words, how do the errors in the estimated optical flows affect the results?

Overall, as one would expect, accurate optical flow is important for achieving good results. From our experiments, we observed that poorly estimated optical flow will still be able to warp the noise, but it tends to create visual artifacts in the form of temporal misalignment, i.e. fine details in the video will not seem to be perfectly synchronized with the motion of larger structures.

That being said, it is worth noting that the method is still relatively robust to small errors. For the video super-resolution and JPEG restoration task we show in the submission (Figure 4), the optical flow is computed on the degraded, low-resolution video, and our method is still able to produce decent, coherent results.

Lastly, as our noise warping method is agnostic to the way the deformation field is computed, it can also work with more robust methods such as Neural Layered Atlases [2], when flow-based methods struggle to provide accurate motion estimation. These methods map each pixel in each frame to a canonical space common to all frames. In this case, the canonical space is considered the “first frame” and the mapping replaces the accumulated flow map in our algorithm. Please refer to our answer to Reviewer 3 for a more complete discussion about the properties of the warping velocity field.

Comparison with DDIM inversion

I’m wondering how will the noise maps obtained [using DDIM inversion] compare to the ones obtained from the $\int$-noise in terms of the final video quality?

The main issue that prevents a fair and straightforward comparison with DDIM inversion is that the latter only produces one single noise map per image, whereas the applications we show generally use DDPM and require as many noise maps as there are steps in the denoising process.

We included a new visual comparison for the bedroom SDEdit surrogate rendering results with DDIM noise inversion that can be visualized here: https://warpyournoise.github.io/#ddim. There are two ways we can think of to use DDIM inversion with SDEdit.

• The first consists in partially inverting the image to an intermediate noisy version using DDIM inversion and denoising it with forward DDIM. This expectedly leads to a reconstruction of the input synthetic scene with no additional realistic details.
• The second way consists in fully inverting each frame to obtain a set of noise maps, which we treat similarly to the other noise priors we compare to in the original submission. Since the synthetic renders we use as input video are far from the distribution of the LSUN dataset, the inverted noises are not Gaussian. Our results demonstrate this creates noticeable artifacts.

Lastly, DDIM inversion deterministically maps each frame to a noise, which removes any diversity in the results. In comparison, our method can apply the same warping to different noise maps to generate different equally temporally-consistent appearances.

Conclusion / References

We hope this clarified some of the interrogations. We will update the paper to integrate the feedback. Please feel free to reach out for any further questions/comments regarding the submission and our answers above.

References:

[1] Ge, Songwei, et al. "Preserve your own correlation: A noise prior for video diffusion models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[2] Kasten, Yoni, et al. "Layered neural atlases for consistent video editing." ACM Transactions on Graphics (TOG) 40.6 (2021): 1-12.