ZeroPatcher: Training-free Sampler for Video Inpainting and Editing

Weaknesses 1.a: Shorten the preliminary. Show where latent masking occurs in pipeline figure.

Thanks for you advice. We are so pleased to update the paper to make it easier to read.

Weaknesses 1.b: Is there a confusion notation on $\boldsymbol{f}_\theta(\boldsymbol{x}_t)$ and $\boldsymbol{\mu} (\mathbf{x}_t, \boldsymbol{f} _\theta (\boldsymbol{x}_t))$?

In fact, there is no confusion. The notation of our paper follows the style of DDIM paper [3]. $\boldsymbol{f}_ \theta (\boldsymbol{x}_ t)$ estimates $\boldsymbol{x}_ 0$. However, $\boldsymbol{u}(\boldsymbol{x}_ t , \boldsymbol{f}_ \theta(\boldsymbol{x}_ t))$ is the mean of $\boldsymbol{x}_ {t-1}$. As is introduced in line 95, it is a linear combination between $\boldsymbol{x}_ t$ and $\boldsymbol{f}_ \theta(\boldsymbol{x}_ t)$. The exact formulation is shown in Eqn.7 in DDIM paper. It is:

$\boldsymbol{\mu}(\boldsymbol{x}_ t, \boldsymbol{f}_ \theta(\boldsymbol{x}_ t)) = \sqrt{\alpha_ {t-1}} \boldsymbol{f}_ \theta(\boldsymbol{x}_ t) + \sqrt{1 - \alpha_ {t-1} - \sigma_t^2} \cdot \frac{\boldsymbol{x}_ t - \boldsymbol{f}_ \theta(\boldsymbol{x}_ t)}{\sqrt{1 - \alpha_ t}}$

We find readers may be confused with the $\boldsymbol{\mu}_ \theta(\boldsymbol{x}_ t)$ in Eqn.10. As is introduced in line 98, we define $\boldsymbol{\mu}_ \theta(\boldsymbol{x}_ t) = \boldsymbol{\mu}(\boldsymbol{x}_ t, \boldsymbol{f}_ \theta(\boldsymbol{x}_ t))$. We will update this part to make our notation more clear.

Weaknesses 1.c: Explain how the expectation in Eqn.8 is maximized. Is Eqn.10 differentiating Eqn.8 w.r.t. $\boldsymbol{x}_ t$?

Eqn.8 defines a least square problem. Since $\|(\boldsymbol{x}_ {t-1} ^ a, \boldsymbol{x}_ {t-1}^ {b, i}) - \boldsymbol{\mu}(\boldsymbol{x}_ t, \boldsymbol{f}_ \theta(\boldsymbol{x}_ t))\|_ 2 ^ 2 \ge 0$, the term is minimized if and only if $(\boldsymbol{x}_ {t-1}^a, \boldsymbol{x}_ {t-1}^{b, i}) - \boldsymbol{\mu}(\boldsymbol{x}_ t, \boldsymbol{f}_ \theta(\boldsymbol{x}_ t))=0$. With this, we can already define a fixed point iteration problem. In our appendix, we use the analytical solution for least square problem. The solution is shown in Eqn.4 in the appendix. We do not compute the derivative w.r.t. $\boldsymbol{x}_ t$ in our paper.

W 1.d: Latent mask fuser illustration is using two irrelevant videos.

This is very good advice! In practice, we use mask to fuse the model generated masked region with the known background. We will update it for better readability.

W 2.a: Why not use latent optimization?

We tried to optimize the latent throught back-prop with the objective Eqn.9. Below, we show a quantitative comparison on DAVIS video inpainting. We use an adaptive learning rate $\eta_t = 0.02 \sqrt{\alpha_t}$ suggested by COPAINT. The experiment shows, fixed point iteration achieves better performance while using considerable lower cost.

Computation and memory cost One objective of our method is to run it without using more graphic memory than normally sampling videos from the model. As the text-to-video generators becomes larger and video resolution becomes higher, this problem becomes more severe. Latent optimization method requires one or multiple back-propagation of the diffusion model (with gradient checkpointing, in most cases, which leads to even more cost), this introduces significantly more computation and memory usage. As shown below.

Weaknesses 2.b: Why not use pixel space masking? Comparison between LMF and direct masked replacing in latent space.

This is a pity we cannot show you the visual comparison due to the full-text rebuttal this year. A quantitative comparison on DAVIS is shown below. Pixel space masking is prohibitively expensive for latent video diffusion models, as it requires decoding to pixels, applying the mask, and re-encoding to latent space at every denoising iteration. Video VAE encoding/decoding alone takes ~6 seconds per iteration, making this approach impractical.

Our Latent Mask Fuser (LMF) eliminates this overhead, reducing latency to 0.07s versus 5.67s for pixel-space masking while maintaining near-identical quality (PSNR: 34.49 vs. 34.67; SSIM: 0.9774 vs. 0.9789). Direct latent-space mask replacement (0.01s) causes severe boundary artifacts due to spatial-temporal compression (e.g., 4×8×8 downsampling in CogVideoX), and the video latent is somewhat delicate. It degrade PSNR to 31.64 and SSIM to 0.9578.

Question 1: Compared to RePaint and COPAINT theoretically

In the paper of RePaint. The key algorithm is described in Eqn.(8a), Eqn.(8b) and Eqn.(8c). This is identical to back projection method described in DDNM [1]. Its limitation is shown in line 126 in our paper.

COPAINT proposes to optimize latent $\boldsymbol{x}_t$ by maximizing the conditional posterior in each denoising step. As Eqn.10 in COPAINT paper says, we translate the notation to the style of our paper:

$\boldsymbol{x}_ t ^ * = \arg \min_{\boldsymbol{x}_ t} \|\boldsymbol{x}_ 0 ^ a - \boldsymbol{f}_ \theta ^ a (\boldsymbol{x}_ t)\|_ 2 ^ 2$ where $\boldsymbol{f}_ \theta ^ a (\boldsymbol{x}_ t)$ is the unmasked part of model prediction. A foundamental difference between COPAINT objective and the EM objective is that, our objective includes the term $\boldsymbol{x}_ t$ instead of only $\boldsymbol{f}_ \theta ^ a (\boldsymbol{x}_ t)$. Only then, we can write the solution in this formation $\boldsymbol{x}_ t ^ * = g(\boldsymbol{x}_ t ^ *)$, and fixed point iteration can be applied.

Note that we also extend the objective for conditional flow matching generative models, as shown in our appendix. Crucially, as reported by COPAINT in Fig.3, the latent optimization method requires siginicantly more time to compute than DDNM (about 5 to 20 times more depending on hyparameter choice). Please see W.2a to check how latent optimization is working in the video model. Our fixed-point iteration strategy is essential for efficient sampling.

Question 2: Visual illustration on convergency of the fixed point.

We show how CD-EM reaches a fixed point by evaluating how much $\boldsymbol{f}_ \theta (\boldsymbol{x}_t)$ is different from the previous iteration. The similarity is evaluated using PSNR on 10 sampling videos. In this experiment, we use $N=1, P=1$. Note that, each row show the convergency of a certain denosing timstep. As K (iteration steps) becomes larger, the fixed point iteration will tend to not to change the content, and reaches a fixed point.

Question 3: A pseudo-inverse example for Eqn.5.

To apply Eqn.5. One must flatten the image to a 1d vector (e.g, flatten using raster order). Now, I show the case where A is a 2x2 mean pooling operator on a 4x4 image. We have:

0.25 & 0.25 & 0 & 0 & 0.25 & 0.25 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\\\ 0 & 0 & 0.25 & 0.25 & 0 & 0 & 0.25 & 0.25 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0.25 & 0.25 & 0 & 0 & 0.25 & 0.25 & 0 & 0 \\\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0.25 & 0.25 & 0 & 0 & 0.25 & 0.25 \end{matrix}\right].$$ Then the pseudo-inverse of it is a upsampling operator $$\mathbf{A}^ \dagger = \left[\begin{matrix} 1 & 1 & 0 & 0 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\\\ 0 & 0 & 1 & 1 & 0 & 0 & 1 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 & 0 & 0 & 1 & 1 & 0 & 0 \\\\ 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 1 & 0 & 0 & 1 & 1 \end{matrix}\right]^ T.$$ > Question 4: Visual comparison with BP. CD-EM's improvements over BP. **Visual comparison with BP** We will add a visual ablation comparison to our paper. Due to the full-text rebuttal this year. We recommend looking to Fig.4 for a qualitative comparison for now (you are right only using BP is equal to DDNM). **CD-EM is the major contributor to video quality** In Tab.3, the four metrics are evaluating the performance from very different perspectives. In PSNR, and SSIM, the majority improvements are brought by BP because they are evaluating whether the background region of the inpainting results looks similar to the ground truth. BP is directly replacing the known background to the generated samples. This makes it naturally the most significant contributor to PSNR and SSIM. In contrast, CD-EM is mainly working on the masked region, which is not considered by PSNR and SSIM. As shown in Tab.3, CD-EM contributed the most on VFID and $E_{\mathrm{warp}}$, the video quality metrics. Therefore, CD-EM is in fact the major contributor to video quality. > Question 6: Shadow in the car-roundabout example. As shown in the first row of Fig.4, the official mask is not including the car shadow. Then all examples in the comparison is not removing the shadow. We rework it by inflating the car mask by 64 pixels to get an extended mask covering the shadow. Then our method is able to render that region. We are sorry for not being able to show you the visual result due to the regulation this year. [1] Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model [2] Video diffusion models [3] Denoising Diffusion Implicit Models

W 1: Is our method fragile for using the captioner? Is our method training-free with LMF?

Usage of captioner

We are giving text-to-video generators the ability for video inpainting and video editing. As the feature of a text-to-video generators, the model depends a lot on the input prompts. Luckily, the LLavaNext + GPT4-o captioning strategy works well for our experiments. We first use LLavaNext to caption the video and extract the main objective within the mask.

For example, in the car-roundabout video, the original prompt is: "The video captures a sequence of images showing a dark-colored Mini Cooper car in motion on a city street. The vehicle is positioned at different angles, indicating it is turning or navigating through the traffic. The backdrop includes buildings with classical architecture, parked cars, and blue parking signs. The sky is clear, suggesting fair weather conditions."

We use GPT4-o to image a video without the masked object to produce the inpainting prompt. "The video shows a city street with buildings, parked cars, and a blue traffic sign. The sky is clear, suggesting fair weather conditions." The performance for removing the main objective from video caption is satisfying for DAVIS and Youtube-VOS. We believe during the practical use of ZeroPatcher, a real user will not be troubled by this prompt dependency.

Is our method strictly training-free

The usage of LMF will not compromise our training-free claim. The main reasons are:

1. LMF is our computation optimization patch for latent video diffusion models. If ZeroPatcher is applied to pixel space video diffusion models. There is no need for LMF.
2. For latent video diffusion models. Our method can achieve the same performance at the cost of additional computation for mask fusing operation. LMF is the way to erase the additional cost.

There are three ways to achieve mask replacing opeartion for latent video diffusion models:

• Method 1: Directly downsample pixel space mask with the scale 4x8x8. Then, we apply the downscaled mask for direct latent space masked replacing.
• Method 2: First, decode the latent to pixel videos using VAE decoder. Then, apply pixel space masked replacing. Finally, encode the replaced result to latent space using VAE encoder.
• Method 3: Train and use LMF.

To compare these three methods. We randomly sample 50 video pairs from DAVIS, each pair use a randomly sampled video mask. The methods are used to fuse the video pairs. The ground truth is obtained by using masked replacing in pixel space.

As shown by the table. Method 2 achieves very good masked replacing performance. It requires no training, but will use a lot additional computation. One can replace LMF with method 2 to achieve strict training-free.

W 2: Comparison with training-free video editting method such as AnyV2V and ReVideo.

AnyV2V is a training-free editing method that improves over the framework VideoP2P. ReVideo, however, is a requires the training of a ControlNet-like architecture to inject content and motion. We compare ZeroPatcher with them on DAVIS using automatic metrics for a comprehensive evaluation. The result is shown below:

W 3: Formating problem.

We update our script for improved readability. Thanks for you advise.

Q 2: Sensitivity to caption quality.

It is very valuable to introduce a caption quality sensitive analysis in our paper. We are working on it.

W 1: Inference speed overhead, possile trade-offs or optimization.

Inference speed overhead Please refer to our response to W 1 for Reviewer U6KC. Our method is not considerably slower than the editing and inpainting baselines. For your convenience, we copy it here

Compared to the baseline DDNM, both DDNM (using 100 inference timesteps) and ZeroPatcher (50 timesteps, for the first 25 steps, use CD-EM with $N=1, K=1, P=3$) uses 100 NFEs. DDNM consumes 241.7 seconds, and ZeroPatcher 250.9 seconds for video inpainting. Our method is not using significantly more time to run compared to the training-free inpainting baseline. For video editing, VideoComposer and VideoP2P are running on smaller resolution (VideoComposer: 16x256x256, VideoP2P: 8x512x512) compared to ours (49x720x480). If we simply use them to perform 49-frame editing clip by clip, our method is in fact not using more time than them. Meanwhile, it would be unfair to compare ZeroPatcher (250.9 seconds) with the text-to-video foundation model (78.3 seconds) on inference speed, since the foundation model do not have video inpainting and editing capability.

Possible trade-offs or optimization

One can choose to use smaller $P, K, N$ to achieve faster sampling. An extreme situation is when $P = 1, K = 1, N = 1$, we reduce the NFEs to 75. The inference time is reduced to 191.4 seconds (22% speed up). As Fig.6 shows, RFID will degrade to 0.102.

W 2: Visual ablation studies.

Due to the text-only rebuttal regulation this year, we are sorry the visual ablation cannot be shown to you now. We are additing it to the appendix. In alternative, we provide some evidences on how each tecnical component contributes to the final result.

Contribution of CD-EM When the model uses PF only, it is identical to DDNM. In Fig.4, we show a visual comparison between DDNM and ZeroPatcher. It is showing what the results look like without CD-EM.

Contribution of LMF When not using LMF, the model adopts direct latent space mask fusing. Given a dynamical video mask in the pixel space. We first downsample it using the scale 4x8x8. Then, we apply masked replacing in latent space directly. Below, we show the performance comparison on LMF and the naive method. We randomly sample 50 video pairs from DAVIS, each pair use a randomly sampled video mask. The methods are used to fuse the video pairs. The ground truth is obtained by using masked replacing in pixel space. It shows LMF achieves much better fusing performance.

We cannot show visual comparison due to the full-text rebuttal rule this year. But if you look into the results, latent space mask fusing is doing okay in the centeral area of the mask. But very severe artifacts are observed in mask boundaries.

W 3: ZeroPatcher on Wan 2.1.

This is very good advice to adopt ZeroPatcher on Wan 2.1. We are now working on it and ZeroPatcher + Wan 2.1 will be available when we open-source our project. Then, we can add a comparison between ZeroPatcher and VACE.

Thank you to the authors for preparing the rebuttal. I have also read the other reviews. My concerns have been well addressed, and I find the additional comparisons with latent optimization methods to be particularly impressive. The proposed approach offers a practical trade-off between computational cost and performance. I would like to raise my rating.

The WAN-based variant and the comparisons with VACE are valuable contributions that could benefit the research community. Additionally, as noted by other reviewers, clarifying the relationship between the latent mask fuser training and the training-free claim is important for understanding the method's scope and limitations.

Thanks for your professional review. The considerations are very valuable in making our paper better. The typos are now corrected in our script. Our response is shown below.

W 1: Inference speed compared with baselines.

Compared to the baseline DDNM, both DDNM (using 100 inference timesteps) and ZeroPatcher (50 timesteps, for the first 25 steps, use CD-EM with $N=1, K=1, P=3$) uses 100 NFEs. DDNM consumes 241.7 seconds, and ZeroPatcher 250.9 seconds for video inpainting. Our method is not using significantly more time to run compared to the training-free inpainting baseline.

For video editing, VideoComposer and VideoP2P are running on smaller resolution (VideoComposer: 16x256x256, VideoP2P: 8x512x512) compared to ours (49x720x480). If we simply use them to perform 49-frame editing clip by clip, our method is in fact not using more time than them.

Meanwhile, it would be unfair to compare ZeroPatcher (250.9 seconds) with the text-to-video foundation model (78.3 seconds) on inference speed, since the foundation model do not have video inpainting and editing capability.

W 3: notation in Eqn.8, Eqn.9, Eqn.10, and Eqn.11

Thanks for the advice. We are pleased to improve our notation to make it easier to follow. We replace the notation $(\boldsymbol{x}_ {t - 1} ^ a, \boldsymbol{x}_ {t-1} ^ {b, i})$ with $\tilde{\boldsymbol{x}}_ {t-1} = \\{\boldsymbol{x}_ {t - 1} ^ a, \boldsymbol{x}_ {t-1} ^ {b, i}\\}$. We will explain the bracket means masked fusing as shown in Eqn.16. It is also consistent to our notation in Eqn.16.

Q 1: Release code.

Yes, we are now working on it. The project will open-source soon.

Q 2: Performance on extreme masking conditions.

It is a valueable to test our method on extreme masking conditions. We are working on the outpainting and large region inpainting test. The visual results will be added to the appendix.

Q 3: How inference speed change according to P, K, N

In Tab.1 in the appendix. We show a theoretical boundary of computation complexity given $P, K, N$. We use $T$ timesteps to sample a video among them, the first $\tau$ steps will use CD-EM. The NFEs of ZeroPatcher can be computed using the following formula. $\mathrm{NFEs} = \tau (PN + P(K-1)) + (T - \tau), \qquad P \ge 1, N \ge 1, K \ge 1, \tau \le T$ Below, we show the measured inference speed ($T = 50, \tau = 25$).

Q 4: Human evaluation details.

We create a questionaire containing 50 questions using all video editing samples computed by all methods on DAVIS (But skip VideoP2P on background consistency, because it do not preserve background, comparing it is meaningless). Each question shows the editing results from all 5 methods along with the editing input. The question ask the respondent to answer which one achieves the best background preservation, and which one matches the editing prompt most. The editing results are labeled anonymously (By A, B, C, D, E). During the human evaluation, the participant will randomly receive 1 of the 50. The labeling and display order of the editing results are shuffled each time we show the participant the question. We ensure the participant will not receive repeated questions. the participant can answer up to 50 questions. We collect 1,310 answers from 171 different participants.