DiffIR2VR-Zero: Zero-Shot Video Restoration with Diffusion-based Image Restoration Models

We thank Reviewer ttiq for their positive assessment and constructive feedback. We address each point below:

[Q1] Regarding adaptation to various degradation types:

The combination of hierarchical latent warping and hybrid token merging enables robust handling of different degradations through complementary guidance at different scales and adaptive correspondence mechanisms. Our approach achieves this through several key mechanisms:

1. Global-level warping between keyframes handles large-scale structural consistency, while local-level propagation addresses fine-scale details, creating a multi-scale approach that helps maintain consistency regardless of degradation type.
2. Early in the denoising process, when latents are noisy, optical flow provides reliable structural guidance. As denoising progresses, later stages transition to similarity matching to preserve restored details. This dynamic adaptation, as illustrated in Fig. 5, allows the method to effectively handle both structural and textural degradations.

We will expand this explanation in the paper to better clarify how these components work together.

[Q2] Benefits of similarity-based guidance:

A key challenge in our setting is that optical flows must be derived from low-quality inputs, making sole reliance on flow correspondence suboptimal. Our hybrid approach addresses this limitation in several ways:

In UNet upsampling blocks of ControlNet-based diffusion models, the latents become progressively more reliable - either through injected ControlNet features or as the denoising process brings them closer to the final state. At this stage, similarity-based matching can provide more accurate correspondences. We further enhance these similarity-based correspondences by incorporating spatial information. The effectiveness of this combined approach is supported by multiple pieces of evidence:

1. Our first ablation study demonstrates that integrating both optical flow and similarity achieves better results than using flow alone
2. As shown in Figure 5, flow and similarity mechanisms can identify different types of correspondences, providing complementary information that helps maintain both structural consistency and fine details

We will expand this analysis with additional technical details in our revision.

[Q4] Inference efficiency:

Below, we report the inference time for processing 10 video frames at 854$\times$480 resolution on a single 4090 GPU:

Our method adds a reasonable overhead compared to per-frame inference (around 26s for SDx4 and 63s for DiffBIR) while maintaining strong temporal consistency. This is notably more efficient than training-based methods like Upscale-A-Video, which requires 32 A100 GPUs and encounters out-of-memory (OOM) issues even during inference on our test setup (one A6000 GPU with 48 GB memory). Furthermore, when applied to lightweight models like Marigold with 4-step sampling, our method achieves very fast inference at just 10 seconds total.

We appreciate these suggestions for improving the paper's completeness and clarity.

References:

• [1] Chan, Kelvin CK, et al. "Basicvsr++: Improving video super-resolution with enhanced propagation and alignment." CVPR. 2022.

• [2] Liang, Jingyun, et al. "Recurrent video restoration transformer with guided deformable attention." NeurIPS. 2022.

[Q3] Comparison with BasicVSR++ and RVRT:

Thank you for suggesting these important baselines. We have now completed comprehensive evaluations comparing our method with BasicVSR++ and RVRT across all test sets. While these traditional CNN-based methods achieve higher PSNR/SSIM scores and lower warping errors (e.g., on DAVIS ×4, BasicVSR++: 26.576/0.743 and RVRT: 26.595/0.744), their results tend to be overly smooth. This is reflected in their relatively poor LPIPS scores (BasicVSR++: 0.383, RVRT: 0.388) compared to our method (0.262 with DiffBIR), indicating that our results are more perceptually pleasing with better detail preservation. As demonstrated in Figure 6 and Figure 13 of our revised paper, the qualitative comparisons clearly show that our method produces sharper and more realistic details, while BasicVSR++ and RVRT tend to generate blurry results. The complete comparison results are shown in the following table:

These numbers demonstrate an important trade-off in video restoration: CNN-based methods like BasicVSR++ and RVRT tend to optimize for pixel-wise metrics at the cost of perceptual quality, while our diffusion-based approach better preserves realistic details and texture. The visual comparisons in our figures clearly illustrate this difference in output quality.

We appreciate your reviews and comments. We hope our responses address your concerns. Please let us know if you have further questions after reading our rebuttal.

We aim to address all the potential issues during the discussion period.

Thank you!

Thank you for your follow-up comments and for giving our paper a positive score!

We appreciate your recognition of the importance of our work for zero-shot video restoration. We would like to clarify a few important points:

Regarding PSNR/SSIM metrics, we note that BasicVSR++ and RVRT achieve even higher scores than recent CVPR'24 methods like FMA-Net. However, as demonstrated in Figure 6 and Figure 13 of our revised paper, their results are significantly blurry - a critical quality issue that PSNR and SSIM fail to capture. Only LPIPS, which better correlates with human perception, reflects this blurriness.

The key advantage of our method is its training-free nature - it can improve the temporal consistency of any diffusion-based model without training and, in many cases, even improves perceptual quality (LPIPS). While we acknowledge the additional computational overhead for some models, the efficiency greatly depends on the base diffusion model. For example, when applied to distilled models like Marigold (4-step), our method achieves impressive results in just 10 seconds for 10 frames. This demonstrates that computational overhead becomes minimal with optimized diffusion models.

Other advantages include:

1. No training is required (compared to methods needing 32 A100 GPUs)
2. Improved visual quality and temporal consistency
3. Flexibility to work with any diffusion model

We hope these clarifications help address your concerns about the metrics and efficiency trade-offs. Thank you again for your positive assessment of our work!

[Q3] Regarding performance variations across different models:

We provide a detailed comparison of improvements for both models in the following table, demonstrating that while DiffBIR with our method shows larger gains, the improvements on SD×4 are also significant. Our method consistently improves both models on almost all metrics, indicating its general applicability.

We sincerely thank Reviewer JKQ8 for their thorough review and insightful questions. Below, we address each point in detail:

[Q1] Comparisons between VidToMe and Upscale-A-Video:

VidToMe is primarily designed for video editing and heavily relies on video token merging. However, when applied directly to restoration tasks, it produces blurry results, as shown in Figs 6 and 7. Upscale-A-Video, a fine-tuned SDx4 upscaler model, requires 32 A100-80G GPUs for fine-tuning. Its ablation study indicates that the flow-based merging submodule contributes minimally to its performance compared to the impact of fine-tuning. In contrast, our method is specifically tailored for restoration tasks and requires no additional training. It achieves good results by utilizing hierarchical latent warping and hybrid flow-guided spatial-aware token merging.

[Q2] Regarding comparisons with VidToMe and objective metrics:

Important clarification: While VidToMe was originally implemented on Stable Diffusion, that version cannot perform restoration tasks. For a fair comparison, we reimplemented VidToMe using DiffBIR as the base model (same as one of our baselines), as mentioned in Sec. 4.1, L460-461. We apologize for any confusion and will emphasize this implementation detail in our revision. Using the same restoration model (DiffBIR), the performance differences stem from our distinct architectural choices:

1. As shown in Tab. 1, our method achieves clear improvements in image quality metrics compared to both VidToMe and per-frame DiffBIR. While VidToMe prioritizes pixel-level consistency, it often produces over-smoothed results (visible in Figs. 6 and 7), leading to degraded perceptual quality and higher LPIPS scores. In contrast, our approach better balances detail preservation with temporal coherence, achieving superior perceptual quality, as demonstrated in Fig. 10.
2. The key hyperparameter affecting this balance is the merging ratio r (Eq. 8). Our ablation studies show the following trade-offs:

These results demonstrate that our annealing strategy (gradually reducing r from 0.9 to 0) achieves a good balance between fidelity metrics (PSNR/SSIM) and perceptual quality (LPIPS), while fixed ratios tend to favor one over the other.

The different improvement margins can be attributed to both architectural and sampling differences:

1. Model architecture: DiffBIR employs a more sophisticated UNet architecture with additional cross-attention layers, providing more opportunities for our token merging strategy to enforce consistency.
2. Sampling strategy: SD×4 uses a DDIM sampler, which is inherently more stable due to its deterministic nature. In contrast, DiffBIR uses DDPM, which introduces additional stochastic noise at each denoising step. This initially makes DiffBIR's per-frame results less temporally consistent, creating more room for improvement when applying our method.
3. Base performance: While DiffBIR's per-frame results show higher quality, they suffer from more temporal inconsistency due to stochastic sampling. Our method effectively maintains high quality while significantly improving temporal consistency, leading to more noticeable gains.

[Q4] Regarding Upscale-A-Video comparison:

Thank you for bringing this to our attention. First, we should note that Upscale-A-Video only released their complete source code on September 27, 2024, very close to the ICLR submission deadline (October 1, 2024). We were not aware of this release and did not have sufficient time to conduct comparisons before submission.

Since then, we've encountered significant technical challenges with their released code:

1. Despite multiple attempts, we are unable to run their inference code on our available hardware (one A6000 GPU, 48GB memory) due to persistent out-of-memory (OOM) issues, even with their default configuration. This appears to be a common issue, as documented by multiple users in their GitHub repository's issues section.
2. Following the reviewer's suggestion to compare on their datasets, we have conducted experiments on the same test cases used in their paper. The qualitative comparisons are now provided in Figure 7 of our revised paper. The results demonstrate that our method produces more detailed outputs that better preserve the content of input frames. This advantage stems from our approach of leveraging pre-trained diffusion priors and zero-shot adaptation to video, compared to their fine-tuning strategy.

Our training-free approach also offers broader applicability - it can be directly adapted to various diffusion models without training, as demonstrated across super-resolution, denoising, and depth estimation tasks (using Marigold). In contrast, Upscale-A-Video requires extensive fine-tuning (32 A100 GPUs) for each specific task and cannot be easily extended to other applications.

Thank you for helping us improve the paper's clarity and completeness.

We appreciate your reviews and comments. We hope our responses address your concerns. Please let us know if you have further questions after reading our rebuttal.

We aim to address all the potential issues during the discussion period.

Thank you!

We thank Reviewer 12WH for their detailed feedback and thoughtful questions. We address each point below:

[Q1] Regarding optical flow dependency and robustness:

We agree that heavy reliance on optical flow is a limitation of our method, which we will explicitly discuss in our revision. However, we have implemented several safeguards specifically designed to maintain robustness when flow estimation becomes challenging:

1. Our forward-backward consistency check (Eq. (5)) effectively filters unreliable flow estimates, helping identify and exclude problematic regions where the flow might fail.
2. We only merge the r most confident token pairs, automatically excluding unreliable matches. This selective approach ensures that only high-confidence correspondences influence the result.
3. Beyond optical flow, our hybrid approach integrates spatial information and similarity matching, providing complementary guidance mechanisms. These additional cues become particularly crucial when flow estimation fails in challenging scenarios like textureless regions or severe degradations.
4. We strategically reduce flow dependency in upsampling blocks by relying more on similarity-based matching, which proves more robust to various degradation types.

While these strategies help mitigate flow-related issues in many cases, we acknowledge this remains an inherent limitation of our approach that deserves explicit discussion in the revised paper. Future work could explore ways to further reduce dependency on optical flow while maintaining temporal consistency.

[Q2] On novelty and technical contributions:

While we build upon existing concepts, our technical contributions go beyond simple combination:

1. We introduce the first zero-shot framework enabling any pre-trained image diffusion model to handle video restoration without training.
2. Our hierarchical approach is novel - operating at both global and local scales provides more comprehensive temporal consistency than previous frame-to-frame methods.
3. Our hybrid correspondence mechanism adapts dynamically through the denoising process (Fig. 5 demonstrates how flow and similarity focus on different regions).

[Q3] Specific technical questions: Choice of GMFlow:

GMFlow was selected for its strong performance on degraded inputs and efficient inference. Our method's modular design allows easy substitution with other flow estimators if needed.

[Q4] Specific technical questions: Equation 5 formulation:

It's not actually repeated; the norm includes two distinct terms: $f_{src->tar}(X(T_{src}))$ and $f_{tar->src}(X(T_{src}) + f_{src->tar}(X(T_{src})))$. The first term represents the flow from source to target tokens at the spatial location $X(T_{src})$. The second term computes the backward flow from target to source tokens at the warped spatial location $X(T_{src}) + f_{src->tar}(X(T_{src}))$. Together, these terms enable proper cycle consistency checking - if the forward and backward flows are consistent, their sum should approach zero. We will clarify this mathematical formulation and its purpose further in the revision.

[Q5] Specific technical questions: Additional datasets:

Thank you for suggesting these datasets. Regarding [2], while it is a valuable benchmark for image denoising, it focuses on single images rather than videos, making it less suitable for evaluating our video restoration framework. However, we agree that [1] is an excellent suggestion for a video face restoration benchmark. Due to time constraints, we have not yet completed inference on the entire dataset. However, we have provided qualitative comparisons in Figure 14 of our revised paper, which shows that our method generates face videos with richer details than FMA-Net and better temporal consistency than per-frame DiffBIR. We will include complete quantitative comparisons on this video face restoration dataset in the final version.

[Q6] Specific technical questions: Inference time:

Below, we report the inference time for processing 10 video frames at 854$\times$480 resolution on a single 4090 GPU:

Our method adds a reasonable overhead compared to per-frame inference (around 26s for SDx4 and 63s for DiffBIR) while maintaining strong temporal consistency. This is notably more efficient than training-based methods like Upscale-A-Video, which requires 32 A100 GPUs and encounters out-of-memory (OOM) issues even during inference on our test setup. Furthermore, when applied to lightweight models like Marigold with 4-step sampling, our method achieves very fast inference at just 10 seconds total.

Thank you for helping us improve the clarity and completeness of our paper.

References:

• [1] Chen, Ziyan, et al. "Towards Real-world Video Face Restoration: A New Benchmark." CVPRW. 2024.

• [2] Guo, Shi, et al. "Toward convolutional blind denoising of real photographs." CVPR. 2019.

We appreciate your reviews and comments. We hope our responses address your concerns. Please let us know if you have further questions after reading our rebuttal.

We aim to address all the potential issues during the discussion period.

Thank you!

We thank Reviewer NZxb for their careful review. We respectfully address each concern below:

[Q1] Regarding the architectural heuristics and theoretical guarantees:

We agree there is no theoretical guarantee that our method won't fail under certain conditions. In fact, we explicitly discuss these limitations in our main paper (L537-539) and provide failure case examples in Appendix Figure 17. However, our method is carefully designed with multiple safeguards and empirical validations:

• Our hierarchical latent warping operates at different temporal scales with principled mathematical formulations - Eq. (4) for consistency enforcement and Eq. (5) for confidence measurement to identify and handle uncertain correspondences.
• The hybrid token merging strategy is designed based on extensive empirical analysis, as demonstrated in Fig. 5. We found that early in denoising, when latents are noisy, optical flow provides more reliable structural guidance, while later stages benefit from similarity matching. This insight helps prevent failures by leveraging complementary strengths at different stages.

Additionally, we have designed several fallback mechanisms and conducted comprehensive ablation studies to validate our design choices. While theoretical guarantees remain an important direction for future work, our extensive experiments across diverse datasets and degradation types demonstrate the practical robustness of our approach.

[Q2] Regarding novelty:

While we build upon existing concepts, our contributions are distinct and significant:

• We are the first to enable zero-shot video restoration using pre-trained image diffusion models without any training or fine-tuning, making it immediately applicable to any image restoration model.
• Our hybrid approach combining flow and similarity-based matching is novel - previous methods rely on either flow (e.g., Rerender-A-Video) or similarity (e.g., VidToMe) alone, not both.
• The hierarchical warping strategy operating at both global and local scales is a new contribution specifically designed for video restoration.

[Q3] Regarding improvements:

Our method shows substantial improvements in challenging scenarios, with varying margins of improvement depending on the task:

• For 8$\times$ super-resolution on the DAVIS dataset, we achieve significant gains in both perceptual quality (LPIPS from $0.470$ to $0.434$) and temporal consistency ($E_\text{warp}$ reduced from $2.199$ to $1.759$) compared to per-frame inference. A large improvement margin in SR is expected as this task requires generating substantial high-frequency details.
• For high noise levels ($\sigma=150$) on Set8, while the improvements are more modest (LPIPS from $0.449$ to $0.402$, $E_\text{warp}$ from $1.207$ to $0.832$), this is expected because denoising inherently requires less detail generation compared to SR. The denoising task focuses more on noise removal and structure preservation rather than detail synthesis, naturally limiting the potential improvement margin from diffusion models.
• Most importantly, our method generalizes across different restoration tasks (SR, denoising, depth) and models without any task-specific training, which is a significant practical advantage. This versatility demonstrates the effectiveness of our approach even in tasks where the potential for improvement varies.

We will clarify these points and strengthen the theoretical analysis in our final version. Thank you for helping us improve the paper.

We appreciate your reviews and comments. We hope our responses address your concerns. Please let us know if you have further questions after reading our rebuttal.

We aim to address all the potential issues during the discussion period.

Thank you!