One-Step Diffusion for Detail-Rich and Temporally Consistent Video Super-Resolution

We sincerely thank this reviewer for the constructive comments. We hope our following point-to-point responses can address this reviewer's concerns. And we adopt VideoLQ4, as detailed in Appendix Section2, as our default test set for the reported experiments.

[W1]: Clarity of contributions.

[Response]: Thanks for the question, and we are sorry if we did not make our contributions clear enough in the main paper. Regarding the CSD loss [1], it draws on the idea of score distillation from pre-trained text-to-image diffusion models. It compares two predictions from the frozen base model: one conditioned on the text prompt and one unconditioned. The difference guides optimization toward outputs that better align with the textual semantics and exhibit higher visual quality. This loss is applied only in the enhancement stage, which focuses on detail enhancement; while in the consistency stage, we emphasize on temporal coherence, making CSD less relevant.

In addition to the design of loss functions, the novelty of our proposed two-stage framework lies in the use of distinct training data, the Cross-Frame Retrieval (CFR) module, the integration of dual LoRA parameter spaces, and the alternative training strategy. Specifically, the consistency stage is trained on real-world videos with complex motion, enabling the use of consistency loss and focusing on temporal coherence. While the enhancement stage uses high-quality image data with synthetic motion. It is guided by inherited consistency loss components to retain learned priors, and also by the CSD loss to enhance fine-grained details. This decoupling allows us to define two specialized and trainable parameter spaces, each leveraging the strengths of different data types and loss functions, which is rarely explored in prior works. By alternating the two phases during iterative optimization, the framework works collaboratively to achieve both temporal consistency and high-quality detail enhancement.

We acknowledge that the description of the CSD loss is not sufficiently detailed in the original manuscript, and we will add a clear explanation of its motivation and implementation in the revised version. We note that the contributions of dual stages can be reflected in the ablation study provided in our response to W2. We acknowledge that this key experiment is missing from the original manuscript, and we will include it in the revised version to improve clarity.

[W2]: Ablation of each stage.

[Response]: We thank the reviewer for the insightful suggestion regarding an ablation study on the two stages. We conducted a dedicated experiment with all models trained from scratch, comparing (i) consistency-stage only, (ii) enhancement-stage only, and (iii) dual-stage design (ours). Due to the policy of NeurIPS rebuttal, we are unable to include visual results here. We will incorporate these results in the revised manuscript to provide a more comprehensive evaluation.

These results confirm that the temporal consistency is primarily enhanced in the consistency stage, while fine spatial details are recovered in the enhancement stage; combining both yields the best overall performance. These findings substantiate the complementary contributions of each stage and justify the proposed dual‑stage design.

[W3] and [W4]: Suggestions of visual presentation.

[Response]: We thank the reviewer for these valuable presentation suggestions. We will (i) label our method explicitly as “ours” in relevant figures, and (ii) adjust the demo video for better visual presentation.

[Q1]: The reason of choosing T2I model.

[Response]: Thank you for the thoughtful question. We agree that recent video generation models (T2V) [2,3] have made remarkable progress and can offer stronger temporal consistency. However, we chose to train our model based on the T2I model primarily due to its superior image quality, which is important for VSR.

To support our choice, we conduct a direct comparison by training our VSR model with a T2I model (SD2.1) and a T2V model (Wan2.1-1.3B). Both VSR models are trained on the same video dataset with identical training settings (i.e., joint training with composite consistency and enhancement losses). And the results are shown below:

We see that the T2I model (SD2.1) achieves much higher image quality while maintaining comparable temporal consistency (measured by warping error). This result may stem from the following two factors:

1. Training data: The per-frame quality of video data used to train T2V models is generally lower than that of images used to train T2I models so that T2V models are harder to generate fine-grained visual details.

2. Model architecture: T2V models typically adopt video VAEs that compress both spatial and temporal dimensions. This leads to a heavier compression compared to the VAE used in T2I models. causing more information loss.

While T2V models can offer better temporal coherence, VSR tasks require both high visual quality and temporal consistency. In our experiments, T2I-based models provide a better trade-off between the two goals. STAR [4] in Table 1 of the main paper is based on a T2V model (i.e., CogVideo [5]), which also reflects this quality gap.

Reference:

[1] L. Sun, R. Wu, Z. Ma, S. Liu, Q. Yi, L. Zhang, "Pixel-level and Semantic-level Adjustable Super-resolution: A Dual-LoRA Approach," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.

[2] W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang, "HunyuanVideo: A Systematic Framework for Large Video Generative Models," arXiv preprint arXiv:2412.03603, 2024.

[3] T. Wan, A. Wang, B. Ai, B. Wen, C. Mao, C.-W. Xie, D. Chen, et al., "Wan: Open and advanced large-scale video generative models," arXiv preprint arXiv:2503.20314, 2025.

[4] R. Xie, Y. Liu, P. Zhou, C. Zhao, J. Zhou, K. Zhang, Z. Zhang, J. Yang, Z. Yang, Y. Tai, "STAR: Spatial-temporal augmentation with text-to-video models for real-world video super-resolution," Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025.

[5] W. Hong, M. Ding, W. Zheng, X. Liu, J. Tang, "CogVideo: Large-scale pretraining for text-to-video generation via transformers," Proceedings of the International Conference on Learning Representations (ICLR), 2023.

Dear Reviewer 1Rdb,

Many thanks for your time in reviewing our paper and your constructive comments. We have submitted the point-to-point responses. We appreciate if you could let us know whether your concerns have been addressed, and we are happy to answer any further questions.

Best regards,

Authors of paper #8760

We sincerely thank this reviewer for the constructive comments. We hope our following point-to-point responses can address this reviewer's concerns. And we adopt VideoLQ4, as detailed in Appendix Section2, as our default test set for the reported experiments.

[[W1] and [Q1]]: The reason of choosing T2I model.

[Response]: We agree that video generation models (T2V) [1,2] have made remarkable progress and can offer stronger temporal consistency. However, we employ the text-to-image (T2I) model primarily due to its superior image quality in the context of Video Super-Resolution (VSR).

To support our choice, we conduct a direct comparison by training our VSR model using the SD 2.1 T2I model and the Wan2.1-1.3B T2V model on the same VSR training data with identical training settings (i.e., joint training with composite consistency and enhancement losses). The results are shown below:

We see that SD2.1 achieves much higher image quality while maintaining comparable temporal consistency (measured by warping error). This result may stem from the following two factors:

1. Training data: The per-frame quality of video data used to train T2V models is generally lower than that of images used to train T2I models so that T2V models are harder to generate fine-grained visual details.

2. Model architecture: T2V models typically adopt video VAEs that compress both spatial and temporal dimensions. This leads to a heavier compression compared to the VAE used in T2I models. causing more information loss.

While T2V models can offer better temporal coherence, VSR tasks require both high visual quality and temporal consistency. In our experiments, T2I-based models provide a better trade-off between the two goals. STAR [3] in Table 1 of the main paper is based on a T2V model (e.g., CogVideo [4]), which also reflects this quality gap.

[W2] and [Q2]: Adaption to DiT.

[Response]: To answer this concern, we implement our approach on a DiT-based T2I model (i.e., SD 3) and train the VSR model with and without our proposed strategies. The experimental results are shown below.

We see that our method can be successfully integrated into the DiT architecture and consistently improves both perceptual quality and temporal consistency. This demonstrates the generality of our approach beyond UNet-based models (i.e., SD 2.1).

[W3] and [Q3]: Decoupling of dual LoRA.

[Response]: Thanks for the question. We would like to clarify that the term decoupling in our method refers to functional specialization, rather than entirely separate training processes.

Please kindly note that D-LoRA is trained on high-quality images from the LSDIR dataset, not on real-world video data as used by C-LoRA. It enables D-LoRA to focus on enhancing perceptual details. The temporal loss inherited from C-LoRA serves to retain the learned temporal priors and prevent forgetting, rather than to model motion.

We further validate this design with two experiments. First, as shown in the table below, adding D-LoRA to a fixed C-LoRA during inference significantly improves visual quality (higher MUSIQ), while maintaining comparable temporal coherence (similar warping error). This demonstrates that D-LoRA effectively enhances perceptual quality without interfering with C-LoRA in maintaining temporal consistency.

Second, the table below shows that removing synthetic motion and the consistency loss ($L_{opt}$) from D-LoRA training results in worse temporal consistency (i.e., higher warping error), even though the visual quality (e.g., MUSIQ) remains comparable.

Therefore, though C-LoRA and D-LoRA share some common training elements, their functional roles are decoupled, where C-LoRA maintains the temporal consistency and D-LoRA enhances the details.

[W4] and [Q4]: Pseudo-video training data for enhancement stage.

[Response]: While existing video datasets [5, 6, 7, 8] offer diverse motion patterns, they often suffer from limited visual quality compared to image datasets. To prioritize perceptual quality in the enhancement stage, we use high-quality image data with synthetic motion as the training data. This design allows D-LoRA to focus on visual detail enhancement without being hindered by the lower quality of real video frames.

However, this does not mean we ignore realistic motion. The consistency stage is trained on real videos with complex, natural motions, ensuring strong temporal priors. This decoupled design allows us to leverage the strengths of both data sources — real motion from videos and fine details from images — achieving a better overall performance of temporal coherence and perceptual quality.

To assess the impact of this choice, we retrain the enhancement stage on the Pexel video dataset [5], one of the highest-quality public video datasets. As shown in the table below, training on real video datasets leads to reduced image quality (e.g., lower MUSIQ) while maintaining comparable temporal consistency (i.e., similar warping error). This confirms that pseudo-video training offers better perceptual quality without sacrificing consistency, and is thus more suitable for the enhancement stage.

[W5] and [Q5]: Ablation of each module.

[Response]: Thank for the nice suggestion. In response, we add a new ablation study to analyze the contribution of each of our proposed modules: (i) C-LoRA, (ii) D-LoRA, and (iii) CFR. We compare the full model (denoted as ours) with three variants, each removing one module while keeping the others unchanged.

As shown in the table above, removing either CFR or C-LoRA leads to weaker temporal consistency (i.e., higher warping error), indicating their complementary roles in maintaining temporal coherence. In contrast, removing D-LoRA significantly impairs all perceptual metrics, confirming its core contribution to fine-grained detail enhancement.

[W6] and [Q6]: Adaption to multi-step diffusion models.

[Response]: We thank the reviewer for the question. If we understand correctly, the concern is whether our faster inference speed mainly comes from using a single-step diffusion model. This is correct — our method is built on a one-step framework, which naturally runs faster than its multi-step counterparts. While this design provides clear efficiency benefits, our main contribution lies in enabling detail-rich and temporally consistent video super-resolution within this lightweight framework.

To ensure a fair comparison, we also evaluate against other single-step baselines. As shown in Tables 1 and 2 of the main paper, our method achieves comparable inference speed to OSEDiff, a state-of-the-art single-step model, while delivering better perception quality and temporal consistency.

Reference:

[1] Kong et al. HunyuanVideo: Large Video Generative Models. arXiv, 2024.

[2] Wan et al. WAN: Large-scale Video Generation. arXiv, 2025.

[3] Xie et al. STAR: Text-to-Video Models for VSR. ICCV, 2025.

[4] Hong et al. CogVideo: T2V Generation via Transformers. ICLR, 2023.

[5] Pexels. https://www.pexels.com/

[6] Nan et al. OpenVid-1M: High-Quality T2V Dataset. ICLR, 2025.

[7] Bain et al. Frozen in Time: Joint Video-Image Encoder. ICCV, 2021.

[8] Zhou et al. Upscale-A-Video: Diffusion for Real-World VSR. CVPR, 2024.

Dear Reviewer Bv1v,

Many thanks for your time in reviewing our paper and your constructive comments. We have submitted the point-to-point responses. We appreciate if you could let us know whether your concerns have been addressed, and we are happy to answer any further questions.

Best regards,

Authors of paper #8760

Thank the authors for the rebuttal and additional experiments, which have addressed my concerns. If the authors commit to adding all the experimental results in the rebuttal to the revised version, I will raise the score to borderline accept.

We sincerely thank this reviewer for the constructive comments and suggestion. We hope our following point-to-point responses can address this reviewer's concerns. And we adopt VideoLQ4, as detailed in Appendix Section2, as our default test set for the reported experiments.

[W1] and [Q1]: Clarification about the training complexity.

[Response]: We appreciate the reviewer's comment. While our method involves two training stages and two datasets, we would like to clarify that each stage is purposefully paired with a single task-specific dataset and the dedicated losses, while the two stages are trained independently in an alternating manner. This design is actually not that complex, but rather a deliberate choice to decouple the optimization of temporal consistency and spatial detail via a dual-LoRA framework.

Specifically, the C-LoRA is trained exclusively on degraded real-world videos to learn robust temporal priors. The D-LoRA is trained only on high-quality images (with synthetic motion) to enhance spatial detail. This decoupled training results in a stable training process and improved final performance than the joint-training approach with a mixed dataset and a combined loss, as shown in our experiments in Appendix Section2: Ablation studies on training strategies in DLoRAL. Observed from the Figure 1 of the Appendix, while a joint-training scheme with a carefully balanced loss may appear simpler, it will result in conflicts between competing objectives, leading to slower convergence and weaker overall performance. In contrast, our dual-space, dual-stage design separates the learning of detail and consistency, enabling higher visual quality and more stable temporal alignment.

[W2] and [Q2]: Concerns on CFR.

[Response]: We sincerely thank the reviewer for raising this important concern. For challenging scenes where the $t{-}1$ frame provides limited useful information, the absolute attention mechanism in our CFR module selectively integrates relevant features while effectively filtering out noise from the previous frame. This design ensures that the reconstruction of the current frame remains robust without being adversely affected by irrelevant information.

Regarding the temporal window size, we evaluate the use of a larger temporal window in the CFR module. And the corresponding results are shown below.

From the table, we observe that a temporal window size of 1 achieves a good balance between temporal consistency and visual quality. When the temporal window is increased to 5, there is a slight drop in the warping error, indicating better temporal consistency. However, this comes at the cost of noticeably reduced visual quality, as reflected by the decrease in metrics such as MUSIQ and CLIP-IQA. On the other hand, when the temporal window is set to 0 (i.e., without CFR), the warping error is significantly worse, highlighting the importance of cross-frame interactions introduced by the CFR module.

Furthermore, larger temporal window corresponds to higher computational cost. This is due to the increased number of cross-frame interactions (i.e., cross-attention operation), which leads to quadratic growth in memory and computation requirements.

In conclusion, while a larger temporal window can slightly improve temporal consistency, the trade-offs in visual quality and computational efficiency make it less attractive. Therefore, we adopt a temporal window of 1 in our final design.

[W3] and [Q3]: Training data of enhancement stage.

[Response]: We appreciate the reviewer's thoughtful attention to the design of our training data for detail enhancement. Simulated motion is used to prevent the model from forgetting previously learned temporal consistency. To further validate its effectiveness, we retrain the model by adopting high-quality still images and removing the optical flow loss in the second stage. The results are shown in the table below.

The two models achieve comparable scores on visual metrics, indicating similar perceptual quality. However, the model trained without simulated motion exhibits a noticeably higher warping error, suggesting degraded temporal consistency. This gap confirms that—even though the synthetic motion is simplistic—it plays an important role in preventing the model from forgetting previously learned temporal consistency during detail enhancement.

Dear Reviewer mZuh,

Many thanks for your time in reviewing our paper and your constructive comments. We have submitted the point-to-point responses. We appreciate if you could let us know whether your concerns have been addressed, and we are happy to answer any further questions.

Best regards,

Authors of paper #8760

Thank you for the careful rebuttal and new ablations. The added comparisons convincingly show that (i) the dual-stage, dual-dataset schedule outperforms a joint-training baseline, (ii) a temporal window = 1 in the CFR module offers the best trade-off between perceptual quality and temporal coherence without excessive memory/time cost, and (iii) the lightweight synthetic-motion curriculum is crucial for preserving temporal consistency during detail enhancement. These results directly address my earlier concerns about training complexity, window size, and data realism.

Some open questions remain—e.g., performance on extremely rapid-motion scenes and practical guidance for hyper-parameter tuning—but the paper now provides solid evidence of a state-of-the-art VSR method that combines high visual quality with single-step inference speed. I therefore raise my score to borderline accept.

We sincerely thank this reviewer for the constructive comments and suggestion. We hope our following point-to-point responses can address this reviewer's concerns. And we adopt VideoLQ4, as detailed in Appendix Section2, as our default test set for the reported experiments.

[W1 and W2]: Limitations of VAE.

[Response]: We sincerely thank the reviewer for pointing out the limitation of using the pre-trained VAE from the Stable Diffusion (SD) model. This is indeed a well-known issue, which we have also explicitly discussed and identified as our limitation in our manuscript (Page 9, Lines 339–343). The aggressive 8× downsampling in the VAE can compromise the reconstruction of fine structures (e.g., small text and textures). Additionally, a heavily compressed VAE may struggle to extract accurate temporal priors. Therefore, improving the VAE is an important research direction, which we plan to explore in future work.

In this paper, however, we aim to strike a balance between leveraging the natural image prior of SD model and achieving temporal consistency for video super-resolution (VSR) task; thus, the original VAE is kept for compatibility with the pre-trained UNet. To jointly maintain visual quality and temporal consistency within this framework, we propose Cross-Frame Retrieval and Dual-LoRA Learning. As shown in Table 1 and Figure 4 of the main paper, our method achieves strong results across both consistency and quality metrics, demonstrating its ability to maintain temporal coherence while preserving high visual quality under the heavily compressed VAE.

Looking forward, recent works, such as TVT [1], have attempted to employ a VAE with a lower downsampling rate in SD-based models. These approaches significantly mitigate the challenges of reconstructing fine structures and alleviating temporal coherence disruption caused by severe compression, while retaining the generative capability of pre-trained diffusion models. We find this direction highly promising and plan to explore VAE designs specifically tailored for Real-VSR, such as TVT-like strategies, in our future work. This line of research is complementary to our current work and may further enhance the applicability of diffusion-based VSR models in real-world scenarios.

[W3]: Generalization capability.

[Response]: We sincerely thank the reviewer for raising this important concern about the generalization ability of our method. Our approach builds upon the pre-trained SD model to leverage its strong natural image generative priors, which have been validated to have high generative capacity in prior SR tasks [2,3,4].

We fully acknowledge that the SD model was trained on large-scale image datasets, which may not cover many real-world video degradation types. This potential distribution mismatch could indeed affect generalization in unseen scenarios. To address this limitation and enhance the robustness of our model, instead of using SD’s original training data, we construct dedicated datasets for Real-VSR training. These datasets simulate diverse degradation types and scene content, aiming to better align the Real-VSR needs. To further evaluate the generalization performance of our method, we conduct the following experiments:

1. Cross-dataset evaluation:
   We extensively test DLoRAL on datasets with varying degradation characteristics, including synthetic benchmarks (UDM10, SPMCS) and real-world datasets (RealVSR, VideoLQ). RealVSR contains mobile-captured videos, while VideoLQ features authentically degraded Internet content. As shown in Table 1 and Figure 3, our method consistently achieves strong performance across all datasets, demonstrating robustness under a wide range of degradation patterns.

2. User study:
   We also perform user studies (Figure 3 of the Appendix) to assess the perceptual quality in challenging conditions, such as low light, fast motion, and compression artifacts. The results indicate that DLoRAL maintains good perceptual quality and generalizes well in these difficult scenarios.

Nonetheless, we agree that there can still be failure cases for degradations outside of our training data.

[Q1]: More ablation studies of each LoRA.

[Response]: We thank the reviewer for the insightful suggestion regarding an ablation study on the two LoRA branches. We conduct a dedicated experiment with all models trained from scratch, comparing (i) C-LoRA only, (ii) D-LoRA only, and (iii) the full C+D configuration. The results are presented in the table below.

The above results confirm that C-LoRA primarily enhances temporal consistency, while D-LoRA recovers fine spatial details; combining both yields the best overall performance. These findings substantiate the complementary contributions of each LoRA and justify the proposed dual‑stage design.

[Q2]: Application to other networks.

[Response]: If we correctly understand the reviewer's question, it concerns whether our proposed design can be applied to other diffusion networks, such as DiT-based models (e.g., Stable Diffusion 3). To verify this, we conduct additional experiments using Stable Diffusion 3 as the pre-trained model, one model trained with our proposed strategies (i.e., CFR and Dual-LoRA learning) and the other model trained without the strategies.

The results show that our method can be successfully integrated into other diffusion models and leads to improved VSR results.

Reference:

[1] Q. Yi, S. Li, R. Wu, L. Sun, Y. Wu, L. Zhang, "Fine-structure Preserved Real-world Image Super-resolution via Transfer VAE Training," Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025.

[2] J. Wang, Z. Yue, S. Zhou, K. C. Chan, C. C. Loy, "Exploiting Diffusion Prior for Real-world Image Super-resolution," International Journal of Computer Vision (IJCV), 2024.

[3] X. Lin, J. He, Z. Chen, Z. Lyu, B. Dai, F. Yu, Y. Qiao, W. Ouyang, C. Dong, "DiffBIR: Toward Blind Image Restoration with Generative Diffusion Prior," Proceedings of the European Conference on Computer Vision (ECCV), 2024.

[4] L. Sun, R. Wu, Z. Ma, S. Liu, Q. Yi, L. Zhang, "Pixel-level and Semantic-level Adjustable Super-resolution: A Dual-LoRA Approach," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025.

Dear Reviewer dmAC,

Many thanks for your time in reviewing our paper and your constructive comments. We have submitted the point-to-point responses. We appreciate if you could let us know whether your concerns have been addressed, and we are happy to answer any further questions.

Best regards,

Authors of paper #8760

Dear AC,

Many thanks for your kind reminder. We are eager to hear from the reviewers for their feedback and further questions to our responses.

Best regards,

Authors of paper #8760