AverNet: All-in-one Video Restoration for Time-varying Unknown Degradations

Q1：Evaluation on degradations with variable intervals.

As suggested, we conduct new experiments by synthesizing new test sets with variable degradation intervals. The results show that our method effectively handles degradations with variable intervals. Specifically, the test sets are synthesized based on DAVIS-test and the intervals are randomly sampled from [t-v, t+v].

Table 1. Quantitative results on the test sets with variable intervals.

Q2: The performance on single type of degradation.

As suggested, we synthesize new test sets, each containing only a single type of degradation, and evaluated the models on these sets. The results, as shown in Table 2, demonstrate that AverNet consistently outperforms RVRT across all types of degradation.

Table 2. Quantitative comparisons on single type of degradation.

Q3: The differences between the proposed AverNet and ViWS-Net.

The differences between AverNet and ViWS-Net [1] are discussed below and will be included in the related works. First, AverNet aims to handle time-varying unknown degradations, whereas ViWS-Net specifically focuses on a single type of degradation within one video. Additionally, AverNet employs prompt-guided modules to conditionally restore videos, while ViWS-Net explicitly optimizes a weather discriminator to classify the degradations and guide the restoration.

[1] Video Adverse-Weather-Component Suppression Network via Weather Messenger and Adversarial Backpropagation.

Dear Reviewer 2PV4,

Thank you for your positive feedback and for raising your score. We appreciate your approval of our revisions and are glad that the additional experiments and discussions have addressed your concerns. Your suggestions throughout the review process have been invaluable.

Q1&Q2：Increase the severity of degradations over time.

As suggested, we conduct new experiments by synthesizing four test sets with progressively worsening degradations and the results prove the effectiveness of our method. Specifically, in the first test set, different types of degradations are gradually introduced, with their intensities increasing over time. In the other three test sets, only one type of degradation is added, with its intensity increasing over time. Since the models have seen various degradations and their variations during training, we directly apply them to these test sets without retraining. The results are presented in the table below. From the tables, one could observe that our AverNet is effective in handling various degradation changes and outperforms RVRT in the settings where degradations worsen over time.

Table 1. Quantitative results on increasing degradation severity over time. Multiple degradations denote noise, blur, and compression are gradually added and their severity worsen with time. Noise, Blur, and Compression denote single type of degradation that worsen with time.

Q3：Evaluations on realistic degraded videos.

As we do not find the NoisyCity4 dataset, we can only conduct evaluations on the realistic video dataset VideoLQ [1]. The results in Table 2 show that our AverNet is more effective in dealing with realistic and complex degradations.

Table 2. Quantitative results on realistic video dataset VideoLQ.

[1] Investigating Tradeoffs in Real-World Video Super-Resolution.

Q4：Parameters for PromptIR and AirNet.

To comprehensively compare the models, we use the PyTorch model profiling API THOP to calculate the parameters. THOP calculates parameter counts using hooks on modules, which may result in lower values than those officially reported. We recalculate and update the results, i.e, PromptIR has 35.60M parameters, and AirNet has 8.93M parameters.

Q5：Clarification on our degradation pipeline and that of BSRGAN and Real-ESRGAN. The performance under pipelines of BSRGAN and Real-ESRGAN.

We argue that the degradation pipelines used in BSRGAN [1] or Real-ESRGAN [2] are not necessarily more complex than ours. In fact, our method has a comparable level of complexity with them. To provide a clearer comparison, we have summarized the key components of our pipeline alongside those of BSRGAN and Real-ESRGAN in Table 3 below. Additionally, our pipeline is designed for all-in-one video restoration that addresses time-varying unknown degradations in videos, whereas these pipelines are developed for blind image super-resolution.

Table 3. Comparison of degradation pipelines.

As suggested, we synthesize new test sets based on the pipelines of BSRGAN and Real-ESRGAN to evaluate the performance of our models. The results, presented in Table 4, show that our AverNet achieves comparable or even superior performance on both test sets synthesized through BSRGAN and Real-ESRGAN. Note that downsampling operation was removed to maintain the frame size.

Table 4. Quantitative comparisons on the test sets synthesized through BSRGAN and Real-ESRGAN.

[1] Designing a Practical Degradation Model for Deep Blind Image Super-Resolution.

[2] Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data.

Q6：Clarification on the degradation pipeline.

Actually, our pipeline adds multiple degradations simultaneously to the same frame snippets. Specifically, in our pipeline, each type of degradation has a 0.55 probability of being sampled and applied to per snippet. In other words, each frame snippet in our test sets usually involves multiple degradations. Consequently, our experiments indeed evaluate the models on test sets containing multiple degradations per snippet, rather than just a single degradation per snippet.

Dear Reviewer Zzy9,

We appreciate your approval of the TUD problem raised in the paper and would like to address your concerns as follows.

As you noted, addressing the TUD problem is a crucial next step in the filed of video restoration. To study this problem, we developed the pipeline to simulate the data of TUD. The previous pipelines of BSRGAN and Real-ESRGAN are not applicable as their goal is to generate images with mixed degradations for blind image super-resolution. In contrast, our pipeline aims to synthesize videos with time-varying degradations, which is well-aligned to our research purpose, i.e., all-in-one video restoration for TUD problem. Experimental results demonstrate that our method can effectively address the TUD problem compared with the baselines.

We hope these clarifications could address your concerns. Thank you once again for your feedback and for helping us improve our work.

Q1：Effectiveness of prompts in longer video Set8 with complex degradation changes.

Actually, our prompt-based AverNet is more effective in dealing with complex degradation changes in longer video Set8. To highlight how well the prompts can adapt to changing degradations, we synthesize new test sets with increasing degradation severity based on Set8. The results are shown in Table 1, from which one could observe that the prompts endow AverNet with a greater capacity to handle degradation changes in long videos. In detail, we synthesize four test sets where the degradations progressively worsen over time. In the first test set, different types of degradations are gradually introduced, with their intensities increasing over time. In the other three test sets, only single type of degradation is added, with the intensity increasing over time.

Table 1. Quantitative results on Set8 with increasing degradation severity over time. Multiple degradations denote noise, blur, and compression are gradually added and their severity worsen with time. Noise, Blur, and Compression denote single type of degradation which worsen with time.

Q2：Effectiveness on weather-induced degradations.

As suggested, we synthesize a new dataset with time-varying weather degradations based on the video dataset REVIDE [1]. The results in Table 2 demonstrate that our prompt-based AverNet effectively handles weather-induced degradations. Specifically, we introduce three types of weather degradations (i.e., haze, snow and rain) through our data pipeline and the synthesis approaches similar to [2,3]. Due to time limitation, we train models on the dataset with 200k iterations, and use a fast baseline BasicVSR++ for comparison. RVRT is not compared since its training is too time-comsuming to finish in the rebuttal phase.

Table 2. Quantitative results on weather-induced degradations.

[1] Learning to Restore Hazy Video: A New Real-World Dataset and A New Method.

[2] Blind Image Decomposition.

[3] Relationship Quantification of Image Degradations.

Q3：Comparisons with all-in-one video restoration methods.

As suggested, we compare our method with ViWSNet [1] and present the results in Table 3. The results show that ViWSNet struggles with handling time-varying degradations and produces unsatisfying results. We speculate that this is because ViWSNet imposes a strong assumption through loss function, i.e., only single type of degradation exists in single video.

Note that Diff-TTA [2] is not compared since its code is not available, and we were unable to reproduce it during the rebuttal phase.

Table 3. Quantitative results of ViWSNet on DAVIS-test and Set8.

[1] Video Adverse-Weather-Component Suppression Network via Weather Messenger and Adversarial Backpropagation.

[2] Genuine Knowledge from Practice: Diffusion Test-Time Adaptation for Video Adverse Weather Removal.

Q4：Clarifications on variation intensity t and key frame interval T.

The confusions may arise from the misunderstanding betweeen the variation intensity 't' in the data pipeline and the keyframe interval 'T' in the PCE module. Specifically, the lowercase ‘t’ controls the interval of degradation changes in the data pipeline. A smaller t corresponds to a higher variation intensity of degradations. For instance, t=6 indicates degradation changes every six frames. Additionally, the uppercase 'T' is the hyperparameter in the PCE module that controls the interval of keyframes. For example, T=12 indicates PCE module select one keyframe every twelve frames.

Q5：Non-prompt ablation studies.

We carry out non-prompt ablation study as suggested. As shown in Table 4, non-prompt baseline shows a significant drop in both the PSNR and SSIM metrics, highlighting the effectiveness of the two prompt-based modules.

Table 4. Ablation studies on the proposed prompt-based modules.

Q6：Experiments on degradations worsen with time.

As suggested, we carry out experiments under the above setting. The results are presented in Table 5, from which one could observe that our prompt-based AverNet significantly outperforms RVRT, which demonstrates the prompts effectively adapt to the changing degradations.

Table 5. Quantitative results on test sets with increasing degradation severity over time. Multiple degradations denote noise, blur, and compression are gradually added and their severity worsen with time. Noise, Blur, and Compression denote single type of degradation which worsen with time.

I thank the authors for the thorough rebuttal. I have gone through the other reviewers' comments, and authors' rebuttal. My comments are addressed, and therefore I am raising my score to borderline accept. I think the work is important, but exploring more complex degradations (such as rain/haze/snow) in more depth in the TUD setting would have been more interesting.

Q1: Evaluations on the realistic video dataset.

We further evaluate the effectiveness of our pipeline and network on realistic video dataset VideoLQ [1]. The results show that the models trained on our pipeline generalizes well on the realistic degradations. As shown in Table 1, our network and RVRT are not re-trained on the realistic video datasets such as RealVSR [2] but still shows great performance on VideoLQ.

Table 1. Quantitative results on the realistic video dataset VideoLQ.

[1] Investigating Tradeoffs in Real-World Video Super-Resolution.

[2] Real-world video super-resolution: A benchmark dataset and a decomposition based learning scheme.

Q2：Other degradation intervals.

As suggested, we conduct experiments on different intervals to show the generalization of AverNet. From Table 2, one could observe that AverNet shows better performance on two additional intervals.

Table 2. Quantitative results on different intervals t=9 and t=15.

Q3：Adaptive keyframe selection.

Following [1,2], we adopt an adaptive strategy to select the keyframes, which chooses the frames with largest changes as keyframes. The results are presented in Table 3, which show that the adaptive strategy only brings slight improvement in SSIM. However, the computational burden of the adaptive strategy is not negligible in practice.

Table 3. Quantitative comparisons between fixed and adaptive keyframe strategy.

[1] Low-Latency Video Semantic Segmentation.

[2] Dynamic Video Segmentation Network.

Q4：More related works [1,2] should be discussed.

As suggested, we will discuss the all-in-one image restoration methods [1,2] in the following and include them in the related works. MPerveiver [1] proposes a multimodal prompt learning approach to exploit the generative priors of Stable Diffusion to achieve high-fidelity all-in-one image restoration. ProRes [2] introduces additional visual prompts to incorporate task-specific information and utilize the prompts to guide the network for all-in-one restoration.

[1] ProRes: Exploring Degradation-aware Visual Prompt for Universal Image Restoration.

[2] Multimodal Prompt Perceiver: Empower Adaptiveness, Generalizability and Fidelity for All-in-One Image Restoration.

Q5：Discussion on limitations.

The training data for AverNet is based on the video synthesis approach, which generates videos with time-varying unknown degradations close to real-world scenarios. However, the corruptions in the real-world videos are complex and difficult to be simulated. Therefore, in real-world applications, our AverNet may need further validation and improvement.

Dear Reviewer hhCd, I have noticed that you have not yet responded to the authors' rebuttal. I kindly urge you to engage in a discussion with the authors at your earliest convenience to help advance the review process.