ReBotNet: Fast Real-time Video Enhancement

We thank the reviewer for their comments. We are encouraged by their positive comments on novelty, uniqueness, and validating our practical compute-efficient solution. In what follows, we address all the concerns:

1) Comparison with other datasets:

We have now added results on two more public datasets where our performance is similar to that of RVRT while being much faster and obtaining real-time settings. It can be seen in the table below:

We would also like to point out that our curated datasets are comparable in size to the existing datasets while dealing with more challenging cases due to multiple degradations operating on the same video. Also, it emulates live video call and live streaming scenarios unlike existing datasets.

2) Demonstration Videos:

We provide some demonstration videos of comparison with previous methods as well as direct slider comparison with real videos in this link. Please look at the left hand side to access all the files from the window that pops up from this link. We are worried right now that releasing a lot of demonstration videos might have a problem with protecting our anonymity. We promise to release a link to more demonstration videos after the reviews.

3) Optical Flow and Computation:

We apologize to the reviewer for that sentence and thank the reviewer for pointing out those papers. We have removed them in our revised version but highlight that our contributions or work does not have any effect from this as our method takes a completely different approach and provides a working solution for the problem of video enhancement.

4) Enhancement in Dynamic Scenes:

While we understand the reviewer’s concerns on static scenes, most of the videos in our FullVideo dataset PortraitVideo in fact have dynamic scenes. This is reflected in the demo videos we have now linked (in point 2). In the paper, while we mention “video actually contains the torso/face of the person”, we do not say that the background is always static. The background can be dynamic and we have included those cases in our datasets as well.

5) Fair Comparison with existing methods:

During our experimental setup, we made sure to look into the code-bases of previous methods like RVRT, VRT, and BasicVSR++ and found that they also train the network across 8 GPUs for their experiments making our comparison with those methods fair.

After reading the rebuttal and other reviews, I generally support the acceptance of this paper and raised my rating. I hope the author can continue to push the results of the paper into applications.

Because of the real-time goal, similar papers often cannot adopt designs that look fancy but have poor hardware support. This limits the author to make the paper "look novel". After reading the paper carefully, I feel that the overall contribution of this paper is worthy of recognition. The extreme efforts towards computational efficiency make this paper different from many previous works.

We thank the reviewer for the comments and address their concerns below:

1) Contributions:

We highlight our important contributions here:

• proposing the first video enhancement network that can work in real-time settings (more than 30 FPS) and with a good performance.
• curating two new datasets that resemble actual real-world settings like live video calls and video streams.
• developing an architecture with ConvNext feature extractors and bottleneck Mixers and showing that this configuration has the best latency-performance tradeoff for video enhancement networks.
• ensuring temporal consistency by integrating with other techniques like recurrent training and temporal bottleneck branch.

To address the particular point of adding more innovation to this work, we would like to point out that our goal for this work was to obtain a strong video enhancement that works in real-time settings rather than solely introducing novelty. Previous works like RVRT provided more than satisfactory performance for our video enhancement but had an issue with latency which was our main motivation as we targetted live video call enhancement. With ReBotNet, we solve this issue and provide the community with a low-compute model that can also perform well; and our goal was accomplished. With that being said, our work is still novel in its architecture design our architecture is completely different from any of the works in the literature for video enhancement.

2) Integrating Tubelet and Image Tokens:

We did explore alternate techniques for the feature fusion, but found out that a simple addition was the most compute effective option. The performance improvements we obtained from more complicated processes were not significant enough for us to consider them as the fusion strategy. We compare the other alternatives we considered in the following table on PortraitVideo dataset:

It can be seen that although using attention to fuse the features did provide some performance improvement, the model was suboptimal in terms of latency, which is our main focus for this work.

3) Difference with previous datasets:

We tabulate the difference of our curated datasets with existing datasets in the following:

We will release the dataset by releasing a script to curate and generate the videos from the web similar to TalkingHead dataset.

4) Everyday life scenarios:

To make the model work on everyday things like mobile phone videos, we need to scale the data that we train the model on. We consider this our future work. Our aim would be to ensure that the model's training data comprehensively encompasses diverse and representative samples from everyday scenarios, enabling its effective application to diverse content. Currently, our contribution is more towards the network architecture and not towards an generalizable model. In fact, this is also an active field of research in video restoration.

5) Table 3:

We thank the reviewer for pointing out this error and we have corrected it in the revised version.

We thank the reviewer for the comments. We are glad that the reviewer pointed out several useful aspects of our work. In what follows, we clear all the 4 points the reviewer had in the weakness section:

1) Novelty of our work:

We agree with the reviewer that there are other works presenting dual-branch networks but we would like to highlight that we never pointed out our key novelty to be the dual-branch part. Our novelty lies in:

• proposing the first video enhancement network that can work in real time settings with a good performance. Please note that all the previous networks proposed do not focus on this aspect at all and so have computationally complex solutions focussed only on performance making them unsuitable for real-time applications.

• developing an architecture with ConvNext feature extractors and bottleneck Mixers and showing that this configuration has the best latency-performance tradeoff for video enhancement.

• curating two new datasets that resemble actual real-world settings like live video calls and video streams and also perform user studies to prove the effectiveness of our method.

2) Comprehensive experiments leading to our design choice:

We experimented with ConvNets, ConvNexts, Transformers, and Mixers individually before finalizing our design choice for ReBotNet. These experiments yielded analytical insights crucial in shaping our decision. We started our experiments with a setup altering the encoder design while maintaining a fixed decoder with simple transposed convolution layers. Our objective was to match the performance of these models, aiming to gauge the computational requirements needed to achieve equivalent performance among these different encoders. The performance in terms of PSNR on the Portrait Video dataset and the latency is reported in the table below:

We can observe that while Transformers take more inference time and complexity to match ConvNet’s performance while both ConvNext and Mixers are more efficient than their predecessors.

After considering these insights, our focus shifted towards leveraging the strengths of both ConvNext and Mixer. Combining a convolutional encoder for initial feature extraction with a mixer network in the bottleneck has demonstrated effectiveness. Convolutional layers excel in efficient low-level feature extraction, while mixers, following transformers, showcase strong representative abilities in extracting deep features, as also highlighted in Srinivas et al. (2021). We performed experiments where we slowly introduce Mixer in the bottleneck of ConvNext which can be seen in the following table where the PSNR is reported on PortraitVideo dataset:

Notably, the last row presents an alternative configuration with early mixer layers and bottleneck ConvNext layers, resulting in sub-optimal performance. We also point the reviewer to Table 4 of the main paper where we conduct further analysis validating our design choice.

3) Performance on open datasets:

We perform experiments on two public restoration datasets DVD and GoPro and tabulate the results below. It can be observed that ReBotNet performs as good as RVRT while having low latency and so enabling real-time use-cases.

4) Failing to effectively demonstrate its significance?

We respectfully disagree with the reviewer's comment on this and emphasize that our datasets simulate challenging real-world scenarios by combining multiple degradations unlike previous works in this literature. These experiments simulate multiple video restoration tasks due to the diversity of the degradations. Also, PSNR and SSIM metrics are the widely used metrics in restoration literature. Mainly, we focus on the latency, which is also not demonstrated in prior literature and demonstrates the significance of our contributions.

Our work also includes a user study revealing where our predictions were found to be more favorable to users, as detailed in Section 4.4 which is an effective way of evaluation. We kindly request the reviewer to provide additional guidance on how to effectively showcase the significance further and we are happy to work on it.

We thank the reviewer for the review and comments. We have addressed all the comments in the following:

1) Analytical Experiments leading to our design choice:

To comprehend the advantages of encoder design, we started our experiments with a setup altering the encoder design while maintaining a fixed decoder with simple transposed convolution layers. Our objective was to match the performance of these models, aiming to gauge the computational requirements needed to achieve equivalent performance among these different encoders. The performance in terms of PSNR on the Portrait Video dataset and the latency is reported in the table below:

We can observe that while Transformers take more inference time and complexity to match ConvNet’s performance while both ConvNext and Mixers are more efficient than their predecessors.

After considering these insights, our focus shifted towards leveraging the strengths of both ConvNext and Mixer. Combining a convolutional encoder for initial feature extraction with a mixer network in the bottleneck has demonstrated effectiveness. Convolutional layers excel in efficient low-level feature extraction, while mixers, following transformers, showcase strong representative abilities in extracting deep features, as also highlighted in Srinivas et al. (2021). We performed experiments where we slowly introduce Mixer in the bottleneck of ConvNext which can be seen in the following table where the PSNR is reported on PortraitVideo dataset:

Notably, the last row presents an alternative configuration with early mixer layers and bottleneck ConvNext layers, resulting in sub-optimal performance. These analytical experiments were pivotal in finalizing the design choice of ReBotNet.

2) Key Takeaways / Influence of ConvNext and Mixers:

Expanding on the tables previously discussed, the key takeaways for the community are:

• Early ConvNext layers are good low-level feature extractors; better than transformers/mixers.
• Bottleneck Mixers are effective at extracting strong deep features; better than ConvNets/ConvNexts.
• Temporal branch adds temporal consistency and makes inter-frame transitions smoother.
• A combination of ConvNext feature extractors and bottleneck Mixers proves highly effective for video enhancement, offering an optimal tradeoff between latency and performance.

3) Comparison with RVRT:

Although ReBotNet's performance is on par with RVRT, the primary difference lies in the model's latency. Unlike RVRT, which is unsuitable for real-time applications and remains challenging to scale down while maintaining a favorable latency-to-performance tradeoff (as demonstrated in Table 1), ReBotNet is an entirely different architecture design. This core difference grants ReBotNet the capability for real-time processing, facilitating its effective utilization in live calls and streams, while delivering comparable performance to RVRT.

4) Temporal Consistency:

We would like to note that quantitatively evaluating temporal consistency is an actively researched field. Usually, one is able to evaluate temporal consistency by just looking at the video. We attach a sample predictions link with and without the temporal branch and one can see that the one with the temporal branch is smoother with transitions in between frames (we recommend the reviewer to focus on the eye region to clearly see the difference).

We quantify temporal consistency via SSIM differences between consecutive frames in the Portrait Video dataset. The configuration with the temporal branch shows lower differences, indicating improved smoothness.

5) Performance on other datasets:

We perform experiments on two public restoration datasets DVD and GoPro and tabulate the results below. It can be observed that ReBotNet performs as good as RVRT while having low latency and so enabling real-time use-cases.