VEnhancer: Generative Space-Time Enhancement for Video Generation

RE1: VEnhancer is proposed for achieving frame interpolation, spatial super-resolution, and video refinement in a single model. This is a challenging task as the proposed model should have sufficient generative ability to obtain high-quality videos and also preserve the original information well. Thus, we base our model on a fixed pretrained video diffusion prior to supply generative ability, and propose a space-time controller to condition the video generation across space and time axes for achieving generative enhancement. In the generative enhancement domain, we are the first work that adopts such design and significantly outperforms previous methods in generative enhancement for video generation. Besides, we propose space-time data augmentation and video-aware conditioning for achieving spatial & temporal super-resolution and video refinement in a single model and enabling arbitrary upscaling factors, interpolation ratios and refinement strengths for various user needs. While your mentioned "copying the encoder of the UNet model to the control branch and initializing the output layer to zero" are only very small parts of our whole design, which is used for weight initialization for our proposed space-time controller.

ControlNet is for controllable image generation, and the conditions are depth, edge, et.al. It cannot be directly used for video generation tasks, let alone video super-resolution, frame interpolation, and video refinement tasks.

RE2: The noise augmentation in our work also considers encoding this information into ST-Controller. More specifically, we reuse the MLP for time embedding and add a zero linear layer for efficient encoding. From the table below, we can find that the performance drops significantly without our proposed noise augmentation embedding (video super-resolution task on AIGC2023 test set). We have carefully designed this technique to improve the performance, and it is surely a contribution and a complement to the noise augmentation technique.

Conditioning on the downscale factor is extremely different from conditioning on FPS, since the former is about spatial control while the latter is about temporal control. It is not fair to compare such different things. By the way, our proposed conditioning on the downscaling factor is very effective as shown in Figure 5 (manuscript). It is no doubt a new contribution for the community.

RE3: Thanks for your suggestion. Our work only focuses on enhancement for AI-generated videos. This is a challenging task, as reconstruction-based video super-resolution and space-time super-resolution methods often fail in removing AI-generated artifacts/distortions and regenerating faithful contents. By the way, VEnhancer is not just for video super-resolution, but can also deal with frame interpolation and video refinement, which are very important for producing high-quality videos.

Thank you to the authors for their response. After carefully reading the reply and considering other reviewers' comments, I believe the main concerns remain unresolved:

1. Core Design Similarity to ControlNet: The authors argue that "copying the encoder of the UNet model to the control branch and initializing the output layer to zero" is only a minor aspect of their method. However, from my perspective, this design represents the core innovation of ControlNet and is also central to the proposed space-time controller. The only difference is that ControlNet uses an image generation model (SD) with 2D CNNs as the base model, whereas this work employs a video generation model with 3D CNNs. Reviewer LBLJ also raised a similar concern about this point.

2. Lack of Novelty in Video-Aware Conditioning: The techniques used in the so-called video-aware conditioning, such as noise augmentation and conditioning on the downscale factor, are borrowed from existing work (see my Weakness 2). These do not, in essence, constitute a novel contribution.

3. Limited Applicability of the Super-Resolution Model: As a super-resolution model, its applicability is quite narrow—it can only super-resolve AI-generated videos and cannot handle real-world videos. This significantly limits its practical use.

RE1:

Core design differs from ControlNet significantly

“Copying the encoder of the UNet model to the control branch and initializing the output layer to zero" is not the core design of VEnhancer, and we have emphasized in our previous response. Now we rephrase it with three key points:

1. VEnhancer is designed for achieving frame interpolation, spatial super-resolution, and video refinement simultaneously in a single video diffusion model. This is a huge advancement since previous methods achieve these functionalities separately, that is, to train separate video diffusion models and conduct time-consuming one-by-one inference.

2. VEnhancer adopts fixed pretrained video diffusion prior to supply generative ability, and propose a trainable space-time controller to condition the video generation across space and time axes for achieving generative enhancement. This approach is novel and differs significantly from previous state-of-the-art methods, where full finetuning or zero-shot refinement is adopted.

3. We propose the novel space-time data augmentation and video-aware conditioning to train the space-time controller for achieving multi-functional space-time enhancement and enabling arbitrary upscaling factors, interpolation ratios and refinement strengths for various user needs. This is achieved for the first time, and yields to be effective through extensive experiments.

RE2:

Video-Aware Conditioning is Novel and Effective

1. Regarding noise augmentation, we don’t just borrow it, we actually improve it -- by incorporating our designed noise augmentation embedding which reuses the MLP in time embedding and newly introduces a zero linear layer. This strategy has been validated to be very effective according to our provided quantitative comparison (table in RE2).

2. Plus, at the first time, we devise a novel technique that enables conditioning on downscaling factor, that yields to be very effective in modifying texture details generation (Figure 5 in manuscript).

3. More importantly, these conditioning techniques are aware of which frame has such conditions, and which one does not. While your mentioned works does not have such ability.

To conclude, they do, in essence, constitute a novel contribution.

RE3:

Strong Applicability for Generative Space-Time Enhancement

It is too narrow that you keep saying VEnhancer is just a super-resolution model. VEnhancer is a generative enhancement model that can achieve video super-resolution, frame interpolation, and video refinement simultaneously in a single model. While these three functions are significant in producing high-definition videos in previous research on video generation, and now we devise a single model to handle all three functions simultaneously. Isn’t it a powerful and practical toolkit for video generation?

RE1:

1. "Copying the weights from the base model as the control branch" is only a small part of our method. We use this strategy because it provides a better initialization for the space-time controller and helps convergence. Compared to state-of-the-art generative enhancement methods, VEnhancer is novel in architecture design since it is based on fixed video diffusion prior with trainable control branch, while previous methods directly full-finetune the base model. Our architecture design yields to be extremely effective in obtaining high-quality videos without sacrificing too much fidelity, leading to superior performance over previous methods.

2. The choice of key frame information integration is explored and discovered by experiments, not just an intuitive design. We have tried another alternative [Jonathan Ho, et al., 2022, Andreas Blattmann, et al., 2023a], that is to repeat the key frame and use the repeated ones as the condition for predicted frames. However, such approach fails when dealing with different interpolation ratios, as each frame could either receive correct condition (key frame) or inaccurate condition (the repeated ones) so the model gets confused during training and produce terrible results. On the other hand, some works [1, 2] concat key frame information for key frames, and concat zero images for predicted frames, as well as masking instruction for every frame. While our strategy is extremely simple and elegant, since we do nothing for predicted frames, but only perform feature summation for key frames.

[1] Guo, Yuwei, et al. Sparsectrl: Adding sparse controls to text-to-video diffusion models. European Conference on Computer Vision. Springer, Cham, 2025.

[2] Chen, Xinyuan, et al. Seine: Short-to-long video diffusion model for generative transition and prediction. The Twelfth International Conference on Learning Representations. 2023.

RE2: Thanks for your suggestions.

1. Space-time augmentation plays a vital role in producing stable and high-quality results. We remove the time augmentation part and only perform downscaling and noise augmentation, and we train VEnhancer based on this newly created data augmentation. The comparison for video super-resolution (4x) on AIGC2023 is presented in the table below.

It is observed that the performance w/o time augmentation is obviously worse than VEnhancer in DOVER (video quality assessment metric) and MUSIQ (image quality assessment metric). This indicates that key frames as conditions will boost the final video quality. On the one hand, as video frames are usually redundant, only using key frames for conditioning might be enough. On the other hand, randomly masking some frames will make the model generate the masked frames based on the generative video diffusion prior, leading to sharper results.

2. The position embedding encoded from the noise augmentation is effective.

We remove the noise augmentation embedding and train VEnhancer as a comparison. The quantitative result is presented in the above table. It is obviously observed that there is a significant performance drop in DOVER and MUSIQ compared with VEnhancer. We checked the visual results and found obvious artifacts and noises. This indicates that the noise embedding could improve the augmented noise removal, leading to more stable results.

RE3: Thanks for your suggestions. Compared with previous SOTA methods, our method have the following contributions.

1. VEnhancer is designed for achieving frame interpolation, spatial super-resolution, and video refinement simultaneously in a single video diffusion model. This is a huge advancement since previous methods achieve these functionalities separately, that is, to train separate video diffusion models and conduct time-consuming one-by-one inference.

2. VEnhancer adopts fixed pretrained video diffusion prior to supply generative ability, and propose a trainable space-time controller to condition the video generation across space and time axes for achieving generative enhancement. This approach is novel and differs significantly from previous state-of-the-art methods, where full finetuning or zero shot refinement is adopted.

3. We propose the novel space-time data augmentation and video-aware conditioning to train the space-time controller for achieving multi-functional space-time enhancement and enabling arbitrary upscaling factors, interpolation ratios and refinement strengths for various user needs. This is achieved for the first time, and yields to be effective through extensive experiments.

RE4&5: Thanks for your suggestions, and we have reorganized our paper and fixed the typos.

RE1: Thanks for your suggestion. Here we provide the table that compares VEnhancer with state-of-the-art generative enhancement methods in terms of inference time for video super-resolution (4x). Note that the input shape is 16x3x320x512.

From the above table, we can find VEnhancer is already the most efficient among state-of-the-art generative enhancement methods. The main reason is that VEnhancer only requires 15 inference steps, while other methods require at least 30 steps.

RE2: Thanks for your suggestion. In fact, the data collection in VEnhancer is not that difficult. The most important thing about data collection for video enhancement is high-resolution. There are some websites (e.g., pexels) where you can download free videos with 4k/8k resolution.

RE3: Thanks. VEnhancer is able to enhance different state-of-the-art text-to-video results (e.g., VideoCrafter2, CogVideoX, Kling, et.al.), which is already practical and can be regarded as a video super-resolution toolkit.

RE4: We compare Upscale-a-video for video super-resolution (4x) on AIGC2023. The quantitative comparison is illustrated in the following table.

It is observed that our VEnhancer is significantly better than Upscale-a-video in video quality assessment metric (DOVER), image quality assessment metric (MUSIQ) and Aesthetic Quality. This comparison demonstrates the superiority of our VEnhancer over state-of-the-art generative enhancement methods.

RE Questions: Regarding dataset filtering, please filter those videos with high resolution (>1080p), and use video quality assessment for further filtering.

RE1: Thanks for your suggestions. The notations and illustrations have been updated in the manuscript according to your comments.

RE2: Thanks for your suggestions. The initialization of z_t should be Gaussian noise, while z{s, \sigma} is fixed for each sampling step.

The inference time and performance across different SSR scales and TSR scales are illustrated in the following Table. When evaluating inference time, the input shape is 16x3x320x512. When evaluating different TSR scales, the SSR scale is fixed to 4. When evalutating different SSR scales, the TSR scale is fixed to 1.

For different TSR scales, the performance in DOVER (video quality assessment) and MUSIQ (image quality assessment) remains stable. For different SSR scales, the performance decreases slightly when SSR increases. It is because larger SSR scale will lead to smoother results. We will consider patch-based enhancement to deal with large SSR scales in the future.

For different TSR or SSR scales, the inference time grows when TSR or SSR scale increases. Here we also provide another table that compares VEnhancer with state-of-the-art generative enhancement methods in terms of inference time for video super-resolution (4x).

From the above table, we can find VEnhancer is already the most efficient among state-of-the-art generative enhancement methods. It is because VEnhancer only requires 15 inference steps, while other methods requires at least 30 steps.

RE Questions: The shapes of inputs to ST-Controller are different because of different interval m. During training, the max target frame length is set to a constant (e.g., 25). While during inference, the target frame length might be very large with higher TSR scale (i.e., interval m). Therefore, we split the video into multiple chunks with overlaps, and each chunk is processed independently for each sampling step. And we apply linear blending for the overlap part after each sampling in the latent space.