VADER: Video Diffusion Alignment via Reward Gradients

Q 6.1: Lacks a quantitative evaluation of the temporal coherence over V-JEPA and other models.

Thanks for pointing this out, we have added temporal dynamics and temporal coherence evaluation over VADER trained using V-JEPA, ViCLIP, PickScore, HPS and Aesthetics reward functions on our webpage. We use VBench [1] and EvalCrafter [2] for conducting this eval.

VBench Evaluation while using *image to video models, while trained using V-Jepa reward model:

VBench Evaluation while using text to video models, while trained using various reward models:

EvalCrafter Evaluation using text-to-video models can be found in the Table below:

We also visualize the generated videos after training on each of these reward models in the project website.

Overall we find that VADER beats the baseline over all these metrics. We find that video reward models such as ViCLIP help achieve higher temporal dynamics, while image-reward models such as PickScore help achieve higher temporal coherence and subject consistency. Lastly, we find that fine-tuning via V-JEPA improves both temporal coherence and motion quality of the original Stable Video Diffusion.

Q 6.2: Lacks a detailed examination of how each specific reward model influences alignment objectives.

Thanks for the suggestion. We show a detailed comparison on how finetuning using a specific reward model affects the results on the other reward models: https://vader-anonymous.github.io/#reward-correlation . We find that there is a strong positive correlation between PickScore and HPS reward models because both of them contribute to text-video alignment while there is a somewhat negative correlation between Aesthetic Score and ViCLIP Score. We visualize the correlation matrix on the webpage and in Figure 11 of the paper.

Q 6.3: How are the visualizations selected? The examples were not selected entirely random. To show an entirely random selection we visualize a lot more videos here: https://vader-anonymous.github.io/Video_Gallery.html. We have also conducted a human eval study and our results are shown in Table 2.

Q 6.4 Why aren’t popular metrics, such as FID/FVD, used in the experiments? FVD or FID requires access to a real video dataset to compute the distance between generated and real distributions. VADER is data-free and does not rely on any real video data, so FID or FVD are not applicable.

[1] VBench: Comprehensive Benchmark Suite for Video Generative Models [2] EvalCrafter: Benchmarking and Evaluating Large Video Generation Models

I thank the authors for their response. They have addressed all my questions and concerns. I have no further questions.

Q 5.1 How does K=1 in VADER affect the outcomes?

We run comparisons with K=10, by offloading the GPU memory to the CPU memory. We find different tradeoffs for using a higher value of K.

In the earlier steps of training, higher values of K results in more semantic level changes, while K=1 results in more fine-grained changes. Further, we find as we train longer, both the models start exhibiting semantic level changes.

We also find, training a model at K=10 is much slower than training it at K=1, for instance with 12 GPU hours, K=10 takes 50 gradient update steps, while K=1 takes 350 gradient update steps. We also find that K=1 is much easier to optimize, for instance with 200 gradient update steps, K=1 gets a reward of 5.49, while K=10 gets a reward of 5.26.

This result is shown here: https://vader-anonymous.github.io/#truncated-backpropagation-ablation

Overall, we think the right value of K, might be system specific, higher values are preferred when the training system is not GPU VRAM bottlenecked.

Q 5.2 Misalignment between text and image content in Figure 7 in two bears example.

Figure 7 of the paper was generated by a model fine-tuned using the Aesthetic Reward Model. This reward model is trained to predict the aesthetic quality of pictures, so it does not contribute much to image-text alignment. Fine-tuning by utilizing the HPS or PickScore or ViCLIP Reward Models would help text alignment much more. More visualizations for video-text alignment are available at https://vader-anonymous.github.io/.

Q5.3: Lack of Textual description alignment

We thank the reviewer for their prompt response. VADER improves the textual alignment score of VideoCrafter from 0.256 to 0.267 when evaluated via HPS score, when evaluated via human eval for text alignment, VADER's generations are preferred 72% of the times over the baseline. These numbers are averaged over 700 videos generated by both the model. We therefore think this is a more accurate metric to evaluate the text-alignment of VADER than evaluating specific videos shown in the paper. We never claim VADER has perfect video-text alignment, however we stand with the claim that on average it's considerably better than the videos generated by VideoCrafter or ModelScope in terms of text-alignment.

We thank the reviewer for their quick response!

Q 4.5: Related work - T2V-Turbo

Thanks for sharing this work! We were not aware of the work of T2V-Turbo [1]. We plan to cite them and have also started working on comparing our results against them.

However, we would like to point out some major differences between the two works:

• The goal of the T2V-Turbo is very different from goal of VADER. Their goal is to improve the inference speed of T2V models, by distilling a teacher model (eg: VideoCrafter) into a few-step video consistency model. The goal of VADER is instead to align existing video-models to various downstream tasks. For instance, we show results on removing objects from the scene while using object detectors as reward models - https://vader-anonymous.github.io/#object-removal-reward . This could be a very useful downstream task for removing explicit content from video generation pipelines but this doesn't align with the goal of T2V-Turbo, as again their goal is to improve the speed of video generation. Further as both approaches have very different goals, this gives rise to very different evaluations studied in both the approaches, for instance we compare against other forms of RL techniques such as DDPO or DPO and we focus on training efficiency of VADER in terms of compute and samples used during finetuning. T2V-Turbo on the other hand focuses on inference time compute efficiency, and hasn't reported any numbers studying finetuning efficiency.

• T2V-Turbo requires having access to an external video-text training dataset, VADER on the other hand is entirely data-free. Explicit requirement on an external training dataset, could potentially make it difficult to adapt to different base video models, as the base models considered might have never been trained on certain datasets.

• Lastly our approach is not restricted to T2V models and can be easily adapted to I2V mdoels as we show here: https://vader-anonymous.github.io/#v-jepa-reward

Irrespective of the major differences between the two works, we still think it's not fair to consider T2V-Turbo's work, when evaluating the novelty/originality of VADER. We consider T2V-Turbo to be a concurrent work due to the following reasons:

• T2V-Turbo was arxiv-only and was not released as a NeurIPS paper until the abstract deadline for ICLR (Sept 27th)
• Our methods and initial results were submitted to conferences and our project webpage was available by end-February, which is way before the arxiving date for T2V-Turbo. Unfortunately we can't share exact details on this, while preserving the sanctity of double blind reviewing.

Having said that we do acknowledge that T2V-Turbo indeed does RL on video diffusion models. We have therefore corrected our previous statement stating that: "we are the first work to successfully align video diffusion models using reinforcement learning".

Q 4.6: Table 1 Clarification

Thanks for pointing this out. We believe there is a misunderstanding here, In Table 1 we do not use multiple reward models at once, instead the results are shown while using each reward model in the column independently for fine-tuning. We consider each reward model to be a different downstream task, and show how effectively VADER can fit to a variety of downstreams tasks. We hope this clarifies the comparison with DPO. We will include this clarification in the paper.

Q 4.1: Lack of novelty.

Please refer to the global answer.

Q 4.2: Unfair comparison between an on-policy strategy and off-policy strategies.

We agree that VADER is more on-policy compared to DDPO and DPO. However we do not think this is the reason why VADER is more sample-efficient. In-fact off-policy methods are generally considered to be more sample efficient than on-policy methods [1].

To further investigate this, we created an on-policy version of DPO and DDPO, we do this by taking a single gradient update per video sampled. In our experiments we don’t find the results to improve the sample efficiency of the baselines. We plot these results in the following link: https://vader-anonymous.github.io/#training-efficiency-comparison and in Figure 12 in the main paper. Overall we think VADER is more sample efficient, because it backpropagates dense feedback to the model weights, while DDPO or DPO backpropagate scalar feedback.

Q 4.3: More methods should be considered for comparison.

There are not many methods in video-diffusion alignment, therefore we adapt state-of-the-art methods in image diffusion alignment space to videos. Since the deadline we have also added other baselines such as DDPO-on policy, DPO-on policy and DOODL , the results for these methods can be found in https://vader-anonymous.github.io/#training-efficiency-comparison and https://vader-anonymous.github.io/#doodl-vader.

Q 4.4: Experimental details on DPO and DDPO.

We implement DDPO [2], by closely following the official code from https://github.com/kvablack/ddpo-pytorch. We adopt the same code structure, which computes the log probability for each ddim denoising step, for the video diffusion model. This log probability is used to compute policy gradients, following the method code. We use the PPO version of the method as opposed to only using policy gradient, since this is reported to give slightly better performance in their paper. We use the same LoRA and gradient checkpointing approach that is used in VADER for updating the video diffusion model

We implement DiffusionDPO [3] based on the official code from https://github.com/SalesforceAIResearch/DiffusionDPO. We alternate between sampling from the diffusion model, and training the model via the DPO objective, similar to the process in DDPO. Samples are added to a replay buffer (since offline samples can be used for DPO training), and we use batches sampled from the buffer for training. Given pairs of video generations, we assign them as $V^w$ and $V^l$ based on rewards from the reward model, where $V^w$ obtains higher reward. We then use the same loss function as the original paper, which increases the likelihood of the $V^w$ sample. We set β, the KL penalty in DPO to be 5000 following the standard. Just like DDPO, the model is updated using same LoRA training and gradient checkpointing approach as in VADER.

Thanks for pointing this out, we have added the explanation in the paper.

References:

[1] Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables

[2] Training Diffusion Models with Reinforcement Learning

[3] Diffusion Model Alignment Using Direct Preference Optimization

[4] End-to-End Diffusion Latent Optimization Improves Classifier Guidance

Thanks for the clarification in Table 1.

For the related work, first of all, T2V-Turbo has already demonstrated in the ablation study the impact of image model rewards and video model rewards on aspects such as aesthetics, dynamic quality, and motion smoothness, etc. , rather than merely using RL to improve inference speed. Furthermore, one concern I have with this paper is that T2V-Turbo was trained and evaluated using a larger-scale dataset and compared against numerous well-known methods, making its results more reliable than those presented in this paper.

I agree with the authors that this work is a concurrent work with T2V-Turbo. But I think the authors need to carefully review more papers before claiming a very strong argument.

I will raise my score to 5.

Evaluation against T2V-Turbo while using Standard Prompt Suite of VBench

We conduct an additional experiment while using the standard prompt suite of VBench. For the baselines, we copy-paste the numbers reported by T2V-Turbo in Table 1 of their paper. We follow the standard evaluation pipeline on VBench, for evaluating VADER (https://github.com/Vchitect/VBench?tab=readme-ov-file#evaluation-on-the-standard-prompt-suite-of-vbench).

We find that, VADER achieves the best Quality Score, amongst all the baselines. Quality Score is the weighted sum of the normalized score of each metric, as reported in VBench and T2V-Turbo.

We report the results in the Table below and the following link: https://vader-anonymous.github.io/#vbench-evaluation-standard-prompt-suite.

Evaluation against T2V-Turbo on EvalCrafter benchmark

To further evaluate temporal coherence and motion quality, we use the official Evalcrafter benchmark while using their standard prompts. We find that VADER outperforms T2V-Turbo on these metrics, as shown in the table below and the following link: https://vader-anonymous.github.io/#evalcrafter-evaluation

We will include these results in the revised version of our paper.

Sorry for the confusion, we use the same evaluation pipeline/benchmark as them i.e VBench, which involves the same evaluation metrics such as Subject Consistency,Background Consistency, Motion Smoothness,Dynamic Degree, Aesthetic Quality and Imaging Quality.

However, To ensure consistency with our Human Eval and VBench results, we use the same set of 700 prompts officially released from the EvalCrafter benchmark (https://github.com/evalcrafter/EvalCrafter/tree/master/prompts, ). Note the same set of 700 EvalCrafter prompts were also used by T2V-Turbo for human-evaluation in Section 4.2.

Further we also follow the VBench custom prompt pipeline for evaluation: https://github.com/Vchitect/VBench?tab=readme-ov-file#new-evaluate-your-own-videos, and use the officially released checkpoints from T2V-Turbo.

We will mention all of these details in the revised version.

Q3.1: Clarity of Contribution

Please refer to the global answer.

Q 3.2: Step-by-step ablation study is needed to show how each component of VADER addresses memory overhead problem

We conduct a step-by-step ablation study to demonstrate how each component of VADER addresses the memory overhead problem. To study the contribution of each component to memory reduction, we conduct experiments on a single gpu, with batch size of 1. For this experiment, we offload the memory to the CPU main memory to prevent GPU out-of-memory error. The results are shown in the Table below.

We find that the total memory is reduced by a significant amount, while using the above components.

Q 3.3: Include comparisons with DPO and DDPO results over longer training times

Thanks for the suggestion. We trained DPO and DDPO for longer. We found a meaningful improvement in DPO after long training, however we couldn’t see the same improvement in DDPO. We think this is because the number of gradient accumulation steps in DDPO’s implementation are very large i.e they scale linearly wrt the number of diffusion timesteps, thus the number of parameter updates done are relatively very little even with large amounts of compute. A better study on DDPO’s parameter update rate for video training, could improve its sample efficiency. The results of our experiments can be found here: https://vader-anonymous.github.io/#training-efficiency-comparison, and also in Figure 12 in the main paper.

Q3.4: Comparison with Existing Guidance Methods

Thanks for the suggestion, the guidance methods such as DOODL or Universal Guidance are applied for each example separately. VADER on the other hand updates the weights of the model, thus not requiring per sample adaptation at test time. In the following link: https://vader-anonymous.github.io/#doodl-vader, we compare VADER with DOODL, we find DOODL improves with more gradient update steps, however the improvement is still relatively less and scales linearly wrt number of examples we use for evaluation, thus making it difficult to be used in practice.

Q3.5 Clarification in Table 1. Our experiments in Table 1 are based on ModelScope, we have clarified this in the paper. We have also conducted further quantitative experiments with other base video models, which are shown in VBench Evaluation, Eval Crafter diversity test.

Q2.1 Lack novelty

Please refer to the global answer.

Q2.2 Diversity of the generated videos using the aligned model.

Aligning a model for a specific use-case, very often results in a reduction of diversity and has been well-studied in prior works [1,2]. For instance, a pre-trained base model has the ability to generate videos of various types such as aesthetic, non-aesthetic, compressed, text-aligned or non text-aligned etc. Fine-tuning the model for alignment forces the model to generate a specific type of videos (eg: only aesthetic and text-aligned) thus reducing its overall diversity.

We find a similar result in the Table below, where we study the diversity of the models.

We study diversity by generating multiple videos given a text prompt, we embed these videos via VideoMAE, and then calculate the variance of the embedding. We find that the overall diversity of the base model reduces as we align them using specific reward models. We also find that the highest diversity is achieved with VADER trained via ViCLIP+Aesthetic reward among all other reward models.

Q2.3: No video visualizations for video reward function

Thanks for pointing this out. We have added visualizations for

ViCLIP here - https://vader-anonymous.github.io/#aesthetic-and-viclip-reward

V-JEPA here - https://vader-anonymous.github.io/#v-jepa-reward.

To further investigate, any sacrifice in terms of temporal variations, we study temporal dynamics using VBench (https://arxiv.org/abs/2311.17982) in the table below.

We find that VADER-ViCLIP achieves the highest dynamic-degree amongst all the baselines. Dynamic degree in Vbench is calculated by checking the dynamicness in the 2D flow predicted by RAFT (https://arxiv.org/abs/2003.12039).

We also find that temporal coherence and motion quality is not hampered while using Image-based reward models such as PickScore or HPS when benchmarking using EvalCrafter (https://arxiv.org/abs/2310.11440), as shown in the Table below:

EvalCrafter calculates Motion Quality using Video Action classification models and flow prediction models such as RAFT, further it calculates Temporal Coherence using warping error and semantic consistency.

[1] - Understanding the Effects of RLHF on LLM Generalisation and Diversity https://arxiv.org/abs/2310.06452

[2] - One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversity https://arxiv.org/abs/2411.04427

Q1.1 Novelty:

Please refer to the global answer.

Q1.2 Image-based Reward function don’t adequately encapsulate the unique characteristics of video data:

We agree that the image-based reward models do not capture the temporal dynamics.

We therefore, run experiments with the following video reward models:

i) Vi-CLIP - since deadline we have included Vi-CLIP, which is a video-text model trained in a similar fashion to CLIP. We show that this improves our video-text alignment and overall temporal dynamics of VADER. We visualize the results in https://vader-anonymous.github.io/#aesthetic-and-viclip-reward.

ii) V-JEPA- we use the base V-JEPA model trained using SSL objective as our reward function. We show that masked autoencoding reward functions can improve the temporal consistency of the generated video. These results are shown in the https://vader-anonymous.github.io/#v-jepa-reward, and Figure 9 in the main paper.

iii) VideoMAE - we use VideoMAE fine-tuned for action classification as our reward model, Our results are described in Figure 10, and Table 1 in the paper.

Further we ran evaluations using VBench (https://arxiv.org/abs/2311.17982) and found that using a video based reward function such as ViCLIP indeed improves the dynamic degree of the generated videos. The results can be found in the following link and the table below:

As can be seen in the Table above: the dynamic degree for VADER-ViCLIP is the highest amongst all the baselines. Dynamic degree in Vbench is calculated by checking the dynamicness in the 2D flow predicted by RAFT (https://arxiv.org/abs/2003.12039). However as can also be seen, VADER-ViCLIP doesn’t score high on other categories such as Image Quality, Aesthetic Quality.

To address the above concern: We have trained our own video reward model by fine tuning ViCLIP. Our video reward model can better capture various dimensions of improvements such as Dynamic Degree, Image Quality, Subject Consistency and Motion Smoothness. We are currently conducting HumanEval using our video reward model.

We also find that temporal coherence and motion quality is not significantly hampered while using Image-based reward models such as PickScore or HPS when benchmarking using EvalCrafter (https://arxiv.org/abs/2310.11440), as shown in the Table below:

EvalCrafter uses Video Action classification models and flow prediction models such as RAFT to evaluate Motion Quality, further it uses warping error and semantic consistency to evaluate Temporal Coherence.

Q1.3 An error in Equation (231) and ordering missing in Equations.

Thank you for pointing this out. We have better clarified the equation you mentioned and have updated the manuscript with numbered equations. References to equations in the text are also cross-checked and updated accordingly.

Q1.4) Beneficial if the authors developed and trained a reward model specifically tailored for video content

To investigate the benefits of training a custom video reward model, We fine-tuned PickScore reward model using custom 16,000 video-text + reward pairs. We constructed this dataset, by generating video-text pairs using VADER, we obtained the reward values using the automated metrics in VBench.

As the PickScore reward model can process only image-text pairs, we adapt it to process video-text pairs. We do so by using its image encoder to obtain 1D embeddings for each frame in the video denoted by $[f_0, f_1 .. f_N]$, where $N$ is the number of frames in the video. We then use its text encoder to obtain the text embeddings denoted as $c$. We then train a 5 layer and 8 head decoder-only transformer model that takes in N+1 embedding vectors (frames + text) as input and outputs a k-dimensional vector, where k represents the number of metrics in VBench such as Subject Consistency,Background Consistency etc. We train this model via supervised learning using MSE loss to predict the normalized scores of VBench metrics. We freeze the PickScore Image and Text encoders, and only train our custom transformer from scratch. We visualize the training curves for the reward models here - https://vader-anonymous.github.io/#vbench-reward-model-train. Note that we train this reward model as the VBench metric is not directly differentiable and thus cannot be directly used in VADER. Further even if made differentiable somehow, it will not be feasible computationally as they use a lot of models.

We then use this VBench-distilled reward model discussed above to finetune VADER-Aesthetic+HPS, we plot the reward curve for fine tuning here - https://vader-anonymous.github.io/#vader-vbench-loss

We find that the VBench metrics (EvalCrafter prompts) when using unseen EvalCrafter prompts improves as a result. We find that we get a better Weighted Average Score (across all metrics), over all the baselines.

We visualize the qualitative results against our base-model of VADER-Aesthetic+HPS Model here - https://vader-anonymous.github.io/#vbench-distilled-reward-model . We observe that fine tuning using the VBench-Distill reward model consistently improves the dynamicness of the generated videos.