VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models

We thank the reviewer for the comprehensive feedback and the useful points for discussion. Please find below our response.

VBench prompts: Both VBench and Movie Gen are common benchmarks for general video evaluation. The VBench prompts are not used in our work since out of the 11 dimensions for evaluation, 6 purposefully contain prompts with no motion at all (e.g., “object class” contains objects: “a person”, “a bicycle”, “temporal flickering” explicitly demands no motion, “a laptop, frozen in time”). Therefore, similar to recent leading models (Step-Video-T2V, Veo2), we employ Movie Gen, which evaluates all aspects of the generation (similar to VBench) while also incorporating motion (for more details, see our response to reviewer RHDu).

VideoJAM scalability: We appreciate the opportunity to clarify. While it is true that (only) the automatic metrics favor the 4B model, please note:

1. It is well-documented that automatic metrics often fail to align with human preferences [1,2,3]. For instance, Movie Gen dedicates a section (3.5.3) to metrics, highlighting that "across all tasks, we find that existing automated metrics struggle to provide reliable results”.
   Automatic metrics tend to prefer static motion since it is more “smooth” (as noted in the VBench paper, "a completely static video can score well in the aforementioned temporal quality dimensions"). Indeed, the 4B model scores better than the 30B model in the automated metrics, despite the 30B model being overwhelmingly superior (see human evaluation below). This anomaly can be attributed to the 4B model's tendency to produce videos with reduced motion (DiT-4B scores only 38.3 in its dynamic degree), which aligns with the biases of automated metrics discussed above. |Model|Text Faith.|Quality|Motion| |:------------:|:----------:|:-------:|:------:| |VideoJAM-4B|21.9%|21.1%|18.7%| |VideoJAM-30B|78.1%|78.9%|81.3%|

[1] Polyak et al., Movie Gen.
[2] Bartal et al., Lumiere.
[3] Girdhar et al., Emu Video.

2. The official VBench leaderboard shows similar inconsistencies. Models ranked highly by the HuggingFace (HF) human leaderboard are ranked significantly lower by VBench. For example, Kling is the leading baseline by both our human evaluation (Tab. 2) and HF. However, it is ranked 21st in the VBench leaderboard, below even much smaller models like CogVid-5B and Wan2.1-1.3B, both ranked much lower in the human leaderboard.

3. To further show robustness and scalability, we enclose a study in the same setting as Tab. 2, conducted on DiT-30B at a 512x512 resolution. VideoJAM maintains its significant improvement with a 71.1% lead in motion, which is less than 2% below the improvement for the lower resolution. |Model|Text Faith.|Quality|Motion| |:------------:|:----------:|:-------:|:------:| |DiT-30B (512)|30.5%|26.6%|28.9%| | +VideoJAM |69.5%|73.4%|71.1%|

Additional references: Thank you for the useful suggestions, the related works section will be revised to include them.

Scene cuts: We hypothesize that it is still possible to use VideoJAM by applying RAFT on each scene separately and stitching the scenes with a “white” (no motion) frame.

Inner-Guidance latency: Inner-Guidance is applied to only 50% of the steps and performed in parallel using a batch size of 3 (L.726-731). This results in a 1.15x slowdown compared to standard CFG, which is a significantly lower slowdown than other multi-guidance methods (e.g., Instruct Pix2Pix incurs a 1.25x slowdown as it operates on all steps).

Physics benchmark: Thank you for this suggestion. We arbitrarily used the first benchmark suggested, and conducted an evaluation in the same setting as Tab. 2. The results below indicate a clear advantage to VideoJAM (62.7% preference in motion).
Note that it is expected for the improvement in this benchmark to be somewhat decreased since optical flow is not an explicit physics representation, therefore it does not solve all physics incoherences (L. 412-418). As we mention in the conclusions, incorporating physics priors in future works is a fascinating direction. Samples from the comparison are provided in this link. We observe that these prompts are quite different from the ones employed by other benchmarks, and do not describe typical natural scenes.

Optical flow calculation on data: The calculation takes 13 hours in total (~1-2 seconds per sample with a batch size of 128). VideoJAM is a generic approach that can theoretically work with any representation, as long as there is a reliable way to encode it (e.g., RGB representation or a specialized VAE).

We are happy to address any remaining questions.

We thank the reviewer for the comprehensive review of our work and the insightful suggestions. Please find below our response to the points raised in the review.

Related work on text-to-3D: Thank you for bringing this work to our attention. We will revise the related works section to reference this paper in the final version of the manuscript.

VBench metrics: Thank you for this point for discussion. VBench has two different contributions: a benchmark and a set of metrics. The benchmark is not used in our work since a significant part of the prompts there intentionally describe static scenes. Most categories of the benchmark (e.g., architecture, food, scenery, plants) are dominated by prompts that do not elicit any motion (e.g., “an apartment building with balcony”, “video of a indoor green plant” (grammar mistake in the original prompt), “landscape agriculture farm tractor”). Even the categories that entail motion (e.g., humans and animals) tend to focus on prompts that do not require meaningful temporal understanding (e.g., “people sitting on a sofa”, “a man with a muertos face painting”, "a black dog wearing halloween costume”).
Additionally, out of the 11 dimensions for evaluation, 6 purposefully contain prompts with no motion at all (e.g., the class “object class” simply contains a list of objects such as “a person”, “a bicycle”, the class “color” contains prompts that assign colors to objects such as “a red bicycle”, “a green bicycle”). Therefore, similar to concurrent leading video models (Step-Video-T2V, Veo2) we opt to employ the Movie Gen benchmark as a general quality evaluator since it is carefully curated to evaluate all aspects of the generation (similar to VBench) while also explicitly incorporating motion instructions in all the prompts (see Section 3.5 of the Movie Gen paper for details on the Movie Gen benchmark).

The set of metrics proposed by VBench is divided into two: those that are applied by VBench to external benchmarks (see official implementation) and a set that includes additional metrics that apply only to the VBench prompts. An example of such a metric is the VBench metric to estimate temporal flickering which requires frozen videos, thus the prompts explicitly encourage no motion at all. Naturally, we used all metrics that are supported for external benchmarks to evaluate the performance of VideoJAM in the setting that it is intended for. This allows us to benefit from the generic and meaningful prompts from Movie Gen with the disentangled axes of evaluation from VBench.

Notably, both benchmarks are not suitable for measuring motion coherence, as both tend to focus more on appearance and less on temporal understanding (L. 330-357). Thus, we construct an additional benchmark (VideoJAM-bench) to estimate temporal coherence and employ Movie Gen for a general assessment of the video quality.

We are happy to address any remaining questions by the reviewer.

We thank the reviewer for the comprehensive feedback and the interesting points for discussion. Please find below our response to the points raised in the review.

VideoJAM adaptability: We appreciate the feedback. While all concepts of our work can be easily generalized to any backbone, we acknowledge that this work focuses on the de-facto leading architecture, DiT. Following the review, the writing will be revised and refined to reflect this and avoid confusion.

Higher resolution and longer videos: The duration of our generated videos is limited by that of the base model we employ, which is 5 seconds (128 frames at 24 fps). However, to showcase the scalability of VideoJAM, we repeat the experiment from Tab. 2 on VideoJAM-30B with a higher resolution of 512x512. The results of the human study are enclosed below. As can be observed, VideoJAM is, once again, superior to the base model in all aspects, and very significantly improves motion coherence, further establishing the scalability of our method.

Inner-Guidance sensitivity test: Thank you for this suggestion. In accordance with common practice in diffusion papers, we selected the best scale empirically. Following the review, we enclose a qualitative sensitivity test with different Inner-Guidance scales in the following link (best viewed in full screen). All results are extracted in the same setting described in the paper.
As can be observed, removing Inner-Guidance causes a noticeable degradation to the motion coherence (e.g., the legs of the man, the helicopter is flying backward). Importantly, Inner-Guidance demonstrates robust results across all reasonable scales (3,7) and does not display unusual sensitivities.
When significantly increasing the scale (50), the results are out of distribution and cause a degradation in the video quality and motion, which is to be expected with any other guidance signal as well (e.g., text with CFG).

Human evaluation protocol: Thank you for raising this point, we appreciate the opportunity to clarify this aspect of our work. The human evaluators selected to participate in our study have extensive experience with evaluating generative visual models. As a filtering criterion, each evaluator has performed at least 1000 evaluations before, where at least 90% of those evaluations have been approved by third-party examiners. The two videos in each comparison are randomly shuffled to ensure an unbiased comparison, and all videos are generated without watermarks to avoid identification.
Additionally, note that the rating of the baselines by our evaluators (Tab. 2) is very much in correlation with the public HuggingFace video leaderboard based on human evaluations, where Kling 1.5 is the leading baseline, with Sora being the second strongest baseline.

We enclose below our instructions for the evaluators for transparency.

Hello! We need your help to read a caption, and then watch two generated videos. After watching the videos, we want you to answer a few questions about them:
Text alignment: Which video better matches the caption?
Quality: Aesthetically, which video is better?
Motion: Which video has more coherent and physically plausible motion? Do note, it is OK if the quality is less impressive as long as the motion looks better.

We are happy to address any remaining questions.

We thank the reviewer for the comments and points for discussion. Please find below our response.

Human evaluation: Thank you for your feedback. There appears to be a misunderstanding in the review. Importantly, VideoJAM appears in all human evaluations. As highlighted in all table captions, the human evaluation scores indicate the percentage of votes that favor VideoJAM, where each result uses the Two-Alternative Forced Choice (2AFC) protocol (Tabs. 1-3, L. 371-377). For example, a motion score of 85.9% for CogVid-5B indicates that evaluators prefer the motion by VideoJAM over CogVid-5B in 85.9% of votes. Since all rows are inherently comparative to VideoJAM, the final row ("VideoJAM vs. VideoJAM") contains minus signs (-). To further improve clarity, we will add visual graphs of the comparisons in the Appendix, and replace the minus signs with upper arrows.
As noted in the paper (L. 400-403, L. 413-419), human raters consistently and significantly prefer VideoJAM in terms of motion over all baselines.

Impact of Inner-Guidance: Inner-Guidance is a highly beneficial component of our method. As shown in Tab. 3, human evaluators significantly prefer the results with Inner-Guidance across all categories (68.9% vs. 31.1% for text faithfulness, 64.4% vs. 35.6% for quality, and 66.2% vs. 33.8% for motion). Also, note that removing the optical flow prediction inherently removes Inner-Guidance, as Inner-Guidance depends on the motion prediction. Thus, eliminating optical flow is an ablation that includes both Inner-Guidance, and the flow prediction and, as expected, results in a greater performance drop.
To further demonstrate the benefits of Inner-Guidance, we enclose a qualitative ablation test (best viewed in full screen). The results, generated in the same setting as in the paper, illustrate that removing Inner-Guidance often degrades motion coherence and introduces artifacts.

Inner-Guidance latency: As noted in the implementation details (L.726-731), Inner-Guidance is applied to only 50% of the steps (following Sec. 3) and performed in parallel using a batch size of 3. This results in a 1.15x slowdown compared to the standard CFG, which is a significantly lower slowdown than other multi-guidance methods (e.g., Instruct Pix2Pix incurs a 1.25x slowdown since it operates on all steps).

Additional related works: We appreciate the suggestion. The related works section will be revised to include all references proposed by the reviewers, ensuring a more comprehensive review of existing literature.

Automatic metrics vs. human evaluation: Thank you for raising this point. We appreciate the opportunity to clarify.

1. It is well-documented in literature that automatic metrics often fail to align with human preferences [1,2,3]. For instance, the Movie Gen paper dedicates an entire section (3.5.3) to metrics, and highlights that "across all tasks, we find that existing automated metrics struggle to provide reliable results”. Specifically, temporal metrics tend to prefer either static motion since it is more “smooth” (as noted in the VBench paper "a completely static video can score well in the aforementioned temporal quality dimensions."), or completely incoherent videos that contain a lot of movement (Appendix, L. 767-768). Appearance metrics inherently prefer static videos, since they are typically based on image models, for which video frames that contain motion are out of distribution. All of the above underscores the inherent limitations of automated metrics in capturing the dynamic and nuanced nature of video content, and the necessity for a human-based evaluation.
   [1] Polyak et al., Movie Gen.
   [2] Bartal et al., Lumiere.
   [3] Girdhar et al., Emu Video.

2. The official VBench leaderboard further highlights these inconsistencies. Models ranked highly by the HuggingFace (HF) human leaderboard are ranked significantly lower by VBench. For example, Kling 1.5 is the leading baseline by both our human evaluation (Tab. 2) and HF. However, it is ranked 21st in the VBench leaderboard, below even much smaller models like CogVid-5B and Wan2.1-1.3B, both ranked much lower in the HF human leaderboard.

We enclose both automated metrics and human evaluations for completeness and believe continued research on reliable automatic metrics is essential for video generation assessment.

Supplementary materials (SM): We noticed that the reviewer indicated “no” to viewing the SM. While we understand that reviewing SM is optional, we believe that the actual video results are very important to fully appreciate the improvement enabled by VideoJAM. We kindly ask the reviewer to consider viewing the results on our SM website.

We are happy to address any remaining questions, and respectfully ask the reviewer to reconsider the score if our response has addressed all the questions.