Direct Motion Models for Assessing Generated Videos

Thank you for your constructive comments and feedback.

there appears to be a contradiction between interpretations of Figure 5.

TRAJAN is sensitive to differences in motion between videos (whether they be from a camera, or from an object). In Figure 5(a), the generated and reference video motions are the same for both the camera and the objects – the camera pans to the left in both cases, and none of the objects move independently of the camera, even though they have different appearances between the two videos. In Figure 5(b), the camera does not move, but the man moves in a distinctly different way. As a result, TRAJAN predicts a small distance in the first case (all motions are the same), and a large distance in the second case (the motion of the object is different). There is therefore no contradiction in Figure 5.

is the TRAJAN embedding more sensitive to camera motion or subject motion?

We separately analyze how well TRAJAN captures human evaluations of generated videos when they have either high camera motion (but low subject motion) or high subject motion (but low camera motion) for the EvalCrafter dataset. The results are provided here in the last section.

In both settings, TRAJAN outperforms all baselines. TRAJAN also captures human judgments better in the high camera motion vs. high object motion setting (motion consistency 0.53 vs. 0.30; appearance consistency 0.56 vs. 0.29; realism 0.45 vs. 0.24). However, we also found that human raters were less consistent within the high object motion setting, suggesting that this may be a generically harder task (for both humans and models).

Table 3 shows that the RAFT score outperforms TRAJAN-Len. in measuring object speed (...). This finding potentially diminishes the value of the proposed metric.

The authors should provide more insights on their motivation. Is the proposed TRAJAN feature intended to serve as a standard metric to evaluate motion magnitude or plausibility for video generation models? The paper would benefit from clearer articulation of the broader purpose and application of this work.

The proposed metric was not primarily intended for measuring object and camera speed. We included this result as an interesting addition to the main arguments of the paper: TRAJAN is useful as a metric for measuring motion quality and motion differences in videos by simultaneously providing a compact latent space representation of motion and reconstruction errors for evaluating individual videos.

Importantly, in this work, and for the first time, we study metrics across distribution-level comparisons, video-video comparisons and evaluations of individual videos in terms of their motions. This involved adapting several prior methods, such as I3D, to the per-video setting as well as exploring new approaches, such as MooG, for evaluating videos. We also conducted a human study that includes fine-grained motion categories for evaluating metrics. All in all we identify numerous shortcomings of existing approaches (including RAFT) and propose a new metric based on point tracks (TRAJAN) that works well across all of the settings we tested.

To our knowledge, TRAJAN is the first metric that can operate across the distribution-, video-video comparison-, and single-video evaluation- levels, providing a compelling metric for motion in all cases.

The authors conducted comprehensive experiments [...]. However, they did not provide results on using the TRAJAN metric to compare existing video generation models.

Our human evaluations on VideoPhy and EvalCrafter are using video samples from existing video generation models. Based on your feedback we have recomputed the results from Table 1 comparing human judgements and metric scores separately for each model and then aggregating. We found that TRAJAN correlates significantly better with human evaluations of which video model creates the most realistic videos: 0.76 (TRAJAN) 0.53 (RAFT) 0.43 (VideoMAE) 0.51 (I3D) 0.58 (MooG).

The paper is poorly written. Figure Quality. Citation Inconsistency. Capitalization. References. Formatting Issues. Metric Description.

We were surprised that you found this to be the case and listed “poor presentation” as one of the reasons for recommending weak reject. The examples you listed, such as citation forms, subsection capitalization, and spacing, can be easily addressed for a camera-ready paper. We will ensure the camera-ready version of our manuscript will have each of these addressed. Thank you for pointing these out.

We provide videos for Figure 1, with additional experiments, here. We hope that by ensuring that the formatting concerns are addressed in a camera-ready version of the manuscript, the “comprehensive evaluations” and “insights to the community” that you highlight might lead you to reconsider your score.

Thank you for your constructive comments and feedback. We were pleased to see that you found our approach “methodologically sound and well-motivated” and that “All claims are thoroughly validated by extensive empirical evaluation”.

You also mentioned how “The paper [...] provides strong conceptual justification for modeling motion independently from semantics via point track encoding.” and how “This work fills a critical gap in the video generation literature by emphasizing motion quality assessment over frame-based metrics.”

“TRAJAN performance still misses edge cases where semantic context is crucial”, “Reliance on point tracks may limit applicability in occlusion-heavy … scenes”

We highlight and discuss this first point about semantic context in Figure 6, where none of the metrics can capture the unexpected disappearance of the beer glass. Fully solving this challenge would require learning a fully accurate world model. This is an exciting future direction, but it is outside the scope of our work.

Occlusion-heavy scenes are similarly challenging for all existing automated metrics which we compare to here (optical flow, action recognition, and masked auto-encoding will similarly suffer in occlusion-heavy scenes, although possibly less so than point tracking).

“Reliance on point tracks may limit applicability in … textureless scenes”

Working with textureless scenes is a general challenge for point tracking. In practice, many of the videos we evaluate are textureless. In such cases, the base point tracker does produce relatively poor point tracks which can drift and be inappropriately marked as occluded. However, because TRAJAN is trained to do point track reconstruction, it can learn this from data, and the reconstruction errors on textureless scenes are still, on average, reasonably small. Some examples of this can be seen here (second section) where point tracks for textureless parts of the background are still well reconstructed (shown in blue). This is an advantage of using a trained autoencoder over the point tracks instead of calculating a metric directly on the point tracks themselves. Thank you for raising this point, we will add a discussion of this to the paper.

Thank you for your constructive comments and feedback. Please see https://sites.google.com/view/trajan-videos-anonymous/home for additional experiments on score interpretation.

“The corrupted images as demonstrated in figure.4 do not seem to be low-level distorted (like blurred or locally rotated and shifted). That casts doubt on whether this proposed TRAJAN metric can indeed capture the distortions of real motions”

The distortions in Figure 4 are taken from prior work by Ge et al. (2024) which were chosen to highlight challenges faced by existing metrics (like FVD) in addressing temporal above and beyond appearance-based distortions. These distortions include local noise (Corruption 2.3, Figure 4e) and local warping (Corruption 1.2, Figure 4b) and were designed to capture realistic distortions from cameras or other sensors. This experiment primarily highlights the advantage of TRAJAN over prior work (including Ge et al, 2024) for being sufficiently sensitive to temporal rather than appearance-based distortions (i.e. motion biased), and we therefore matched the experimental conditions from prior work.

However, we agree with the reviewer that synthetic distortions are not necessarily representative of real motion distortions. This informed our choice to investigate different model’s sensitivities to the kinds of distortions introduced by state of the art video generation methods.

Our evaluations on VideoPhy and EvalCrafter (as referenced in your review) are doing precisely this. For example, in Appendix B.1.3 we describe how “[...] we randomly select 104 generated videos from 11 of the text-to-video models (the original 5 from the EvalCrafter human evaluation dataset, and 6 additional models).” to conduct a human study. In Table 1 and 2, we report how TRAJAN is the top-performing model that best correlates with human judgements of motion/appearance consistency and realism.

”Some other methods also include motion distortion such as EvalCrafter [1] , VBench [2], or SIFT-sora [3]. This paper should compare against those methods.”

The RAFT-Warp error, which we report in our paper, is the one used in EvalCrafter, with a related RAFT-based metric in VBench for assessing motion quality and distortions (“Temporal Quality - Dynamic Degree”). Across all of our experiments, TRAJAN performs better than or equal to RAFT-Warp for assessing motion quality and realism including on the EvalCrafter dataset.

SIFT-sora is a compelling method for interpretability, but can only be applied to videos where the motion solely comes from the camera. If any of the objects move independently of the camera (which is the case in many generated videos), SIFT-sora cannot be applied.

“Some work like [3] works on pure geometry for estimating motion score, but this work still uses representation learning, which lacks explainability a little. Including more analysis on score interpretation will be helpful.”

Thank you for this suggestion – we have added a detailed investigation of whether TRAJAN scores can provide more interpretability. Since TRAJAN measures the reconstruction of individual point tracks (Figure 1), it is possible to localize errors in both space and time (with badly reconstructed tracks, shown in red, indicating where the largest errors are). We provide 3 further examples of this kind of score interpretation here.

In each example, we show on the left the full video, in the center the Average Jaccard across all points for each frame of the video, and on the right a clip from the video centered on the frame with the worst overall Average Jaccard. In the first two cases, there is a major change in object appearance partway through the video, which is picked up in the time course plots. Looking at individual points, the greatest errors are centered on the part of the video that unnaturally changes in appearance, as indicated by the red and white colored point tracks. In the last case, this video maintains consistent motion throughout, as evidenced by the consistently high Average Jaccard. Most points are well reconstructed, with the only exceptions being points on the border of the moving object, which are generally ambiguous (their motions could go either way).

We will update the paper to discuss this aspect of our approach in greater detail and include more examples of localizing motion errors in this way.