Stable Cinemetrics : Structured Taxonomy and Evaluation for Professional Video Generation

We thank the reviewer for their constructive feedback of our work. Please find below our responses to your questions:

Professional Nature of Prompts

The videos and prompts shown in the Camera Trajectory section of the supplementary material were selected as illustrative positive/negative examples, not as a representative sample of the full prompt set. To clarify, we report below the actual distribution of values from all prompts labeled cinematographer in our SCINE_VISUALS.json.

As this shows, our prompt set spans a wide range of values, across different nodes. Importantly, these values do not occur in isolation; controls often co-occur, e.g., "push in + low angle" or "push in + dutch angle", yielding a rich and combinatorially diverse testbed for video generative models.

Additionally, as demonstrated in Figure 4, our prompts closely align with real-world screenplays. Figures 25–27 further show that our prompt set offers broad and diverse coverage across all four taxonomies. These findingss collectively support that our prompts faithfully reflect the professional intent of our taxonomy.

Additional visual examples

We apologize for this. We were initially restricted by the 100 MB NeurIPS supplementary size limit; the size of each video is quite large, for example a video generated by Wan 14B, is on average ~7 MB.

Thus, as requested by the reviewer, we have now created a fully anonymous web-page with 200+ videos, presenting side-by-side comparisons across models on SCINE Scripts and Visuals, and detailed pairwise comparisons with categories, questions, and scores.

However, we are unable to share this link with the reviewer, due to the NeurIPS rebuttal policy. Therefore, we have shared this fully anonymized link with the AC; we hope you are able to consider the web-page during your internal discussions. Thank you!

Organization of ranked_videos in Supplementary

Thank you for pointing this out. The ranked_videos directory in the supplementary material was not intended to contain paired examples generated from the same prompts. Rather, it provides representative samples of high- and low-scoring videos across different categories. For example, videos located at good_examples/all_models/Setup__Text Generation are videos that received high scores in the Text Generation node of our taxonomy.

To eliminate any ambiguity, we have now added both : (i) sets of videos produced from the same prompt and (ii) explicit pairwise comparisons to the anonymous link.

Scores of Human annotation videos

For each video in the Supplementary, we present the individual annotations and the reason of it being a positive or negative case.

Additional annotation scores for more videos across multiple categories are also available at the anonymous webpage link.

Visual examples justifying Taxonomy Categories

We apologize for this. The webpage link now contains visual examples justifying the categories introduced in our taxonomy.

Additionally, the taxonomy is grounded in well-established references from professional filmmaking and cinematography literature:

• A Shot in the Dark by Jay Holben offers a deep dive into lighting practice in film. It covers concepts such as light quality (hard vs. soft), light position, the use of practical lights on set, and the role of the exposure triangle in audience perception.
• Film Directing: Shot by Shot by Steven Katz outlines key visual storytelling principles, including props, makeup, shot sizes, and camera movements, which directly map to our Setup and Camera categories.
• Cinematography: Theory and Practice by Blain Brown discusses visual language elements such as lines, space, contrast, blur, and noise.
• The Visual Story by Bruce Block explores how depth, shapes story structure, and composition contribute to narrative meaning in film-making.
• The Tools of Screenwriting by Howard and Mabley analyzes how actions, dialogues, and emotions are intentionally crafted during the writing process, informing our Events taxonomy and its sub-nodes.

In addition to these references, we also conducted multiple iterative feedback rounds with professionals to validate the taxonomy’s relevance and completeness. We hope these revisions clarify both the motivation behind the taxonomy and the real-world grounding of each of its 76 categories.

Training process of VLM

While our training setup is simple, the design goal was not novelty, but practicality and scalability. Human evaluation is costly and does not scale to the thousands of generations required for rigorous benchmarking. Our lightweight VLM enables scalable, high quality evaluation with a single forward pass and outperforms much larger baselines.

Specifically, we fine-tune Qwen2.5-VL-7B as a Bradley-Terry–style evaluator, replacing the LM head with a regression head. Unlike traditional autoregressive models that require seq_len forward passes per evaluation, ours requires only 1, making it roughly 100× more efficient than a 72B model, while also being more accurate.

Our results show that even state of the art models perform poorly when used directly for evaluation, underscoring the need for task specific tuning. Additionally, this design enables downstream applications. For example, our evaluator can be plugged in as a learned critic during video model training, offering a path towards fine-grained, control optimization.

We thank the reviewer for their constructive feedback of our work. Please find our responses to your questions:

Clarification on Controllability and Visual Quality.

Thank you for raising this point. We considered this extensively as well. It's important to clarify that cinematic controls represent the "input" dimensions specific knobs or dials that professionals prioritize, while video quality represents the "output" fidelity of the model. Correctness in elements like human anatomy and physical laws is inherently assumed in real-world professional videos. Our initial evaluations indicated a strong correlation between control metrics and video quality. Specifically, we observed that human annotators implicitly factor in visual quality while rating for controls. For example, if a video shows an "interactive" action, but the "interaction" is not grounded in physical laws, humans inherently tend to rate the video low. To validate this empirically, we performed the following experiment, at scale :

1. Augmented each SCINE Scripts prompt with extra questions evaluating visual quality (VQ) and temporal smoothness (TS).
   • Example prompt:

     "In an airport during the day, passengers form a line, checking tickets as they prepare to board. Outside the large window, ground crew load suitcases onto a baggage cart, while another airplane taxis slowly across the tarmac in the distance."

   • VQ question: "How smooth and artifact‑free is the rendering of the airplane taxiing in the distance?"
   • TS question: "Is the motion of the ground crew loading suitcases and the taxiing airplane fluid and natural, with consistent transitions between frames?"
2. Collected independent ratings from annotators on all control, plus VQ and TS questions.
3. Computed per‑model averages and compared Control vs. VQ and TS, with results shown below :

This denotes a Spearman correlation of 0.93 (p < 1e-5), confirming that VQ and TS are implicitly embedded within control evaluations.

Although video quality and controllability are fundamentally different problems, our findings demonstrate that our primary metric for controllability is an effective proxy for overall model performance, as it implicitly factors in the visual quality of a generated video. However, as models evolve and video quality approaches a plateau, we anticipate that fine-grained controllability will increasingly become the decisive differentiator for professional applications. Our focus on controllability is thus a forward looking evaluation of this critical frontier.

Limitation to Single Shot Evaluation

We agree that filmmaking involves more than generating a single independent shot. That said, our current work specifically focuses on a single shot control because even a single shot involves coordinating multiple elements to translate a creative vision to screen.

We identify a critical gap in current literature: despite the growing capabilities of generative video models, a principled understanding of what constitutes a well-controlled single shot in a professional setting, remains underexplored. Our contribution lies in filling this gap by proposing taxonomies that define and evaluate these elements in a structured way. Further, we believe that if we cannot reliably control a single shot using video generative models, building coherent sequences or shots becomes an even greater challenge. As our evaluations show, current models struggle in setting up a single shot.

We agree that extending to multi-shot setups is an important future direction. Multi-shot setups will add new dimensionalities to our taxonomy, where two axes become relevant:

1. Intra-shot correctness
2. Inter-shot consistency

Our taxonomy already provides the foundation and structure for evaluating both. In a multi-shot setup, each node remains relevant while additional factors such as consistency will link identical nodes across shots.

Consistency of nodes such as character, style, story, and color become important in a multi-shot professional filmmaking setup; all of which are already defined in our taxonomies :

• Attributes such as Costume and Hair, associated with Character identity, are under the Setup → Subjects axis.
• Style and Color are captured under Set Design and Scene Texture, respectively.

Multiple baseline approaches can then be explored to measure inter-shot consistency, across different controls:

• Facial consistency: via facial‐embedding similarity (RetinaFace, FaceNet) [1–3]
• Costume consistency: via DINO feature similarity [4]
• Hair consistency: via perceptual (SSIM, PSNR) + semantic (CLIP-I) metrics [5]
• Color consistency: via sliced Wasserstein/histogram in CIELAB space + palette recoloring and transfer methods [6–8]
• Style consistency: via Contextual Style Descriptor (CSD) [9]

In conclusion, our current work concentrates on defining a structure of controls in a single shot setting; an understanding of which is still missing from the generative video literature. By formalising this space with our structured taxonomy, we lay a solid foundation for future research on multi‑shot filmmaking; our framework can be extended with additional dimensions such as "consistency", that are crucial when moving beyond single shots.

[1] Wang et al. "CharaConsist: Fine-Grained Consistent Character Generation."

[2] Serengil et al. "A Benchmark of Facial Recognition Pipelines and Co-Usability Performances of Modules"

[3] Schroff et al. "Facenet: A unified embedding for face recognition and clustering."

[4] He et al. "VTON 360: High-fidelity virtual try-on from any viewing direction."

[5] Zhang et al. "Stable-hair: Real-world hair transfer via diffusion model."

[6] He et al. "Multiscale sliced wasserstein distances as perceptual color difference measures."

[7] Liu et al. "Temporally consistent video colorization with deep feature propagation and self-regularization learning."

[8] Chang et al. "Palette-based photo recoloring."

[9] Somepalli et al. "Measuring style similarity in diffusion models."

T2V and its scope in professional applications

We use text as our input modality as it is the most common source of control across video models today. Further, a single text prompt can combine coarse ideas ("dolly zoom at dusk") with exact parameters (35 mm, 4000K). T2V models enables us to cover the breadth of the entire taxonomy in a single, unified workflow. Furthermore, our control taxonomy is input‑agnostic; it defines what a creator needs to steer, regardless of how those controls are fed in.

While T2V models are not the only tools in a professional workflow, they have largely been integrated into creative pipelines. A recent report titled, "Darren Aronofsky Launches AI-Driven Storytelling Venture, As Google Unveils New Gen AI Video Tool" - states that T2V models have been widely used by highly creative professionals. Furthemore, the "AI in Content Creation 2025" report by Wondercraft, states that the rise of T2V workflows is opening new doors; platforms that allow users to start with a written script and quickly generate narrated videos or animated explainers are in high demand. A recent film, titled "We Tested Google Veo and Runway to Create This AI Film" by the Wall Street Journal describes using text prompts to create specific shots. Short films titled, "THE CLEANER" and "SWAT", have been created entirely by using text prompts as input.

In conclusion, T2V models allow complete traversal of our control taxonomy and align with professional pipelines. Evaluating through this lens therefore covers the taxonomy’s full scope without constraining future extensions to other input modalities.

Extensions to I2V

Yes, our evaluation framework can be extended to I2V pipelines. Below, we propose and perform a simple experiment to validate the same.

1. Reference Image: Since reference images have widely been used to control character appearance, we base our current experiment on them.
2. Build Prompts for respective models :

• T2I prompt – contains only the character description, conditioned on (Setup -> Subjects) from our taxonomy. e.g. "Male punk with a neon-green mohawk (Subjects -> Hair), nose ring, tattoos (Subjects -> Accessories), black tank-top (Subjects -> Costume), arms crossed."
• I2V prompt – Adds more controls from our taxonomies. "A punk man leans forward (Events) while filmed from a static low-angle shot (Camera) under warm light (Lighting)."

3. Evaluate :

• Control: "Is the clip low-angle while the punk leans forward under warm light?"
• Character Consistency: "Do the mohawk, nose ring and tattoos remain identical to the reference image?"

We create 20 prompts and generate reference images with Stable Diffusion 3.5 Large and generate videos with two I2V models - Hunyuan and Wan-13B - and rate the models on consistency and control. We present their win rates :

As shown, our taxonomy and evaluation framework are inherently flexible and can be naturally adapted to additional modalities.

Space below Figure 4.

Thank you for the suggestion; we will improve our presentation in the camera-ready version.

We thank the reviewer for their constructive feedback of our work. Please find below our responses to your questions:

Generalization to unseen videos

Thank you for your suggestion. We have now evaluated our VLM‑based evaluator on previously unseen data from Veo 3 Fast, a model released after our submission.

We sample 27 Veo 3 Fast Videos (restricted due to high rate limits), evaluating a total of 100 questions, ensuring equal distribution across categories. We follow a similar setting to our previous experiments and use human annotation as the ground truth. Below we present VLM accuracy, on unseen Veo 3 Fast videos :

As shown above, our model still outperforms existing baselines and generalizes well to out-of-distribution data not seen during training.

Open-Source commitment

We are fully committed to open-sourcing the data and models associated with our work. Post legal approvals and additional safety checks, we will release all the artifacts associated with Stable Cinemetrics.

Quality issues in Supplementary

Thank you for this question. Our evaluation framework assesses each cinematic dimension independently using our 4 taxonomies, Camera, Lighting, Setup, and Events. And a generated video can be good in one aspect, while being poor in another.

The presented example, ltx_00569.mp4 is considered a good example only for Camera Movement, which is expected to be static (as mentioned in the prompt : A sun-dappled camping ground framed by towering pine trees....The camera static on a tripod, shallow depth of field isolating....mirroring the vertical trunks rising behind them | full prompt in metadata.csv), in this case.

As shown above by the annotator ratings, and as indeed seen in the generated video (present in Camera__Trajectory__Camera Movement folder in the good_examples folder of the Supplementary), the video correctly depicts a video generated with a static camera movement. This is the only reason the video is listed as a good example, but only for Camera Movement.

Like you correctly pointed out, the generated video has issues related to the movement of the axe and its corresponding effects. These are evaluated and captured under the Events taxonomy, in this case. Below, are the evaluation scores :

These low scores align with your observation that the physical realism of the axe swing is inadequate.

In conclusion,

• ltx_00569.mp4 is only a positive example for Camera‑Movement (Static), in this case.
• Under Events, the same video would be a negative example, and is treated as such during evaluation.

Concerns about visual examples

We apologize for this. We were initially restricted by the 100 MB NeurIPS supplementary size limit; the size of each video is quite large, for example a video generated by Wan 14B, is on average ~7 MB.

Thus, as requested by the reviewer, we have now created a fully anonymous web-page with 200+ videos, presenting side-by-side comparisons across models on SCINE Scripts and Visuals, and detailed pairwise comparisons with categories, questions, and scores.

However, we are unable to share this link with the reviewer, due to the NeurIPS rebuttal policy. Therefore, we have shared this fully anonymized link with the AC; we hope you are able to consider the web-page during your internal discussions. Thank you!

I have read the author's rebuttal, most of my concerns are solved, and I will raise the score.

We thank the reviewer for their constructive feedback of our work. Please find below our responses to your questions:

Quantitative Metrics

We appreciate the reviewer’s suggestion to supplement our human and VLM‑based assessments with quantitative, objective driven metrics.

Below we first explain why such metrics remain challenging to be directly applied to state‑of‑the‑art video‑generation systems :

• Out‑of‑distribution content — Monocular depth and camera‑pose estimators are trained and evaluated almost exclusively on real‑world data (e.g. KITTI, RealEstate10K). Their behaviour on synthetic videos has not been systematically benchmarked.
• Poor alignment with human annotators — Widely‑used automated video metrics (e.g. FVD) have been shown to poorly correlate with human judgments [1], prompting a shift towards human and VLM‑assisted evaluations [2,3].

To unify evaluations of all of our 76 control nodes, and because professionals are the ultimate judge of the video's quality, we treat professional human raters as the gold standard and train a VLM proxy to approximate them.

In response to the reviewer’s suggestion, we carried out baseline experiments that demonstrate how geometry‑based computer‑vision metrics can serve as an additional, objective signal for future evaluations :

Camera Pose and Depth Estimation

We generate two 81‑frame videos (Wan 1 B vs. Wan 13 B) for the below prompt :

"A figure walks along the beach shore at sunset as the sun hangs low, ... The camera pushes in smoothly on a dolly from an eye-level perspective, ... drama in sharp relief."

Using CUT3R [4] we recover per‑frame camera parameters and depth maps, then derive the baseline metrics below (ideal dolly path = best‑fit straight line through the 81 camera centres) :

Insight: Wan 1 B adheres more closely to the intended dolly motion and maintains a tighter depth distribution, whereas 13 B introduces significant pitch drift.

Global and Temporal Contrast

Across 9 models, we compute standard contrast metrics to 50 videos per model, each video generated from prompts activating the Setup → Scene → Texture → Contrast branch of our taxonomy.-

In summary, our current work benchmarks model capabilities through expert human evaluations, which remain the gold standard, and trains a vision–language model (VLM) to approximate them. Additionally, our initial experiments reveal potential directions for grounding geometry-based metrics in the evaluation of video generative models.

[1] Ge Songwei et al., "On the Content Bias in Fréchet Video Distance"

[2] Wu, Jay Zhangjie, et al. "Towards a better metric for text-to-video generation."

[3] Bansal, Hritik, et al. "VideoPhy 2: Challenging Action-Centric Physical Commonsense Evaluation of Video Generation"

[4] Wang, Qianqian et al., "CUT3R: Continuous 3D Perception Model with Persistent State"

Extension to multi-shot sequences

Evaluating multi-shot sequences are an important future direction. Multi-shot setups will add new dimensionalities to our taxonomy, where two axes become relevant:

1. Intra-shot correctness : the per‑shot controls defined and evaluated in our current work;
2. Inter-shot consistency : coherency of shots across sequences.

Our taxonomy already provides the foundation and structure for evaluating both. In a multi-shot setup, each node remains relevant while additional factors such as consistency will link identical nodes across shots. Consistency of nodes such as character, style, story, and color become important in a multi-shot professional filmmaking setup; all of which are already defined in our taxonomies :

• Attributes such as Costume and Hair, associated with character identity, are under the Setup → Subjects axis.
• Style and Color are captured under Set Design and Scene Texture, respectively.

Multiple baseline approaches can then be adopted to measure inter-shot consistency, across different control elements:

• Facial consistency: calculated using facial embedding similarity [1] computed with models like RetinaFace [2] and FaceNet [3].
• Costume consistency: measured using visual feature similarity methods such as DINO embeddings as shown in [4].
• Hair consistency: evaluated using perceptual and semantic metrics e.g., SSIM, and CLIP-I, as shown in [5].
• Color consistency: evaluated using sliced Wasserstein distances [6] or histogram comparisons [7] in the CIELAB color space. Discrete color palettes can also be compared using palette-based photo recoloring and color transfer methods [8].
• Style consistency: measured using the Contextual Style Descriptor (CSD) [9], which evaluates style similarity between reference and target sets.

In conclusion, our current work concentrates on defining a structure of controls in a single shot setting; an understanding of which is still missing from the generative video literature. By formalising this space with our structured taxonomy, we lay a solid foundation for future research on multi‑shot filmmaking; our framework can be extended with additional dimensions such as "consistency", that are crucial when moving beyond single shots.

[1] Wang, Mengyu, et al. "CharaConsist: Fine-Grained Consistent Character Generation."

[2] Serengil, Sefik et al. "A Benchmark of Facial Recognition Pipelines and Co-Usability Performances of Modules"

[3] Schroff, Florian, et al. "Facenet: A unified embedding for face recognition and clustering."

[4] He, Zijian, et al. "VTON 360: High-fidelity virtual try-on from any viewing direction."

[5] Zhang, Yuxuan, et al. "Stable-hair: Real-world hair transfer via diffusion model."

[6] He, Jiaqi, et al. "Multiscale sliced wasserstein distances as perceptual color difference measures."

[7] Liu, Yihao, et al. "Temporally consistent video colorization with deep feature propagation and self-regularization learning."

[8] Chang, Huiwen, et al. "Palette-based photo recoloring."

[9] Somepalli, Gowthami, et al. "Measuring style similarity in diffusion models."

Failure modes of VLM Evaluator

Please find the nodes on which the VLM has the strongest and weakest performance, respectively. Score indicate the accuracy of the VLM's choice with most common preference among human annotators.

Top 10 Nodes

Bottom 10 Nodes

Computational cost of VLM

Our VLM, trained as a Bradley-Terry style evaluator starting from Qwen2.5-VL-7B adds a regression head in place of a LM head. It needs just one forward pass instead of  seq_len passes for a traditional autoregressive generative model. This makes our model roughly 100 × than the 72 B baseline, while also offering better performance (72.36 % vs 62.50 %) since it's trained for the objective to work like a preference model.

Additional Visual Results

We apologize for this. We were initially restricted by the 100 MB NeurIPS supplementary size limit; the size of each video is quite large, for example a video generated by Wan 14B, is on average ~7 MB.

Thus, as requested by the reviewer, we have now created a fully anonymous web-page with 200+ videos, presenting side-by-side comparisons across models on SCINE Scripts and Visuals, and detailed pairwise comparisons with categories, questions, and scores.

However, we are unable to share this link with the reviewer, due to the NeurIPS rebuttal policy. Therefore, we have shared this fully anonymized link with the AC; we hope you are able to consider the web-page during your internal discussions. Thank you!