ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models

Thank you for the support and for recognizing the novelty and impact of our work. We have carefully reviewed your questions and provide point-by-point responses below:

Q1: Was Inter-Annotator Agreement (IAA) computed?

While we did not compute a formal inter-annotator agreement score, we employed a rigorous multi-stage annotation and verification protocol to ensure quality and consistency (as described in Section 3.2). All annotations were evenly divided into batches, each of which was audited by experts. A batch was only accepted if it achieved an accuracy above 90%; otherwise, it was sent back for re-annotation. Whenever a batch was rejected, annotators and experts engaged in discussion to reach a consensus. We believe this expert-in-the-loop approach offers stronger guarantees of annotation quality than relying solely on IAA scores.

Q2: Is it possible that clips from the same movie appear in both training and test sets?

We confirm that there are no duplicate samples between the training and test sets. To ensure this, we compute CLIP features for all samples and remove any training sample with cosine similarity greater than 0.95 (following [1,2,3]) to a test sample.

To clarify, our benchmark aims to evaluate fine-grained cinematic understanding, which often requires distinguishing between visually similar but cinematographically distinct shots (e.g., different shot sizes or compositions from the same scene). Therefore, we allow different shots from the same movie to appear in both splits, as this better reflects real-world challenges in cinematic understanding.

We will clarify these points in the revised paper.

Q3: What does "low-quality sample" mean?

The "low-quality samples" mentioned in Line 125–126 refer to those filtered out using the LAION aesthetic predictor. These are samples with low aesthetic scores, often characterized by poor composition, motion blur, or chaotic visual content. Such samples typically lack well-defined cinematic attributes and were excluded to maintain the overall quality and clarity of our dataset.

Minor comment on references

Thank you for your careful proofreading. We will merge the duplicate references (17 and 18) accordingly.

References

[1] Gadre S Y, Ilharco G, Fang A, et al. Datacomp: In search of the next generation of multimodal datasets[J]. Advances in Neural Information Processing Systems, 2023, 36: 27092-27112.

[2] Caron M, Iscen A, Fathi A, et al. A generative approach for wikipedia-scale visual entity recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 17313-17322.

[3] Caron M, Fathi A, Schmid C, et al. Web-scale visual entity recognition: An LLM-driven data approach[J]. Advances in Neural Information Processing Systems, 2024, 37: 34533-34560.

Please let us know if there are any remaining questions or suggestions, we sincerely value your feedback.

Thank you for the careful, constructive, and detailed review. We sincerely appreciate your recognition of the systematic methodology and practical value of our work. We have carefully reviewed your concerns and provide point-by-point responses below:

Q1: On dataset availability, construction process, and train-test split integrity.

Open-sourcing Datasets

We confirm that we will publicly release all resources used in this paper, including the datasets ShotBench and ShotQA, model checkpoints, training hyperparameters, and evaluation code. All image and video samples in our datasets are obtained from publicly accessible sources, such as YouTube (for videos) and professional cinematography platforms (for images), and each sample is directly downloadable via its corresponding URL. To ensure transparency and long-term reproducibility, we will also release detailed metadata for each sample, including IMDb IDs, timestamps, and QA annotations, allowing users to trace or reconstruct the data even if source URLs become unavailable.

ShotQA Construction

ShotQA is constructed in a similar manner to ShotBench (Section 3.2), including:

Data filtering: We apply quality filtering based on aesthetic scoring, content safety checks, and automated shot segmentation.

Annotator training: Before labeling, annotators are trained using curated cinematography tutorials and domain-specific guidelines.

QA annotation: Question–answer pairs are generated using structured templates and professional metadata.

Expert verification: Annotations undergo multi-round expert review to ensure consistency and quality.

Except that only video samples are totally manually annotated and verified by trained annotators and experts. For large-scale image annotations sourced from expert cinematography websites, we conducted random sampling checks and found their quality adequate for training use.

Dataset Splitting

We adopt a two-stage filtering strategy to ensure no overlap between training and evaluation sets:

1. We first remove duplicate samples based on IMDb IDs and timestamp as a coarse-level filtering step.
2. Then, we extract CLIP features for all samples and exclude samples from the training set whose feature has a cosine similarity greater than 0.95 with any sample in ShotBench.

This strategy ensures no duplicate or near-duplicate samples are shared across splits. The 0.95 threshold is supported by existing works[1,2,3] and our own visual inspection. Additionally, we find that this choice allows us to retain samples that are visually similar yet differ in key cinematic attributes (usually adjacent shots within one scene), such as shot size or composition, which are central to our task. This makes ShotBench more challenging and realistic.

The GRPO sub-dataset consists of a combination of mid-difficulty samples and uniformly sampled QA pairs across all eight dimensions. We identify mid-difficulty samples by prompting Qwen2.5-VL-3B to answer a subset of the data multiple times and selecting those for which both correct and incorrect answers are observed across different runs. The remaining QA pairs were uniformly sampled again and used for SFT.

We will add these details to Appendix in the revised version.

Q2: On copyright and data licensing concerns.

All movie clips used in our dataset are sourced from publicly available YouTube videos, and we will provide:

• Downloadable clips (in processed form)
• Corresponding YouTube URLs
• Metadata (title, timestamp, etc.)
• Scripts for downloading and processing

This ensures full transparency and reproducibility. We will also add a clarification in the paper to address potential copyright concerns directly and explicitly.

Q3: No comparison with existing cinematography understanding methods like SGNet.

We thank the reviewer for this suggestion. Unfortunately, HunyuanVideo’s camera movement classifier and SGNet are not open-sourced. Moreover, these methods use limited and inconsistent taxonomies that do not align with standard cinematography principles.

To ensure fair comparison, we instead evaluated CineScale [4], an earlier open-source model for shot size classification. Note that CineScale only supports three coarse-grained categories (close-up, medium shot, wide shot). On the ShotBench shot size dimension, we tested CineScale in two settings:

The overall performance is much lower than most existing VLMs (regard that Shot-VL achieves 82.5 accuracy in this dimension), which highlights both the difficulty of ShotBench and the limited transferability of prior shot classification methods.

Q4: How are multi-keyword answers evaluated?

All ShotBench questions are designed as four-option single-choice questions. For questions involving multiple keywords (e.g., lighting condition), each option contains the same number of keywords to maintain balance. Only the correct option includes all the correct keywords, while the distractors may contain a mixture of correct and incorrect keywords to enhance the challenge and maintain fairness. A prediction is considered correct only when the model selects the fully correct option. This design follows existing works[5,6,7] and provides a clear and consistent evaluation protocol.

We will explicitly state this in the Evaluation section (Sec 3.3).

Q5: Have you tried applying your approach to larger models?

Yes, we have fine-tuned Qwen2.5-VL-7B-Instruct following the same two-stage training setting (SFT + GRPO), producing ShotVL-7B, it also achieves consistent improvements on ShotBench (53.1 $\rightarrow$ 68.0) and outperforms our previous ShotVL-3B (65.1).

We are also conducting additional experiments on InternVL3-8B, and will report these results in the next version of the paper.

Formatting and grammar issues:

Thank you for the careful and helpful proofreading. We will comprehensively review the paper to correct all identified issues.

References

[1] Gadre S Y, Ilharco G, Fang A, et al. Datacomp: In search of the next generation of multimodal datasets[J]. Advances in Neural Information Processing Systems, 2023, 36: 27092-27112.

[2] Caron M, Iscen A, Fathi A, et al. A generative approach for wikipedia-scale visual entity recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 17313-17322.

[3] Caron M, Fathi A, Schmid C, et al. Web-scale visual entity recognition: An LLM-driven data approach[J]. Advances in Neural Information Processing Systems, 2024, 37: 34533-34560.

[4] Savardi M, Kovács A B, Signoroni A, et al. CineScale: A dataset of cinematic shot scale in movies[J]. Data in Brief, 2021, 36: 107002.

[5] Chow W, Mao J, Li B, et al. Physbench: Benchmarking and enhancing vision-language models for physical world understanding[J]. arXiv preprint arXiv:2501.16411, 2025.

[6] Fu C, Dai Y, Luo Y, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 24108-24118.

[7] Yue X, Ni Y, Zhang K, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 9556-9567.

Please let us know if there are any remaining questions or suggestions, we sincerely value your feedback.

Thank you for the detailed and constructive review, as well as for the positive evaluation of our contributions. We have carefully reviewed the concerns and provide point-by-point responses below:

Q1: The eight core dimensions are primarily based on Wikipedia, which may not be the most authoritative source.

We appreciate the reviewer’s comment. To clarify, the Wikipedia page we referenced also cites a number of authoritative cinematography textbooks (e.g., The Focal Dictionary of Photographic Technologies [6], The Art Direction Handbook for Film & Television [7]), and our intention is to provide readers with a broad and accessible entry point into the topic. Our definitions are likewise grounded in established academic and professional sources. As detailed in Table 9, annotators were trained using materials from industry platforms like StudioBinder, and our taxonomy aligns with widely adopted cinematography textbooks, including Blain Brown’s Cinematography: Theory and Practice [1] and David Bordwell’s Film Art: An Introduction [2]. We will clarify this in the revised manuscript to enhance the credibility of our benchmark.

Q2: The dataset of ShotBench is not considered large-scale.

We apologize if the distinction between our datasets was unclear. To clarify:

• ShotQA (~70k QA pairs) is our large-scale dataset used for training.
• ShotBench (3.6k QA pairs) is our diagnostic evaluation benchmark.

While ShotBench is compact by design, this reflects the typical nature of benchmarks, which prioritize evaluation efficiency and diagnostic precision over scale. ShotBench is fully annotated and verified by trained annotators and experts, making large-scale expansion costly but ensuring high reliability. Despite its size, it provides diverse and comprehensive coverage across the eight core dimensions of cinematography and associated terminology (as detailed in Table 7), enabling fine-grained model analysis. Moreover, its size is comparable to or larger than several widely used VLM benchmarks (e.g., MMVU [3], Video-MME [4]).

We will clarify these roles and the rationale behind our design choices more explicitly in the revised manuscript.

Q3: Lack of detail on the data construction process

To clarify, we have provided a detailed data construction process in Section 3.2, including:

Data curation&filtering: We apply datacuration and filtering based on aesthetic scoring, content safety checks, and automated shot segmentation.

Annotator training: Before labeling, annotators are trained using curated cinematography tutorials and domain-specific guidelines.

QA annotation: Question–answer pairs are generated using structured templates and professional annotations.

Expert verification: Annotations undergo multi-round expert review to ensure consistency and quality.

and representative reference materials in Table 9. In the revised version, we will expand our Appendix to include more detailed information on the data construction process, including the aesthetic score threshold used for sample filtering (3.0), the number of annotators involved (20), additional materials and documentation used during annotator training, and more details on the expert review and validation procedures.

Q4: Fairness of baseline comparison without fine-tuning and missing DeepSeek-VL models.

To clarify, our goal is to evaluate general-purpose VLMs under zero-shot settings to reflect their out-of-the-box understanding of cinematography. While fine-tuning all 24 baselines is infeasible, we agree it is valuable to test whether ShotQA improves other models. We have fine-tuned Qwen2.5-VL-7B on ShotQA and observed consistent gains (53.1 $\rightarrow$​ 68.0). Additionally, we evaluated the performance of DeepSeek-VL[5] on ShotBench, and we found that DeepSeek-VL models perform poorly on ShotBench (35.0 for DeepSeek-VL2-tiny and 36.0 for DeepSeek-VL2-small).

We will include these results in the next revision of the paper to provide a more comprehensive and fair comparison.

Q5: On the attribution of failure cases in Figure 5.

Thank you for highlighting this ambiguity. All failure cases shown in Figure 5 were produced by GPT-4o, the strongest proprietary model in our evaluation. We will revise the caption and discussion to clearly indicate the source model. Additionally, we will include failure cases from other representative models to offer a more balanced view of common challenges across models.

Q6: Language and grammar issues.

We sincerely appreciate your careful reading. We will thoroughly proofread the manuscript and correct all instances of awkward phrasing, grammatical issues, and unclear expressions to improve overall clarity and professionalism.

References

[1] Brown B. Cinematography: theory and practice: image making for cinematographers and directors[M]. Routledge, 2016.

[2] Bordwell D, Thompson K, Smith J. Film art: An introduction[M]. New York: McGraw-Hill, 2004.

[3] Zhao Y, Zhang H, Xie L, et al. Mmvu: Measuring expert-level multi-discipline video understanding[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 8475-8489.

[4] Fu C, Dai Y, Luo Y, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 24108-24118.

[5] Wu Z, Chen X, Pan Z, et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding[J]. arXiv preprint arXiv:2412.10302, 2024.

[6] Spencer D A. The Focal dictionary of photographic technologies[J]. London: Focal Press, 1973.

[7] Rizzo M. The art direction handbook for film & television[M]. Routledge, 2014.

Thank you for the follow-up question and your interest in the data quality aspect of our work. To clarify, ShotBench refers to our evaluation set and it's compact by design. In contrast, ShotQA is our training dataset (∼70k QA pairs). We agree that data quality outweighs scale. During data collection, we found many online images and videos mislabeled with noisy or inaccurate cinematography terms. Our early experiments showed that scaling with such data did not improve performance. We believe similar low-quality supervision exists in current MLLM training corpora, which is a key factor behind their limited performance in cinematic language understanding.

Dear Reviewer K6tX,

Thank you again for your detailed review and for engaging with us after the rebuttal.

We’d greatly appreciate knowing whether our responses have addressed all of your concerns. If any concerns remain, we’d be happy to address them.

We sincerely value your time and feedback.

Thank you for your positive assessment and for highlighting the novelty and extensive experiments of our work. We sincerely appreciate your valuable feedback and constructive suggestions. We have carefully reviewed your concerns and provide point-by-point responses below:

Q1: The multiple-choice (MCQ) format limits the evaluation of generative capabilities.

We agree that open-ended evaluation is an important direction for cinematic understanding. We adopt a multiple-choice format in ShotBench to ensure objectivity, scalability, and reproducibility, which are key qualities for building a reliable benchmark in a new domain. This design follows the common practice in recent representive VLM benchmarks (e.g., Video-MME [1], PhysBench [2], MMBench [3]) and enables fair comparison across models. While it does not capture generative reasoning, we view it as a necessary first step and will clarify this limitation in the revised version.

Q2: Strengthen the claims to show that ShotQA and GRPO training strategies is model-agnostic and may generalize across different VLM architectures of various sizes.

We appreciate the reviewer’s suggestion. In response, we have conducted additional experiments using a stronger backbone, Qwen2.5-VL-7B-Instruct, following the same two-stage training setting (SFT + GRPO). This larger model, ShotVL-7B, achieves better overall performance (53.1 $\rightarrow$ 68.0) than ShotVL-3B (65.1) on ShotBench.

We will include these results, along with more extended comparisons and analysis, in the next revision of the paper.

References

[1] Fu C, Dai Y, Luo Y, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 24108-24118.

[2] Chow W, Mao J, Li B, et al. Physbench: Benchmarking and enhancing vision-language models for physical world understanding[J]. arXiv preprint arXiv:2501.16411, 2025.

[3] Liu Y, Duan H, Zhang Y, et al. Mmbench: Is your multi-modal model an all-around player?[C]//European conference on computer vision. Cham: Springer Nature Switzerland, 2024: 216-233.

Please let us know if there are any remaining questions or suggestions, we sincerely value your feedback.