TUMTraf VideoQA: Dataset and Benchmark for Unified Spatio-Temporal Video Understanding in Traffic Scenes

Dear Reviewer,

We appreciate your valuable feedbacks and address your concerns as follows.

Q1: Clarification on whether TraffiX-Qwen’s performance gain stems from more input frames.

A1: As open-source VideoQA models often adopt model-specific frame sampling strategies, which are tightly coupled with architecture and training, some models (e.g., LLaVA-OneVision, Video-LLaMA2) are not designed to handle longer frame sequences. Hence, we used different frame settings that were in line with their original paper and codebase. We also fully agree that comparisons under a unified frame setting would strengthen the results. Therefore, we conduct extra experiments with 101 frame inputs for the models used in our paper. (Qwen2-VL, whose model structure imposes constraints on input frame numbers, is adapted to 96 frames.)

Results show that: 1. LLaVA-OneVision and Video-LLaMA2 perform far worse with 101 frames, likely due to the lack of long-range temporal modeling. 2. Qwen2-VL shows improved performance with more frame inputs, suggesting that additional temporal information can indeed be beneficial. This also highlights Qwen2-VL’s stronger temporal reasoning capability.

Importantly, even under the same frame input setting, TraffiX-Qwen still consistently outperforms all baselines, confirming that the performance gain stems from multiple aspects, not just access to more temporal information. These results will be added to the revised paper.

Q2: The role of YOLOv8, ByteTrack, etc in the pipeline.

A2: We use 2D detectors and trackers only during the TraffiX-VideoQA dataset construction to generate meta-information aligned with human annotations, improving quality and consistency. They are not used in TraffiX-Qwen’s training or inference. Our baseline is a fully end-to-end VLM that processes raw videos and generates answers directly, without external modules. This design simplifies the pipeline and promotes research toward integrated video understanding in the traffic domain.

Q3: Statistics supporting the claim of dataset diversity.

A3: We add 3 figures illustrating dataset distribution at: https://imgur.com/a/eoZXJEb.

The figures statistically support the dataset’s diversity. We group scenes in our dataset into 3 key types: highways (rural), urban intersections (city), and country roads (rural/urban). They show that the dataset aligns well with real-world traffic distributions. The figures will be added in the revised version.

Q4: Missing information on model efficiency and computational cost.

A4: We provide the training and inference details in the paper:

Inference time is measured as the average to process a 10s–1min video with autoregressive decoding on 1*A100 GPU (no acceleration). We believe further optimization (e.g., quantization, pruning, distillation) is a promising direction for improving deployment efficiency in traffic monitoring. We will include this table and discussion in the revised version.

Response to other suggestions:

1. We have included an initial discussion regarding how sampling strategy impacts spatiotemporal reasoning in Sec 5. We will expand this with new experimental results and discussions.
2. Qualitative results for ST-OG and V-ROC have been added to Appendix B.5, B.6, covering both successful and failure cases. For the MC-QA task, we will provide extra qualitative visualizations and analyses.
3. Initial analysis of frame count effects on TraffiX-Qwen is provided in Appendix B.1. We will expand this with an extra discussion of how frame sampling impacts temporal reasoning capabilities.
4. We agree that including extra AD benchmarks (e.g., Ego4D etc.) brings valuable context and will add them.
5. We will add more details on model complexity, and potential acceleration techniques to support practical deployment.
6. We agree that using fixed camera views may introduce distributional bias in traffic patterns and will include a discussion on this in limitation.
7. We will improve Fig. 6. refine the definition of spatiotemporal grounding, and emphasize the role of human verification during annotation process.

Dear Reviewer,

Thank you very much for your positive feedback and valuable reviews. Our responses to your comments are detailed below:

Q1. There is a need to incorporate the recent work [1] for a more comprehensive analysis.

A1. Thank you for pointing out this important related work. We have discussed a detailed comparison with it in our response to Q3 below, and we will also include this reference in our final manuscript to improve the completeness and introduce our work more clearly.

Q2. Why only focus on multiple-choice video QA, referred object captioning, and spatiotemporal object grounding?

A2. Thank you for the insightful suggestion. We fully agree that video understanding covers many valuable tasks beyond our current focus, such as action recognition, event localization, video captioning, and anomaly detection. Following the survey [2], we broadly categorize video understanding tasks based on the granularity of required information into abstract, temporal, and spatiotemporal levels.

In traffic scenarios, high-level events are typically composed of interactions among traffic participants. As such, fine-grained spatiotemporal understanding and reasoning serve as a crucial foundation for interpreting complex traffic events. Based on this, our benchmark focuses first on spatiotemporal understanding through three core tasks in traffic scene comprehension: multiple-choice video QA, spatiotemporal object grounding, and referred object captioning. We sincerely appreciate your suggestion and agree that using VLMs for higher-level semantic tasks such as traffic event recognition is a promising direction. We will add this to the Future Work section of our paper.

Q3. Compared to [1], what are the strengths of the paper?

A3. Thank you for raising this important question. We have reviewed this work and highlighted the following strengths of our dataset and benchmark compared to the work [1] as follows:

1. Data Source and Data Novelty.

Ref [1] is built upon existing datasets, providing additional textual caption annotations without contributing new video data or organizing raw footage. Moreover, most of the videos used in [1] are sourced from the internet. In contrast, our dataset is collected from real-world traffic scenes, ranging over 2 years using multiple intelligent infrastructure systems installed from over 20 diverse perspectives. From over 1,000 hours of raw video data, we curate our benchmark through a semi-automatic data annotation process. Compared to internet videos, our data more accurately reflects the distribution and characteristics of real-world traffic scenarios, offering higher data novelty and practical value. We include figures to support this point as suggested by Reviewer mL1S, A3: https://imgur.com/a/eoZXJEb

2. Inherent Limitation for Fine-Grained Spatiotemporal Tasks.

Ref [1] focuses on video understanding tasks that require abstract semantic information, such as high-level event descriptions. However, such object reference is not well-suited for fine-grained spatiotemporal tasks in traffic scenes. This limitation arises from several factors: the inherent ambiguity of descriptive natural language expressions, the modality gap between visual and linguistic representations, and the high visual similarity of traffic participants in surveillance footage.

This limitation is also clearly illustrated in Ref [1], Figure 1 (top video). The sentence query states: “A black car drove past the corner, and a gray car followed closely behind it.” However, the video scene contains multiple black cars and multiple gray cars, making the reference inherently ambiguous. Such ambiguity prevents the benchmark from supporting more fine-grained VideoQA tasks, particularly those requiring precise object grounding and temporal reasoning.

3. Unified Benchmark with Structured Object Representation.

Our benchmark unifies three fine-grained tasks, i.e., multi-choice QA, referred object captioning, and spatiotemporal object grounding, via a tuple-based spatiotemporal object expression. This design enables benchmarks on more complex tasks, such as relative spatial reasoning, which are critical in the traffic domain but missing from [1].

4. Unified Model for VideoQA Benchmark.

In [1], each proposed task is evaluated separately using existing models like CoCap or UEDVC. While valuable, these approaches are separate for each task and do not support unified understanding. In contrast, we propose TraffiX-Qwen, a unified VLM-based model trained with multi-task learning across all three tasks. This contributes a solid baseline toward generalizable traffic visual foundation models.

[1] Towards Surveillance Video-and-Language Understanding: New Dataset, Baselines, and Challenges. CVPR 2024

[2] Video Understanding with Large Language Models: A Survey. arXiv:2312.17432, 2024.

Dear Reviewer,

Thank you very much for your positive feedback and valuable comments. Below, we provide point-by-point responses to the concerns raised.

Q1: What are the differences between the proposed multi-choice QA and existing VQA work, aside from the camera view?

A1: Thank you for the thoughtful question. Beyond the camera perspective, our work differs from previous VQA works in several key aspects:

1. The object reference expression in existing video QA works cannot be directly applied to the domain of spatiotemporal traffic scene understanding. We also argue it in reply to Reviewer z4TA, A3-2. This is due to the inherent ambiguity of natural language, the modality gap between visual and linguistic representations, and the need for accurate and unique object referring in traffic scenarios. Purely language-based descriptions often fail to achieve precise cross-modal object association. Hence, our work introduces a structured tuple-based object representation, enabling accurate and interpretable fine-grained analysis of traffic videos.

2. Our benchmark is specifically designed for unified intelligent traffic monitoring and scene understanding, which is currently missing in existing works. We aim to bridge this gap by providing a dataset and benchmark tailored to traffic scene analysis needs. Our dataset reflects real-world traffic distributions, and we include additional figures to support this point in response to Reviewer mL1S, A3: https://imgur.com/a/eoZXJEb

3. We unify three previously disjoint tasks in this domain, i.e., multiple-choice video QA, referred object captioning, and spatiotemporal object grounding, into a single benchmark. This positions our work beyond traditional VQA, offering a more comprehensive and domain-relevant challenge.

Q2: Why investigate different token sampling strategies when they show similar performance across tasks?

A2: Thank you for raising this question. In traffic scenes, particularly in the context of intelligent traffic monitoring and scene understanding, a key domain-specific characteristic is that the visual background tends to remain relatively static within a video.

For existing VLMs, efficient visual representation is critical to overall performance. Therefore, exploring different visual token sampling strategies that remove redundant information while preserving key content is particularly important in this domain. Our study investigates whether tailored visual token strategies can leverage this static background nature to improve efficiency and performance. This motivation is also consistent with some prior works for roadside visual-centric tasks [3][4], where techniques are applied to suppress background redundancy and enhance downstream task performance.

Q3: Why emphasize “traffic anomalies” in the abstract and contributions without dedicated benchmark analysis?

A3: Our intention was to reflect real-world distribution, where traffic anomalies (e.g., accidents) are rare but critical cases. While our benchmark does not focus solely on anomaly analysis, we still provide high-level annotations indicating whether each video contains anomalies. This supports valuable future research focused on anomaly detection and emphasizes the diversity and realism of our dataset. We will revise the content to clarify this.

Q4: Define the difficulty levels of QA questions (e.g., single-hop vs. multi-hop) in the supplementary material.

A4: Thank you for highlighting this point. We currently include a subset of QA templates in Supplementary Material C.2. To improve clarity, we will introduce an additional subsection that clearly defines and categorizes the difficulty levels of QA questions.

Q5: Clarify why there are multiple bars for a class of questions in Figure 4(a).

A5: In Figure 4(a), we plot the distribution of question word counts for each question task/type. Since questions within the same task type can vary significantly in length and structure, we chose to visualize this distribution to better reflect the linguistic diversity of the dataset, rather than simply reporting the total number of questions per type. We will revise the caption to make this clearer.

Q6: L55 - The statement justification for the importance of QA in traffic scenes is unclear.

A6: Thank you for pointing out the unclear phrasing. We will revise this part to emphasize the motivation and importance of introducing LLM/VLM for the intelligent traffic scene understanding domain and the development of AV in our revised version.

Q7: Missing highly relevant work on traffic event recognition [1,2] in the Related Work section.

A7: We appreciate your suggestion. We will include the missing references and expand the section to properly acknowledge prior work on traffic event recognition.

[3] Zekun Zhang et al., Object Detection With Self-Supervised Scene Adaptation, CVPR 2023.

[4] Shuo Wang et al., The 8th AI City Challenge, CVPR-W 2024.

Dear Reviewer,

Thank you very much for your positive feedback and insightful questions. We provide detailed responses to your concerns below.

Q1: Would the authors provide a brief discussion on how this dataset, especially the QA part, will remain challenging for upcoming new multimodal LLMs given the high baseline results?

A1: During our literature review, we also observed that some VQA benchmarks in vertical application domains, such as earth observation, autonomous driving, and smart cities, tend to achieve relatively high baseline accuracy (generally > 60%) after fine-tuning (e.g., EarthVQA[1], nuScenes-QA [2], City-3DQA [3]). Our dataset shows a similar trend, indicating that large VLMs can more easily adapt to constrained, domain-specific tasks. However, we believe that the high baseline accuracy does not indicate that the task is simple or lacking in meaningful challenges. Our benchmark involves multiple complex tasks that remain crucial for developing future multimodal LLMs, especially those for traffic-related applications. Below, we provide a detailed discussion:

1. Our benchmark unifies three core tasks, i.e., multi-choice QA, referred object captioning, and spatiotemporal object grounding through tuple-based spatiotemporal object expressions for traffic scene understanding. While the QA task demonstrates relatively high accuracy, the other two tasks still exhibit substantial room for improvement. Importantly, as discussed in the paper, techniques that enhance QA performance, such as visual token strategies, do not consistently benefit the other tasks. This reveals that optimizing for QA alone is insufficient, and highlights the need for holistic model designs that jointly address all tasks.

2. Even within the multi-choice QA task, simply scaling model size yields diminishing returns. Although the 7B model achieves higher accuracy, the improvement over smaller models (e.g., 0.5B) is marginal. This suggests that simply increasing LLM size or replacing it with newer variants alone is unlikely to bring large improvement in this domain. Together, these findings emphasize the importance of domain-specific model design and training strategies that consider multiple tasks tailored to traffic-centric understanding.

3. In real-world traffic applications, where computational resources are often limited, developing lightweight models becomes a key priority. Rather than focusing solely on improving benchmark scores, we encourage future work to explore trade-offs between performance and efficacy with lightweight LLMs and techniques (i.e., pruning, quantization, and knowledge distillation). Our benchmark and baseline models provide practical references to guide and evaluate such efforts toward building efficient, real-time multimodal systems.

In summary, we believe our dataset and benchmark remain highly important for the development and evaluation of upcoming multimodal LLMs, particularly those tailored to traffic domains. It presents practical challenges, encourages efficient model development, and supports a holistic evaluation across multiple multimodal tasks.

[1] Junjue Wang, et al., “EarthVQA: Towards Queryable Earth via Relational Reasoning-Based Remote Sensing Visual Question Answering”

[2] Tianwen Qian, et al., “NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario”

[3] Penglei Sun, et al., “3D Question Answering for City Scene Understanding”