TRACE: Temporal Grounding Video LLM via Causal Event Modeling

Thank you for your valuable suggestions. We have addressed the questions you raised below.

Autoregressive modeling.

Thank you for the insightful suggestion! After carefully examining the submission and the papers you listed, we acknowledge that the concept of "event" in our paper may differ from that in [1, 3], which could lead to some confusion. To address this, we have revised the illustration and Eq. 1 of our paper to provide a clearer understanding of our approach.

• In summary, conditioned on prompts/instructions and all the video frame tokens, TRACE formats its responses using event triplets that consist of timestamps, scores, and text. This structured approach aligns well with the inherent structure of videos and provides a comprehensive understanding of their content.
  • In our work, we define an "event" as a triplet comprising timestamps, scores, and text. This triplet serves as a model output unit, providing textual descriptions (answers) along with the corresponding timestamps and matching scores.
    • For dense video captioning tasks, the event triplets serve as descriptions that summarize the video contents (i.e., "events" in [1,3]).
    • For moment retrieval tasks, the event triplets provide descriptions of specific moments along with their corresponding timestamps.
    • For general video QA tasks, the event triplets may encompass the answer, accompanied by the relevant timestamps and matching scores. We believe that constructing these data is an important future work direction for TRACE.
  • Importantly, each event $e_{k}$ is dependent on previous events $e_{1:k-1}$ and all the video visual contents $F$ (see Eq. 2). By observing all the video visual inputs $F$ and previous generated events $e_{1:k-1}$, the model has sufficient information to generate the event $e_{k}$.
  • In [1, 3], the events are considered as the fundamental units of video content, i.e., video clips. The authors primarily focus on identifying the causality between these video clips. We believe that this approach is relatively orthogonal to ours. Moreover, the outputs generated by causality discovery models have the potential to be integrated as inputs of TRACE, which may further enhance its performance.
• The design of "casual event modeling" makes it easy to leverage pretrained LLMs. Since current LLMs are typically decoder-only models, "casual event modeling" aligns closely with their pretraining objectives, allowing it to benefit from (1) instruction-following capabilities for handling diverse tasks simultaneously in a zero-shot manner, and (2) performance-boosting techniques like KV-cache and flash-attention. While "casual event modeling" could be adapted to "masked event modeling" by taking into account future events, doing so would likely disrupt the pretrained knowledge of LLMs and result in a loss of the benefits.

Inference speed.

Thank you for the suggestion! We have provided the inference speed of TRACE (128 frames) compared to VTG-LLM (96 frames), which is a typical video LLM with a standard architecture. The results demonstrate that TRACE does not incur any additional inference cost.

LLM backbone.

Thank you for the valuable comments. Due to time and resource constraints, we have not conducted experiments using the llama2. However, we have conducted experiments without causal event modeling, also using Mixtral-7B as the LLM backbone.

As shown in the "w/o causal event modeling" section of Table 4, following the previous SOTA [5], we (1) use slot-based compression, (2) add time tokens and score tokens to the text vocabulary, and (3) use natural language outputs instead of event-structured outputs. The training data, vision encoder, and LLMs remain the same as in the original TRACE. The results show that TRACE significantly outperforms this ablation, even with fewer sampled frames.

[5] VTG-LLM: Integrating Timestamp Knowledge into Video LLMs for Enhanced Video Temporal Grounding. Arxiv 2024.

Thank you for your prompt response and insightful suggestions. We fully acknowledge the importance of modeling the complete causal relationship between events for a comprehensive understanding of video content, as you have pointed out. We would like to provide further clarification regarding our study.

• Firstly, while modeling the complete relationship may pose a challenge for decoder-only LLMs, this is a general limitation, not specific to TRACE. Given that most current video LLMs also rely on decoder-only architectures, we believe that TRACE's structure does not diminish its capacity for general VQA tasks compared to these approaches. We have previously reported this to reviewers p7QK and MMw2 during the rebuttal phase.
• Secondly, we agree with your observation that causality discovery is not orthogonal to VTG tasks. We believe that it is possible to design causality discovery models in an 'orthogonal' manner, which can provide a comprehensive understanding of video content. The outputs of these causality discovery models can then be used as supplementary inputs for TRACE, potentially overcoming some of the limitations associated with decoder-only LLMs. We have discussed these related works and future directions in the 'Conclusion and Future Works' section of revised paper.
• Lastly, despite TRACE's potential limitations in modeling the complete causal relationship, we firmly believe that it still provides significant benefits. By representing the response of video LLMs as a series of event triplets (timestamps, scores, and captions), TRACE excels in VTG tasks. Furthermore, we envision that TRACE can be extended to other video understanding tasks, such as VQA, by expanding its annotations in the future. Therefore, we believe that TRACE remains a valuable contribution to the field of video understanding.

Thank you once again for your detailed review, which has greatly enhanced the clarity of our paper.

Thank you for your valuable suggestions. We have addressed the questions you raised below.

While the motivation for TRACE is clear, the use of multiple task-specific heads may limit the model’s generalization. A primary appeal of Video-LLMs lies in their ability to handle a variety of tasks without specific fine-tuning. TRACE’s focus on VTG may narrow its versatility, making it less effective for general video understanding tasks.

Thank you for the insightful suggestion! We have also recognized this limitation and conducted additional experiments before the rebuttal. The results show that the TRACE architecture is still capable of handling general video understanding tasks and excel in VTG tasks.

• Despite NOT being trained on extensive multi-task datasets, TRACE is still highly effective in handling general video understanding tasks. For example, the TRACE outperform generalist video LLMs like VideoChat2, ShareGPT4Video, and ST-LLM on VideoMME benchmark.
• We train TRACE-uni by incorporating additional general video understanding data from a subset of LLaVA-Video-178k (specifically the perceptiontest and YouTube parts). TRACE-uni shows both improved general video understanding and stronger VTG performance without additional VTG training data. Notably,
  • TRACE-uni performs on par with, or even outperforms, generalist video LLMs that use the same LLM backbone and vision encoder (VideoLlama2) using only about 2M training data.
  • TRACE-uni surpasses TRACE in VTG performance across all three evaluation datasets.

---

In most cases, lightweight VTG-specific models with stronger performance could be more suitable for VTG scenarios.

Thank you for your insightful comments. We would like to clarify a few points and provide further context:

• Video LLMs offer distinct advantages that traditional VTG-specific models cannot match. Specifically, they provide:

  1. Zero-shot capability, which allows the models to perform VTG tasks without the need for task-specific training.
  2. The ability to handle multiple VTG tasks simultaneously, offering a level of versatility that traditional models lack.

  We believe these features are crucial for real-world applications, where scalability and adaptability are keys.

• While VTG-specific models generally exhibit strong performance on VTG tasks, there is an emerging research trend exploring how video LLMs can address VTG challenges [1, 2, 3, 4, 5]. TRACE introduces a novel solution that significantly enhances the performance of video LLMs on VTG tasks, and we believe this contribution is valuable to the community.

Some clarity is not clear. For example, the paper does not adequately explain slot-based compression, which is not a widely known technique. Moreover, compressing each frame to just 8 visual tokens might lead to significant information loss, raising concerns about the trade-off between efficiency and accuracy.

Thank you for your valuable suggestion! We would like to provide the following clarification

• We compress the visual tokens to address efficiency and context length limitations. Since TRACE samples 128 frames, without compression, the ViT would produce over 70K visual tokens. To handle this, we compress the visual tokens to 8 tokens per frame, resulting in a total of 1,792 visual tokens after incorporating the time tokens corresponding to each frame. This compression allows us to effectively handle VTG tasks within the 4K context length limit.
• We choose slot-based compression for its lightweight architecture and high performance on VTG tasks. Introduced by [5], slot-based compression uses only one-third of the parameters of a single cross-attention layer, while outperforming both cross-attention and sampling-based methods on VTG tasks.
• As per your recommendation, we have conducted ablation studies on the number of tokens per frame. However, the training will take more than a week to complete. We will post the results here once the ablation training is finished.

It is unclear whether the same set of number tokens is used for both timestamps and scores. If so, this could blend the two types of information, contradicting the authors' claim (lines 45–46) that the model preserves the distinct structure of video events.

We apologize for any confusion. To clarify, timestamps and scores in TRACE use separate sets of tokens. As explained in lines 235-245, the timestamps and scores are processed by distinct encoder-decoder pairs, though both pairs share the same model structural design.

[1] Vtimellm: Empower llm to grasp video moments. CVPR 2024.

[2] Timechat: A time-sensitive multimodal large language model for long video understanding. CVPR 2024.

[3] Lita: Language instructed temporal-localization assistant. ECCV 2024.

[4] Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning. ICML 2024.

[5] VTG-LLM: Integrating Timestamp Knowledge into Video LLMs for Enhanced Video Temporal Grounding. Arxiv 2024.

Dear Reviewer MMw2,

We have finished the experiments that compressing each frame into 16 tokens. Due to time constraints, we adopted the same settings as in Table 3, using VTG-IT only in Stage 2 and sampling 64 frames. Our findings are as follows: Increasing the number of slots per token significantly enhances TRACE's performance. Therefore, if computational or efficiency constraints are not a concern, we recommend using a larger number of slots per frame.

Additionally, we evaluated TRACE on the causality reasoning benchmark [6], as shown in Table 11 of the revised paper. TRACE outperformed open-source video LLMs and achieved performance comparable to GPT-4o on tasks such as event description, contextual reasoning, and episodic reasoning. This result demonstrates the potential of the TRACE architecture for handling complex reasoning tasks.

We hope our responses have sufficiently addressed your concerns. Please feel free to reach out if you have any further questions. Thank you again for your time and effort in reviewing our paper.

[6] Towards Event-oriented Long Video Understanding. ArXiv 2024.

Dear reviewer MMw2,

To further clarify our points, we evaluate Qwen2-VL on the event-level video understanding benchmark E.T.Bench [8]. Our results show that, despite being trained with less data, shorter context, and an older LLM backbone

• On the E.T. Bench, TRACE
  • outperforms VideoLlama2 across all tasks;
  • achieves performance comparable to GPT-4o on RAR, ECA, RVQ, and DVC tasks;
  • achieves similar performance to Qwen2-VL on RVQ and GVQ tasks;
  • outperforms both GPT-4o and Qwen2-VL on TVG, EPM, TAL, EVS, SLC, and TEM tasks.
• While Qwen2-VL demonstrates advanced performance on multiple-choice QA tasks such as the RAR, ECA, and RVQ evaluations from E.T.Bench, outperforming even GPT-4. But its performance on other tasks, ranging from TVG to TEM, remains subpar. This highlights the ongoing challenge of developing a generalist video LLM capable of effectively handling a wide variety of video understanding tasks.

We hope the additional numerical results address your concerns. Please don't hesitate to reach out if you have any further questions. Thank you again for your review.

[8] E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding. NeurIPS 2024.

Thank you for your valuable suggestions. We have addressed the questions you raised below.

While the paper compares TRACE with other video LLMs, it presents limited comparison and may not adequately address how it stands against traditional non-generative and task-specific models.

Thank you for the detailed comment. We have provided TRACE’s results compared to traditional non-generative and task-specific models after fine-tuning in Table 5. The results show that:

• TRACE sets a new SOTA on the YouCook2 dataset without audio inputs, outperforming existing SOTA by a large margin, and even surpassing Vid2Seq (with audio) on F1 score.
• On the Charades-STA dataset, TRACE performs comparably to non-generative methods and outperforms strong baselines such as VDI, Moment-DETR, and UnLoc-L.
• TRACE significantly outperforms other video LLMs after fine-tuning, highlighting the potential of video LLMs for VTG tasks.

Beyond fine-tuned results, TRACE’s strong zero-shot performance, surpassing existing methods and handling multiple VTG tasks simultaneously, offers significant value—something traditional non-generative and task-specific models cannot achieve.

The extent to which TRACE can be applied to other types of video tasks beyond VTG is unclear. Its design may be highly specialized, which could limit its applicability across diverse video understanding tasks. Authors should present more results on other video-understanding tasks since the design seems generalizable by building such causal event relations

Thank you for the insightful suggestion! We have also recognized this limitation and conducted additional experiments before the rebuttal. The results show that the TRACE architecture is still capable of handling general video understanding tasks and excel in VTG tasks:

• Despite NOT being trained on extensive multi-task datasets, TRACE is still highly effective in handling general video understanding tasks. For example, the TRACE outperform generalist video LLMs like VideoChat2, ShareGPT4Video, and ST-LLM on VideoMME benchmark.
• We train TRACE-uni by incorporating additional general video understanding data from a subset of LLaVA-Video-178k (specifically the perceptiontest and YouTube parts). TRACE-uni shows both improved general video understanding and stronger VTG performance without additional VTG training data. Notably,
  • TRACE-uni performs on par with, or even outperforms, general video LLMs that use the same LLM backbone and vision encoder (VideoLlama2) using only about 2M training data.
  • TRACE-uni surpasses TRACE in VTG performance across all three evaluation datasets.

---

Thank you for your valuable suggestions! We have addressed the concerns you raised below.

While causal event modeling is presented as a core contribution of this work, the related work section does not address any prior research on similar methodologies. It would be helpful to clarify whether comparable approaches have been explored in the field of video understanding, or if this approach is entirely novel within this domain. Providing this context could strengthen the argument for the method’s originality and situate it more clearly within existing research.

Thank you for pointing that out! To the best of our knowledge, TRACE is the first video-LLM method to model the response of video LLMs as a series of event triplets (timestamps, scores, captions). To illustrate:

• Existing video LLMs typically only using text tokens—either directly representing times/scores by text [1, 2, 3], or adding new tokens to the text vocabulary [4, 5, 6]. However, these methods still generate output in human language, which limits their ability to effectively capture the underlying structure of the video. Furthermore, introducing new tokens into the vocabulary could potentially degrade the pretrained captioning capabilities of the LLMs [6]. TRACE addresses these challenges by directly modeling video LLM responses as a series of events, using separate encoders and decoders to handle timestamps, scores, and text independently.
• Some non-LLM methods use event-like structures, but they lack causal modeling and are difficult to adapt to video LLMs. For example, PDVC [7] uses distinct heads to decode timestamps and text, while UniVTG [8] reformats VTG tasks and employs three separate heads. While these methods share some intuitive similarities with the design of TRACE, they are challenging to integrate with LLMs pretrained using causal language modeling. In contrast, TRACE introduces causal event modeling, enabling easier utilization of pretrained LLMs' reasoning capabilities and knowledge.

Overall, we believe TRACE is a significant contribution to advancing the field of video LLMs, particularly in addressing the challenges of VTG tasks. Its novel approach could pave the way for future research in effectively integrating video and language models.

It is unclear whether compressing visual features to 8 tokens is sufficient for preserving critical information in complex video scenes. The paper does not provide an analysis or experimental results on the trade-off between the number of tokens and model performance, which would be valuable in understanding the potential impact of this compression choice.

Thank you for your valuable suggestion! We would like to provide the following clarification

• We compress the visual tokens to address efficiency and context length limitations. Since TRACE samples 128 frames, without compression, the ViT would produce over 70K visual tokens. To handle this, we compress the visual tokens to 8 tokens per frame, resulting in a total of 1,792 visual tokens after incorporating the time tokens corresponding to each frame. This compression allows us to effectively handle VTG tasks within the 4K context length limit.
• We choose slot-based compression for its lightweight architecture and high performance on VTG tasks. Introduced by [6], slot-based compression uses only one-third of the parameters of a single cross-attention layer, while outperforming both cross-attention and sampling-based methods on VTG tasks.
• As per your recommendation, we have conducted ablation studies on the number of tokens per frame. However, the training will take more than a week to complete. We will post the results here once the ablation training is finished.

There are several grammatical and spelling errors throughout the manuscript, which impact readability and may detract from the paper’s clarity. For example: Line 22: "processes" should be corrected to "process". Line 44-45: The phrase "...which,..." should be rephrased, and "lacks" should be changed to "which lack".

Thank you for your detailed suggestions! We have corrected the typos in the revised paper, which are highlighted in blue font.

Refine Prompt Design Explanation

Thank you for the suggestion! We have provided the examples of QA pairs in the Appendix of our submission.

Explore Custom Scene Parsing Techniques

Thank you for the suggestion! In TRACE, since timestamps, scores, and captions are decoded using separate heads, the parsing process is relatively straightforward. This allows us to independently collect timestamps, scores, and text, and then directly use these collected results for evaluation.

[1] VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs. Arxiv 2024.

[2] Vtimellm: Empower llm to grasp video moments. CVPR 2024.

[3] Timechat: A time-sensitive multimodal large language model for long video understanding. CVPR 2024.

[4] Lita: Language instructed temporal-localization assistant. ECCV 2024.

[5] Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning. ICML 2024.

[6] VTG-LLM: Integrating Timestamp Knowledge into Video LLMs for Enhanced Video Temporal Grounding. Arxiv 2024.

[7] End-to-End Dense Video Captioning with Parallel Decoding. ICCV 2021.

[8] UniVTG: Towards Unified Video-Language Temporal Grounding. ICCV 2023.