Fine-Grained Captioning of Long Videos through Scene Graph Consolidation

1. Categorization of algorithms

Given the diversity of zero-shot video captioning approaches, their categorization is not straightforward, and we acknowledge that alternative perspectives exist. However, since SGVC consistently outperforms LLM summarization throughout our experiments, its improved performance is not merely due to the use of LLMs but rather the effectiveness of the proposed algorithmic components. In the revised manuscript, we will reorganize Table 1 to better position SGVC and clarify the sources of its performance gains.

2. Comparison against LLaVA-NEXT-7B itself and VideoPrism-B

Since LLaVA-NEXT-7B [1] is an image-based captioner, directly evaluating its performance on video captioning is not straightforward. A meaningful approach is to compare LLM summarization and SGVC, as both share the same pipeline for extracting image-level captions using LLaVA-NEXT-7B but differ in their subsequent processing. As shown in Tables 1 and 2, our results clearly demonstrate the superior performance of SGVC over the simple combination of LLaVA-NEXT-7B and LLMs.

VideoPrism-B [2] indeed achieves outstanding performance on MSR-VTT, but it has been trained on an extensive collection of public and private videos with paired captions. As a result, its encoder is specifically optimized for video data, contributing to its superior performance. Additionally, it employs a powerful decoder (PaLM-2) for caption generation, which further enhances its results. Given these factors, we argue that a direct comparison between VideoPrism-B and our method is not entirely fair. Since VideoPrism-B’s model and source code are not publicly available, we instead selected InternVL2.5-1B [3], a video captioner, to assess SGVC’s performance with a video-based model. Our results in Tables D and E demonstrate that SGVC outperforms LLM summarization when integrating a clip-level captioner, highlighting the effectiveness of our graph-based approach.

Table D. Experiments on MSR-VTT, using InternVL2.5-1B as VLM backbone.

Table E. Experiments on ActivityNet Captions, using InternVL2.5-1B as VLM backbone.

4. Efficiency

Please refer to our response to #1 for Reviewer JfkZ.

5. Additional ablation studies

Please refer to Tables A, B, and C for the ablation studies regarding VLM captioner backbones and hyperparameter $\tau$, respectively.

Additionally, we analyze the impact of the hyperparameter $k$, which determines the size of the extracted subgraph. As shown in Table F, lower $k$ values yield more concise subgraphs centered on salient objects, enhancing precision-oriented metrics (CIDEr, $P_\text{BERT}$). In contrast, higher $k$ values produce richer subgraphs that capture broader contextual information, leading to improvements in recall-oriented metrics (METEOR, $R_\text{BERT}$).

Table F. Ablation study on the hyperparameter $k$, on the MSR-VTT test set.

6. Clarification on the graph-to-text model

Our graph-to-text model consists of a BERT-based encoder and a T5-base decoder, totaling 235M parameters. Given that the other two-stage video captioning models often rely on LLMs, our approach is significantly more lightweight. We will include detailed information about our architecture in the revised version of our paper. More importantly, we plan to release our source code and models upon acceptance of our paper.

7. Error propagation case study

As mentioned in our response to #3 for Reviewer JfkZ, hallucinations in frame-level caption generation introduce errors in the final video-level captions. We will discuss cases of error propagation in the revised manuscript.

[1] Liu, H. et al. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge. https://llava-vl.github.io/blog/2024-01-30-llava-next/, 2024.

[2] Zhao, L. et al. VideoPrism: A Foundational Visual Encoder for Video Understanding. In ICML 2024.

[3] Chen, Z. et al. Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling. In arXiv, 2024.

1. Inference cost

SGVC is efficient in terms of latency, comparable to Decap and C3. However, direct comparisons are challenging as each method utilizes different backbone models for image captioning (SGVC: BLIP, BLIP2, LLAVA-NEXT-7B; Decap: CLIP; C3: ImageBind). In contrast, Video ChatCaptioner and [Tewel et al.] are significantly slower, by at least an order of magnitude, due to the former requiring multiple evaluation rounds and the latter involving test-time optimization.

For the performance of SGVC with BLIP and BLIP2, please refer to our response for Tables A and B in #2. These models actually outperform our implementation with LLAVA-NEXT-7B.

2. Different VLM captioning models

We evaluate our approach using various VLM backbones, including image-based models (BLIP, BLIP-2, LLaVA-NEXT-7B) and a video-based model (InternVL2.5-1B [1]), on the MSR-VTT (Table A) and ActivityNet Captions (Table B) datasets. The results consistently demonstrate that SGVC outperforms its LLM summarization counterpart, regardless of the chosen VLM backbone. Note that the SGVC results with LLaVA-NEXT-7B in Table B differ slightly from those in the original manuscript, as we extended fine-tuning on Visual Genome paragraphs from 500 to 3200 iterations.

Table A. Ablation of VLM backbones on MSR-VTT test set.

Table B. Ablation of VLM backbones on ActivityNet Captions val set.

3. Failure cases

We provide two failure cases given by hallucinations in the initial image-based caption:

Example 1.

• Reference captions: ["A group of people dressed in all of the colors of the rainbow sing a happy song.", "Two elderly women dancing with a group of men.", …]
• SGVC output: "Two guys in multi-colored tops dance in front of a wall."

While the caption accurately captures specific visual details such as "multi-colored tops", "wall", and "dance", the VLM hallucinates the number of individuals (reporting "two guys" instead of the actual group of five people).

Example 2.

• Reference captions: ["A man fixes a piece of machinery that appears to be a miniature tank.", "A guy fixing his camera equipment.", … ]
• SGVC output: "A man is holding a drill in his hand while working on machinery."

The object in the person's hand is a camera, but the initial frame-level captioner incorrectly identified it as a "drill", influenced by the surrounding context. This hallucinated detail was propagated to the final consolidated caption.

4. Impact of threshold $\tau$ in graph merging

We conducted an ablation study by varying the cosine similarity threshold, $\tau$, as shown in Table C. The results demonstrate stable performance within the range $\tau \in [0.80, 0.95]$.

Table C. Ablation study on the threshold $\tau$.

5. Frame selection criteria

In our experiment for Table 3, we uniformly sampled frames to ensure coverage of the entire video content.

6. Quantitative evaluation of hallucination reduction

Thank you for your suggestion. However, conducting a quantitative evaluation within a short time frame is challenging due to the lack of necessary annotations and evaluation metrics.

[1] Chen, Z. et al. Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling. In arXiv 2024.

1. Reliance on the quality of frame-level captions and vulnerability to cumulative parsing errors in scene graphs

• Error propagation is common in other algorithms: The primary objective of this work is to understand video content and generate fine-grained, video-level captions without encoding entire videos at once. This approach is especially effective for handling long videos, which require division into multiple segments or frames. However, consolidating information from these segments inevitably leads to cumulative errors—a challenge common to various algorithms despite their efforts to minimize it.

• Error quantification is difficult: Due to the zero-shot nature of our method, directly quantifying the cumulative error from abstracting segment-level (or frame-level) visual information into textual captions and extracting video-level representation based on a scene graph generation and consolidation is challenging. This is because existing methods employ fundamentally different algorithms, including variations in encoder and decoder choices for representation and captioning.

• An attempt to estimate the impact of cumulative parsing error: One way to isolate the parsing error in scene graph construction is to compare performance between LLM summarization and our approach, SGVC. Note that LLM summarization follows the same pipeline as SGVC up to the segment-level (or frame-level) captioning stage but consolidates multiple captions directly using an LLM. As shown in Tables 1, 2, and 3 of the main paper, SGVC significantly outperforms LLM summarization, indicating that scene graph consolidation effectively integrates information without introducing substantial errors.

• Artifact from the generation of segment-level (frame-level) captions: Analyzing the impact of segment-level (or frame-level) caption generation is more complex. However, converting video segments (or frames) into text is sufficiently effective for merging semantic information across multiple segments (or frames), facilitating more coherent video-level understanding. Note that the counterpart, maintaining visual representations for individual segments (or frames), is not interpretable and requires an additional module for consolidating the information from multiple segments (or frames).

2. Limited ability to capture subtle temporal dynamics between frames

As discussed in our response to #1, our core contribution lies in developing a pipeline for fine-grained video captioning without encoding entire videos at once, which involves scene graph generation and its consolidation. This allows for a variety of captioning models to be used; our approach is not restricted to frame-level captioners but can also incorporate video-level ones. Notably, the combination of a video-level encoder (InternVL2.5-1B [1]) and SGVC demonstrates promising performance, outperforming LLM summarization, although comparisons with external algorithms are not available at this time. For detailed results, please refer to our response to Reviewer 3YCD for #2.

3. Additional reference

We appreciate the reference to HIG [2] for video scene graph generation. This study offers a valuable technique for constructing more comprehensive scene graphs. However, our work is not a competitor to HIG; rather, it has significant potential to benefit from integrating HIG’s scene graph generation techniques to enhance performance. Additionally, HIG requires extensive training to learn a scene graph generation model, but since pretrained models are not publicly available, immediate integration into our approach may not be feasible.

[1] Chen, Z. et al. Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling. In arXiv, 2024.

[2] Nguyen, T. et al. HIG: Hierarchical interlacement graph approach to scene graph generation in video understanding. In CVPR 2024.

1. Reliance on the quality of frame-level captions and vulnerability to cumulative parsing errors in scene graphs

• Error propagation is common in other algorithms: The primary objective of this work is to understand video content and generate fine-grained, video-level captions without encoding entire videos at once. This approach is especially effective for handling long videos, which require division into multiple segments or frames. However, consolidating information from these segments inevitably leads to cumulative errors—a challenge common to various algorithms despite their efforts to minimize it.

• Error quantification is difficult: Due to the zero-shot nature of our method, directly quantifying the cumulative error from abstracting segment-level (or frame-level) visual information into textual captions and extracting video-level representation based on a scene graph generation and consolidation is challenging. This is because existing methods employ fundamentally different algorithms, including variations in encoder and decoder choices for representation and captioning.

• An attempt to estimate the impact of cumulative parsing error: One way to isolate the parsing error in scene graph construction is to compare performance between LLM summarization and our approach, SGVC. Note that LLM summarization follows the same pipeline as SGVC up to the segment-level (or frame-level) captioning stage but consolidates multiple captions directly using an LLM. As shown in Tables 1, 2, and 3 of the main paper, SGVC significantly outperforms LLM summarization, indicating that scene graph consolidation effectively integrates information without introducing substantial errors.

• Artifact from the generation of segment-level (frame-level) captions: Analyzing the impact of segment-level (or frame-level) caption generation is more complex. However, converting video segments (or frames) into text is sufficiently effective for merging semantic information across multiple segments (or frames), facilitating more coherent video-level understanding. Note that the counterpart, maintaining visual representations for individual segments (or frames), is not interpretable and requires an additional module for consolidating the information from multiple segments (or frames).

2. Low performance of 'LLM summarization’ in Table 2

We appreciate the suggestions for additional references [1, 2, 3] to compare on ActivityNet Captions. However, we clarify that these methods are extensively trained on large-scale video-text datasets, including ActivityNet Captions, whereas our approach operates in a zero-shot setting without using target dataset annotations during training. Therefore, direct comparisons between [1, 2, 3] and SGVC are not meaningful.

3. Evaluation on YouCookII

We agree that evaluating on a dataset with longer videos, such as YouCookII [4], is meaningful. We are currently conducting experiments on YouCookII and expect that the benefits of SGVC will be particularly evident in datasets with long videos, similar to ActivityNet Captions. However, due to time constraints, we were unable to include these results in our initial response.

[1] Huang, B. et al. VTimeLLM: Empower LLM to Grasp Video Moments. In CVPR 2024.

[2] Tang, Y. et al. Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding. In AAAI 2025.

[3] Kim, M. et al. HiCM2^2 2: Hierarchical Compact Memory Modeling for Dense Video Captioning. In AAAI 2025.

[4] Zhou, L. et al. Towards Automatic Learning of Procedures From Web Instructional Videos. In AAAI 2018.