Grounded Video Caption Generation

1. There are concerns regarding the technical soundness of the AUTOMATIC ANNOTATION METHOD section. I understand that the grounded video caption generation task requires high precision; however, due to the richness and ambiguity of language, I am skeptical about the consistency of the language-based tracking method in object localization. For example, while the case of the woman drinking a beverage in the authors' paper is unambiguous, many instances will likely present significant ambiguity. If, in the third frame of Figure 2, the image caption still states 'holding a glass,' how can one determine whether 'beverage' and 'glass' refer to the same object?

2. The writing quality is poor, containing a lot of redundancy, and some phrases read awkwardly and lack fluency. For instance, the phrase 'Differently than the image-based task, where bounding boxes...' could be improved. Additionally, some sentence structures need adjustment, and paragraph transitions require attention, particularly when describing the GROC construction process. There are also typos to address, such as in Section ?? on page 5. Overall, the writing and illustrations need further polishing.

I am primarily concerned with the automatic data generation pipeline. In this process, the authors used GLaMM to generate frame-wise captions and then converted these into Subject-Verb-Object (SVO) triplets. Since there is not alignment between generated SVOs, if there are two different instances with the same label in different frames, it is possible that the LLM might treat them as a single instance, potentially leading to inaccurate conclusions and relationships between them.

Similarly, in the tracking by language process, instances are tracking with only textual description of the phrases. This cannot guarantee 2 different instances with same label are mistakenly treated as the single instances. For example, imagining a video that there are 2 different people appearing in different frames but all drinking beers, the process is likely to link them as one person by just considering its textual description. The strategy described in Supp. is not very convincing as there can be 2 different people both with white shirts and drinking.

The process appears to capture only a portion of the video content, potentially overlooking valuable concurrent events within the video clip.

I am also curious about the visual encoders used in VideoGLaMM. Is there any specific reason or necessity for using two different image backbones separately for captioning and bounding box generation?

the reference in Line 230 is missing.

• The authors claim to introduced a new task: grounded video caption generation. However, there is prior research on this task, such as PG-VIDEO-LLAVA[1]. How do the authors justify their assertion of novelty?

[1] PG-Video-LLaVA: Pixel Grounding Large Video-Language Models.

• For videos that contain two or more events, how should the automated annotation pipeline be applied? For example, consider a scenario where a man picks up a knife to chop vegetables and then puts the vegetables into a pot. If the knife and pot appear simultaneously in the frame but the action hasn't progressed to the pot yet, how can we avoid generating a bounding box for the pot prematurely?

• The case studies (e.g., Figure 4) presented by the authors seem overly simplistic. Can VideoGLaMM be effectively applied to videos that involve multiple events?

• There are very few comparative methods included in the study. The authors should compare more methods, such as the integration of video captioning models with object detection models, and PG-Video-LLaVA.

• The first paragraph of the introduction feels more like a related work section, making it hard to follow. I would suggest starting by clearly stating the task you aim to solve and providing a high-level summary of the existing work.
• The method for automatically creating the dataset uses SVOs, which can lead to important details being omitted from captions, such as the color of the shirt in Figure 2. This omission could hinder downstream tasks. For instance, a model aiming to find videos of orange shirts might miss this information, which is otherwise captured in current captioning models like GlaMM.
• The paper’s main contribution is the creation of the HowToGround dataset. However, the choices made during dataset creation are not justified, nor does the paper include any ablation study of the creation steps. Including such a study, showing alternative choices for each step, would greatly strengthen the paper. For example, how about using SAM-2 or an open-world object detector to identify object locations? How would VideoGLaMM perform if step 3 were omitted from the dataset creation process? How well does the resulting VideoGLaMM for different train set sizes?

Typos/Errors:

• L230 - Section reference is broken.

I would say the most important weakness of this paper is over-claim, fails to discuss and compare previous published research and show no respect to previous work. Let's start with the beginning:

1. The authors claim that they propose a new task called grounded video caption generation (GROC). However, I don’t think so at all. Please refer to paper [1] titled "Grounded Video Description." I think that, in vision-and-language research, most researchers use the terms "video description" and "video captioning" interchangeably (the slight difference may be that "video description" sometimes includes more details). Paper [1] is the first to collect grounded video-text datasets and propose a model that uses grounding information to generate better video descriptions. They didn’t even claim that this task is a so-called new task because it is viewed as using alternative guidance to improve video description generation, not as a new task. One quick and intuitive way to compare these two is to check Figure 1 in this submission and Figure 2 in paper [1]. Thus, I completely reject the claim that this task is newly proposed by the authors of this submission.

2. The more interesting thing is that the authors actually cited paper [1] but did NOT mention it at all in the main pages. Instead, they "secretly" placed it in the Appendix. In my opinion, a paper with so much overlap should be carefully discussed in the related work section, in section 3 where the authors claim their novelty, and also at the end of section 5, where the authors describe how they construct their datasets.

3. As for the experiments, the proposed model is also lacking of novelty. Put more relevant papers here, like [2, 3, 4], which are also not properly cited and discussed in this submission. Thus, I also clearly reject the claim in line 055-056 that "At the same time, producing natural language video descriptions where the described objects are spatio-temporally grounded with bounding boxes in videos has received much less attention". From my understanding, the authors replace the previsous LSTM networks with large language models. For sure, LLMs can get better results than LSTMs. However, these methods and models are not compared in their experiments. I don't believe that, in the era of LLMs, simply replacing the language module in previous methods with an LLM can be considered a big innovation in the overall model structure.

In summary, I clearly think this submission needs major revisions and a complete reframing in order to meet the publication standards of a conference like ICLR.

References:

[1] Grounded Video Description, Zhou et al. CVPR 2019.

[2] Learning to Generate Grounded Visual Captions without Localization Supervision, Ma et al. ECCV 2020.

[3] Attend and Interact: Higher-Order Object Interactions for Video Understanding, Ma et al, CVPR 2018.

[4] Comprehensive Visual Grounding for Video Description, Jiang et al. AAAI 2024.

Limited Model Comparison: It would be better for VideoGLaMM to include comparisons against additional models like InternVL and Qwen-VL, which also produce bounding boxes in text format, to strengthen its evaluation. Since the paper aims to establish a new task, a more comprehensive benchmark would enhance the reliability of its proposed contributions.

Discussion Scope: Additional discussion is needed on alternative methods for VLM bounding box outputs, specifically addressing why GLaMM was prioritized over other methods like [1,2]. This would help clarify its selection rationale within the broader context of existing approaches.

[1] Bannur, Shruthi, et al. "MAIRA-2: Grounded Radiology Report Generation." arXiv preprint arXiv:2406.04449 (2024). [2] Zou, Ke, et al. "MedRG: Medical Report Grounding with Multi-modal Large Language Model." arXiv preprint arXiv:2404.06798 (2024).