Dense Video Object Captioning from Disjoint Supervision

We thank the reviewer for the positive feedback on our model, training pipeline, comparison to the baselines, and versatility on related tasks. We address the concerns below:

Is it possible to unify the multi-dataset training process within a robust code infrastructure?

Great point! We already include the ** full code ** to reproduce our work in the supplementary, including the data-preprocessing scripts and multi-dataset training. Adding more datasets with a different annotation type is straightforward in our infrastructure (<20 lines, as exemplified in code/input_utils.py L711), as the different annotation types are conveniently handled by a loss mask.

Compute resources needed in training and inference.

We would like to highlight that our computational resources required by our model are not onerous, as the entire model fits into a single GPU, and we simply use more GPUs for data-parallelism to speed up training and inference.

We trained our model using V100 GPUs with 16 GB of memory, as mentioned in Appendix B.2 (L873). We train our model with data-parallelism, using a local batch size of 1. In our experiments, we used a global batch size of 32, and therefore used 32 GPUs. Importantly, we verified that our training is still effective with the linear learning rate scaling rule [A]: Namely, we can train our model with a global batch size of 8 or 16 (therefore with 8 or 16 GPUs respectively) by linearly increasing the number of training steps and linearly decreasing the learning rate, with minimal performance change.

For inference, we currently use GPUs with 32GB of memory, as we process a maximum of 200 frames concurrently on the VidSTG dataset. It is possible to use less GPU memory, whilst maintaining the identical result, by running the tracking part of our model in a sliding window fashion following GTR [B], but we avoided this to avoid introducing additional implementation complexity.

Inference efficiency across different stages.

Thank you for the suggestion! We report the runtime on a 16 GB V100 GPU below for a 64-frame video below.

Our model takes 2.36 seconds to process a 64-frame video, which means 27.1 frames per second (amortizing the global trajectory captioning time over all frames). The majority of the runtime is in the autoregressive text decoding which is an inherently sequential process.

The absence of specialized, annotated data for Dense VOC remains a challenge.

We agree. As the first work in this direction, the goal of our paper is to set up benchmarks, provide a solid baseline, and attract the attention of the community. We believe pushing better performance of our Dense VOC requires curating a training dataset with complete annotations (either manually or using an automatic/ semi-automatic data pipeline). We leave this as a future work, and we are actively working on this.

The term “Dense” is somewhat confusing and seems ambiguous.

We used “dense” following the literature for “Dense object captioning” [C], as discussed in L122-132, which localizes objects spatially in an image and describes them with text, and “Dense video captioning” [D], L130-131, which localizes events temporally in a video and describes them with text. As our task localizes both in space and time, we named our task “Dense video object captioning”.

We are happy to rename our task and title to “Spatio-Temporal Video Object Captioning” if the reviewer thinks that is clearer.

Is there an analysis of different event types in this task?

As we are taking captioning annotations from existing datasets, we take the event statistics from the original dataset papers.

Video Localized Narratives focuses on the “story” of the actor object. The events include object actions, interaction with other objects, and attribute changes.

VidSTG dataset focuses on human-human or human-object relation. The events highlight changes in spatial relations (e.g., move towards/away from) and action relations (e.g., watch, hold, kick, drop etc).

References

[A] Goyal et al., Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour. arXiv:1706.02677, 2017

[B] Zhou et al., Global Tracking Transformers, CVPR 2023

[C] Johnson et al., DenseCap: Fully Convolutional Localization Networks for Dense Captioning, CVPR 2016

[D] Krishna et al., Dense-captioning events in videos. ICCV 2017

We thank the reviewer for the positive comments on our task and our extensive experiments. We address the concern below:

How does the model compare to query-propagation-based models?

That’s a great question! We explored query-based detectors and trackers during our development. However, despite our best efforts, we did not get close performance as our detection-and-association model. We identify two main reasons:

• The Hungarian matching used in DETR-style architectures is significantly more expensive for captioning than “standard” tasks like detection and segmentation. This is because captioning outputs are conditioned on the ground truth during training due to teacher forcing. Therefore, to compute the captioning cost for one example, we need to forward the text decoder with every pair of ground-truth and predicted objects (typically $Q = 100$ queries x $N = 50$ ground-truth objects per image in Visual Genome). This means $Q \times N = 5000$ forward passes for just a single training step. “Standard” tasks require just $Q$ forward passes, and typically use a much lighter linear head as well, so it is not a bottleneck there. Due to this compute limitation, we did not manage to train a reasonable DETR-based model on the Visual Genome dataset for image-object captioning.
• Performing detection and tracking jointly makes it more difficult to train individually for these tasks, as we do with our disjoint, multi-dataset pretraining (Section 3.4).

As a result, it is difficult for query-based models to take advantage of the disjoint multi-dataset training like us.

In addition, we compared to a concurrent query-based model OW-VisCap [A] (which could not leverage disjoint pre-training for the reason mentioned above) in Table 5, and showed an improvement of 3.9 CHOTA.

References:

[A] Choudhuri et al., OW-VISCap: Open-World Video Instance Segmentation and Captioning, NeurIPS 2024.

We thank the reviewer for the positive feedback on our benchmark, training framework, and experiments. We reply to each concern below:

It would be advantageous to illustrate the task's relevance in more challenging or representative application scenarios

Thank you for this suggestion. We have updated the introduction to mention that models trained for DenseVOC can be used for diverse real-world applications including sports analysis and commentary, animal monitoring, security and behaviour analysis among others.

Rationale for task decomposition, and in-depth discussion on the domain differences between datasets

Great point! In our paper, we chose the task decomposition and datasets based on availability of existing annotations: we decompose our task into video captioning (SMiT, 500k videos), image object captioning (VisualGenome, 80K images), and tracking (COCO, 108K images), as large-scale datasets are available for these tasks. We do realize the domain differences in these datasets, for example, the object definitions in VisualGenome are different from COCO (discussed in L454 - L458 of the original paper), and we mitigate this by ignoring the losses from datasets that have incompatible annotations. The results in Table 3 also confirms that our mixed training yields the best results in both zero-shot and finetuning evaluation, suggesting that the domain differences in our training datasets are migrated by the large-scale training.

Typos

Thank you for pointing out the typos! We have corrected them and uploaded a new version of our paper.

How does the model handle interactions between targets? Is it from the entire-frame training in SMiT?

Good question! Our answer is that it can be from using either SMiT or Visual Genome during pretraining. It is common in Visual Genome that an object caption mentions another object, for example, “a woman riding a horse” as exemplified in Figure 1 of the GRiT paper. While our model does not explicitly model the object interaction, the region feature operator RoIAlign may have sufficient receptive fields to capture information in the context. We also agree that training on SMiT amplifies this effect, and the annotations in SMiT contain more descriptions of relations that may shift the output caption distribution towards including relations.

As the discussion phase for ICLR 2025 is nearing its conclusion, I wanted to bring to your attention that I have posed some questions and comments regarding your submission. These points are crucial for a comprehensive evaluation of your work.

Please note that if I do not receive a response soon, I may need to reconsider the quality of this submission, which could potentially impact the score I assign to your submission.

We thank the reviewer for their positive feedback on our task, model, and results. We address the weaknesses below.

Evaluation on a broader range of benchmarks.

We kindly remark that our paper presents results on 3 datasets (VidSTG, Video Localized Narratives, and BenSMOT) and 2 tasks (Dense VOC and grounding). We also remark that evaluation datasets on our new task are scarce, due to the cost of collecting such annotations. In this rebuttal, we additionally evaluate our disjoint-trained model on the image object captioning dataset, Visual Genome ,that we have trained on, and compare to single-task trained models and the SOTA baseline GRiT[A] (which uses an MAE backbone rather than CLIP backbone as in our model). The results suggest that our disjoint-pretraining maintains the performance.

Analysis of failure cases.

Thank you. We briefly discussed the failure cases in Appendix A. The failure cases include misrecognizing object classes in the output captioning, and wrong object boundaries (captioning an action before it happens, as our model predicts a single caption for the whole object trajectory). While our tracking is not perfect, we do not observe a bottleneck in tracking. It is also reflected in our high AssA scores in Table 3 and Table 4 (~70 AssA on VidSTG, ~90 AssA on VLN and BenSMOT), whose numbers are higher than typical MOT datasets (~50 AssA on MOT-17 datasets as shown in the HOTA paper) [B].

How does the model perform with unseen objects?

Our model is open-vocabulary by its design (i.e., outputs free-form text), and our training set covers a diverse range of object classes. However, for objects that never appear in the training text vocabulary, our model will not have the capability to output them. Using stronger, pre-trained text decoders will mitigate this issue, as language models output texts which they have encountered previously during training.

Rationale behind the chosen evaluation metrics.

We choose an evaluation metric that measures the three primary components of our DenseVOC task: namely detecting objects in a video, tracking them throughout the video, and captioning these objects.

The first two, detecting and tracking objects, are typically measured in the Multi-Object Tracking (MOT) community with HOTA [B]. We therefore adopt HOTA, which consists of detection and association terms (Detection Accuracy (DetA) and Association Accuracy (AssA) respectively), and extend it with a captioning term, which we name Captioning Accuracy (CapA).

We decided to extend HOTA, as it is already the standard in the MOT community, and therefore facilitates fairer comparison with these models, and makes use of the lessons learned by the MOT community in evaluating trackers.

Finally, to evaluate Captioning Accuracy, we use an average of three common captioning metrics, namely CIDEr, METEOR and SPICE. We adopt these three metrics as they are common in the community, and known to be complementary to each other. Specifically, CIDEr is common for image captioning (it is the default metric for the COCO dataset), METEOR is the main metric for object captioning (ie the Visual Genome dataset [C]). SPICE [D] is known to be complementary to CIDEr and METEOR as it highlights semantics [E].

We will use other captioning metrics if the reviewer has other alternate suggestions. Note that we report individual values of CIDEr, METEOR and SPICE in Tables 11 and 12 of the appendix.

References:

[A] We et al., GRiT: A Generative Region-to-text Transformer for Object Understanding. ECCV 2024.

[B] Luiten et al., HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking. IJCV 2020.

[C] Krishna et al., Visual genome: Connecting language and vision using crowdsourced dense image annotations, IJCV 2017

[D] Anderson et al., SPICE: Semantic Propositional Image Caption Evaluation, ECCV 2016

[E] Kilickaya et al., Re-evaluating Automatic Metrics for Image Captioning, ACL 2017