Learnability Matters: Active Learning for Video Captioning

Thank you for your valuable comments. We will provide more details on the motivation behind our design and include additional supportive observations in the final version. Please review our feedback below. We are happy to address any further questions during the discussion period.

1. The usage of LVLM is debatable.

• Currently, LVLMs that support video are still in their infancy. Their zero-shot predictions on complex videos can be unreliable, e.g. performance on VATEX in Tab.10 of mPLUG[1]. Similarly, directly applying BLIP[2] gives an unsatisfactory performance on VATEX. For instance, 37.4 with CIDEr metrics according to [1]. Therefore, instead of directly exploiting predictions LVLM, we exploit LVLM to 1. generate human-like captions at the same granularity so that the abstraction inconsistency can be measured 2. approach granularity inconsistency by comparing predictions from LVLM and $f$(Line 176-187). Overall, while LVLM itself may not be entirely reliable, it can provide valuable information through inter and intra comparisons.

2. If the reduction of human annotation costs justifies the extensive computational cost.

• Our additional computational costs arise from the active learning algorithm, primarily due to applying BLIP2 to the unlabelled dataset and generating scene graphs from its predictions. This process occurs only once on the unlabelled set. On our hardware, consisting of 4 RTX 4090 graphics cards with a power capacity of 2000 kWh, it takes no more than 9 minutes to run BLIP2 and 5 minutes for scene graph generation on the MSVD training dataset. In contrast, according to [3], annotating the full dataset in 2010 required hundreds of annotators, around 2 months, and less than 5000 USD in total. Even if the highest annotation efficiency claimed by the author is always followed (10K captions per day), it still takes about 5 days to finish MSVD without considering the cost of Quality Control. Therefore, we argue that active learning is more efficient in terms of both time and cost.

3. The technical innovation for each component appears incremental, and the terms in the acquisition function are heuristic, without mathematical proof.

• Our approach incorporates learnability, uncertainty, diversity, and a caption-wise active learning protocol. To the best of our knowledge, we are the first to explore collective outliers in video captioning tasks while also formulating them as learnability in an active learning framework. Although uncertainty and diversity are common strategies in active learning, we have adapted them specifically for our video captioning setting. For example, uncertainty measures the consistency between the scene graph generated from the most confident caption by $f$ and that from LVLM (Line 229-233).

References:

[1] Li, Chenliang, et al. "mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections." Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.

[2] Li, Junnan, et al. "Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation." International conference on machine learning. PMLR, 2022.

[3] Chen, David, and William B. Dolan. "Collecting highly parallel data for paraphrase evaluation." Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies. 2011.

Once again, your time and effort is more than appreciated.

I thank the authors for preparing the rebuttal. After reading the rebuttal, I intend to keep my original rating, which is positive.

Thank you for your encouraging comments. In addition to including the cross-dataset experiments and limitations in our final version, we will double-check the text and redraw some figures to enhance clarity and aesthetics. Please see our responses to your concerns below. More figures are provided in the PDF file. We are happy to address any further questions during the discussion period.

1. Lack of cross-dataset experiments.

• Due to time constraints, we were unable to complete the experiment on large-scale datasets such as VATEX or WebVid. To simulate a cross-dataset setup, we used the small annotated dataset MSVD (1,200 videos with 40 captions per video) and treated MSR-VTT as the large unannotated dataset, which includes 6,513 videos paired with 20 captions each. The results using the official SwinBERT implementation are shown below.

• We report the overall performance on MSVD test set. "Data Per." is the percentage of human annotations on MSR-VTT. "Random" refers to the random selection baseline described in Lines 268-269 of the main paper. We also report the performance of directly selecting 20% of MSR-VTT (Row 4) and iteratively adding 5% of MSR-VTT four times (Rows 5-8). As shown in the table, our AL algorithm significantly improves the overall performance of MSVD and is a more effective choice compared to random selection. Furthermore, directly selecting 20% of data is slightly less effective than iterative selection, demonstrating the benefits of curriculum learning. Notably, the overall performance peaks at two iterations, or 10% of human annotations on MSR-VTT, according to CIDEr and SPICE. Beyond this point, the performance saturates and slightly declines. This is expected, as 20% of MSR-VTT includes 26K captions and at least 1.3K videos, which is comparable to the original training set of MSVD. Adding more data from a different dataset can degrade performance, as the training may deviate from the original dataset.

2. Lack of limitation analysis.

• There are several limitations that we believe are worth further exploration. Firstly, our paper only briefly touches on the relationship between curriculum learning and learnability. Beyond provoking the design of the learnability term, we believe curriculum learning can enhance the interpretability of learnability terms and even active learning. Secondly, we found that current evaluation metrics, such as CIDEr, do not always align with human evaluations. More human analysis is needed for video captioning tasks. Thirdly, we made some preliminary attempts to combine knowledge from LVLMs in a semi-supervised learning manner (Line 389-394). Although we did not see a significant improvement, we believe further efforts are warranted. Lastly, our experiments with ChatGPT-4 can be improved with more refined designs. We will include these limitations in our final version.

3. Some writing issues.

• We appreciate the reviewer pointing out the writing issues in our manuscript. For Figure 5, we will enlarge the font size and ensure consistent font size across all figures. Additionally, we will double-check the text and redraw some figures to enhance clarity and aesthetics.

We are grateful for your constructive feedback and will revise our manuscript accordingly.

Thank you for your constructive comments. We are pleased to clarify our claims and will update our manuscript with additional supportive results. Below are our responses to your concerns. We are committed to addressing any further questions.

1. Low baseline performance, resulting in overclaims on performance.

• We would like to clarify that the "SOTA methods" mentioned in "our algorithm outperforms SOTA methods by a large margin" refers specifically to SOTA active learning methods. We will enhance this clarification in our final version. Regarding the baseline methods, SwinBERT and CoCap were selected as they offer a good balance between accuracy and efficiency (Line 288). In contrast, other SOTA video captioning methods, such as COSA[1], require extensive pre-training. In addition, fine-tuning is also required when adapting to downstream tasks to achieve SOTA video captioning performances. To validate this, we downloaded the pre-trained mPLUG-2[2] model from its official site and applied it to the MSR-VTT test set. Without fine-tuning on the MSR-VTT training set, it achieved a CIDEr score of 24.2. In contrast, mPLUG-2 ranks first on the MSR-VTT leaderboard with a CIDEr score of 80.3 after fine-tuning, demonstrating the significant performance improvement that comes from including data from the target task. Our active learning method remains highly valuable as it focuses on selecting the most informative cases for annotation rather than requiring annotation of all data, thereby leading to a more effective fine-tuning process.

2. The claim of "25% of human annotations" is questionable as "annotating two captioning for the same video can be more efficient"

• When reconstructing a video captioning dataset, it is common for an annotator to provide just one caption per video. Although this approach is less efficient than having multiple annotations per person per video, it ensures data quality and diversity. For instance, VATEX[3] states that "Each video is paired with 10 English and 10 Chinese diverse captions from 20 individual human annotators." (Introduction). Further details in Section 3.1.1 highlight that "Towards large-scale and diverse human-annotated video descriptions, we build upon Amazon Mechanical Turk(AMT)2 and collect 10 English captions for every video clip in VATEX, where each caption from an individual worker.” Similarly, Section 2 of MSR-VTT[4] notes, "Therefore, we rely on Amazon Mechanical Turk (AMT) workers(1317) to annotate these clips. Each video clip is annotated by multiple workers after being watched. ... As a result, each clip is annotated with 20 sentences by different workers. " Therefore, our claim of "25% of human annotations" is both practical and meaningful.

References:

[1] Chen, Sihan, et al. "COSA: Concatenated Sample Pretrained Vision-Language Foundation Model." Proceedings of the Twelfth International Conference on Learning Representations, 2024.

[2] Xu, Haiyang, et al. "mPLUG-2: a modularized multi-modal foundation model across text, image and video." Proceedings of the 40th International Conference on Machine Learning. 2023.

[3] Wang, Xin, et al. "Vatex: A large-scale, high-quality multilingual dataset for video-and-language research." Proceedings of the IEEE/CVF international conference on computer vision. 2019.

[4] Xu, Jun, et al. "Msr-vtt: A large video description dataset for bridging video and language." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

Once again, we sincerely thank the reviewer for your time and effort in reviewing our manuscript.