Youku Dense Caption: A Large-scale Chinese Video Dense Caption Dataset and Benchmarks

Thank you for your comprehensive review and insightful questions. We greatly appreciate your feedback and would like to address your concerns as follows:

Regarding Weakness 1.1:

Unlike datasets such as HD-VILA-100M or Panda-70M that use ASR or image captioning models for annotation, our dataset is entirely manually annotated. Human annotators describe the videos and select informative segments for segmentation. This approach ensures that our annotations align with natural human language usage and undergo manual screening to maintain high quality.

Regarding Weakness 1.2:

While advanced language models like ChatGPT could indeed improve Chinese translation quality, our dataset's primary motivation extends beyond language differences to encompass cultural and content disparities between Chinese and foreign contexts. As mentioned in Section 3.2.2, unique Chinese cultural elements like tea ceremonies, traditional festivals (Enjoy the moon on Mid-Autumn Festival and pay New Year's greetings on New Year's Day), etc., are rarely found in public English datasets but are common in Chinese contexts. This difference is reflected not only in annotation text distribution but also in video content distribution. As noted by widely-used Chinese CLIP[1], bridging the gap caused by cultural differences in annotation and video content is our dataset's core value.

Regarding Weakness 2:

In Section 6, we demonstrate the effectiveness of our dataset by training Qwen2-VL-2B on the VATEX[2] Chinese dataset for dense video captioning and ViCLIP on the Youku-mPLUG[3] Chinese dataset for video retrieval. The results show improved performance on their own benchmarks, validating the utility of our dataset.

Regarding Weakness 3:

As mentioned in Appendix A, we've implemented strict requirements for annotation length and quantity. All annotation texts must exceed 10 characters, and each video must have more than 3 annotation texts, ensuring sufficient detail and comprehensiveness.

Regarding Weakness 4:

Appendix A details our annotation process. Video segment division is entirely manual. The color histogram correlation-based filtering algorithm mentioned in Section 4.2 is an additional step to enhance dataset quality by removing potentially similar segments from the manual divisions, thereby ensuring diversity.

Regarding Question 1:

Appendix A provides a comprehensive explanation of our annotation process. After detailed human descriptions of complete videos, annotators manually divide the annotation text and video segments. The division criteria are primarily based on human judgment of information content in video segments.

Regarding Question 2:

In the submitted supplemental materials, we've included all detailed annotation texts for generation, retrieval, and grounding tasks. For the retrieval benchmark, we've already divided and screened similar texts, ensuring that highly similar texts do not repeatedly appear in the dataset.

We hope these responses address your concerns adequately. If you have any further questions or require additional clarification, please don't hesitate to ask. We appreciate your valuable feedback and are committed to improving our research based on your insights.

[1] Yang, An, et al. "Chinese clip: Contrastive vision-language pretraining in chinese." arXiv preprint arXiv:2211.01335 (2022).

[2] Wang, Xin, et al. "Vatex: A large-scale, high-quality multilingual dataset for video-and-language research." Proceedings of the IEEE/CVF international conference on computer vision. 2019.

[3] Xu, Haiyang, et al. "Youku-mplug: A 10 million large-scale chinese video-language dataset for pre-training and benchmarks." arXiv preprint arXiv:2306.04362 (2023).

Thank you for the further feedback on my review. Below are my responses:

"In contrast, traditional video caption datasets typically offer a single, high-level description for the entire video."

This statement is inaccurate. Datasets like HD-VILA-100M and Panda-70M also segment videos into multiple clips and provide caption annotations for each segment. Regarding the Youku Dense Caption annotations, the provided format is same as that of regular video caption datasets. For instance, the annotation format in Panda-70M (as in the csv file: https://drive.google.com/file/d/1BZ9L-157Au1TwmkwlJV8nZQvSRLIiFhq/view) shows the same format as in Youku Dense Caption.

For ActivityNet, it is a dense caption dataset because the annotation includes temporal-related information. Taking the provided sample "v_cp52LdlmlUk" as an example, it has the structure: "People doing something" → "Then doing another thing" → "Then doing yet another thing". This information is what differentiates dense caption datasets from regular video caption datasets.

"We have also provided additional test results on the Youku-mPLUG validation set, utilizing Vatex and Youku Dense Caption for training"

Please include this table and details about the experimental setup in the revised version of the paper. It would also be helpful to provide more comprehensive results, such as using different combinations of Vatex and Youku Dense Caption as training datasets, similar to what was presented in the original manuscript.

Thank you for your valuable feedback. We appreciate the opportunity to address your concerns and provide additional clarification.

Regarding the weakness you mentioned, we sincerely apologize for any oversight. Our open-source license is the same as that of Youku-mPLUG[1], as we mentioned in the ETHICS STATEMENT section. This section comprehensively addresses the open-source license and potential impacts of our work. The dataset will not be used for commercial purposes, will be properly attributed, and will be shared under the same conditions.

Concerning question 1, we acknowledge your point about an English version of the Youku dense caption dataset. Initially, our focus was on the Chinese language community, given the abundance of high-quality, influential datasets already available in English. However, we appreciate your suggestion and will seriously consider releasing an English version in the future. If we proceed with this, we would likely utilize more advanced APIs such as GPT-4 to ensure the highest possible quality of the translated data.

We're glad to provide additional details about our annotation process:

1. Our annotation team consisted of 1 annotation leader and 10 annotators. The leader was responsible for aligning annotation standards, conducting trial annotations, training annotators, and quality control. The annotators focused on specific annotation tasks and data revisions.

2. We engaged a professional data annotation agency. The annotation leader had at least 3 years of experience in managing annotation tasks, while the annotators had a minimum of 3 months of experience in CV-related data annotation. All team members were native Chinese speakers.

3. The annotation cost was 0.63 RMB per valid clip caption, with total video costs varying based on the number of valid slices. The entire process took 2 months, with efficiency improving over time as annotators became more proficient.

4. While we don't have detailed information about the internal labor distribution within the annotation agency, we understand from our interactions with the annotation leader that the workload was generally evenly distributed among annotators.

5. The annotation leader conducted quality checks on 10% of each batch of annotations. A 97% pass rate was required for batch approval; otherwise, the entire batch was returned for re-annotation. In the early stages, we also performed secondary reviews of the data to ensure alignment with annotation standards.

We hope this additional information addresses your concerns and provides a clearer picture of our rigorous annotation process. We remain committed to maintaining high standards of data quality and transparency in our research.

[1] Xu, Haiyang, et al. "Youku-mplug: A 10 million large-scale chinese video-language dataset for pre-training and benchmarks." arXiv preprint arXiv:2306.04362 (2023).

Thank you for your thorough review and valuable feedback on our paper. We greatly appreciate your insights and would like to address your concerns as follows:

Regarding Weakness 1:

We understand your concern about the similarity of annotation texts. Figure 1 only shows examples of the original annotations in the dataset, which are in their raw form before being filtered by the algorithm proposed in Section 4. This is not the actual data we used in our experiments. While it's challenging to completely avoid similar texts during the annotation process without significantly increasing difficulty, we've implemented rigorous measures to address this issue. As shown in Algorithm 1, we use the state-of-the-art Chinese embedding model (xiaobu_embedding[1]) to filter out texts with over 90% similarity. This ensures that our final dataset doesn't contain highly similar texts, thus maintaining diversity and quality.

Regarding Weakness 2:

We apologize for any confusion our previous statement may have caused. In the soon-to-be-released dataset, we will retain all original content, including audio and high-resolution video quality. The removal of audio in our experiments was solely due to the limitations of the models used, which don't process speech. This decision doesn't affect the integrity or value of the dataset itself.

Regarding Question 1:

We appreciate your insightful question. As our dataset is the first Chinese dense video dataset, it's indeed challenging to find a directly comparable dataset for vocabulary analysis. However, we've provided a comparative statistical analysis of the original annotation text and the processed data in Section 4, as shown in the table. This comparison demonstrates the changes in word frequency distribution, addressing the issue of repetition.

Regarding Question 2:

Your question about the self-BLEU threshold is crucial. As mentioned in Section 4.2, our choice of self-BLEU threshold aims to balance data diversity and suitability for generation and moment retrieval tasks. After manually observing hundreds of video samples under different threshold settings, we determined that a self-BLEU threshold of 0.15 provides the optimal balance between text distinctiveness and task applicability.

We hope these responses address your concerns adequately. If you have any further questions or require additional clarification, please don't hesitate to ask. We're committed to improving our research based on your valuable feedback.

[1] https://huggingface.co/lier007/xiaobu-embedding-v2

We sincerely appreciate your thorough review and insightful comments. We would like to address your concerns as follows:

Regarding Weakness 1:

We acknowledge that our use of "fine-grained Chinese captions" may be misleading. A more accurate description would be "Chinese dense captions", as we divide videos into fine-grained segments and provide annotations for each segment. We will revise this terminology in the relevant sections to ensure clarity.

Regarding Weakness 2:

We have attempted to elaborate on the reasons for the discrepancies between Chinese and English annotations in Section 3.2.2. This includes differences in word usage (e.g., "切" vs. "剁"), cultural nuances (such as tea ceremonies), and unique vocabulary (like "新闻联播"). These factors likely contribute to the suboptimal performance of translated data in our experiments. We believe these distinctions underscore the importance of a native Chinese dataset.

Regarding Weakness 3:

In Section 6, we demonstrate the value of our dataset through experiments on video captioning and retrieval tasks. By augmenting the Vatex dataset with our data, we achieved state-of-the-art performance on both the Vatex[1] validation set and our Youku dense caption benchmark. Similarly, for video retrieval, gradually incorporating our dataset into training with the Youku-mPLUG dataset led to state-of-the-art results on the Youku-mPLUG[2] validation set. These experiments provide empirical evidence of our dataset's positive impact on existing benchmarks.

Regarding Weakness 4:

We have provided a detailed analysis of the differences between Chinese and English linguistic characteristics in Section 3.2.2, theoretically justifying the necessity of the first Chinese dense video dataset. In Section 5, experiments across retrieval, generation, and grounding tasks show that Chinese-specific models like Chinese-CLIP[3] outperform models trained on translated English data, further validating our dataset's importance. Our ablation studies on Vatex and Youku-mPLUG datasets demonstrate that incorporating our dataset improves performance on other benchmarks, substantiating its value.

We hope these clarifications address your concerns and highlight the contributions of our work. We are committed to improving the presentation of our research based on your valuable feedback. If you have any further questions or require additional information, please don't hesitate to ask.

[1] Wang, Xin, et al. "Vatex: A large-scale, high-quality multilingual dataset for video-and-language research." Proceedings of the IEEE/CVF international conference on computer vision. 2019.

[2] Xu, Haiyang, et al. "Youku-mplug: A 10 million large-scale chinese video-language dataset for pre-training and benchmarks." arXiv preprint arXiv:2306.04362 (2023).

[3] Yang, An, et al. "Chinese clip: Contrastive vision-language pretraining in chinese." arXiv preprint arXiv:2211.01335 (2022).

Thank you for your reply and efforts on the rebuttal! My current response is based on the latest version of the paper.

For Weakness 1: Your response and revisions have not addressed my concerns. My main issue is that for English dense caption video datasets, translating these caption into accurate Chinese could also constitute a Chinese video caption dataset. For reference, the sentence in lines 205–207: “For example, the English phrase ‘The woman is cutting vegetables in the kitchen’ translates to ‘女人在厨房里切菜’, where ‘切菜’ can mean ‘cutting vegetables’ or ‘chopping vegetables’, each with different nuances” is difficult to understand as an explanation of so-called Chinese characteristics. Furthermore, examples like “一家人在吃饭，其中一位女子帮他端了一碗酒” (“The family is having dinner, and one of the women brings him a bowl of wine”) can also be considered fine-grained video captions in English; it’s just a different level of fine-grainedness. Claiming this difference stems from different linguistic communities lacks sufficient evidence.

For Weakness 2: My question has been resolved. Thanks for the reply!

For Weakness 3: The introduction of YDC shows limited improvement on the VATEX dataset, and there is a performance drop on the METEOR metric (comparing 100% VATEX + 0% YDC to 100% VATEX + 50% YDC). Additionally, for YDC data, as mentioned in the paper, the OOD setting leads to a performance drop of 13% with the same amount of data. This makes the value of fine-grained captions appear limited, as most of the improvement seems to come from the increased data volume. Moreover, on the 100% Youku-mPLUG dataset, adding 13% YDC data leads to a drop in video-to-text R10, text-to-video R5, and text-to-video R10 metrics. This does not support the claim that “based on the Youku-mPLUG training set, the video retrieval performance steadily improves with the addition of YDC training data.”

For Weakness 4: My concerns regarding this have already been stated under Weakness 1.

From my perspective, I appreciate the value this dataset brings to the Chinese community, but I believe the value primarily lies in the localized data. It would be better to focus more on reducing the gap caused by differences in localized data distribution.

Thank you for pointing out the issues in our paper. We have taken your feedback seriously and have rewritten Section 3.2.2. Specifically, we have condensed the discussion on language translation to one paragraph. Additionally, we have added three paragraphs at the beginning of the section to introduce and describe unique aspects of Chinese culture and explain how these contribute to the differences in data distribution between Chinese and English. At the end of this section, we have summarized our findings, emphasizing that the distinctiveness of our dataset’s Chinese characteristics stems from various cultural contexts, which involve not only differences in annotations but also in video content.

Regarding weakness 3, as discussed in our ablation study, the Vatex dataset and YDC dataset have training sets of similar size for the dense video captioning task. This implies that training on YDC, with a dataset of the same scale, can achieve better results on the Youku-mPLUG validation set, thereby demonstrating the higher quality of our data.

We sincerely appreciate your thorough review and constructive suggestions. We hope these revisions meet your expectations and look forward to your further feedback.