ACAV-1M: Data Curation and Benchmarking for Audio-Visual Representation Learning

1. There are inconsistencies in the writing that hinder the paper’s readability. For example, in the caption for Figure 1, the authors refer to a "Large Language Model (LLM) (OpenAI, 2023) to generate multiple captions." However, in Section 3.1 on data curation, they mention "VideoLLaVA (Lin et al., 2023) generates several sentence-level descriptions." These contradictions create confusion and make the paper harder to follow. Furthermore, the structure of Section 3.2 could be clarified to improve the overall flow.
2. The authors use the open-source Video-LLaVA framework to generate comprehensive captions for audio-visual content, which forms a critical part of their work. However, the Video-LLaVA framework does not include an audio encoder and processes only images and video. The paper does not clearly explain where or how the audio captions are generated, which raises concerns about whether it is appropriate to use Video-LLaVA for audio-visual captioning. This is a key technical issue that should be addressed.
3. The paper mentions that each audio segment is standardized to a 10-second duration. However, it does not provide sufficient detail on how these segments are cut or how the semantic integrity of the data is maintained after cutting. It is unclear whether the alignment between audio and video is preserved during this process. A more thorough analysis of these technical aspects would strengthen the paper and reassure readers about the dataset's quality.
4. The paper employs automated alignment and filtering techniques to evaluate the synchronization between audio, video, and text. While this approach is efficient, it may not be equally effective for all types of content. For example, music or ambient sounds may not require perfect alignment with visual elements. The authors could consider adjusting filtering parameters based on the content type to improve the alignment process for different scenarios.
5. While the alignment process enhances data quality, overly strict filtering thresholds may cause valuable samples to be discarded. The paper does not discuss or quantify potential data loss resulting from this filtering, which could lead to a dataset biased towards content that is "easy to align." This reduction in sample diversity might limit the dataset's utility in certain tasks. The authors should explore a balance between strictness and diversity, or at least provide an analysis of the data loss caused by filtering.

Although the paper demonstrates the strength of the audio-visual alignment through various downstream tasks, could more intuitive representations be provided? For instance, a statistical similarity comparison calculated via ImageBind.

1. The data collection paradigm uses ImageBind to measure the alignment between audio-visual-language. However, as discussed in [1], ImageBind actually is inferior to aligning audio and language. Better models or more in-depth comparative discussions are needed.
2. The main baseline in the paper should be AudioSet, which includes twice the amount of data and has been widely used. Table 1 claims that ACAV-1M can support more tasks, while AudioSet cannot. However, in actual experiments, ACAV-1M is mostly used as pre-training data, which can also be replaced by AudioSet, so the claim in Table 1 is not convincing. A fairer comparison with AudiosSet is needed to highlight the necessity of ACAV-1M.
3. Lack of some more advanced baselines, such as using ImageBind for audio-visual retrieval.

[1] FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion[C]//Forty-first International Conference on Machine Learning.

• Addressing the issue of audio-video misalignment is crucial, as most misaligned samples in common datasets involve voice-over narration and background music that lack synchronization with the visual content. In a random inspection of samples from the video list provided by the authors, I identified a lot of cases with audio-video misalignment, such as “kCb7VJgc1Zc,” “_QQHFllvooU,” and “JUChESSdblA.” These samples include voice narration but lack a visible talking head in the video, which generally suggests inconsistency between the audio and video content. This raises concerns about whether ACAV-1M can reliably ensure effective alignment of audio and video. Please detailed the quality control process - for instance, what percentage of samples were manually checked, and whether there's a process for addressing misalignments identified after curation.
• The paper does not seem to outline steps to verify the audio-visual consistency of the filtered dataset. This omission leads to an open question: are the model’s performance gains due to the dataset’s size or improved audio-visual alignment? Would a similar boost be observed with models trained on ACAV-100M, even without alignment assurance? Can you compare performance of models trained on ACAV-100M without alignment process and ACAV-1M with alignment process?
• The authors currently do not provide metadata for the dataset. I believe that various types of metadata, such as video captions, timestamps, and cluster categories in AVAC-100M, are essential. Similarity information could allow users to customize subsets and filter the samples they need. As captions are one of the supported modalities in this dataset, they should be included, while timestamps would enable users to segment data more effectively.

Data Distribution Issues:

• Lack of distribution statistics. The paper fails to provide detailed breakdowns of video types, categories, and their proportions in the dataset, making it impossible to assess dataset balance and potential biases.

• Missing temporal distribution. No information about the temporal spread of collected data, which could affect the dataset's representativeness across different time periods.

Data Source Issues:

• Undisclosed sources. The paper doesn't specify the origins of the video data, raising concerns about data quality and reliability. No explanation of how videos were selected or filtered during the initial collection phase.
• Licensing ambiguity. Lack of discussion about copyright and usage rights, potentially limiting the dataset's practical applicability.

Filtering Methodology Concerns:

• Limited performance comparison. The implementation of instance and temporal filtering isn't well-justified, especially given that most professional audio-visual content is naturally synchronized. The study only explores similarity thresholds for filtering. However, there is no comparative analysis between filtered and unfiltered datasets. On the other hand, how much data is filtered out is not reported, and it is unclear whether the filtering process actually unnecessarily reduces the dataset size, therefore, it is hard to judge whether using the unfiltered dataset, which could have much larger size, might yield better results.

• Uniform threshold application. The study applies the same similarity threshold (50%) across language alignment, instance alignment, and temporal alignment. This one-size-fits-all approach may be problematic as different types of alignment may require different thresholds for optimal filtering. The lack of independent threshold optimization for each alignment type could lead to suboptimal filtering results.

Foundation Model Evaluation Concerns:

• MLLM and ImageBind reliability. The paper relies heavily on VideoLLaVA and ImageBind for data processing but lacks rigorous evaluation of these models' performance (e.g., using human annotated subsets for evaluation):

  • No human evaluation of MLLM-generated captions to verify their accuracy and relevance

  • Missing analysis of ImageBind's cross-modal alignment accuracy on this dataset