OMCAT: Omni Context Aware Transformer

We thank the reviewer for their positive feedback and comments that will greatly help in improving this work. Below, we address all the concerns raised.

The use of GPT-4 for evaluation introduces an element of subjectivity, especially in grading responses on a 0-5 scale. Although this approach is common, the absence of standardized quantitative metrics could complicate comparisons with future studies. Additionally, human evaluation is recommended to achieve a more reliable and robust assessment.

A key advantage of using GPT-4 for evaluation over traditional metrics is its ability to handle open-ended answers, where a response may be correct but not identical to the ground truth. While human evaluation is the gold standard, it is highly expensive and time-intensive. Recent works [1] have shown reliability of GPT-4 for evaluation under different scenarios and it has proven robust to response types, prompts, style changes and iterations. Furthermore, evaluating responses with GPT-4 has been a very effective tool for reinforcement learning with AI feedback to align model responses closer to human responses [2,3].

Given that the dataset is primarily constructed using GPT, I am concerned about its quality control. It would be beneficial to incorporate human annotation or validation to ensure greater accuracy and reliability.

Thanks for raising this. For assessing the quality of our generated dataset, we trained OMCAT without the OCTAV dataset and compared the results of this method on the final model trained with OCTAV. As shown in the results below, training with the OCTAV dataset leads to a gain in performance not only on the OCTAV benchmark but also across temporal understanding tasks (e.g. Charades-STA).

Moreover, the QA task in OCTAV is closely tied to the ground-truth video and audio time-stamped annotations, both of which are human annotated, providing a solid foundation for quality control.

Furthermore, the GPT-4-assisted generation heavily relies on prompt tuning and in-context examples (see Appendix B.1 and B.2). This approach effectively minimizes the risk of errors or hallucinations in question-answer generation, delivering reliable quality without the need for extensive human validation.

While the paper compares OMCAT to several state-of-the-art models, it would be beneficial to include more baselines from diverse modalities to fully illustrate its strengths and weaknesses across different scenarios.

In Table 9 (in the Appendix) we compare OMCAT’s video understanding ability on three benchmark datasets MSVD-QA, MSRVTT-QA and ActivityNet QA. Many multimodal LLMs, such as Video LLaMA 2, leverage vast amounts of video-text paired data (around 12 million pairs), which boosts their video understanding capabilities. In contrast, OMCAT is trained on a much smaller subset of video-text data (about 2 million pairs), which naturally limits its video understanding performance. Despite this, OMCAT demonstrates competitive results in video understanding tasks.

Furthermore, we evaluate OMCAT on the audio question-answering task using the Clotho-AQA dataset [4], where it achieved an accuracy of 57.4%, closely matching the 57.9% achieved by Qwen-7B Audio [5], a model specifically designed for audio only understanding tasks. This demonstrates that OMCAT excels not only in cross-modal and temporal tasks but also sustains strong performance in modality-specific audio understanding. We will add these results in the paper.

The paper relies heavily on GPT-4 for scoring responses with a subjective grading scale. How consistent and reliable is this approach across different response types? Did the authors test the robustness of the GPT-4 evaluation process, such as through inter-rater reliability tests or additional scoring methods?

Following previous work, we use the GPT-4 based evaluation for consistency and fair comparison. Recent works [1] have shown reliability of GPT-4 for evaluation under different scenarios and it has proven robust to response types, prompts, style changes and iterations.

We thank the reviewer for their positive and insightful feedback and comments that will greatly help in improving this work. Below, we address all the concerns raised.

The synthetic nature of the OCTAV dataset may limit the performance of the model in real-world applications.

Real-world applications can be a bit ambiguous, and hence we address this concern by:

i) providing a benchmark on "natural" videos. We evaluated the model on the UnAV-100 dataset (refer to Table 6, last row) in the main paper, which features real-time audio-visual events. Although the annotation style in UnAV-100 differs somewhat from our proposed OCTAV dataset, it still offers a robust benchmark for testing cross-modal and temporal understanding. As demonstrated in the example provided in the appendix (Figure 7), UnAV-100 dataset has complex, dynamic, real-world scenarios, making it a good testbed for real-world applications.

ii) providing an example with a movie trailer. We updated the section “Temporal Question Answering on natural videos” in the demo website: https://om-cat.github.io/ to capture this. Specifically, the first two examples show the capability of OMCAT model to identify musical capabilities on movie trailers from Whiplash and Wicked. The model is able to correctly identify which musical instruments are being played in the video.

iii) providing an example with an unedited, natural video from youtube. We provide an example from a random youtube video (third example in the section "Temporal Question Answering on natural videos”) on the demo website that captures cross-modal and temporal understanding. OMCAT is able to successfully “predict the activity happening in the video before the man starts speaking” and “temporally identify what is happening in the video in the second half.”

We can hence show OMCAT's suitability for real-world applications.

The technical details of methods such as RoTE are not explained in detail, which may affect understanding and reproducibility.

We have added an algorithm section in Appendix B for better understanding of RoTE. Moreover, we will make our implementation publicly available for reproducibility.

RoTE is claimed to have a better ability to capture actual time elapsed between frames while lacking more comparison of how fine-grained it can do.

RoTE’s capacity to capture fine-grained information depends on two things:

i) Sampling intervals for video and audio frames. In our setup, we sample 64 uniformly spaced video frames and 3-second audio windows, enabling RoTE to process fine details effectively within videos up to one minute long. For longer videos, this approach becomes less granular—a limitation we plan to address in future work and,

ii) Training data. As for the dataset, there is a lack of an existing fine-grained dataset that fully captures the nuances of real-world scenarios. This gap presents an opportunity for future work, where developing such a dataset could provide an even more robust benchmark for evaluating models like OMCAT.

The dataset is comprehensive but including also the evaluation dataset. This raises the concern of proposed methods.

The evaluation split of OCTAV dataset is the same as the eval split of Youcook2 and ActivityNet. Additionally, we would like to point out that for a fair comparison, we trained and evaluated the baseline models on our proposed versions of the dataset similar to OMCAT to assess the ability of OMCAT. As demonstrated in Tables 5 and 6, OMCAT significantly outperforms these baselines on the OCTAV benchmark.

You mentioned the ability to process music. Could you show some result of out-of-distribution results in the event-intensive data like movie trailers?

We updated the section “Temporal Question Answering on natural videos” in the demo website: https://om-cat.github.io/ to clarify this. Specifically, the first two examples show the capability of OMCAT model to identify musical capabilities on movie trailers from Whiplash and Wicked. The model is able to correctly identify which musical instruments are being played in the video.

Furthermore, we also provide an example from a random youtube video (third example in the section “Temporal Question Answering on natural videos”) that captures cross-modal and temporal understanding. OMCAT is able to successfully “predict the activity happening in the video before the man starts speaking” and “temporally identify what is happening in the video in the second half.”

Audio tracks often overlap. Have you considered the relevant design?

Overlapping audio tracks (e.g., a man speaking with birds screeching in the background) represents a very realistic scenario and is an important consideration for future work. We plan to extend the OCTAV dataset and the OMCAT model to address this problem in the future.

Video and music or video often have strong coupling or delay. Have you considered how to find the time node of multi-modal events? For example, due to broadcasting issues, the time node of the ball landing and the time node of the sound in a tennis video may not be synchronized. So I ask which mode should be prioritized when the tennis ball lands?

In cases where there is a timing mismatch between audio and visual signals, it’s essential to consider the specific task at hand. For tasks where accurate synchronization is crucial, such as in video-to-audio generation models, this lag in the training data can be detrimental. In our setup, we can address this by enabling the model to respond to queries such as, “when can we see the tennis ball landing?” and “when can we hear the tennis ball landing?” This dual-query approach allows us to pinpoint and evaluate any misalignment between the two modalities and act accordingly.

We thank the reviewer for their positive feedback and comments that will greatly help in improving this work. Below, we address all the concerns raised.

The experimental results are insufficient, and there is no comparison regarding the understanding ability of audio.

Thank you for pointing this out. We evaluated OMCAT on the audio question-answering task using the Clotho-AQA dataset [1], where it achieved an accuracy of 57.4%, closely matching the 57.9% achieved by Qwen-7B Audio [2], a model specifically designed for audio only understanding tasks. This demonstrates that OMCAT excels not only in cross-modal and temporal tasks but also sustains strong performance in modality-specific audio understanding. We will add these results in the paper.

How can the quality of the generated dataset be determined?

Thanks for raising this. For assessing the quality of our generated dataset, we trained OMCAT without the OCTAV dataset and compared the results of this method on the final model trained with OCTAV. As shown in the results below, training with the OCTAV dataset leads to a gain in performance not only on the OCTAV benchmark but also across temporal understanding tasks (e.g. Charades-STA).

Moreover, the QA task in OCTAV is closely tied to the ground-truth video and audio time-stamped annotations, both of which are human annotated, providing a solid foundation for quality control.

Furthermore, the GPT-4-assisted generation heavily relies on prompt tuning and in-context examples (see Appendix B.1 and B.2). This approach effectively minimizes the risk of errors or hallucinations in question-answer generation, delivering reliable quality without the need for extensive human validation.

In the appendix, the video understanding experiments, such as MSRVTT-QA, MSVD-QA, and ActivityNet-QA, show average performance.

Our work primarily aims to advance model performance in cross-modal and temporal understanding tasks. Many multimodal LLMs, such as Video LLaMA 2, leverage vast amounts of video-text paired data (around 12 million pairs), which boosts their video understanding capabilities. In contrast, OMCAT is trained on a much smaller subset of video-text data (about 2 million pairs), which naturally limits its video understanding performance. Despite this, OMCAT demonstrates competitive results in video understanding tasks and excels at fine-grained cross-modal understanding, demonstrating strong adaptability even with significantly less data.

The comparative experiments in Table 8 are not entirely fair. The data used for OMCAT with only LP, WC, and V is 430.7k less than the total dataset, so the performance of the first row being worse than the second row does not necessarily prove that the improvement is due to the characteristics of the 430.7k data rather than the difference in data volume.

We conducted preliminary experiments to evaluate the impact of audio-video-text paired data on alignment tuning. Initially, we tried adding an additional 300k video-text pairs, but this resulted in no significant improvement in model performance. In contrast, incorporating audio-video-text paired data proved highly beneficial, significantly enhancing performance on cross-modal tasks. We will clarify this in the paper.

The dataset consists of artificially added audio clips that are not aligned with the visuals and are unrelated to the overall video. However, there are no annotations regarding audiovisual misalignment in the dataset. Wouldn't this affect the model's understanding ability?

Thanks for raising this. The ability of the model is not affected, as we demonstrate its effectiveness on natural videos from the UnAV-100 dataset in Table 6, last row of the main paper.

Additionally, the challenge is to teach the network anchoring and thus we do not believe that artificial audio-visual misalignment causes an issue as also illustrated by the demo which we perform on a youtube video updated on the demo website https://om-cat.github.io/.

We updated the section “Temporal Question Answering on natural videos” in the demo website to show the performance of OMCAT on out-of-distribution videos from youtube. We provide an example from a random youtube video (third example in the section “Temporal Question Answering on natural videos”) that captures cross-modal and temporal understanding. OMCAT is able to successfully “predict the activity happening in the video before the man starts speaking” and “temporally identify what is happening in the video in the second half.”

We thank the reviewer for their positive feedback and comments that will greatly help in improving this work. Below, we address all the concerns raised.

The method innovation is somehow mediocre. The RoTE over RoPE seems to be a small but effective improvement. However, given the recent MLLM progress, the staged training approach seems standard. Interleaving tokens is effective, but others are also employing this technique.

We agree that interleaving tokens is effective but due to its computational complexity, we address these challenges with the effective improvement of RoTE. RoTE is model-agnostic, easy to implement, and optimized for efficient training without increasing computational demands. We highlight its importance through multiple experiments and demonstrate its impact in improving performance (see Table 5, 6 and 7).

Furthermore, we agree that combining diverse training data—image-text, audio-text, video-text, and audio-video-text —is a standard approach, and we do not claim it as our unique contribution. However, we emphasize that it is a critical part of our training strategy to ensure exceptional results across a range of cross-modal tasks. We respectfully disagree with any notion that the method is limited or mediocre. Benchmark results clearly demonstrate OMCAT’s adaptability and effectiveness, significantly advancing fine-grained multi-modal understanding in the audio-video domain.

The evaluation doesn’t fully support the claims that OMCAT has achieved significant gains in temporal reasoning and cross-modal alignment. First, a lot of entries in Table 5 are missing. This is quite unfortunate to draw a well-rounded comparison. Second, among them, AVSD and zero-shot Music-AVQA have relatively more methods benchmarked. On AVSD, a few methods outperform OMCAT by a 3-4% margin. On zero-shot Music-AVQA, OMCAT outperforms the second-best by a 1.6% margin.

Most of the models in Table 5 are either not open sourced (AVLLM, AVicuna) or do not provide pre-trained models for evaluation (PandaGPT, MacawLLM). Consequently, we trained the most recent AV model, GroundingGPT, from scratch for a fine-tuned and zero-shot evaluation and tested VideoLLaMA 2 using their pre-trained model as their full training code is not open-sourced.

As both GroundingGPT and VideoLLaMA 2 are state-of-the-art models in audio-video understanding, we can safely set them as strong baselines for a direct comparison.

For other benchmarks (AVSD, Music-AVQA, AVQA) which are quite widely used, we report results from other methods directly on these tasks.

Some claims were exaggerated. For example, “OCTAV dataset addresses all the above mentioned limitations and provides a comprehensive benchmark for interwoven and fine-grained audio-visual understanding.” It is not a very convincing comparison to other datasets since OCTAV doesn’t collect new videos or new annotations.

The video datasets available in literature consist of videos from various domains and realistic scenarios, making it redundant to collect new videos. However, a primary limitation in the existing datasets listed in Table 3 is the absence of timestamped, cross-modal annotations as proposed in our OCTAV dataset. Hence, our main contribution lies in defining a new task to enhance future research.

Our pipelined approach addresses this gap by carefully selecting videos with clear event transitions and supplementing them with corresponding audio events, ensuring comprehensive cross-modal alignment in the most efficient way. We will clarify this in the main paper.