Watch and Listen: Understanding Audio-Visual-Speech Moments with Multimodal LLM

First of all, we would like to sincerely thank the four reviewers for their constructive feedback on our manuscript: "The paper is clearly written and easy to follow, with well-organized sections and clear explanations of both the method and experiments", "The introduction of the new TriSense 2M dataset is a valuable contribution to the community" (Reviewer ThZF); "This paper studies a very interesting problem which seeks to build an omni-modal large language model for comprehensive multi-modal understanding" (Reviewer 1DNL); "The authors propose a groundbreaking adaptive weighting framework validated by TriSense-2M" (Reviewer NaTe); "Make substantial data contributions, which are significant for omni-model exploration" (Reviewer 9tN8). Next, we will carefully address each of the reviewers’ comments.

Q1: The authors focus only on simple and basic benchmarks/tasks, without considering implications for general video or audio-visual understanding.

We would like to clarify that our task is NOT simple, we considered not only audio-visual-speech tasks and all their combinations (including audio-visual understanding), but also general understanding tasks (Table 10).

Firstly, our proposed task requires the model to natively support full-modality input and perform temporal omni-modal captioning and moment retrieval, while also supporting modality dropout—features that most VLMs do not support. Secondly, as shown in Figure 3, we provide a detailed breakdown of video durations in TriSense-2M, where 83.5% of the videos are long-form (10–20 minutes), demanding accurate temporal understanding in long-form challenging settings. Finally, as shown by our experiments in Section 5, even models like Qwen2.5-Omni, which benefit from higher-quality training data and more training resources, struggle to perform well on our proposed task. This further demonstrates that our task presents a meaningful challenge for existing VLMs.

Q4: Why do the authors only focus on temporal video understanding tasks rather than more general video understanding benchmarks?

We would like to kindly emphasize that the dataset and model proposed in this paper are specifically designed to address the poor performance of current general VLMs in omni-modal temporal understanding scenarios. It is the temporal understanding that we aim to address, rather than general video understanding tasks. This is why we chose to focus on this direction.

Additionally, we have shown that the TriSense model not only outperforms existing VLMs in temporal understanding, but also remains competitive in general understanding tasks, even when trained with only 55% of the general data, as we have included the VideoMME results in the Appendix C for your reference. We plan to include more general data in the future for a stonger model with both solid temporal & general understaning abilities.

Q2 & Q5: The authors argue that existing models struggle with audio fusion but do not explain why. Since similar query-based connector designs have appeared in prior work, is it appropriate to consider this a core technical contribution, or is the main novelty in the dataset?

Analysis was added in Sections 1 (challenges) and 2 (last paragraph) of the original paper, and more will be added in the supplementary material. Besides, we would like to respectfully claim that the proposed Query Based Connector has NOT been used by Video-LLaMA and VideoLLaMA 2, especially for the modality reweighting mechanism, with detailed explanations below:

• Architectural differences:

  1. Video-LLaMA [1] uses QFormer [2], which levergaes fixed, learnable query vectors to perform cross-attention across modalities, generating instruction-aware, fixed-length representations. In contrast, our method uses the actual user-input prompt as the query, making it flexible and only used in the initial stage. Because the cross-attention facilitates cross-modal interaction, placing the query alongside the target modality is a natural and intuitive design choice. Therefore, despite similarities, our approach differs significantly from QFormer in both the nature of the query and our overall objectives, as cross-attention is not the central contribution of our work.
  2. VideoLLaMA 2 [3]: Although VideoLLaMA 2 also incoporates audio stream, the audio connector is just a simple MLP to project the audio feature, which is totally different from the Query-Based Connector proposed in our paper.
  3. Query-Based Connector: The query attends to each modality (visual, audio, speech) separately via cross-attention to obtain query-relevant features (note: although this is similar to QFormer, it’s only our initial step). Then, all modalities pass through an MLP and softmax to obtain dynamic weights for each modality according to the query, enabling modality reweighting and temporal-aligned fusion that naturally masks out missing modalities. Note that both of the Video-LLaMA or VideoLLaMA 2 have no modality reweighting mechanism.

• Design Intentions Difference: The primary purpose of QFormer is to extract single-modality features in order to bridge the modality gap. The self-gated Multimodal Query Fusion proposed by CREMA [4] is designed to prevent the query token size from growing linearly with each new modality. In contrast, our Query-Based Connector is designed to provide a universal, simple, and effective way to extract features from all modalities and dynamically weight and fuse them. This design philosophy naturally favors a simple connector structure, that is, making it too complex would risk reduced generalizability, overfitting, and optimization difficulties.

Lastly, we want to emphasize that our key contribution lies in demonstrating that such a concise and adaptable architecture can perform well across diverse modalities, which has not been shown in prior work. We believe this offers meaningful insights for future research. The proposed TriSense-2M dataset is intended to serve as a robust pretraining foundation for large multimodal models. Additionally, the TriSense model trained on this dataset can act as a strong foundation model for future studies, supporting fine-tuning and enhancement while substantially reducing training costs. Therefore, while our dataset is a key contribution, the Query-Based Connector is also a core technical contribution.

Q3 & Q6: The authors introduce audio and speech modalities to enhance temporal video understanding, but provide no ablation studies or case examples to show whether these modalities actually improve performance.

We provide the following ablation study trained from scratch. We selected 20K samples and trained for 3 epochs, removing certain branches of the Query-Based Connector. For example, AV-Only in the table means we discarded the Speech branch. During trainning, all the weigths are initialized from [4], and LLM backbone is set to frozen. Results show that all three modalities contribute positively to all tasks, with the full AVS configuration performing best. Due to resource and time constraints, we are unable to conduct larger-scale experiments at this time, but we plan to include them in future versions of the paper.

References:

[1] Zhang, Hang, et al. "Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding." EMNLP, 2023.

[2] Li, Junnan, et al. "Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models." ICLR, 2023.

[3] Cheng, Zesen, et al. "VideoLLaMA 2: Advancing spatial-temporal modeling and audio understanding in video-llms." arXiv, 2024.

[4] Guo, Yongxin, et al. "TRACE: Temporal Grounding Video LLM via Causal Event Modeling." ICLR, 2025.

Dear reviewer 9tN8,

As you may have seen already, the authors have responded to the questions in your initial review. Can you please share your thoughts post rebuttal, once you've had a chance to see the author responses and also the other reviews?

This is critical in arriving at an informed decision.

Best, AC

Dear Reviewer 9tN8,

We appreciate your thoughtful feedback and have done our best to address all of your concerns thoroughly. If there’s anything else we can clarify or expand on, we’d be happy to do so.

Since only two days remain, we look forward to your response at your early convenience.

Best regards, The Authors

Dear reviewer 9tN8,

Thank you for improving your score and recognizing the value of our paper!

Since we are unable to provide more general understanding experiments in such short time, we will incorporate these experiments in the final version to further strengthen the paper and improve its overall quality. Furthermore, the clarification and ablation study in poin 3 and point 4 will also be included in the final version.

We sincerely appreciate your efforts, your valuable suggestions have helped up improve the quality of our paper.

We thank all four reviewers for their constructive feedback: "The paper is clearly written and easy to follow, with well-organized sections and clear explanations of both the method and experiments", "The introduction of the new TriSense 2M dataset is a valuable contribution to the community" (Reviewer ThZF); "This paper studies a very interesting problem which seeks to build an omni-modal large language model for comprehensive multi-modal understanding" (Reviewer 1DNL); "The authors propose a groundbreaking adaptive weighting framework validated by TriSense-2M" (Reviewer NaTe); "Make substantial data contributions, which are significant for omni-model exploration" (Reviewer 9tN8). Next, we will carefully address each of the reviewers’ comments.

Q1 & Q4: Training requires 16×A100 GPUs for 7 days; inference on 11.4K videos takes 8–10 hours (64–128 frames). Despite Slot-Based Compression, LLM and temporal modeling remain costly. No lightweight or quantized variants are offered. Table 4 lacks broader efficiency metrics.

Stage 3 involves full fine-tuning a 7B VLM on 2M samples, so using 16×A100 GPUs for 7 days is standard to training times reported in recent video-LLM works [1,2]. After CUDA/driver upgrades, training now reduces to ~5 days. We plan to explore lightweight variants (e.g., quantization) moving forward. Table 4 is to demonstrate that TriSense can achieve strong performance on public benchmarks with fewer frames, so it is only related to frames.

To further clarify the computational efficiency, we conducted a comparison of inference performance against recent VLMs on a single H100 GPU using 64-frame input over 500 samples. Despite two additional modalities, our model achieves competitive inference speed and outperforms Qwen2.5-Omni with Flash Attention. Importantly, the inclusion of these modalities leads to significant omni-modal performance gains, thus a marginal trade-off in inference speed is worthwhile.

Q2: TriSense-2M uses GPT-generated captions for training, risking bias. Sparse manual filtering (500 samples/batch) may let inconsistencies remain, with no analysis of bias impact provided.

Since each event has four different caption versions and we set a high acceptance threshold of 80%, our actual workload was 4–5 times higher than the basic sample count, so 500 samples per batch already resulted in a heavy workload for us. To overcome the potential bias, the pass rate for the judger was set relatively low. As mentioned at the end of Section 3, we started with 5 million samples and ultimately selected only 2 million after filtering.

To further validate annotation quality, we conducted a human evaluation on 100 randomly selected samples generated by the generator. Each was scored by a human annotator who has not seen the model output according to the rules in Figure 5. Lastly, 91% samples are passed by human. The results show that while human captions were often more fluent, model-generated captions showed strong semantic adequacy and scalability advantages. Due to text constraints, we cannot provide more samples here, but we have provided scoring samples in the response of Q3 for Reviewer 1DNL.

Q3: The model performs poorly in visual-only settings because it is optimized for multimodal synergy rather than single-modality refinement. This limits its applicability when audio or speech is missing. Although this limitation is acknowledged, no mitigation strategy is provided.

The model achieves SOTA performance on 62.5% of visual-only benchmarks, as reported in Section 5, indicating strong competitiveness. Therefore, the visual-only performance is not degraded, in the meantime, it is also optimized for omni-modal flexibility. To further strengthen visual-only capability, possible mitigation includes using a balanced ratio of omni-modal and visual-only samples (e.g., 90% and 10%), for which we are already taking actions. This may avoid overfitting to multimodal synergy while preserving single-modal ability.

Q5: Using closed-source GPT-o1 for Judger training limits reproducibility. It’s unclear if open-source LLMs were tested or how different LLMs affect the data.

We would like to clarify that, similar to previous works [1,2], closed-source models such as GPT have been widely adopted, as their instruction-following and captioning capabilities are generally recognized as superior to those of open-source models. To further improve reproducibility, we have included all our prompts in Figure 5 of Appendix B. Given the strong instruction-following ability of GPT-o1, and more importantly, the incorporation of human review, we believe that reproducibility should not be a concern.

To further evalute the generation quality of different models, we generated 100 omni-modal captions (AVS, VS, AV) with four different models and had human evaluators review each model’s outputs according to the standards in Figure 5. The results show that although open-source models like DeepSeek and Qwen3 can finish the task, their pass rates are not as high as closed models, and their outputs are not sufficiently fluent or natural in terms of English expressions. Therefore, it is necessary for us to manually select high-quality data from closed-source models and fine-tune the open-source models accordingly.

Q6: In Section 3, have you calculated the percentage of missing modal data, and what specific modal training data affects the final model performance results?

Approximately 10% of videos in TriSense-2M lack audio, in these cases, both audio and speech inputs are replaced with zero tensors. For the videos with audios, the model is forced to learn how to handle such dropout via dynamic modality reweighting, allowing robust performance under partial input conditions, as proven by our results in Section 5.

To assess individual modality contributions, we conducted an ablation study using 20K samples over 3 epochs training the model from scratch, selectively removing branches of the Query-Based Connector, benchmarking on randomly selected 500 samples. For example, AV-Only in the table below means we discarded the Speech branch. We initialize the LLM from [1] to avoid potential biases in TriSense’s LLM towards omni-modal connector.

Results show that all three modalities contribute positively to all tasks, with the full AVS configuration performing best. Visual-only and dual-modal variants perform reasonably well, confirming graceful degradation. Due to resource and time constraints, we are unable to conduct larger-scale experiments at this stage, but we plan to include more in future versions of the paper.

References:

[1] Guo, Yongxin, et al. "TRACE: Temporal Grounding Video LLM via Causal Event Modeling". ICLR, 2025.

[2] Guo, Yongxin, et al. "VTG-LLM: Integrating timestamp knowledge into video llms for enhanced video temporal grounding". AAAI, 2025.

Dear reviewer NaTe,

As you may have seen already, the authors have responded to the questions in your initial review. Can you please share your thoughts post rebuttal, once you've had a chance to see the author responses and also the other reviews?

Best, AC

First of all, we would like to sincerely thank the four reviewers for their constructive feedback on our manuscript: "The paper is clearly written and easy to follow, with well-organized sections and clear explanations of both the method and experiments", "The introduction of the new TriSense 2M dataset is a valuable contribution to the community" (Reviewer ThZF); "This paper studies a very interesting problem which seeks to build an omni-modal large language model for comprehensive multi-modal understanding" (Reviewer 1DNL); "The authors propose a groundbreaking adaptive weighting framework validated by TriSense-2M" (Reviewer NaTe); "Make substantial data contributions, which are significant for omni-model exploration" (Reviewer 9tN8). Next, we will carefully address each of the reviewers’ comments.

Q1: The overall quality of the proposed TriSense-2M dataset is not very clear. Could you provide some randomly picked examples from the dataset.

Due to NeurIPS and OpenReview rebuttal policies, we are unable to include any visual samples or external links at this stage. However, we fully acknowledge the importance of illustrating dataset quality through representative examples, we have also included a certain number of image samples in Appendix D of the initial version of the manuscript. In the meanwhile, to address this, we provide qualitative examples and evaluations in our response to Q3, where we include a selection of text-based samples spanning multiple modality combinations (audio, visual, and speech). These include source captions, generated captions, human-written references, and judger scores. To further promote transparency and reproducibility, we will release the code and data on GitHub.

Q2: According to the recommendation of the NeurIPS conferences, the authors should discuss the limitation of their proposed models and potential future improvements.

Below, we outline the current limitations of our work and future directions for improvement from two perspectives:

1. Dataset Quality: While extensive filtering reduced the dataset from 5M to 2M samples, full manual verification at this scale remains challenging. Moving forward, we anticipate further quality improvements through integration of stronger models such as GPT-o3 or Gemini 2.5 Pro. Additionally, selectively expanding the volume of human-reviewed samples and scaling the dataset to 5–10M entries without compromising quality are promising avenues that will enhance the dataset’s utility in both pretraining and transfer learning tasks.
2. Model Design (Query-Based Connector): Potential refinements include the adoption of 2D-RoPE in place of absolute positional encodings, the substitution of LayerNorm with RMSNorm for improved stability, and the incorporation of activation functions like SwiGLU, which have shown efficacy in large language models. Moreover, our ongoing work includes optimizing the dynamic modality weighting mechanism to further strengthen multimodal alignment and representation fidelity. We are already implementing these improvements and will open-source them on GitHub.

Lastly, while we acknowledge the limitations, we believe that the contributions of this paper outweigh its limitations. These limitations are not fundamental flaws but rather natural outcomes of working at the frontier of model and dataset development. We view them as opportunities that will enrich the future evolution. The proposed TriSense-2M dataset is expected to serve as a strong pretraining foundation for large multimodal models. Furthermore, the TriSense model trained on this dataset can act as a powerful foundation model for future research, supporting fine-tuning and further improvements while significantly reducing training costs. To promote reproducibility and encourage continued development, we will release the full codebase, including the training pipeline and connector implementation on GitHub.

Q3: As mentioned in Appendix B, in the dataset construction pipeline, only 10,000 and 3,000 samples are used to train the generator and the judger, respectively. This raises my concerns about their captioning and judging capabilities for reliable data annotation.

The 10,000 and 3,000 samples refer to manually reviewed sets, however, due to the multiple-caption-per-sample design (AVS, AV, VS), the actual review workload was 4–5× larger, which is already a heavy workload for us. To overcome the potential bias, we applied a strict acceptance threshold (80%) and set a low pass rate for the judger model, resulting in rigorous filtering. Out of 5M initial samples, only 2M were retained.

To further validate annotation quality, we conducted a human evaluation on 100 randomly selected samples generated by the generator. Each was scored by a human annotator who has not seen the model output according to the criterias in Figure 5. Lastly, 91/100 samples are passed by human, yielding a 91% acceptance rate. The results show that while human captions were often more fluent, model-generated captions showed strong semantic adequacy and scalability advantages. Due to text constraints, we cannot show all of them to you, but we do our best to show you some in the table below, and we will include more samples in the future version.

Q4: Could you provide a detailed explanation of the feature shapes for the vision, audio, and speech modalities? Additionally, how many tokens are ultimately fed into the large language model?

As described in Section 4.1, we use slot compression to reduce the number of tokens for each modality to 16. If we set the number of frames to 64, then for the vision, audio, and speech modalities, the feature shape will be $64$ frames $\times$ $16$ tokens of 4096 dimensions, i.e., [$64$, $16$, $4096$]. The query typically consists of 40–50 tokens, i.e., [$40$, $4096$]. After inputting these four types of features into the Query Based Connector, it outputs a multimodal feature of [$1024$, $4096$]. If the number of sampled frames is set to 128, then the modality features will be [$128$, $16$, $4096$], and the final multimodal feature will increase to [$2048$, $4096$].

Dear reviewer 1DNL,

As you may have seen already, the authors have responded to the questions in your initial review. Can you please share your thoughts post rebuttal, once you've had a chance to see the author responses and also the other reviews?

Best, AC

First of all, we would like to sincerely thank the four reviewers for their constructive feedback on our manuscript: "The paper is clearly written and easy to follow, with well-organized sections and clear explanations of both the method and experiments", "The introduction of the new TriSense 2M dataset is a valuable contribution to the community" (Reviewer ThZF); "This paper studies a very interesting problem which seeks to build an omni-modal large language model for comprehensive multi-modal understanding" (Reviewer 1DNL); "The authors propose a groundbreaking adaptive weighting framework validated by TriSense-2M" (Reviewer NaTe); "Make substantial data contributions, which are significant for omni-model exploration" (Reviewer 9tN8). Next, we will carefully address each of the reviewers’ comments.

Q1: Limited novelty in modality fusion techniques. While the Query Based Connector is positioned as the key modeling innovation of the paper, its design closely resembles existing approaches[1,2]. Specifically, the query based cross attention mechanism is highly similar to the instruction-aware QFormer proposed in InstructBLIP and X-InstructBLIP[1]. In addition, the dynamic reweighting of modalities resembles the self-gated query fusion strategy used in CREMA[2].

We would like to respectfully claim that the proposed Query-Based Connector is NOT the combination of these techniques, with detailed explanations below:

• Architectural differences:
  1. QFormer [1,2,3,4] uses fixed, learnable query vectors to perform cross-attention across modalities, generating instruction-aware, fixed-length representations. In contrast, our method uses the actual user-input prompt as the query, making it flexible and only used in the initial stage. Because the cross-attention facilitates cross-modal interaction, placing the query alongside the target modality is a natural and intuitive design choice. Therefore, despite similarities, our approach differs significantly from QFormer in both the nature of the query and our overall objectives, as cross-attention is not the central contribution of our work.
  2. Self-gated Multimodal Query Fusion [4]: In CREMA, video is treated as the primary modality, while other modalities are linearly compressed as supportive modalities, gated using a sigmoid function, and fused into a fixed-length token alongside the primary modality. In contrast, our Query-Based Connector treats all modalities equally, without designating any as primary or supportive, and does not aim to align them to the primary modality. Therefore, our proposed Query-Based Connector is totally different from Self-gated Multimodal Query Fusion from both architecture and concept.
  3. Proposed Query-Based Connector: The query attends to each modality (visual, audio, and speech) independently via cross-attention to extract query-relevant features, this is only an initial step that is similar to QFormer, but only superficially. Next, all modalities are processed through an MLP followed by a softmax layer to generate dynamic weights conditioned on the query. This enables adaptive modality reweighting and temporally aligned fusion, which naturally suppresses missing modalities. Unlike QFormer, our method goes beyond basic query-to-modality cross-attention. Unlike CREMA, we do not designate a major modality or attempt to merge other modalities into the major one. Notably, none of the referenced works incorporate modality reweighting.
• Design Intentions Difference: The primary goal of QFormer is to extract single-modality features to mitigate the modality gap. In contrast, CREMA’s self-gated Multimodal Query Fusion [4] is designed to prevent the query token size from scaling linearly with the number of modalities. Our Query-Based Connector, however, is built with a different design philosophy: to offer a universal, simple, and effective mechanism for extracting features from all modalities while enabling dynamic weighting and fusion. This approach naturally favors a concise connector architecture to maintain generalizability, avoid overfitting, and reduce optimization challenges

Lastly, we want to respectfully emphasize that the connector itself is carefully designed, incorporating design elements such as inter-frame and inter-token positional encoding, the placement of LayerNorm, and the adjustment of softmax temperature—all of which contribute to its effectiveness in omni-modal settings. Our key contribution lies in demonstrating that such a concise and adaptable architecture can perform well across diverse modalities, a capability that, to our knowledge, has not been shown in prior VLMs. We believe this offers meaningful insights for future research. The proposed TriSense-2M dataset is intended to serve as a robust pretraining foundation for large multimodal models. Additionally, the TriSense model trained on this dataset can act as a strong foundation model for future studies, supporting fine-tuning and enhancement while substantially reducing training costs.

References:

[1] Dai, Wenliang, et al. "Instructblip: Towards general-purpose vision-language models with instruction tuning." Advances in neural information processing systems 36 (2023): 49250-49267.

[2] Panagopoulou, Artemis, et al. "X-instructblip: A framework for aligning image, 3d, audio, video to llms and its emergent cross-modal reasoning." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

[3] Zhang, Hang, Xin Li, and Lidong Bing. "Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 2023.

[4] Yu, Shoubin, Jaehong Yoon, and Mohit Bansal. "CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion." The Thirteenth International Conference on Learning Representations. 2024.

Q2: Missing citations and related work discussion. Given that the Query Based Connector is the central technical innovation of the paper and even emphasized in the abstract, the lack of a thorough discussion of related work on modality fusion is a notable weakness. A more comprehensive related work section discussing these and clarifying how the proposed connector differs or improves would strengthen the paper and help position it more clearly in the landscape of multi modal fusion research.

The discussion of the Query-Based Connector in relation to existing connectors was concise in the original submission. We have now provided a detailed comparative analysis addressing this point in your last concern and will incorporate it into future revisions to further clarify the distinctions and strengthen the paper’s contribution.

Dear reviewer ThZF,

As you may have seen already, the authors have responded to the questions in your initial review. Can you please share your thoughts post rebuttal, once you've had a chance to see the author responses and also the other reviews?

Best, AC