Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities

We thank you for your thorough review and constructive feedback. We have tried to address each of your concerns point by point.

Will the dataset be released? It seems to be a huge project if the followers of this paper want to reproduce the dataset.

Ans.: Yes, absolutely! As stated on Page 1 of our paper, we will open-source all code, data, and model checkpoints. We have all approvals and plan to release the dataset publicly at the time of ICML notifications (if accepted).

For the purpose of this rebuttal, we are providing access to a part of the QA data at this anonymous link. We kindly ask that you do not share this dataset for now.

Why the authors filter out segments with audio-visual similarity above a threshold? Why a high relevance on audio and vision is not a good signal indicating a good data? How to filter out the video whose image are not quite relevant to the audio (e.g., a video with background music added)?

Ans.: We would like to clarify a potential misunderstanding. As stated in Lines 158-160 of the paper, audio-visual similarity is not used to discard low-quality segments—it is used to discard redundant or highly similar consecutive segments. Our goal is to promote diversity in the training data. Here's how the method works:

1. We segment each video (and its corresponding audio) into 10-second clips.
2. For each 10-second segment, we compute a visual embedding as follows:
   • We extract the middle frame from every 1-second interval (10 frames total).
   • Each frame is passed through CLIP to get an embedding, resulting in $\mathbb{R}^{10 \times 768}$.
   • These are mean-pooled to obtain a single visual embedding of shape $\mathbb{R}^{1 \times 768}$.
3. We compute the audio embedding of the 10-second audio using LAION-CLAP, also resulting in $\mathbb{R}^{1 \times 768}$.
4. We then average the audio and visual embeddings to obtain a single audio-visual embedding per segment: $\mathbb{R}^{1 \times 768}$.
5. Finally, we compute the cosine similarity between consecutive audio-visual embeddings and discard those with high similarity. This ensures diversity in both content and modality for CLAP training.

To address the last part of your question: if a video has low audio-visual correspondence (e.g., background music over unrelated visuals), the embeddings tend to be highly similar across segments. As a result, such segments are filtered out, since they lack useful diversity.

We will rewrite this line to make it clearer in the final version of our paper.

The audio feature seems to have a length of 64*2048, how to use it to calculate similarity with T5 feature? Mean-pooling or a linear?

Ans.: Thank you for the question. We would like to clarify a potential misunderstanding.

We follow a similar setup to LAION-CLAP for training our model, using HTSAT-Large as the audio encoder. The original HTSAT-Large model produces a feature embedding of shape $64 \times 2048$ for any 10-second audio clip.

For CLAP training, these embeddings are mean-pooled and passed through a linear projection layer. Specifically:

• Audio representations of shape $n \times 2048$ (after pooling) are obtained for a batch of size $n$.
• These are then compared to text features from CLAP, also of shape $n \times 2048$, to compute similarity.

This linear layer is trained jointly with the rest of the CLAP model.

To integrate our trained CLAP with the LLM and build Audio Flamingo 2, we remove the final linear layer and instead use the raw $64 \times 2048$ embeddings from the last layer of HTSAT as input to the LLM. This provides the model with richer temporal representations for downstream multimodal understanding.

Currently the AF2 seems to be an expert model on audio understanding. Can it be extended to 1) multiturn, 2) speech scenario (i.e., accept a spoken question and then answer)?

Ans. Thank you for the question. Yes, absolutely!

1. Multi-turn Extension: Just like the original Audio Flamingo, extending to multi-turn interaction is straightforward. We simply include the dialogue history in the text context and condition on prior audio inputs. These are separated using the special <sep> token, which is already a learned component in AF2.

2. Speech Scenario Extension: To support speech input, we add speech data to the training corpus. Empirically, we found that:

• Incorporating ASR data during Stage 1, and
• Including Speech QA data during Stage 2 significantly improves AF2’s speech understanding capabilities.

Furthermore, we observed additional benefits when replacing CLAP with a custom Whisper-based encoder capable of handling both speech and music. This modification specifically enhances performance on ASR and speech-to-text translation tasks. Extending AF2 to better handle speech is an active area of our ongoing and future work.

We thank you for your thorough review and constructive feedback. We have tried to address each of your concerns point by point.

Data Accessibility: While the authors mention open-sourcing code and data, providing a clear timeline or repository link would enhance accessibility.

Ans. As stated on Page 1 of our paper, we will open-source all code, data, and model checkpoints. We have all approvals and plan to release the dataset publicly at the time of ICML notifications (if accepted). This will include all QAs and audios released on GitHub and HuggingFace.

For the purpose of this rebuttal, we are providing access to a part of the QA data at this anonymous link. We kindly ask that you do not share this dataset for now.

Limited discussion on the model's ability to understand speech content.

As mentioned in the Abstract and Introduction, Audio Flamingo 2, like its predecessor, primarily focuses on sounds and music. Speech content understanding is intentionally out of scope for AF2, aligning with other models in this family such as Pengi [1], GAMA [2], and LTU [3].

That said, extending AF2 to handle speech is part of our ongoing and future work. We would like to share some preliminary insights:

• Minimal changes needed for speech support: Incorporating speech understanding simply requires adding speech data to the training corpus.
• Stage-wise improvements: We observe that including ASR data in Stage 1 and Speech QA data in Stage 2 significantly improves AF2’s performance on speech tasks.
• Replacing CLAP with Whisper: Using a customized Whisper model in place of CLAP enhances AF2’s ability to process speech alongside music, improving performance on ASR and speech-to-text translation tasks.

We are actively working on these extensions and will explore them more fully in future work.

Potential over-reliance on synthetic data for training, which may affect generalization.

Ans. Thank you for the thoughtful question! Most of the audio used in the AudioSkills and LongAudio datasets is sourced from real-world recordings (as mentioned in the paper). The only exception is the Counting skill, which accounts for less than 1% of the total audio data (see Table 14 for detailed statistics). While the question-answer QA pairs are synthetically generated, this modality is inherently artificial—even in human annotations. Importantly, our use of real-world audio ensures there is no sim-to-real gap during training or inference.

However, the reliance on synthetic data and LLM-as-a-judge framework for evaluation could introduce biases or limitations in the results.

Ans. Thank you for raising this concern. We adopt the LLM-as-a-judge evaluation framework following widely used practices in long-form video and multimodal benchmarks such as Video-MME [5], as well as other recent LLM evaluation benchmarks for audio understanding such as CompA-R [1] and AIR-Bench [5]. Prior work [4] has demonstrated that this framework provides a more robust and semantically grounded measure of generation quality, especially for open-ended tasks. It also mitigates common issues in traditional evaluation approaches, such as overly strict regex matching or limited coverage of reference answers.

To reduce potential bias, we carefully designed our evaluation prompts (see Fig. 25) to be neutral, consistent, and faithful to the context of each QA pair. We acknowledge that synthetic evaluation frameworks have limitations, and we view this as an active area of improvement for future work.

Citations

[1] GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities (https://aclanthology.org/2024.emnlp-main.361/).
[2] Pengi: An Audio Language Model for Audio Tasks (https://openreview.net/forum?id=gJLAfO4KUq).
[3] Listen, Think, and Understand (https://openreview.net/forum?id=nBZBPXdJlC).
[4] AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension (https://aclanthology.org/2024.acl-long.109/).
[5] Video-MME The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (https://arxiv.org/abs/2405.21075).
[6] Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMs (https://openreview.net/forum?id=3PRvlT8b1R)

Thank you for the encouraging review. We are happy you liked our paper. We'd just like to clarify that, in addition to the dataset contributions, our paper also presents several modeling insights that we believe are novel and impactful:

• Dynamic batching for efficient training: As described in Appendix H.2, we introduce a dynamic batching strategy based on audio length. This significantly reduces padding, improves training efficiency, and yields faster convergence with better model performance.

• Efficiency in long audio modeling: Section 3.2 explains how our AF2 module acts as an effective alternative to prefix-based architectures for long audio inputs. By leveraging cross-attention instead of increasing context length, we avoid the quadratic time complexity associated with standard attention mechanisms.

• Superiority of cross-attention over prefix tuning: Section 6.5 demonstrates that cross-attention outperforms prefix tuning on the same dataset, indicating its effectiveness for audio-text alignment and long-context understanding.

• Curriculum learning strategy: In Section 6.7, we compare 10 training schedules and show that our proposed curriculum learning strategy consistently achieves the best performance. We further highlight a counterintuitive insight: fine-tuning the language model weights degrades performance, and curriculum design is crucial for reasoning over long-form audio.

• High-quality data over large models: Finally, Section 6.6 shows that training smaller LLMs on high-quality audio-text data results in performance that matches or exceeds that of larger LLMs trained on lower-quality data—underscoring the importance of data quality over model size for audio reasoning.

We thank you for your thorough review and constructive feedback. We have tried to address each of your concerns point by point.

Thus, I would like to ask the authors to explicitly explain their plan in releasing the training data and the evaluation benchmark during the rebuttal period.

Ans. As stated on Page 1 of our paper, we will open-source all code, data, and model checkpoints. We have all approvals and plan to release the dataset publicly at the time of ICML notifications (if accepted). This will include all QAs and audios released on GitHub and HuggingFace.

For the purpose of this rebuttal, we are providing access to a part of the QA data at this anonymous link. We kindly ask that you do not share this dataset for now.

I believe that a clearer narrative would have focused on the dataset and the evaluation, while benchmarking several existing models on this data, which would leave even more room for discussing in depth the challenges of creating audio understanding datasets.

Ans. Thank you for the suggestion. We did benchmark the strongest fully open-source LALM with available training code—GAMA—on our dataset. As shown in Table 5, simply introducing new data is insufficient to significantly improve audio understanding performance.

Our results highlight that it is the combination of better audio perception (via stronger audio encoder), the cross-attention architecture, and curriculum learning that leads to meaningful improvements. This holistic approach is critical for the unique challenges posed by long audio understanding. Moreover, our architecture choice of cross-attention also supports our first exploration of tackling long-form audio reasoning . Section 3.2 of the paper details how our cross-attention mechanism offers a more efficient and effective alternative to prefix-based architectures, avoiding the quadratic time complexity that arises from increasing context length in standard attention mechanisms.

On contributions beyond just data (AudioSkills)

• Efficiency in long audio modeling: Section 3.2 explains how our AF2 module acts as an effective alternative to prefix-based architectures for long audio inputs. By leveraging cross-attention instead of increasing context length, we avoid the quadratic time complexity associated with standard attention mechanisms

• Dynamic batching for efficient training: As described in Appendix H.2, we introduce a dynamic batching strategy based on audio length. This significantly reduces padding, improves training efficiency, and yields faster convergence with better model performance.

• Superiority of cross-attention over prefix tuning: Section 6.5 demonstrates that cross-attention outperforms prefix tuning on the same dataset, indicating its effectiveness for audio-text alignment and long-context understanding.

• Curriculum learning strategy: In Section 6.7, we compare 10 training schedules and show that our proposed curriculum learning strategy consistently achieves the best performance. We further highlight a counterintuitive insight: fine-tuning the language model weights degrades performance, and curriculum design is crucial for reasoning over long-form audio.

• High-quality data over large models: Finally, Section 6.6 shows that training smaller LLMs on high-quality audio-text data results in performance that matches or exceeds that of larger LLMs trained on lower-quality data—underscoring the importance of data quality over model size for audio reasoning.

• Long audio modeling: (also acknowledged by the reviewer) We explore long audio reasoning for the first time and make attempts to build training and benchmarking datasets for this purpose.

• A proof that better and robust audio representations can improve performance: Table 3 highlights that AF-CLAP improves Audio Flamningo 2 performance across benchmarks. Beyond the chosen benchmarks (primalrity taken from literature), as also mentioned in Section 3.1.1, AF-CLAP may help Audio Flamningo 2 performance in various real-world scenarios by expanding its breadth of audio understanding (e.g., home sounds). Finally, robust representations also help our UnusualQA task (in demo) as the representations capture fine-grained information about the input audio.