MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark

We thank you for your thorough review and constructive feedback. In the rebuttal, we have tried to address each of your concerns point by point.

Weakness 1 (task may require multiple skills)

We thank you for your observation. To quote from our current paper, “although individual questions may require multiple skills from each respective category”. We apologize if this was confusing and not sufficiently supported. To be precise, MMAU has 18% of questions that require more than one skill to solve and 8% of questions that require one skill each from information extraction and reasoning to solve. For better clarity, we have added more details to Appendix I and also added such examples in Table 12 of the Appendix. Below are a few examples for the purpose of the rebuttal:

{
        "question": "During what time frame can you hear the chord G# in the audio?",
        "choices": [ "0.00 - 2.22", "2.22 - 4.44", "4.44 - 6.67", "6.67 - 8.89" ],
        "answer": "0.00 - 2.22",
        Skills involved: Harmony and Chord Progressions, Temporal Reasoning
}
{
        "question": "What is the nature of the weather in the scene?",
        "choices": ["Snowing heavily", "Sunny day", "Windy", "Heavy rain"],
        "answer": "Heavy rain",
        Skills involved:  Acoustic Scene Reasoning, Eco-Acoustic Knowledge
}
{
       “question”: “What makes the last comment sarcastic in relation to the dialogue?”
       “Choices”: [“Criticism of scientific method”, “Genuine admiration of intelligence”, “Requesting further explanation”, “Mocking exaggerated praise”]
       “Skills Involved”: Multi Speaker Emotion Reasoning, Dissonant Emotion Interpretation
}

We have also included these examples in the Appendix I section of the revised paper.

Weakness 2 (source selection of data)

Thank you for your question. In Section 3.2, we describe our data sourcing process, where we exclusively used audio from test sets or evaluation sets (when test sets were unavailable) to ensure unbiased and robust evaluation. Specifically:

• Music: For music data, we utilized labeled test audio files, ensuring that the corresponding questions were highly relevant to the task at hand.
• Sound: We utilized the evaluation set of AudioSet Strong, selecting audio clips containing a minimum of two and a maximum of five unique acoustic events, with each event lasting at least two seconds. This selection criterion ensured the inclusion of high-quality, distinguishable audio samples suitable for reasoning tasks.
• Speech: For speech data, additional quality checks were applied by considering the lengths of transcriptions and ground truth labels, ensuring that the selected samples were clear and of adequate length for generating meaningful questions and answers. These were preliminary checks conducted before the actual data filtering stage, which involved expert-driven refinement to ensure quality and diversity in the final selection of 10,000 data points. We have added this information in Appendix D.5 in the revised version of the paper.

Weakness 3 (quality check and bias control)

Thank you for your question. Bias control in our annotation pipeline is implemented through a two-stage process:

• Filtering Stage: As outlined in Section D. 1, we engage a diverse group of six expert annotators to mitigate individual biases and ensure consistency. Annotators follow a standardized set of guidelines, such as discarding short audio samples and those with mismatched labels, to establish a uniform understanding and baseline quality for the annotations.
• Expert Review Stage: After the option augmentation process, each annotator independently scores questions generated by other experts on a 1-to-5 scale. Low scores are assigned to questions with misleading or overly correlated options, as well as those with incorrect answers. This scoring ensures the filtering of subpar samples and contributes to the reliability of the dataset.

This structured, multi-stage approach ensures quality control and minimizes bias in the final selection of 10,000 samples in the MMAU dataset.

Weakness 4 and 5 (open-ended vs MCQ evaluation)

Thank You for the question! Our choice to use multiple-choice questions in MMAU is inspired by a wealth of prior research in vision [2] and language [1], which often rely on MCQ format for structured and scalable assessments. We would like to point out several reasons why we, and a plethora of prior work, have preferred MCQ-type questions: Open-ended responses require employing LLM-as-a-judge. While they allow evaluating more free-form responses by LALMs, this method faces several limitations including:

(i) High evaluation costs for powerful judges, which are almost always proprietary

(ii) LLM-as-a-judge evaluation lacks standardization. Proprietary LLMs are taken down by their respective companies, and different researchers using different LLMs might lead to discrepancy in results due to different knowledge in them

(iii) Lack of strong judges that can't (yet) take as input audio and return a faithful score for a response.

By using MCQ-type questions, we not only identify the correct answer but also assess the model’s ability to eliminate plausible but incorrect distractors. This approach provides a more nuanced evaluation of the model’s reasoning capabilities, as it tests not just knowledge but also the process of reasoning through competing options. MCQ questions also allow the usage of standardized metrics like accuracy. This promotes fairness in the evaluation process.

Finally, upon your request, we provide you a correlation between our regex matching and a more sophisticated GPT-based scoring of open-ended generations (more details in Appendix B.5) below:

As we can see, both scores are highly correlated, thus indicating the robustness of our regex evaluation protocol. We have also added this comparison in the revised version of the paper in Appendix B.5.

To also note, for reproducibility, we will open-source all codes. The current code is available in the Supplementary materials. Our evaluation script is adopted from well-established benchmarks MMLU [1] and MMMU [2].

Question 1 (distinction between reasoning and extraction tasks)

Thank you for this insightful question. The distinction between reasoning and extraction tasks was made to establish two broad categories for evaluating the skills of large audio-language models. This separation allows us to assess these models' capabilities individually on different skill sets, providing a structured framework for identifying their strengths and weaknesses. Such a framework is critical for guiding future research and enabling focused improvements in specific areas. As this is the first benchmark of its kind, we intentionally adopted this distinction to simplify the evaluation process and establish baseline performance metrics for LALMs across well-defined skill categories. We recognize that some tasks inherently require a combination of reasoning and extraction skills, and integrating such tasks is a natural next step for improving the comprehensive coverage of MMAU.

Here are some examples of questions from our benchmark requiring both reasoning and extraction skills:

{
        "question": "During what time frame can you hear the chord G# in the audio?",
        "choices": [ "0.00 - 2.22", "2.22 - 4.44", "4.44 - 6.67", "6.67 - 8.89" ],
        "answer": "0.00 - 2.22",
        Skills involved: Harmony and Chord Progressions, Temporal Reasoning
}
{
        "question": "What is the nature of the weather in the scene?",
        "choices": ["Snowing heavily", "Sunny day", "Windy", "Heavy rain"],
        "answer": "Heavy rain",
        Skills involved:  Acoustic Scene Reasoning, Eco-Acoustic Knowledge
}

Question 2 (correct option generation in prompt)

Thank You for the question! In Appendix L, the prompt is designed to instruct the model to generate contrasting options for each question, including the correct answer and distractor options. By providing the correct answer in the prompt, the model is tasked with formatting it alongside the distractor options, streamlining the process, and ensuring consistency in the generated output. This eliminates the need to add the correct answer later. To ensure reliability, we included a check to confirm that the correct answer was present in the generated options.

Dear reviewer 2krx,

Thank you for taking the time to review our paper and thank you for the insightful engagement. Your comments have helped us improve the quality of the paper. This message is to request you to review our last response and the revised paper. We have made all changes suggested by you, responded to your questions and also made revisions to our paper to reflect the changes. Thank you once again for your valuable feedback!

Best,
Authors of Submission3927

We thank you for your thorough review and constructive feedback. We have tried to address each of your concerns point by point in the rebuttal.

Weaknesses

Weakness 1 (About MCQ Questions)

Ans. Thank you for your suggestion regarding open-ended questions. Our decision to use multiple-choice questions (MCQs) in MMAU is motivated by extensive prior research in vision [6,8] and language [5,7], where MCQ formats are widely used for structured and scalable evaluations.

Here are several reasons why we, along with much of prior work, prefer MCQ-type questions:

• Open-ended responses typically require using an LLM-as-a-judge, which presents several limitations: (i) The most powerful (and widely used) judges, such as GPT-4 or Gemini Pro, are proprietary and expensive to use. (ii) Proprietary LLMs may be retired by their companies, and using different LLMs as judges can lead to inconsistent results due to differences in knowledge and behavior. (iii) There are no sufficiently robust LLMs capable of evaluating audio-based inputs and returning reliable scores for open-ended responses.

• MCQs allow us to not only identify the correct answer but also assess the model’s ability to distinguish between plausible but incorrect distractors. This provides a more nuanced evaluation of the model’s reasoning capabilities, testing both knowledge and the process of reasoning through competing options.

• MCQs enable the use of standardized metrics like accuracy, which ensures fairness and comparability in the evaluation process across models and research efforts.

However, in light of the request, we have planned to add a subset of 500 QA pairs in the final version of MMAU post acceptance. We apologize for not being able to accomplish this in the short period of rebuttal but promise to do this in the final version.

Weakness 3 (About scores for Audio -Language Encoders)

Ans. Thank You for the question! The primary reason for separating the results of contrastive Audio-Language Encoders (ALEs), such as CLAP, is that their evaluation strategy differs significantly from that of LALMs, as detailed in Table 3. As explained in Appendix Section B.1, ALEs inherently lack instruction-following capabilities. Consequently, their evaluation requires an ad hoc method similar to the approach described by Deshmukh et al. (2024). In this method, each question-choice pair is transformed into a hypothesis using GPT-4. The audio and hypotheses are then encoded using ALEs, and the hypothesis with the highest cosine similarity to the audio embedding is selected as the answer.

We appreciate the reviewer’s suggestion to summarize the key findings of the ALE experiments in the main paper. We have added this detail to Section 5.1 of our paper, where we provide a concise summary of ALE performance on MMAU.

Questions

Question 1 (About regular expressions for evaluation)

Ans. Thank You for the question. Our regex method to evaluate responses has been inspired by well-established benchmarks like MMLU[1] and MMMU[3]. In addition, our evaluation protocol is standard across benchmarks[1,3,4,5]

At your request, we provide you a correlation between our regex matching and GPT scores below:

As we can see, both scores are highly correlated, thus indicating the robustness of our regex evaluation protocol. We have also added this comparison in the revised version of the paper in Appendix B.5.

Question 2 (About AudioSet strong and possible data leakage)

Ans. Thank You for your question. All questions for MMAU for all datasets were sourced from the test set of all datasets. For AudioSet strong, we employ the eval set. Thus, we can confirm there is no data leakage for any dataset used to evaluate MMAU. We have added this detail to Section 3.2 of our paper.

Question 3 (About synthetic data generated)

Ans. We synthesized audio data only for tasks where no open-source audio datasets were available. For audio generation, we utilized Text-to-Audio models such as Parler-TTS for speech and Stable Audio for other audio types.

For instance, in the task of Multi-Speaker Role Mapping, no open-source speech datasets explicitly provide information about the roles of speakers in a conversation. To address this gap, we collaborated with domain experts to design synthetic textual conversations that clearly define each speaker's role (e.g., doctor, patient, etc.). These conversations were then converted into speech using text-to-audio models. Finally, domain experts reviewed the generated speech and curated reasoning-based question-answer pairs tailored to the specific skill being assessed.

continued

The list of tasks for which synthetic data was generated, along with their respective counts, is as follows:

Detailed information about the synthetic data generation process has been added in Appendix C.

Weakness 2 (About distinction between Information extraction and reasoning)

Ans. We appreciate the reviewer’s concern and acknowledge the limitation of treating the skills required for information extraction and reasoning as entirely separate sets. As highlighted in the limitations section, the current version of MMAU does not include many questions that require a combination of these skills. We recognize that incorporating such overlapping skills is crucial for a more comprehensive evaluation and plan to address this in future benchmark iterations.

However, MMAU has few questions (~8%) with skills needed from both information extraction and reasoning. We present a few below:

{
        "question": "During what time frame can you hear the chord G# in the audio?",
        "choices": [
            "0.00 - 2.22",
            "2.22 - 4.44",
            "4.44 - 6.67",
            "6.67 - 8.89"
        ],
        "answer": "0.00 - 2.22",
        Skills involved: Harmony and Chord Progressions, Temporal Reasoning
}

{
        "question": "What is the nature of the weather in the scene?",
        "choices": [
            "Snowing heavily",
            "Sunny day",
            "Windy",
            "Heavy rain"
        ],
        "answer": "Heavy rain",
        Skills involved:  Acoustic Scene Reasoning, Eco-Acoustic Knowledge
}

We have added this detail to the Appendix I section of the revised version of our paper.

References:

[1] Hendrycks, Dan, et al. "Measuring massive multitask language understanding." arXiv preprint arXiv:2009.03300 (2020).
[2] Meng, Fanqing, et al. "Mmiu: Multimodal multi-image understanding for evaluating large vision-language models." arXiv preprint arXiv:2408.02718 (2024).
[3] Yue, Xiang, et al. "Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
[4] Weck, Benno, et al. "Muchomusic: Evaluating music understanding in multimodal audio-language models." arXiv preprint arXiv:2408.01337 (2024).
[5] Yang, Qian, et al. "AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension." arXiv preprint arXiv:2402.07729 (2024).

We thank you for your thorough review and constructive feedback. We have tried to address each of your concerns point by point in the rebuttal.

Weaknesses

Weakness 1 (About long audios)

Ans. Thank you for raising this important question! We fully acknowledge that long audio could provide a broader evaluation of LALM performance. However, most current open-source and open-access Large Audio-Language Models (LALMs), such as LTU (10 seconds), GAMA (10 seconds), SALMONN (10 seconds), and Qwen2-Audio (30 seconds), are limited to processing short audio inputs. This limitation also stems from the majority of training data available for these models, as well as the audio encoders, which typically support inputs of 30 seconds or less.

Given these constraints, we have focused the current version of MMAU on reasoning over short audio clips, leaving long audio processing outside the current scope. However, we recognize the importance of this area and plan to include long audio reasoning as an immediate goal for future versions of MMAU. This approach is analgous in video understanding research, where initial efforts centered on short video benchmarks [1,2], eventually transitioning to long video understanding [3,4] as more capable long video models emerged [5,6]. We believe similar advancements will occur in the audio domain as the research community continues to push LALM capabilities.

Following are the average durations for MMAU:

We have included additional details and discussion in the Limitations and Future Works section and updated Table 1 of our paper.

Weakness 2 (About MCQ)

Ans. Thank you for your suggestion regarding open-ended questions. Our decision to use multiple-choice questions (MCQs) in MMAU is motivated by extensive prior research in vision [6,8] and language [5,7], where MCQ formats are widely used for structured and scalable evaluations.

Here are several reasons why we, along with much of prior work, prefer MCQ-type questions:

• Open-ended responses typically require using an LLM-as-a-judge, which presents several limitations: (i) The most powerful (and widely used) judges, such as GPT-4 or Gemini Pro, are proprietary and expensive to use. (ii) Proprietary LLMs may be retired by their companies, and using different LLMs as judges can lead to inconsistent results due to differences in knowledge and behavior. (iii) There are no sufficiently robust LLMs capable of evaluating audio-based inputs and returning reliable scores for open-ended responses.

• MCQs allow us to not only identify the correct answer but also assess the model’s ability to distinguish between plausible but incorrect distractors. This provides a more nuanced evaluation of the model’s reasoning capabilities, testing both knowledge and the process of reasoning through competing options.

MCQs enable the use of standardized metrics, such as accuracy, which ensures fairness and comparability in the evaluation process across models and research efforts.

Questions

Question 1 (On synthetic data)

Ans. We synthesized audio data only for tasks where no open-source audio datasets were available. For audio generation, we utilized Text-to-Audio models such as Parler-TTS for speech and Stable Audio for other audio types.

For instance, in the task of Multi-Speaker Role Mapping, no open-source speech datasets explicitly provide information about the roles of speakers in a conversation. To address this gap, we collaborated with domain experts to design synthetic textual conversations that clearly define each speaker's role (e.g., doctor, patient, etc.). These conversations were then converted into speech using text-to-audio models. Finally, domain experts reviewed the generated speech and curated reasoning-based question-answer pairs tailored to the specific skill being assessed.

The list of tasks for which synthetic data was generated, along with their respective counts, is as follows:

Appendix C of the revised version of our paper now includes detailed information about the synthetic data generation process.

Continued

Thanks for the authors' reply. Could you please give some examples of questions with different difficulties? Also, how do humans perform on questions of varying difficulties? Besides, for humans, is there a similar phenomenon to that of models, whereby they perform poorly on certain types of tasks?

We apologize for missing your question earlier. Amidst many queries, we updated the relevant section in the paper (Appendix D.4) but missed addressing it in our response. Here is our response to Question 2:

Question 2 (determining difficulty of a question)

Ans. As mentioned in Section 3.1, each question in MMAU requiring specific skills is manually categorized by domain experts into easy, medium, or hard difficulty levels. Our domain experts rated the difficulty of each question from 1-10. For each question, we then averaged the scores from each expert and assigned the question to be “easy” if the score was between 1-3, medium if 4-6, and hard if > 6. The score for the difficulty level was assigned based on the level of expertise or information required to respond correctly to a question. We have added this detail to Appendix D.4 of our paper.

We thank you for your thorough review and constructive feedback. We have tried to address each of your concerns point by point in the rebuttal.

Weaknesses

Weakness 1

Ans. We appreciate your recognition of the significance of our work. Section 5.1 provides a discussion on the performance comparison of different LALMs, and we would like to take this opportunity to elaborate further:

• Model size is a crucial factor influencing performance on MMAU. Larger LLMs inherently store more knowledge and demonstrate superior reasoning capabilities. As a result, LALMs incorporating larger LLMs perform better. For instance, LTU, which has 6 billion fewer parameters than SALMONN, shows an average performance drop of 14% in Table 3.
• Models trained for specific domains outperform general-purpose models on corresponding tasks. For example, M2UGen, designed for music understanding and reasoning, surpasses general-purpose models like LTU and GAMA by up to 15% on music-related tasks.
• Proprietary and open-access models outperform fully open-source models across all tasks. This suggests that larger datasets and additional training resources likely contribute to enhanced audio perception and reasoning performance on MMAU.

We believe these insights will further strengthen the significance of our findings and we have added these details to the revised version of our paper.

Weakness 2

Ans. Thank you for your question! We did not conduct few-shot (or in-context learning) experiments because most openly available Large Audio-Language Models lack support for multiple audio inputs, a feature essential for enabling few-shot learning. As a result, comparing the limited number of models that support this functionality would have been challenging. However, based on your request, we are sharing results for Qwen2-Audio, which does support multiple audio inputs. Below are the results on the test-mini subset:

Performance comparison of Qwen2-Audio on test-mini across different few-shot settings.

We have also added these results to the revised version of our paper in Appendix B.3.

Weakness 3

Ans. Thank you for your question! You are correct that OpenAI's audio APIs were not available at the time of paper submission. However, upon your request, we explored the recently launched OpenAI audio APIs and found that OpenAI currently provides two ways to interact with GPT models using voice inputs:

• Real-Time Mode: This mode supports speech-to-speech interactions and can also handle text inputs and text outputs. However, it does not allow simultaneous use of text and audio inputs.

• Voice Mode: This mode extends GPT inputs to include audio. Unfortunately, this feature is only accessible to GPT Plus users through the ChatGPT UI and does not have a public API available. Given the size of our benchmark, evaluation through the UI is not feasible.

Due to these limitations, we regret that we are unable to provide GPT scores at this time. However, we will monitor future developments and update our paper for the camera-ready version (if accepted) or in future revisions if OpenAI releases an API that enables evaluation on MMAU.

Questions

Question 1

Ans. The experiment depicted in Figure 5 aims to investigate whether LALMs genuinely attend to audio inputs when answering complex MCQ questions in the MMAU benchmark. In this setup, we present the average accuracy scores across all tasks. Under the noisy condition, the models are not provided with the ground truth audio captions. To ensure the model responds, we explicitly instruct it to select an answer for each question.

continued

Below we provide the task-wise breakdown of the scores for the experiment in Section 5.2:

Overall, we observe that:

• Our benchmark is robust as most models perform close to random chance when addition of white noise.
• When LALMs do not undergo performance degradation with white noise, their scores are ideally close to random chance with and without white noise.
• LALMs like SALMONN and MuLLaMa exhibit minimal performance degradation with white noise. This suggests that these models may be guessing randomly or relying on strong language priors from their LLM counterparts to select an answer.

We have added the complete Table and discussion to the Appendix B.4 of our paper.

Dear reviewer W76o,

Thank you for taking the time to review our paper. We have addressed your concerns in our submitted response and provided a revised version of the paper. As the rebuttal period is nearing its conclusion, we kindly request you to review our rebuttal and share any additional comments or concerns you may have. Thank you once again for your valuable feedback!

Best,
Authors of Submission3927

Dear reviewer W76o,

Thank you for taking the time to review our paper. We have addressed your concerns in our submitted response and provided a revised version of the paper. As the rebuttal period is nearing its conclusion, we kindly request you to review our rebuttal and share any additional comments or concerns you may have. Thank you once again for your valuable feedback!

Best,
Authors of Submission3927