Mellow: a small audio language model for reasoning

We sincerely thank the reviewer for recognizing our contribution and novelty! We hope that our response resolves your concerns. Any follow-up questions are welcome.

Mellow performs poorly on speech-based tasks ... in training. This limits generalizability.
We thank the reviewer for requesting clarification. Mellow is not trained on ASR data and therefore cannot perform speech recognition tasks. As a result, its ability to understand conversation-related information is limited. However, this is consistent with prior ALM literature, where models have focused primarily on audio and music understanding rather than speech content. For example, Pengi (NeurIPS 2023), LTU (ICLR 2024), and GAMA (EMNLP 2024) were all trained solely on audio and music data and thus exhibit similar limitations in speech understanding. While speech-based tasks are important, our current work is aligned with the audio-focused research direction and provides sufficient evidence for the claims made.

70% of ReasonAQA is synthetic; concerns may ... and generalizability of such data
We appreciate this observation and emphasize that we took deliberate steps to ensure the quality and diversity of synthetic data. Specifically, we: (1) reduced the number of generated questions per caption to improve quality; (2) generated both detailed and MCQ formats to cover varied reasoning skills; (3) explicitly enforced audio grounding in prompts to avoid world-knowledge-only questions; and (4) instructed models to incorporate acoustic properties for richer audio-specific reasoning. Evidence of improved quality includes vocabulary analysis and grounded responses (Appendix Figures 6 and Table 13). Furthermore, 30% of ReasonAQA consists of curated reasoning datasets from the literature, ensuring diversity. The effectiveness of this dataset is shown by strong OOD performance on MMAU and MMAR benchmarks, therefore showing good generalization.

While results are impressive, the evaluation would benefit ... on more speech-focused tasks
We acknowledge this point. Our primary focus is on non-speech audio reasoning, consistent with prior ALMs, and our evaluations therefore emphasize sound and music reasoning benchmarks. Nonetheless, following the reviewer's suggestion, we additionally fine-tuned QwenAudio -- a speech-capable model -- on ReasonAQA and report results below:

Even after fine-tuning, Mellow remains competitive and, in many cases, outperforms QwenAudio despite being over 50× smaller. Evaluating against speech-focused reasoning benchmarks is a promising direction for future work as we explore adding speech reasoning capabilities to Mellow.

Thank you reviewer Ae2v for reviewing our paper. We have addressed your questions in our rebuttal response. As the rebuttal period is nearing its conclusion, we kindly request you to review our rebuttal and share any additional comments or concerns you may have. Thank you once again for your valuable feedback!

We thank the reviewer for recognizing our contributions and providing constructive feedback. We address every question and hope that our response resolves your concerns. Any follow-up questions are welcome.

My main concern is that the paper lacks sufficient novelty ... a research paper.
We thank the reviewer for the comment. To provide context from the literature, one of the first ALMs released was Pengi, published at NeurIPS 2023. Pengi was a small audio-language model that achieved state-of-the-art performance at the time. In subsequent years, research in the field primarily focused on increasing the parameter count and training data for ALMs, leading to performance improvements and a shift from small to large audio-language models. However, the necessity of large ALMs has rarely been questioned. Fundamental questions -- such as what are the limits of small audio-language models? How far can we push their performance? What architectural changes improve small ALMs? Do these changes scale as more data is added? -- remain largely unexplored. We believe that answering these questions is essential to understanding both small and large ALMs and to building efficient small models. Therefore, we start with Pengi, the only small ALM in the literature, and subsequently improve upon it while keeping the model size comparable.

We only partially agree with the statement that “many of the conclusions drawn are largely aligned with what is already widely recognized by researchers.” Small Audio-Language Models (ALMs) have not been previously explored prior to this work. Although the mechanisms or architectures we use may be common, our findings are novel and not previously reported in the literature. For example, we found that larger, transformer-based projection layers -- commonly used in prior work -- only improve performance when the language model is frozen. In contrast, with full fine-tuning, simpler projection layers perform better. Our ablation studies reveal a clear performance trend for training the language model backbone: full fine-tuning > LoRA adaptation > frozen LM. This challenges the common practice of using large language models with LoRA adaptation (as done in LTU, Qwen, GAMA, etc.). Moreover, while transformer-based projection layers consistently outperform linear and non-linear projections when the language model is frozen (as in Pengi), this advantage completely disappears when the language model is trainable. In that case, simple linear or non-linear projections achieve comparable performance. This contradicts the widely held assumption that complex projection layers are universally beneficial. Overall, we believe this line of work -- focused on understanding and systematically studying small ALMs -- is as important as traditional notions of novelty that emphasize introducing new components. While some individual ablations may not be entirely unique, it is necessary to study them specifically in the context of small audio-language models before making definitive claims.

The paper attempts to justify the development of ... smaller ones or directly quantizing large models.
We thank the reviewer for bringing up this valid point. We agree that A key motivation for SALM is enabling edge deployment and reducing inference costs. Scaling from 8B to 135M parameters improves tokens-per-second throughput, resulting in faster responses, support for lower-compute devices, and reduced cloud latency and costs. In the NLP domain, this approach has already proven viable with models such as Phi, SmolLM, and MobileLLM. For concrete numbers, we evaluated Mellow and existing ALMs on an Nvidia V100 GPU, where each model was tasked with captioning 100 randomly sampled audio files from the AudioCaps test set. Throughput was computed as the number of tokens produced per second. The results are summarized below:

Higher throughput reflects better performance, with Mellow significantly faster than 7B models. Moreover, models like Qwen-Audio 2 cannot fit on a V100 GPU and require larger memory. We believe that pursuing this research direction will lead to performant yet efficient ALMs.

The fairness of the experimental comparisons ... ability of the proposed approach.
We thank the reviewer for this question. We use the MMAU benchmark, which consists of 10k annotated Audio Question Answering (AQA) pairs spanning the speech, sound, and music domains. This benchmark evaluates a model’s capabilities across 27 distinct skills involving various reasoning tasks. Notably, Mellow is trained exclusively on audio files from AudioCaps and Clotho, whereas MMAU sources its audio from 10 different datasets, with no overlap with Mellow’s training data. This makes MMAU an ideal benchmark for evaluating Mellow’s reasoning performance in an out-of-distribution (OOD) setting. However, we agree with the reviewer that it is still possible to overfit to MMAU.

To address this concern, we additionally benchmark Mellow on the MMAR benchmark. This benchmark consists of 1,000 examples spanning sound, music, speech, and their mixtures, and it focuses on multi-step, deep reasoning beyond surface-level understanding. Moreover, MMAR was released after Mellow, making it a completely independent benchmark with no possibility of overfitting. The results are shown below:

Mellow performs comparably to state-of-the-art reasoning models in the literature while being only one-tenth their size. Specifically, it outperforms all models except SALMONN, remaining competitive with Qwen2 and demonstrating strong performance on sound- and music-based questions despite having only 167M parameters.

The comparison in Table 5 may not be entirely fair ... expected to perform better.
We thank the reviewer for the question. We use linear-probe performance because it is the standard fine-tuning procedure used in the Audio Entailment paper (AAAI 25). We followed the same method to ensure comparability.

The paper emphasizes the high quality of ReasonAQA and ... sufficient to produce significantly better data?
We thank the reviewer for the clarifying question. The full data generation process is described in Appendix Sections C and D; we provide a concise summary here. Below are the some steps we took to ensure high-quality synthetic data generation:

(1) Reduced Question Count: We deliberately reduced the number of generated questions per caption from 10 (as in OpenAQA) to 5, significantly improving question quality. As noted in Section D.1, requiring 10 questions per caption in OpenAQA often led to illogical or poorly structured questions. (2) Varied Question Formats: We generated both detailed and multiple-choice (MCQ) questions, each serving different learning objectives. MCQs enforce precise answer selection and reasoning about incorrect options, while detailed questions encourage rich, descriptive responses. (3) Explicit Audio Grounding: We added specific instructions such as “you are listening to the audio instead of reading the captions” and “generate question-answer pairs about what you just heard” (Figure 5). This discourages world-knowledge-only questions that can be answered without audio (e.g., “Q: What kind of sound is sizzling? A: A hissing sound when frying food”), a common issue observed in OpenAQA (Section D.1, Table 12). (4) Acoustic Property Emphasis: We instructed the model to explicitly use words related to acoustic properties -- semantic relations, spectro-temporal characteristics, frequency, loudness, duration, materials, interactions, and sound sources. We also used few-shot prompting with audio-focused examples.

Evidence of the improved synthetic data quality includes vocabulary analysis (Figure 6), showing richer, more audio-specific terminology, and comparative examples (Table 13), demonstrating more grounded and detailed responses.

Thank you reviewer 9EBg for reviewing our paper. We have addressed your questions in our rebuttal response. As the rebuttal period is nearing its conclusion, we kindly request you to review our rebuttal and share any additional comments or concerns you may have. Thank you once again for your valuable feedback!

We sincerely thank the reviewer for the detailed comments. We reiterate the key contributions of our work: this paper presents Mellow, a small ALM capable of reasoning over audio and text, outperforming much larger models on several benchmarks (MMAU, Entailment, Difference Explanation, Captioning, AQA) while using only ~152 hours of training audio. We also introduce ReasonAQA, a 1M-instance dataset with curated reasoning questions grounded in audio content. Finally, our ablation studies reveal the effectiveness of architectural choices and training strategies in enabling reasoning in compact ALMs, and we present several additional findings.

We have carefully addressed each of the reviewer’s concerns below. Any follow-up questions are welcome.

About the novelty. The model structure seems to ... the novelty of this paper is limited.
We thank the reviewer for the comment. To provide context from the literature, one of the first ALMs released was Pengi, published at NeurIPS 2023. It was a small ALM that achieved state-of-the-art performance at the time. In subsequent years, the field largely shifted toward increasing parameter count and training data for ALMs, leading to performance improvements and a move from small to large ALMs. However, during this progression, the necessity of large models was rarely questioned. Fundamental questions -- such as what are the limits of small audio-language models? How far can we push their performance? What architectural changes improve small ALMs, and do these changes scale with additional data? -- remain largely unexplored. We believe answering these questions is crucial for understanding both small and large ALMs and for building efficient models. Therefore, we start with Pengi, the only small ALM in the literature, and subsequently improve it while keeping the model size comparable.

We disagree with the sentence "the model’s improvement seems to stem from the ReasonAQA data ... through training with higher-quality data." First, high-quality data is indeed important, and we believe ReasonAQA is a valid, novel contribution to the field, particularly since AQA datasets are currently very limited. Second, data is only one factor contributing to model performance. For example, Pengi -- the only small ALM in the literature -- achieves 3.40 on the MMAU benchmark. To test the effectiveness of ReasonAQA, we trained a Pengi-like model using ReasonAQA data alone. This model achieves a score of 30.19 on MMAU, as shown in Table 8 (line 3). We then made subsequent improvements to the projection layer, pretraining strategy, SLM fine-tuning, audio encoder, and data augmentation techniques, ultimately reaching an updated score of 48.40 on MMAU. This 18-point absolute improvement is solely due to the architectural and training improvements proposed in the paper.

Does Mellow understand speech? I believe ... speech understanding ability in Mellow also limits its novelty.
We thank the reviewer for requesting clarification. Mellow is not trained on ASR data and therefore cannot perform speech recognition tasks. As a result, its ability to understand conversation-related information is limited. However, this is consistent with prior ALM literature, where models primarily focus on audio and music understanding rather than speech content. For example, Pengi (NeurIPS 23), LTU (ICLR 24), and GAMA (EMNLP 24) are all trained solely on audio and music data, resulting in similarly limited speech understanding capabilities. While conversation-related abilities are desirable, our current work aligns with the existing audio literature and provides sufficient evidence to support the claims being made.

Regarding the rationality of the results ... the test prompts to avoid underestimating the performance of the tested models.
We understand the reviewer’s concern and thank them for raising this valid point. The numbers and benchmarks reported in Table 4 are borrowed from the Audio Entailment paper (Table 4), published at EMNLP 2024. These are standard literature-reported results with prompts optimized for each task by the respective paper authors. Since these represent state-of-the-art results, we report them directly without further prompt optimization.

Did the authors try to use ReasonAQA data for fine-tuning ... more remarkable results?
We thank the reviewer for the question. We used ReasonAQA to retrain Pengi, which is the only model with a parameter count comparable to Mellow. The results are shown in Table 8, line 3. The retrained Pengi achieves an MMAU score of 30.19, whereas Mellow achieves a score of 48.40. We did not evaluate QwenAudio or SALMONN in the submitted paper. However, following the reviewer’s suggestion, we fine-tuned QwenAudio on the ReasonAQA dataset using both LoRA fine-tuning and full fine-tuning (with no frozen parameters). The results are shown in the table below:

The full fine-tuning version performs worse than the LoRA fine-tuned model. Upon examining model responses, we see that the fully fine-tuned QwenAudio heavily overfits the ReasonAQA dataset, likely due to the mismatch between the dataset size and QwenAudio’s parameter count. In contrast, LoRA fine-tuning allows the model to adapt to ReasonAQA while retaining its broader world knowledge. Scaling ReasonAQA to better match QwenAudio’s parameter count might improve performance, but this is beyond the scope of the current work.

How does Mellow perform on other audio benchmarks besides MMAU?
We appreciate the reviewer's suggestion and assistance in improving our work. We benchmark Mellow on several other audio benchmarks, including Audio Entailment (deductive reasoning), Audio Difference (comparative reasoning), Audio Captioning, and Audio Question Answering. The results and detailed explanations are provided in Sections 4.2, 4.3, and 4.4, respectively. We specifically focus on MMAU benchmark, which consists of 10k annotated AQA pairs spanning the speech, sound, and music domains. This benchmark evaluates a model’s capabilities across 27 distinct skills involving various reasoning tasks. Notably, Mellow is trained exclusively on AudioCaps and Clotho, whereas MMAU sources its audio from 10 different datasets, with no overlap with Mellow’s training data. This makes MMAU an ideal benchmark for evaluating Mellow’s reasoning performance in an out-of-distribution setting.

However, to address concerns about overfitting or in-distribution training, we also evaluate Mellow on the MMAR benchmark. It consists of 1k examples spanning sound, music, speech, and their mixtures, and focuses on multi-step, deep reasoning beyond surface-level understanding. Moreover, MMAR was released after Mellow, making it a completely independent benchmark with no possibility of overfitting. The results are shown below:

Mellow performs comparably to state-of-the-art reasoning models in the literature while being only one-tenth their size. Specifically, it outperforms all models except SALMONN, remaining competitive with Qwen2 and showing strong performance on sound- and music questions.

Thank you reviewer rXRV for reviewing our paper. We have addressed your questions in our rebuttal response. As the rebuttal period is nearing its conclusion, we kindly request you to review our rebuttal and share any additional comments or concerns you may have. Thank you once again for your valuable feedback!

We thank the reviewer for recognizing our technical contributions, thorough experiments, clear writing, and providing constructive feedback. We address every question and hope that our response resolves your concerns. Any follow-up questions are welcome.

While the authors have intentionally constrained ... performance in practical applications where both larger models AND larger datasets are available
The reviewer raises a valid point regarding the choice of experimental setup and its implications. To reiterate the premise: in audio models, reasoning can improve through two approaches. First, expanding data (audio) coverage (e.g., knowledge of audio objects and scenes), which provides a broader foundation upon which the model can apply its existing reasoning capabilities. Second, enhancing the model’s inherent reasoning ability through architectural, learning, or post-training techniques. By fixing the training audio, one can conduct controlled evaluations of an ALM’s reasoning performance, which can later be scaled via data expansion. Therefore, we break down the reviewer’s question into two main parts: (1) do reasoning improvements achieved in isolation translate when trained on larger datasets? and (2) do models with smaller parameter counts retain competitive performance when larger datasets are available?

For point (1), we agree with the reviewer that by constraining data, we limit the model's knowledge of audio concepts and its generalization ability. However, we believe that by identifying a minimal recipe on constrained data, the approach can later be scaled (in both data and parameters) to achieve better performance. One empirical piece of evidence supporting this is shown in Table 8 (the "+ WavCaps" row) and further discussed in Appendix F7. In this experiment, we utilize the popular WavCaps dataset and, using the same generation method as ReasonAQA, generate additional training data. This data provides broader coverage of audio concepts, and the results show an absolute 1.5% improvement in overall MMAU performance. Notably, music-related performance improves by an absolute 10%.

For point (2), the WavCaps experiment provides some indication that improvements hold when the training data is increased. However, how competitive this approach would be compared to larger models trained on larger datasets remains an open question, and this work does not provide evidence to conclusively answer it. Unfortunately, due to resource constraints, we cannot scale the parameter count to conduct the necessary experiments for validation. Nevertheless, we believe this work is an essential step toward building efficient ALMs, which will eventually help close the performance gap between large models trained on massive datasets and smaller ALMs.

Could you please provide Fig. 3 ... weakness for an audio understanding model.
We thank the reviewer for this suggestion. We conducted this experiment for Mellow, where we measured the drop in performance on the music- and sound-only subsets of MMAU. However, due to space restrictions, we did not include it in the paper. The results are shown in the table below:

The results show that performance on sound and music drops significantly, while speech performance changes only marginally. This implies that the model actively utilizes audio information when answering sound and music questions but relies mostly on the language model for speech predictions. This behavior is expected since the model was not trained on speech data. Therefore, in the paper, we state: “(1) Speech-based task constitute nearly 30% of MMAU … limited speech training ..”, which explains why the overall average difference is not higher. We will add this discussion to the Appendix.

While 135M parameters are indeed smaller than cloud-scale models, ... of the intended deployment scenarios would immensely help this paper.
We thank the reviewer for raising this point and helping to improve the manuscript. Mellow, while having 167M parameters, still occupies approximately 670 MB of storage. We agree with the reviewer that most successful on-device or always-on applications typically operate in the 1-5M parameter range. However, there are multiple instances where LLaMA-like models are successfully deployed on devices such as iPhones, laptops, and tablets after applying quantization and other optimization techniques.

By scaling down from 8B to 135M parameters, we can significantly improve the tokens-per-second throughput of these models. This improvement translates to faster responses and better support for older devices on the edge side, while on the cloud side, it results in lower inference costs and reduced latency. Even in server-side deployments, where compute resources can theoretically scale infinitely, minimizing inference costs remains desirable. In the NLP domain, this approach has already proven viable with models such as Phi, SmolLM, and MobileLLM. Similarly, choosing Mellow or other small ALMs offers benefits for both edge and cloud scenarios. For concrete numbers, we evaluated Mellow and existing ALMs on an Nvidia V100 GPU, where each model was tasked with captioning 100 randomly sampled audio files from the AudioCaps test set. Throughput was computed as the number of tokens produced per second. The results are summarized below:

A higher throughput indicates better performance. As shown, Mellow is significantly faster than the representative 7B model. Moreover, models like Qwen-Audio 2 cannot fit on a V100 GPU and require larger memory. We believe that pursuing this research direction will lead to performant yet efficient ALMs.

Thank you reviewer n6cA for reviewing our paper. We have addressed your questions in our rebuttal response. As the rebuttal period is nearing its conclusion, we kindly request you to review our rebuttal and share any additional comments or concerns you may have. Thank you once again for your valuable feedback!