LLark: A Multimodal Foundation Model for Music

Thank you to the reviewer for their thorough analysis of the paper and even the supplementary material. We are pleased that the reviewer acknowledged the modeling improvements in the work, found the data and technical details “comprehensive” and “clearly communicated,” and felt the paper “presents a clear and compelling path for future MIR work in this direction”. We discuss the reviewers’ main concerns below, but would like to highlight up front that we have made several changes and additions to the paper in order to reflect the reviewers’ concerns, and we believe they have improved the paper considerably. These include: a new supplementary section on “Failure Cases” (Section N), a Model Card (Section M) to highlight limitations and concerns about potential downstream use, and updates to the Ethics statement (Section A). We are adding content to the online supplement to highlight the behaviors noted by the reviewer and discussed in these sections as well.

We interpret one of the reviewers’ main concerns to be a need for more clearly highlighting and understanding the ways in which our model’s outputs can be misleading or incorrect. As the reviewer notes, LLMs have a tendency to generate “impressive text” with a “sense of authority” and this can be confusing or downright detrimental in some use cases when the model is incorrect. The persuasiveness of this kind of text only grows as the models’ language modeling capabilities improve. And while we made careful efforts to evaluate the music understanding capabilities of the model (e.g. Section 6.1), even long-form text responses are only helpful insofar as they are also correct, whether users know it or not. We share the reviewers’ concern, and both assessing and handling the (un)reliability of outputs is widely acknowledged to be an open research question [1] and a major challenge for any language mode. This is made more challenging by the fact that there is no current methodology for language models to convey their uncertainty in the way that a human expert would; instead, autoregressive text generation models simply output the sequence of tokens which maximize the estimated probability conditional on the inputs -- regardless of whether the models’ estimates of these probabilities is reliable, or even whether the predictions themselves are confident.

We share this concern, and have made several revisions to both clearly highlight potential issues with LLark’s outputs, make clear potential (mis)uses of the model, and clearly highlight risks and limitations.

In particular, we have made the following changes:

• The reviewer identifies a subset of problematic examples from the online supplement. In order to better highlight and categorize such behaviors across all tasks and datasets, we have introduced a new “Failure Cases” section to the paper (Section N), motivated by similar efforts in recent multimodal model releases (e.g. [2]). We have explicitly analyze some of the examples the reviewer mentions, among many others, in this new section, and we identify and discuss five important failure modes of the LLark model.
• We have added a Model Card (Section M; we discuss this in detail below).
• We have added a clear statement regarding the risks and limitations of using language models in this context to the ethics statement of the paper in (Section A).

Response to question: The reviewer asks “Given the time to do it, how would you design an evaluation to address the kinds of issues raised under "weaknesses" above? How much of that can be communicated within this paper rather than left to future work?”

The reviewer raises an important question. We offer several thoughts:

• (1) One approach we believe is clearly motivated here: to clearly identify and highlight failure cases of the model. This has been used in other multimodal works, for example [2], and is in line with a broader movement toward investigating and publicizing the undesirable outputs of models (“red teaming” [3] is a version of this for safety-critical applications, where the model is explicitly prompted to perform dangerous tasks). We have added this to the paper in a new “Failure Cases” section (Section N), and we discuss five prominent failure cases (N.1-N.5), including examples, potential causes, and offer potential solutions to address each.
• (2) For each failure mode, we propose specific mitigation strategies in Section N. We believe that each of these strategies, if implemented, could significantly reduce these failure modes, and each could be verified with a small set of targeted evaluations specific to the failure mode.

Once again, we thank the reviewer for your time reviewing the paper. As the author-reviewer phase is ending soon, we request the reviewer to please review the author response and let us know whether the comments and revisions to the paper (indicated in red text) have addressed your concerns, or whether we can provide any further clarification before the discussion window closes. Thank you!

• Performance vs. state-of-the-art (SOTA) on MIR tasks [cont]:
  • Clarification regarding linear probing: We believe it is important to note that some of the models the reviewer highlights as SOTA, including MERT and the model of [23], achieve the reported performance only after fine-tuning a probe directly on each target task (this procedure is described in [9], section B.2, and [23], Fig. 1). Thus, while these are useful to provide an upper bound on the best possible performance for the tasks, these represent the performance by a model explicitly trained on those datasets, not generalist models never exposed to the target task or data. Thus, we believe the best comparison for LLark is other audio-text models which can generalize directly to new tasks, without fine-tuning on task-specific data.
  • Discussion of genre on GTZAN: The reviewer highlights the GTZAN dataset. Since the reviewer is undoubtedly familiar with the literature, we expect the reviewer also understands that, while it is used widely, GTZAN is also a problematic dataset that can be unreliable, in isolation, for evaluating deep music understanding. The issues with this dataset have been widely discussed in the literature, as we are sure the reviewer is aware (for example, it assigns only a single genre to each song; genres are assigned by artist, not by track; the 10 genres selected are arbitrary and in fact often different levels of the same branch of a genre hierarchy, such as “metal” and “rock” or “disco” and “pop”). We believe that the detailed results on GTZAN present in the supplement (e.g. Figure 10) show that LLark’s responses on this dataset are in fact reasonable, and in line with responses humans would make (for example, LLark most often labels “metal” songs as “rock” -- an entirely reasonable response for a model that is maximizing the probability of its outputs). We note that in the second genre dataset (MedleyDB), the gap between LLark and the top-performing model (IB-LLM) is less than 1pp, which reflects the uniqueness of GTZAN and the need to conduct a holistic evaluation without overly emphasizing a single data point, particularly for ill-defined tasks such as genre classification.
  • Question about tempo estimation SOTA: The reviewer suggests that [20] is SOTA on Giant Steps tempo estimation. However, we could not find tempo reported in this paper (https://archives.ismir.net/ismir2022/latebreaking/000049.pdf). Could the reviewer please confirm the source of this number? Otherwise, it appears to us that the current best SOTA on Giant Steps tempo is the model of [21], reported in [22] as Acc2 of 92.5.
• Comparison to representation learning models (MERT and CLAP): The reviewer states that “Furthermore, models like MERT, CLAP, and others can extend their capabilities to encompass broader applications, such as music separation and text-to-music generation tasks.” Both MERT and CLAP are music representation-learning models. These models do not directly make predictions for any downstream task; they are either used in a zero-shot fashion by comparing embedding distances between an audio input and a text label (as in CLAP [10], see Fig. 1, and LAION CLAP [11], Fig. 1 middle) or via linear probing (as in MERT [9], sec. B.2, and LIAON CLAP [11], Fig. 1 bottom). LLark is capable of directly outputting text responses for any query or task, and this is because it addresses a fundamentally different task: generating language from music + instructions. However, just as the reviewer suggests that the representations learned by MERT and CLAP could be used for source separation or music generation, the same task could be performed with LLark. For example, LLark’s hidden states could be used as general-purpose embeddings, and LLark could be extended to output discrete acoustic tokens to generate music. In this sense, the model we propose is capable of the same hypothetical extensions as those described for MERT and CLAP, but unlike MERT and CLAP, it is also capable of performing classification, regression, and text generation tasks out of the box without zero-shot adaptation or probing.

Thank you to the reviewer for their detailed, thorough, and expert response. We discuss the main themes below. Then, we answer the reviewers’ questions individually. We would like to acknowledge upfront that the reviewers’ highly constructive responses have been incorporated to the paper in many places, and that they have overall improved the presentation of results and contextualized our work better in comparison to existing models, including the representation-learning methods the reviewer highlighted. The paper has benefited significantly from the reviewers’ expertise in MIR/audio modeling and benchmarking - thank you!

• Lack of novelty (numbered according to reviewers’ original numbering): (1, 2) The reviewer is correct that LLark is built upon existing modality-specific models, and uses instruction-following techniques that have worked well for vision and language. As we discuss in our response to reviewer gHco above, this is both a benefit to our approach (not requiring expensive retraining of language or audio models) and in line with existing multimodal contributions in other domains [1-6]. The use of a straightforward architecture with “no novel adaptations to better suit the domains'' is also in line with recent multimodal models such as [1, 6, 7] which use simple projection layers as multimodal adapters. (3) Instruction tuning is neither used in LAION-CLAP (which uses keyword-to-caption augmentation to train a contrastive model) nor in WavCaps (which uses an LLM for pseudolabeling captioning data). However, we do not claim to be the first to use instruction-tuning, even for music. Instead, our contribution is in line with data-centric machine learning, where our results are distinguished by their scale (1.2M musical query-response pairs), diversity (over 160k tracks), and quality (using high-quality GPT models) to achieve results not demonstrated by any existing multimodal model for music. We believe that this is in line with similar works in the audio space appearing in ICLR; for example, MT3 [8, ICLR’22], also used preexisting, unmodified architecture and datasets.

• Status as foundation model: The term “foundation model” is nebulous; we apologize to the reviewer for any confusion and think they raise a fair objection. We are using the term in the sense of the Stanford Center for Research on Foundation Models (CRFM), which states on its website (https://crfm.stanford.edu/) that “In recent years, a new successful paradigm for building AI systems has emerged: Train one model on a huge amount of data and adapt it to many applications. We call such a model a foundation model.” While we believe LLark meets this definition, we have removed the term “foundation model” from the title and have updated the paper text to avoid characterizing LLark as a “foundation model” (we are not currently able to change the OpenReview submission title, but have changed the title in the PDF).

• Performance vs. state-of-the-art (SOTA) on MIR tasks:

  • Discussion of task-specific vs. generalist models: The reviewer correctly notes that LLark does not achieve SOTA on the MIR tasks in the paper. We have added discussion to the paper highlighting the gap between LLark and the SOTA models mentioned (Section E) and have included a more explicit discussion of representation learning models in the related work (Section D). We specifically added the SOTA metrics the reviewer highlights for tempo, key, and genre. However, not beating SOTA relative to task-specific models is in line with other generalist models for music. This includes MERT [9], which only achieves SOTA performance on 3 of the 23 tasks evaluated in the paper (Tables 1 and 2 of [9]; these tables show that task-specific models still hold the SOTA across 16 of 23 tasks), and even then MERT’s performance is only after linear probing directly on the target task (see below). Like the reviewer, we agree that MERT is still a useful scientific contribution due to its ability to achieve good performance on many challenging tasks with a unified model -- and we believe that LLark makes similar gains (for example, LLark excels on captioning tasks not addressed by any representation learning model). Similar to other generalist models in both the audio- and vision-language domains, our objective was not to achieve SOTA performance on any single task, but instead to build a model which (i) achieves strong performance on many tasks, including unseen tasks (as opposed to the task-specific models the reviewer cites, which will almost always outperform any generalist model) and (ii) outperforms existing audio-language models. We believe our evaluations, including in Section 6, demonstrate that these criteria are met. In fact, the numbers the reviewer provides also highlight how close LLark is to SOTA performance in both the key and tempo tasks, despite being capable of performing both tasks at this level (along with many other tasks).

Once again, we thank the reviewer for your time reviewing the paper. As the author-reviewer phase is ending soon, we request the reviewer to please review the author response and let us know whether the comments and revisions to the paper (indicated in red text) have addressed your concerns, or whether we can provide any further clarification before the discussion window closes. Thank you!

[continued from previous point]

• Additional considerations beyond architectural novelty: We believe that our contribution extends beyond purely proposing a novel model (since other audio-language models do exist, as the reviewer and our paper both note, and as discussed in our point "Lack of novelty" in the original Reviewer p6wE response (1/4)). This is why we have emphasized our contribution as one of data-centric machine learning and encompassing data, models, and evaluation. In addition to a novel model (which is not what we consider our standalone contribution), we believe that the training data generation strategy, strong empirical results, and the quality and rigor of evaluation are important to our work. To our knowledge, LLark contains the most thorough set of musical evaluations of any audio-text model, where we evaluate MIR tasks (tempo estimation, key estimation, genre classification, instrument identification), conduct high-quality human evaluations for both music captioning and reasoning tasks, and also provide a set of objective (non-human evaluated) metrics across these tasks. Our ablation studies and extensive supplementary results provide further insights related to scaling and model components, as we discuss above and in the paper. We believe these evaluations and empirical insights are also critical contributions of our work, as rigorous evaluation is also critical to reliable progress in the field of machine learning [1]. As we noted in our previous response, other music-related works have also appeared in the ICLR conference without proposing novel model architectures, such as MT3 (ICLR 2021), which uses "an off-the-shelf T5 architecture" (see p.2 of [2]).

Once again, we thank the reviewer for their thoughtful engagement. Please let us know if we are able to provide any additional clarifications or further improve the paper to reflect the reviewers' concerns during the author-reviewer discussion window.

References:

[1] Liao, Thomas, et al. "Are we learning yet? A meta review of evaluation failures across machine learning." NeurIPS 2021.

[2] Gardner, Josh, Ian Simon, Ethan Manilow, Curtis Hawthorne, and Jesse Engel. "MT3: Multi-task multitrack music transcription." ICLR 2022.

5. In Figure 4, the matching rates are quite low across all models (around the random guessing level), leading to questions about whether LLark's performance on certain tasks is only marginally better than random guessing. Section 6.4 attributes this result to “limitations in the musical expertise of the (non-expert) raters in our study," further emphasizing my previous concerns about relying solely on non-experts for human evaluation.

   We acknowledge the reviewers’ concern here. Open-ended reasoning tasks are, by far, the most challenging task in our study, as they require combining music understanding, language modeling, and external “world knowledge” to answer correctly. While our model outperforms existing instruction-following audio-text models, there is room for future improvement. We believe that the rigorous formal reasoning evaluation in our study, which has not been attempted for similar models in either music-, audio-, or vision-language models to our knowledge, provides a useful benchmark for future work in this direction. We provide two additional points of context:

   • Our above discussion of objective vs. subjective metrics is relevant here. In particular, the subjective metrics can only tell “part of the story” with respect to reasoning evaluation. We hesitate to rely only on audio-text matching (see next point), and believe that the objective metrics for this task also provide useful context. Specifically, this includes the results in Table 3 and Figure 12.

   • In practice, “random guessing” is not the lower bound for audio-text matching performance for models in our study, and “perfect matching” (100%) may be an impractically high bar. Recall that our design presents raters with one piece of audio + a text query, and three responses to that query (from the same model, in random order).

     Lower bound: we observe that many models produce “convincing hallucinations” that attracted reviewers at a high rate whenever they appeared, regardless of whether they were correct or not. This was a major contributor to the poor performance of ImageBind-LLM (see Figure 12, which shows that ImageBind-LLM’s responses were both extremely long, and had a high number of unique tokens; these often described fictional artists and elaborate visual scenes not associated with the input but were selected by raters at high rates). In contrast, for LTU-AS, the descriptions tended to be short and generic (as shown in Figure 12), which led reviewers to select the more detailed responses at a high rate. Both behaviors can lead to lower-than-chance performance. (Consider the extreme case, not possible with this specific design, where one answer is always chosen regardless of the correct response; in that design the matching rate would plummet to 1 / size of response pool, effectively zero.)

     Upper bound: note that our audio-text matching setup is a very challenging task even under the best of circumstances: it requires that a response addresses a question correctly, and that the response matches the audio in sufficient detail such that a rater can link the textual response to the audio in the presence of other potentially-similar responses from other tracks. We hypothesize that even in the presence of expert-human-provided responses, matching rates would likely not achieve 100%. This is particularly true for non-expert raters, as the reviewer notes, since they may not be able to match attributes such as key or tempo to the audio, and again motivates a more holistic approach to our evaluations (i.e. considering the level of musical detail in Table 3, and the music understanding results in Table 2). As a result, while we acknowledge that performance even higher than LLark’s is achievable, we do not expect that 100% matching rate would be possible. We have also added further analysis summarizing these points to Section G.3.

6. With no clear indication of randomness in option order (or maybe I missed?), such as whether LLark is consistently presented as the first option, this raises concerns about potential biases [6] in the evaluation results of GPT-4 in Sections 6.3 and 6.4.

   The ordering of all models and answer choices in all of our human and GPT-4 studies are always randomized. This is mentioned in the “interface” sections in L.1 (captioning) and L.2 (reasoning); we have also added it to the main text in Sections 6.3 and 6.4.

7. Wrong citation in Appendix F: the ablation study used the CLAP developed by LAION [7], but wrongly cited the Microsoft one [8]. This made me confused for a while. Please correct the citation. We have corrected this mistake; thank you for noting it.

4. In Sections 6.3 and 6.4, it’s ok to rely on human evaluation as we don't have holistic objective metrics to assess captioning and reasoning tasks. But leaning entirely on non-experts for this might not be the best call. It might be better to balance this out with some expert raters rather than going all-in on non-experts.

   The reviewer is highlighting a crucial and unresolved challenge in the evaluation of text generation models in general (discussed in e.g. [1]), and for music in particular. We agree that non-experts are limited in what they can assess. Overall, our approach to evaluation was to let existing works in the MIR and language modeling space be our guides, to maximize our ability to compare results to previous works. To our knowledge, prior work conducting human evaluation of audio-language models has exclusively used non-experts for both musical and non-musical applications [e.g. 2, 3] -- but human evaluations tend to be rare due to the high cost of conducting these studies. We believe there are three separate considerations raised by the reviewers’ point (and others in the review):

   • Objective vs. subjective metrics: Any human evaluation is necessarily subjective. Objective metrics can help to offset the human element from evaluations, helping us to avoid “leaning entirely on non-experts” while measuring different behaviors. While the reviewer is correct that “we don’t have holistic objective metrics to assess captioning and reasoning tasks”, we provide several objective metrics assessing our models’ outputs on the tasks discussed in 6.3 and 6.4, in the supplementary material. For example: (i) Figures 11 and 12, which show that LLark provides captions that are competitive with existing models in terms of both diversity and length, for a fixed prompt (keeping in mind that the higher diversity of tokens in LLaMA-Adapter are largely due to detailed hallucinations, as our user study shows; we can provide examples if the reviewer would find this useful); (ii) Figure 13, which shows that LLark can follow instructions; (iii) Tables 3 and 5, which show (using GPT-4 as judge) that LLark consistently contains more musical detail in its responses; and (iv) Table 6, which shows that, with minimal prompt tuning (i.e. asking for a “short” caption vs. the standard captioning prompt), LLark is competitive with or outperforms captioning models even in its ability to match reference captions in various language metrics (BLEU, BLEU-4, METEOR, ROUGE, CIDER). We caution against overreliance on the language metrics, however, as these only measure adherence to a reference caption, which can reflect the biases of the original captions [4] and penalizes even high-quality captions which deviate from the original caption.
   • Experts vs. non-experts: We acknowledge the reviewer’s concerns about non-experts being used in the subjective evaluations, and attempted to highlight this limitation in Section 7. Our goal is to construct a model which is useful to both experts and non-experts, which motivated our use of non-experts for selected tasks. Our study design (including the use of non-experts) was matched to recent language modeling evaluations, for example [1, 2, 3] which use non-experts for language, music, and audio models. As the reviewer is undoubtedly aware, the cost of assembling musical experts to evaluate our model was also prohibitively high. However, since our intention was to construct a model that is useful for a broad user base, we believe non-expert evaluations are still useful for this. Furthermore, there is not only a binary choice between expert vs. non-expert evaluators -- objective metrics (discussed above) are also a useful tool in assessing model quality without relying exclusively on non-experts.
   • Correctness of musical aspects of text responses: One possible concern with overreliance on non-expert human evaluation is that non-experts cannot easily assess the correctness of musical information (tempo, key, etc.). We share this concern. We point to our formal evaluations on music understanding tasks (Section 6.1), which evaluate the model’s outputs of musical properties that may be otherwise assessed by experts (key, tempo, and instrument estimates) and show that the model’s musical responses regarding these attributes are of high correctness. At the same time, we acknowledge that incorrect, but convincing-sounding outputs may affect the model. As a result, we have (1) updated the ethics section of the paper to reflect these issues with truthfulness of outputs; (2) added a Model Card [11] to the paper (Section M); (3) added a “Failure Modes” section to the paper (Section N) to highlight and categorize cases where these behaviors may emerge with LLark to ensure that users and researchers alike are appraised of the risks unique to music-language models. We also propose strategies to address each failure mode.

We are grateful to the reviewer for their thoughtful and detailed response! We are particularly glad to see that the reviewer observed the “impressive results,” “ potential for practical applications”, the potential to “provides reusable insights within the realm of music information retrieval,” and our efforts to “ensure data accessibility and reproducibility” by utilizing publicly accessible, permissively-licensed datasets.

The reviewer provides a useful list of weaknesses, and questions. We address these individually inline below, but would like to acknowledge up front that we made several additions, clarifications, and improvements to the paper to incorporate the reviewers’ detailed feedback and address the reviewers’ concerns about the paper, framing, and results.

Weaknesses:

1. Section 1 mentions LLark's architecture as a contribution, but the integration of large language models (LLM) and domain-specific models has been previously explored in the literature [1-4], as properly cited in the paper. To strengthen the paper, consider providing a more detailed explanation of what sets LLark's architectural design apart from existing implementations.

   Thank you - we have updated the language in Section 1. To summarize here, the main novelty is in using a generative model as the encoder, but our intended primary contribution is not the novelty of the architectural components, but in demonstrating that this architecture is effective when paired with our data pipeline. As we discuss in depth in our response to Reviewer gHco (points (a), (b) in that response), we believe this contribution -- focused on data-centric machine learning for multimodal modeling -- is in line with the contributions of other studies in the vision-language space [5-10], but our work is the most comprehensive such effort in the music space.

2. Limiting the audio encoder to 25-second clips (mentioned in Sections 4.1 and 7) is a notable drawback, as it lacks the understanding of important contextual information in real-world scenarios (most music audio is longer than 25 seconds), leading to potential misinterpretations and missed insights in audio content analysis.

   We acknowledge that this is a limitation of the work as presented (as discussed in Section 7). However, this is not a fundamental limitation of the approach. The audio encoding size could be extended indefinitely (up to the language model’s context window size; 4096 tokens in the case of Llama 2 7B). This could be achieved by passing chunked audio into Jukebox, and could be further compressed to accommodate longer audio by decreasing the audio frame rate (currently 10fps, or 250 tokens per 25-second clip). We adopt the 25-second limitation due to convenience in this work, and to limit the computational expense of using Jukebox (since audio clips in these datasets can be an hour or more). We leave analysis of longer audio encodings to future work - but note that our code can easily support such experiments, as it can run on arbitrarily-sized audio encodings out of the box.

3. Section 4.2.1 should investigate how LLark's performance is affected by the features extracted from traditional MIR models [5], especially regarding temporal-related music properties like timestamped chords and beat grids. Additional ablation studies should be included to confirm the performance gain of metadata augmentation.

   We agree that this would be a useful analysis, and would have liked to conduct this absent computational or budgetary constraints. However, an ablation study which removed the metadata augmentation would require completely rerunning our entire dataset creation pipeline, withholding the augmented metadata features when the query-response pairs are created (the dataset creation, not the training, was the most expensive component of our work, so this was not tractable for us). Additionally, we felt the results of this ablation would be clear: we would expect a performance degradation on the “music understanding” tasks in particular, due to the lack of any supervision on these tasks during training, and a corresponding decrease in musical detail in free-text tasks. Our open-source code release would enable others to conduct this analysis in the future -- or, perhaps more likely, to extend the metadata augmentation to further improve the model by incorporating more and better pseudolabeling models, as MIR research progresses and new models become available.

Once again, we thank the reviewer for your time reviewing the paper. As the author-reviewer phase is ending soon, we request the reviewer to please review the author response and let us know whether the comments and revisions to the paper (indicated in red text) have addressed your concerns, or whether we can provide any further clarification before the discussion window closes. Thank you!

We are grateful to the reviewer for the thoughtful response, and glad that the reviewer assessed that “[t]he results are promising,” that the work is “well thought out” and the paper is easy to read. As we discuss below, the reviewers’ questions also helped us to improve the clarity of the paper - thank you!

The reviewer’s generally positive feedback seems somewhat at odds with their rating. It appears that the reviewers’ main reservation about our study is the nature of the contribution: in particular, the reviewer appears to feel that the empirical nature of the work, and a belief that it “merely rehashes existing datasets and models,” mean the work is not significant. We would respectfully like to reframe our contribution to the reviewer, and put it into context with existing works. Our paper falls in line with existing highly impactful contributions in the multimodal modeling space, and with work on data-centric machine learning, where the emphasis is on empirically understanding how to build high-performing models and on solving important applied problems, regardless of the architecture used.

We provide a few specific responses below on how our work relates to others in the data-centric multimodal learning space:

• (a) Using existing models is a feature, not a bug: We believe the claim that we use “existing datasets and models” is a benefit, not a drawback, of our approach. Our ablation studies (Figures 6, 7) show the impact of using these specific models, and demonstrates that starting from the highest-quality pretrained (language, audio) models is critical to the quality of the resulting output. Utilizing existing resource-intensive foundation models is also an effective use of training resources. Furthermore, neither our study nor the work of others suggests that new (audio, language) models are necessary for multimodal music understanding (see next point).

• (b) Many impactful multimodal models utilize pretrained modality-specific encoders: It is standard practice, and in line with the contributions in other multimodal models, to use “existing datasets and models” as the building blocks of a multimodal model. This includes recent SOTA multimodal models in the vision-language space, such as LLaVA [1] (Llama, CLIP), Otter [2] and OpenFlamingo [3] (Llama, CLIP), LLaMA-Adapter [4] (Llama, CLIP), ImageBind-LLM [5] (Llama, ImageBind) Mini-GPT4 [6] (Vicuna, ViT).

• (c) Open-source data is a benefit in the music domain: The reviewer also voices a concern about using existing datasets. To this, we offer two considerations:

  • (1) All of the works discussed in the previous paragraph [1-6] start by building a multimodal training set from existing open-source datasets. We follow this widely-used approach, albeit in the music domain. Our contribution is in line with these widely-cited and used works. Using open-source data is a benefit for music researchers in particular, as it allows others to reliably reproduce and extend our results (due to the strict regulations surrounding recorded music, this can be challenging for closed-source datasets!). Reviewer zBmL also highlighted this as a strength of the paper: “The paper leverages publicly accessible, open-source, and permissively licensed music datasets for training, which ensures data accessibility and reproducibility.”
  • (2) We do not simply use unmodified off-the-shelf data to train the model. Indeed, our data preprocessing (via instruction-tuning across 3 task families) and metadata augmentation (which uses music-specific feature extractors to add critical musical information) is a key contribution of our work. Again, this is in line with existing instruction-tuning works in the language and vision domains [1-6]. LLark is the most comprehensive such effort in the music domain.

• (d) Simple multimodal projection architecture is in line with existing work: The simplicity of the multimodal adapter is both a benefit of our approach, and in line with existing work. Recent state-of-the-art models across the multimodal space have shown that simple adapter layers are highly effective: e.g. LLaVA [1], Mini-GPT4 [6], FUYU [7], among others. Given no clear reason to add further complexity to the architecture (and the emerging likelihood that this is the most effective technique for multimodal alignment), we felt this approach was appropriate and is validated by our results.

References:

[1] Liu, Haotian, et al. "Visual instruction tuning." arXiv preprint arXiv:2304.08485 (2023).

[2] Li, Bo, et al. "Otter: A multi-modal model with in-context instruction tuning." arXiv preprint arXiv:2305.03726 (2023).

[3] Awadalla, Anas, et al. "Openflamingo: An open-source framework for training large autoregressive vision-language models." arXiv preprint arXiv:2308.01390 (2023).

[4] Gao, Peng, et al. "Llama-adapter v2: Parameter-efficient visual instruction model." arXiv preprint arXiv:2304.15010 (2023).

[5] Han, Jiaming, et al. "Imagebind-llm: Multi-modality instruction tuning." arXiv preprint arXiv:2309.03905 (2023).

[6] Zhu, Deyao, et al. "Minigpt-4: Enhancing vision-language understanding with advanced large language models." arXiv preprint arXiv:2304.10592 (2023).

[7] Rohan Bavishi et al. Fuyu-8B: A Multimodal Architecture for AI Agents. https://www.adept.ai/blog/fuyu-8b

[8] Schreiber, Hendrik, and Meinard Müller. "Musical tempo and key estimation using convolutional neural networks with directional filters." arXiv preprint arXiv:1903.10839 (2019).

[9] Korzeniowski, Filip, and Gerhard Widmer. "End-to-end musical key estimation using a convolutional neural network." 2017 25th European Signal Processing Conference (EUSIPCO). IEEE, 2017.

[10] Li, Yizhi, et al. "MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training." arXiv preprint arXiv:2306.00107 (2023).

[11] McCallum, Matthew C., et al. "Supervised and unsupervised learning of audio representations for music understanding." arXiv preprint arXiv:2210.03799 (2022).

[12] de Souza, Mila Soares de Oliveira, Pedro Nuno de Souza Moura, and Jean-Pierre Briot. "Music Tempo Estimation via Neural Networks--A Comparative Analysis." arXiv preprint arXiv:2107.09208 (2021).

Once again, we thank the reviewer for your time reviewing the paper. As the author-reviewer phase is ending soon, we request the reviewer to please review the author response and let us know whether the comments and revisions to the paper (indicated in red text) have addressed your concerns, or whether we can provide any further clarification before the discussion window closes. Thank you!