Unified Music-Language Model for Symbolic and Waveform Integration

Thank you for your numerous suggestions.

W1: After careful reconsideration, we acknowledge that our work encompasses five tasks across nine test datasets. These tasks include music theory, waveform-based tasks (captioning and QA), and symbolic tasks (continuation and inpainting). While "generation" may not fully describe certain tasks—such as inpainting, which is typically not classified under generation—our current focus remains on single-track melodies. To better reflect these distinctions, we will adjust the terminology in future revisions.

In fact, from our perspective, we do not intend to overly highlight the generation tasks. In our experiments, continuation and inpainting primarily serve to maximize the utilization of unlabeled symbolic music data. In other words, these are pseudo-tasks designed for training purposes. Our model primarily emphasizes music theory and understanding capabilities. Both symbolic music generation and waveform music generation, as mentioned in the future work section, are considered downstream applications of this work. We will make this distinction more explicit in future versions.

W2: Our current baseline selection primarily considers structural and dataset similarities to our model. We have not yet included GPT-4o, Qwen-Audio2, and SALMONN as baselines because their model scales and training data sizes differ significantly from ours. Also, they have not reported results on music understanding benchmarks, are instead focused on traditional MIR tasks. Consequently, we did not prioritize reproducing their results for this version. However, we will ensure their inclusion in future comparisons to provide a more comprehensive evaluation.

W3: We have evaluated the tasks using additional metrics, including BERT-score, METEOR, and CIDEr. These results will be included in the Appendix of the revised version.

W4: We conducted preliminary experiments on transcription tasks but did not include them in the results for two reasons. First, we initially considered transcription as atypical for MuLMs. Second, specialized models optimized for this task naturally outperform general-purpose approaches like ours. However, as you pointed out, transcription is indeed valuable for demonstrating alignment between waveform and symbolic modalities. We will include this task in future experiments to further validate our model's alignment capabilities.

W5: Thank you for your guidance. In stage 2, training a reliable low-level encoder that minimizes information loss is indeed non-trivial. We relied on empirical observations and experimental results without conducting systematic experiments to support this. Due to resource constraints, we did not progress stage 3 training for some stage 2 results that showed poor performance. We will address this gap in future iterations and provide more rigorous evaluations.

W6: While we do not have a separate ablation study section, we have reported results for "w/o bar-align" in the experimental results tables for each task. However, as you rightly noted, we failed to provide a detailed explanation of its impact. To address this, we will include additional case studies in future versions to better illustrate how bar-level tokenization contributes to specific tasks. However, as you pointed out, transcription is indeed a valuable task for demonstrating the alignment between waveform and symbolic modalities. We will include this task in future experiments to further validate our model's alignment capabilities.

Thank you for your suggestions.

W1: We would like to clarify that in our title, "Unified" is intended to highlight the integration of symbolic music and waveform modalities, rather than a comprehensive unification of music and language. We understand that this phrasing may have caused some misunderstanding, and we will revise it in future iterations. Regarding our problem formulation, since the outputs for ABC notation and music understanding tasks are textual, we believe our definition is accurate and not overstated; however, we will further refine our descriptions to reduce any potential misinterpretations.

The main technical innovation of this work is indeed at the tokenizer level. However, the tokenizer serves as a foundation for the subsequent integration of natural language and music modalities during training. We believe that a key challenge in multimodal language models lies in aligning modalities, resolving ambiguities, and addressing the sparsity or absence of certain modalities—areas where we aim to contribute meaningfully. We would also like to emphasize the role of our bar-level tokenizer, which enables different modalities to benefit from each other’s tasks and datasets during training. This, we believe, is where integration truly happens. Of course, we warmly welcome further discussion with you on this topic.

W2: Thank you for your suggestions regarding the details of our presentation. We will update the framework diagram and provide formal definitions for LM(), Adapter(), and Symbolic-Encoder() to enhance clarity in future revision.

W3: We agree that the integration is currently bounded by the fact that paired audios are synthesized, which may impact robustness. However, given our resource constraints, this is a necessary trade-off. Synthesized audio, with its high accuracy and consistency, remains a reasonable choice for alignment training, as demonstrated by the effectiveness of our approach in experiments. We will enhance the demo page by including more comparisons with other baselines to provide a more intuitive demonstration of the model's advantages. And, we will consider enhancing robustness, as mentioned in our response to Reviewer 2's Weakness 3.

W4: We acknowledge that Figure 1 could be clearer in conveying the five types of tasks and nine test datasets. We will revise its caption to include more explicit explanations. Regarding the experiments, since the improvements stem from additional modality integration and training data, conducting detailed ablation studies may be challenging. We will enhance the demo page and include tasks like music transcription and traditional MIR tasks (e.g., key estimation, tempo estimation) to provide more intuitive evaluations of fine-grained alignment and music understanding capabilities.

Thank you again. We are committed to improving our work based on your suggestions.

Thank you for your numerous suggestions and for acknowledging our idea of bar-level tokenization.

W1: We have been aware of JASCO, and we note the following key differences: 1. Unlike JASCO's joint conditioning approach, we align waveform and symbolics as pre-aligned embeddings. 2. Our approach directly uses raw musical notes, avoiding the resampling and interpolation required in JASCO, making it more faithful to the original structure. We will consider adding it to the literature review. And, we will include transcription experiments to showcase our ability to preserve musical structure.

W2: We believe our model's suboptimal performance stems mainly from two factors: training resource constraints—we used LoRA instead of full fine-tuning, had less data, and trimmed answer lengths—and existing benchmarks often encourage template fitting over genuine understanding. To address this, we plan to analyze this issue in detail and introduce traditional MIR tasks (e.g., key estimation, tempo estimation, genre classification, instrument identification) to evaluate MuLMs' music understanding capabilities.

W3: We agree that relying solely on synthesized waveforms may limit our model's applicability to real-world audio. Your generous suggestions have pointed us in the right direction, and we will dedicate time and resources to implement them. Strategies like rendering symbolic music with diverse soundbanks or using transcription tools are promising avenues for future exploration.

Q1: We will provide a detailed explanation of the dataset construction process in the revised version. Briefly, for MusicQA, we excluded questions unrelated to music. For MusicPile's knowledge-related questions, many were only weakly related to music (e.g., concert etiquette). Similarly, MusicPile-Summary mainly focuses on metadata and lyrics summarization, which we believe has limited relevance to music understanding. We retained a small subset to preserve the model's general language abilities. For the other datasets, we aimed to reduce computational costs under limited resources and balance the proportion of tasks in the training set. These decisions optimized training efficiency while ensuring diverse task coverage.

Q2: Our current tasks focus on single-track music, i.e., melodies, and do not handle multi-track alignment. We believe melody is more crucial for understanding tasks, and multi-track alignment may be challenging, but we will endeavor to explore it in future work.

Q3: Yes, we considered prompt engineering to assess general models' music theory knowledge but found it challenging to design fair comparisons due to the uniqueness of each question in our dataset. We experimented with in-context learning using 5-shot examples, but improvements were limited or even negative. We believe that music inpainting and continuation inherently test ICL capabilities, and that incorrect examples can be detrimental.

Q4: We will provide more detailed numerical analysis in revision. Briefly, due to limited training resources, the dataset responses are truncated, and UniMuLM tends to output shorter answers.

Q5: We surveyed 28 participants, asking, "How much do you engage with music?" on a scale of 1 to 5: 1 (1') minimal exposure; 2 (8') frequent listening; 3 (12') playing an instrument or participating in activities like a choir; 4 (4') proficiency in one or more instruments; 5 (3') formal training in music theory or composition. This distribution aligns with or exceeds general population levels, so we treated all responses equally without weighting. For multitrack generation, while the model is not currently trained for it, we are working on prompt engineering to include multitrack results in the demo and better showcase its potential.

Q6: We acknowledge that the current title may be somewhat misleading. We agree that the complementarity between symbolic and waveform generation is valuable to pursue, but we recognize that the alignment among the two and language is still weak. Therefore, we aim to first focus on music understanding.

Supporting both symbolic and waveform generation is indeed an important goal. When decoding symbolic and text data simultaneously, two primary approaches are feasible: interleaving music tokens, as exemplified by AnyGPT, or leveraging tools for downstream music generation, as seen in NExTGPT. The former often struggles under limited computational resources, while our current efforts focus on the latter approach, using fine-grained control of MusicGen.

Thank you again for your valuable feedback. We are committed to improving our work based on your suggestions.

Thank you for your professional and detailed review. We truly appreciate your insightful comments and have carefully considered each point you've raised.

W1: We would like to clarify that in our title, the term "Unified" was intended to highlight the integration of symbolic music and waveform modalities, rather than to imply a comprehensive unification of music and language. We understand that this phrasing may have caused some misunderstandings, and we will revise it in future iterations to improve clarity. Regarding our problem formulation, since the outputs of symbolic music generation and music understanding tasks are textual, we believe our current problem definition remains appropriate. However, we will refine our descriptions to minimize potential misinterpretations.

We agree with your observation that our model can be partially described as an "audio-informed, text-based music language model." At present, our primary focus is indeed on music understanding. We acknowledge that achieving fine-grained control over waveform music remains a significant challenge, one that requires advances in semantically rich and fine-grained representation learning—an objective that forms the core idea of this work. If possible, we would be very interested in your perspective: Which approach do you think is more promising for achieving a true unification of music and language—interleaving music/audio tokens directly within the generation process (AnyGPT-style), relying on downstream tools for modality transformation (NExTGPT-style), or perhaps an entirely different approach?

W2: We selected ChatMusician, MuPT, and Mu-LLaMA as baselines due to their architectural and setting similarities to our method. We acknowledge your suggestion to include Music Transformer and hierarchical diffusion models as baselines in future work. These models were not included initially due to their lack of support for ABC input and output formats, which made direct comparisons challenging. That said, we now realize that metrics such as FD, FAD, and KL could help address this gap, and we will incorporate these metrics in subsequent evaluations.

For the Music Understanding task, we differentiate Music-Language tasks (e.g., Music Captioning, Understanding, QA) from traditional MIR tasks. The latter typically focuses on specific attributes of music, without considering natural language understanding and reasoning capabilities. Therefore, we believe our evaluation approach is justified. That said, we also see the value in your suggestion to include traditional MIR models and datasets for evaluating MuLM. Indeed, we found in our experiments that some music captioning test sets are constructed from metadata, which may lead models to overfit templates rather than genuinely understanding the music. This is an issue we will address in future work.

Q1: Yes, considering this issue, we chose to train the model using LoRA to minimize the loss of the model's inherent language and instruction-following abilities. By employing a unified LoRA, we aimed to maximize complementary effects within the training set. For future revisions, we will test new types of questions and include examples to further demonstrate the model's generalization capabilities.

Q2: Yes, the aligned data is prepared by synthesizing ABC notation into audio waveforms using the original tempo specified in the notation. We conduct alignment within a single bar, assuming that tempo variations are minimal in such short segments, so we have not explicitly handled unstable tempos. We will statistically analyze tempo distributions.

We also recognize the limitations of our current approach, as highlighted by other reviewers. At present, the system only processes single-track music synthesized with a single instrument. While this setup ensures experimental control, it does not yet account for multi-track compositions, diverse instrumentation, or real-world scenarios involving noisy, non-synthetic music. As you suggested, leveraging MIR tools to extract ABC from real audio is a promising direction. We are excited to explore this in future work, as it could enable the model to handle real-world multi-track compositions and more diverse instrumentation more effectively.

Thank you again for bringing up these important points. We believe these insights will help us improve the robustness and applicability of our method in future iterations.