MuPT: A Generative Symbolic Music Pretrained Transformer

Cons

Mistake

Thank you for pointing out the typos in our manuscript. If the paper is accepted, we will ensure that these issues are corrected in the camera-ready version.

Novelty of this paper

• Leadsheet-based multitrack symbolic music generation is a promising yet underexplored area. Leadsheets effectively compress the semantic information of music, facilitating whole-song-level modeling. In the future, this approach could guide audio music generation models similar to Suno's framework and seamlessly integrate text-based leadsheet generation into the framework of large language models (LLMs).

• Our proposed SMT-ABC notation effectively addresses the alignment challenges in multitrack ABC notation, significantly improving training loss and overall model performance.

• To the best of our knowledge, we have collected the largest ABC notation dataset and successfully scaled model training on this dataset. For the first time, we have derived a scaling law for symbolic music, laying a solid foundation for integrating musical creativity into general artificial intelligence systems in the future.

• While scaling laws have been extensively studied in other domains, limited research exists on scaling laws specific to ABC notation. Due to the relatively small amount of available ABC-notation data, repetitive training is necessary to achieve meaningful performance. Importantly, ABC notation differs significantly from standard language data, both in structure and representation. Our work addresses this gap by providing insights into scaling laws and repetitive training for ABC data, offering a significant contribution to this underexplored area.

Changing sequence of ABC

One major issue is the measure consistency problem. As shown in Appendix B.2, Figure 8, this figure illustrates the measure consistency scores for SMT-ABC and standard ABC notation under the same training iterations and model structures. The results demonstrate that as training progresses, SMT-ABC quickly achieves consistent measures across tracks, indicating effective alignment and coherence. In contrast, models trained solely with ABC notation struggle to maintain measure consistency across tracks, even as training iterations increase. This highlights the advantage of SMT-ABC in addressing alignment issues and improving overall musical structure.

Track coherence vs. Bar coherence

There are two ways to discuss this issue.

1. the final performance of SMT-performance. As shown in Table 5 and the final loss As shown in Figure 4. the training loss and the evaluation metrics are well shown that the performance for SMT-ABC is better than ABC-notation.

2. The concepts of bar coherence and track coherence are useful but challenging to evaluate directly. Instead, we propose the concept of the closeness of related tokens. Since this is a causal model for pretraining, the later tokens in the sequence are less influenced by earlier tokens as the tokens gap increases. The purpose of designing the SMT-ABC notation is to ensure that bars played in each track at the same time are generated as closely as possible. Thus, we focus on discussing the token gaps between different tracks.

In our training dataset, which aligns with the natural distribution of ABC notation, the average tokens per bar is 3.38 tokens, while the average bars per track is 77.40 bars. This means that the benefits of reducing token distances between tracks significantly outweigh those of reducing token distances within a track.

For example, we define the most related bars as:

• The bar with the same index in a different track.
• The next bar in the same track.

In a 4-track piece of music:

• The token distance between Track 1, Bar 1 and Track 4, Bar 1 in standard ABC notation is approximately:
  $77.4$\times$(4-1) = 232.2 tokens$.
• The token distance to the next bar within the same track is much smaller:
  $3.38 tokens$.

In contrast, with SMT-ABC notation:

• The token distance between Track 1, Bar 1 and Track 4, Bar 1 is reduced to:
  $3.38$\times$(4-1) = 10.14 tokens$.
• However, the token distance to the next bar in the same track increases slightly to:
  $3.38$\times$4 = 13.52 tokens$.

This design in SMT-ABC notation brings related bars closer together while slightly increasing the distance for less related bars. This shift in token distribution improves the closeness of related tokens, enhancing the overall coherence and ultimately boosting the model’s final performance.

• To the best of our knowledge, we have collected the largest ABC notation dataset and successfully scaled model training on this dataset. For the first time, we have derived a scaling law for symbolic music, laying a solid foundation for integrating musical creativity into general artificial intelligence systems in the future.
• ABC tokens are similar to the symbols of natural languages, which helps interact with natural languages and can be easily adapted to LLM, such as chatMusician, etc.

Typos

Thank you for pointing out the typos in our manuscript. If the paper is accepted, we will ensure that these issues are corrected in the camera-ready version.

Datasets

Due to the length of the reply, please refer to our response to Reviewer f4e7.

The alignment and difference between ABC notation and MIDI

Alignment between 2 notations

ABC is the music staff/stave in terms of natural language notations, including tune info (e.g. title, composer, meter, key, pitches, rhythms, bar lines & repeats, instrument, etc. [2-3]. The “K: Bb” you mentioned in ABC means the key of the music piece is Bb, which is the same as the key of the MIDI note in the piano roll and is related to the overall pitch of the MIDI note in the piano roll. And “|F2 z G|” corresponds to one music bar that includes a quarter note at pitch F, an eighth-note/quaver rest, and an eighth-note at pitch G. MIDI start and stop times are absolute time slots in seconds and do not include quantized beat/bar information. This given measure corresponds to two MIDI note rectangles on the chart that correspond to the pitch and time (the rest is not shown).

The difference between MIDI and ABC

(1) As you said, the velocity, control information (and also sound effect etc) included in MIDI are not included in ABC notations. ABC notation includes bar and note quantization information (how many beats v.s. how many seconds per note), key, and music/tune structure such as repeat sign, etc., that MIDI does not include. In general, MIDI can keep more information on the styles of performers with acoustic information, and ABC notations are better with modelling repeats and structures, leading to better compression rate and more suitable for LLM training.

[1] Ma Y, Øland A, Ragni A, et al. Foundation models for music: A survey[J]. Foundation models for music: a survey[J]. arXiv preprint arXiv:2408.14340, 2024. [2] https://abcnotation.com/wiki/abc:standard:v2.1 [3] https://abcnotation.com/videos#basics [4] Yuan R, Lin H, Wang Y, et al. Chatmusician: Understanding and generating music intrinsically with llm[J]. arXiv preprint arXiv:2402.16153, 2024.

Thank you for your valuable feedback. We acknowledge that our comparison between ABC and MIDI representations is limited, as our primary focus is to demonstrate that our model（SMT-ABC-notation-based model）has the potential to outperform the state-of-the-art symbolic music generation models such as MMT and chatMusician, as shown in Table 5&6. An in-depth comparison with ABC notation and other symbolic music formats of MIDI (and more in [1] section) is not the main contribution or purpose of our paper. Below, we detail our points and why ABC can be a good choice for symbolic music LLMs:

Advantage of ABC notation

ABC notation offers significant advantages over MIDI in terms of compression ratio, which directly impacts model efficiency and reduces training costs. As highlighted in ChatMusician [4], ABC notation achieves an average of 288.21 tokens per song and 5.16 tokens per second, requiring roughly 38% fewer tokens than various MIDI-based representations. This remarkable reduction in token count stems from ABC's ability to efficiently encode symbolic musical repetition using concise notations such as repeat signs (|: and :|). These symbols effectively represent patterns spanning several seconds to minutes. By reducing sequence length, ABC not only lowers computational overhead but also simplifies the learning complexity, making it an ideal format for music-related tasks.

It is undeniable that MIDI offers more detailed dynamic information compared to leadsheet formats like ABC. However, a major drawback of MIDI is that, within a limited context length, it may not be possible to encode an entire song. While hierarchical modeling methods, such as Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models, have been proposed to address this issue, scaling up such approaches can be challenging. In contrast, ABC notation is compatible with natural language symbols and naturally encoded structures like repetition, benefitting training efficiency, convergence speed, and modelling the overall structure of a song with NLP codebase. Moreover, ABC notation integrates seamlessly with text tokenizers, making it an efficient choice. After carefully weighing the pros and cons of both formats and considering the characteristics of our collected dataset, we decided to use ABC notation.

Our proposed SMT-ABC notation effectively addresses the alignment challenges in multitrack ABC notation, significantly improving training loss and overall model performance.

Additional Contribution for Scaling up SMT-ABC based LLM

Reserach work on ABC notation remains under-exploration

As previously discussed, our contribution lies in exploring how to scale up a music generation model and the rationale behind choosing ABC notation, which is naturally similar to natural language. SMT-ABC is designed to ensure that bars played simultaneously across tracks can be inferred causally by the scaled-up model. Music Transformer, RWKV, and Compound Word Transformer are all causal modeling strategies. We believe the SMT-ABC notation dataset can effectively complement these model architectures. We chose LLaMA because it is one of the most widely used causal models. Furthermore, we believe that these models can greatly benefit from our dataset and the SMT-ABC methodology.

Scaling Laws and Repetition in ABC Notation Pretraining

While scaling laws have been extensively studied in other domains, limited research exists on scaling laws specific to ABC notation. Due to the relatively small amount of available ABC-notation data, repetitive training is necessary to achieve meaningful performance. Importantly, ABC notation differs significantly from standard language data, both in structure and representation. Our work addresses this gap by providing insights into scaling laws and repetitive training for ABC data, offering a significant contribution to this underexplored area.

Foreesee the future roles and applications

• Leadsheet-based multitrack symbolic music generation is a promising yet underexplored area. Leadsheets effectively compress the semantic information of music, facilitating whole-song-level modeling. In the future, this approach could guide audio music generation models similar to Suno's framework and seamlessly integrate text-based leadsheet generation into the framework of large language models (LLMs).
• It is also worth noting that audio can be generated from ABC leadsheets by using a rendering model. For example, Seed-Music: A Unified Framework for High Quality and Controlled Music Generation demonstrates this approach, enabling the generation of high-quality audio based on ABC notation. This allows for the inclusion of performance characteristics and timbre while maintaining the capability to model entire songs with ease.

Thanks for your inputs and feedback. This is super valuable for us. Questions about SMT-ABC

The name SMT-ABC indicates that this model is specifically designed for ABC-notation. While it could potentially be adapted for MIDI data, this does not align with the purpose of our paper. Our focus is on ABC-notation and how to fully leverage it within large models. SMT-ABC has already demonstrated its ability to outperform models trained on the original ABC data, showcasing its effectiveness. We believe SMT-ABC could also be beneficial for enhancing models like GPT-4 and MMT. However, MMT does not release their training pipeline, and GPT-4 is not suitable for direct training or fine-tuning, as it is a closed-form model. Due to these limitations, we are unable to conduct further experiments in this regard.

Scaling law

We appreciate the reviewer’s feedback and clarify that SMS Law is essential for symbolic music LLMs:

Data-Constrained Setting for Music Domains: Music modeling often faces dataset sparsity, where existing laws like the Data-Constrained Law (NeurIPS2023 best paper) fail on prediction. Symbolic Music Scaling (SMS) Law incorporates a novel "overfitting" term tailored for symbolic music and other data-constrained domains, achieving superior predictive accuracy in such scenarios.

Scaling Insights: SMS Law predicts that models beyond 2B parameters may not yield better results due to dataset constraints. Our experiments confirm this, with the 2B model outperforming the 4B model, optimizing efficiency and quality for developping symbolic music LLMs.

ABC Notation Specialization: SMS Law uniquely supports symbolic music tokens like ABC Notation, with learned parameters and mathematical format significantly different from related domains like speech, audio or natural language symbols, making it a critical advancement for music-specific scaling.

These contributions highlight SMS Law’s methodological and practical relevance to symbolic music LLMs.

Reply to your additional questions

• metadata for the training dataset The dataset includes a total of more than 1.8 million songs. For the subset of data with genre metadata, the approximate genre distribution is as follows:

Due to the lack of comprehensive metadata for the entire dataset, we cannot provide precise genre statistics for all songs. Nevertheless, we ensured that the training data is diverse and representative of a wide range of musical styles, enabling robust and comprehensive model training. Besides, to address intellectual property concerns, the datasets are sourced from publicly available repositories, ensuring ethical usage and proper citation. The authors explicitly state that the data is used strictly for research purposes, complying with licenses and copyright laws. The dataset is not redistributed, ensuring adherence to the terms set by the original sources.

• polyphonic music generation Our model can generate polyphonic music. Plz refer to our submission materials and Figure 11, which is the multi-track examples generated by MuPT.
• Architecture comparison As our methods is a pretraining model, I would like to say this is resource extensive, and but we do make a research before training our models. ABC-notation can be generated by GPT-4, Chatmusician is also discussed is benefits with ABC-notation and why this suitable for the Large lanuage model. We should use the widely used model architecture, which is LLama-based architecture, to prove that the LLM can be applied to ABC-notation, And this kind of model is suitable for ABC-notation generation tasks.
• Subject comparision Details are shown in Appendix C and Ethics Statements parts. Here are some details about the participants background, we devide in four categories, beginner, itermediate, advanced and expert. For each category of participants and their propertion are shown as follows: Beginner: you have little to no experience in music and is just starting to learn or explore it. 30.0% Intermediate: you have some knowledge and skills in music, such as being able to play an instrument, read sheet music, or understand basic music theory. 33.3% Advanced: you have a high level of proficiency in music. You have extensive experience in playing an instrument, performing in ensembles, composing music, or have in-depth knowledge of music theory. 26.7% Expert: you are highly skilled and knowledgeable in music, possibly with professional training or experience. You may be a professional musician, music teacher, or have a deep understanding of advanced music theory and performance techniques. 10.0%

Samples from baseline model

Thanks for your suggestions， however, we do share the samples in the supplementary materials, which are called MMT-*.mp3. And in this link https://www.notion.so/MuPT-Demos-vs-MIDI-54c19455438f4368a9a43f8ac10a5c01, you can see there are GPT-4 generation examples. According to the subjective participant's results shown in Table 7, Mupt outperforms MMT and GPT-4 in terms of music generation.

The repetition metrics not clearly be motivated

We show this in Line 449.

Repetition is significant in evaluating how well-structured the music is.

As mentioned in the paper, we intuitively use the repetition rates to measure how often repetition symbols appear in the generated pieces, as a simple metric for evaluating the structure of music. Theoretically, repetition will largely influence the structureness we perceive about the music. It is a fundamental characteristic of what we experience as music, except for certain types like aleatoric music[1-2]. To the best of our knowledge, some empirical research tries to model the multi-level structure of music by recognising repetition but little of them emphasizes the position of repetition[3-4]. In other words, even a randomly selected note sequence can be regarded as music if it repeats as a pattern. We understand your claim that the position of repetition symbol will influence music quality but it depends on personal preference. Therefore, we do ask the participants to consider the structureness which indicates the repetition and its position by mentioning how well music phrases organize when evaluating musicality in the subjective tests.

[1] Ockelford, A. (2017). Repetition in music: Theoretical and metatheoretical perspectives. Routledge. [2] Margulis, E. H. (2013). On repeat: How music plays the mind. Oxford University Press. [3] Nieto, O., Mysore, G. J., Wang, C. I., Smith, J. B., Schlüter, J., Grill, T., & McFee, B. (2020). Audio-based music structure analysis: Current trends, open challenges, and applications. Transactions of the International Society for Music Information Retrieval, 3(1). [4] Dai, S., Jin, Z., Gomes, C., & Dannenberg, R. B. (2021). Controllable deep melody generation via hierarchical music structure representation. arXiv preprint arXiv:2109.00663.

Question: Is there any study to check if the generations are copied from the training data?

Yes, we conducted experiments to investigate whether the music generated by the model exhibits any copying from the training data. Specifically, we sampled 750 pieces of music with 1-15 tracks and 100 pieces with more than 15 tracks from the training set. For these samples, we used their headers and the first bar as prompts and calculated the similarity between the model-generated music and the original music from the training set. The similarity was measured using three metrics: n-grams (character-based segmentation), n-gram-music (note/chord-based segmentation), and the longest common subsequence(LCS) between the two music sequences. The results are shown in the table below.

The results suggest that the model is capable of generating novel sequences rather than simply copying training data. The low n-gram and n-gram-music scores, paired with the moderate LCS score, indicate that the model retains stylistic or structural influences from the training set without overfitting to exact examples.