MuseCoco: Generating Symbolic Music from Text

We sincerely appreciate your valuable comments. In the following response to all the concerns and questions you have posted, we use W for weakness, Q for question, and L for limitation.

W1. The paper writing can be improved.

Thank you for your guidance. We appreciate your input and we enhance the wording for the abstract part in the updated version for conciseness. Regarding the term "two-stage design," we have confirmed its grammatical correctness through dictionary research. However, we are open to specific suggestions on how to improve its usage. Your additional insights would be valuable. Thank you.

Q1. In section 3.1, does position encoding not included for the m classes?

In BERT pre-training, the first position is the first token of the input text rather than the $[CLS]$prepended to the input text. Therefore, to unleash the power of BERT by aligning with pre-training, we also assigned the first text token as the first position and did not compute the position embedding of $[CLS_i]^m_{i=1}$. We prepend $[CLS_i]^m_{i=1}$ to the text after the text is inputted to BERT, instead of concatenating them before the input action. We separately calculate the embedding of $[CLS_i]^m_{i=1}$ (the word embedding and token type embedding) and the text (the word embedding, position embedding and token type embedding), and then concatenate the $[CLS_i]^m_{i=1}$ embedding and the text embedding. This implementation makes sure the order of $[CLS_i]^m_{i=1}$ is unchangeable. We provide the code about this in the line 244-253 of code_submit/1-text2attribute_model/text-attribute_understanding/bert/modeling_bert.py.

W2&W3&Q3. What about the cases where the text prompt is abstract?

In the formulation of attribute conditions, we meticulously incorporated a spectrum of explicit and nuanced descriptors to accommodate the varied preferences of users. For proficient musicians, the inclusion of precise technical music terms optimizes efficiency, ensuring the model generates music precisely in line with their articulated preferences. Conversely, users lacking a musical background often articulate their ideas using more abstract language to convey emotions or stylistic preferences. Consequently, we employ objective attributes to streamline the music creation process for musicians, supplementing them with subjective elements like emotion, artistic style, and genres. This approach facilitates expressive music generation, catering to users with diverse backgrounds and preferences. As you highlighted, acknowledging the crucial role of subjective elements in music generation involves describing a more abstract representation, we incorporate artist, genre, and emotion—elements commonly featured in music retrieval projects [2,3] and previous controlled music generation endeavors that incorporated subjective attributes [4-7].

However, your insight into the exploration of more obscure descriptions adds an intriguing dimension to our approach. Our framework is intentionally designed for expansibility, allowing for the integration of captivating and diverse descriptions, as per your suggestion. In response to your recommendations, we undertook the creation of an additional dataset by employing ChatGPT to refine the text-to-attribute understanding stage. During this phase, we directed ChatGPT to emulate the perspective of a musician, randomly selecting attributes and their corresponding values (which could also be user-provided). Following this, we tasked ChatGPT, now assuming the role of a user without a musical background, to utilize these chosen values in crafting descriptions enriched with more abstract language. To ensure coherence, we instructed ChatGPT to provide an explanation of how each description aligns with the selected values. This augmentation and examples are presented comprehensively in Appendix D.

[1] https://jmir.sourceforge.net/jSymbolic.html

[2] https://mubert.com/render/themes

[3] https://open.spotify.com/search

[4] Hung, Hsiao-Tzu, et al. "Emopia: A multi-modal pop piano dataset for emotion recognition and emotion-based music generation." arXiv preprint arXiv:2108.01374 (2021).

[5] L. N. Ferreira, L. Mou, J. Whitehead, and L. H. Lelis, “Controlling perceived emotion in symbolic music generation with monte carlo tree search,” in Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, vol. 18, no. 1, 2022, pp. 163– 371 170.

[6] C. Bao and Q. Sun, “Generating music with emotions,” IEEE Transactions on Multimedia, 2022.

[7] H. H. Mao, T. Shin, and G. Cottrell, “Deepj: Style-specific music generation,” in 2018 IEEE 12th International Conference on Semantic Computing (ICSC). IEEE, 2018, pp. 377–382.

W1.

We greatly value your guidance on the writing aspect, and we have incorporated your suggestions into the revised version. Continuous enhancements will be made in subsequent versions.

Q1.

As the "positional encoding" aspect is not the central focus of our work and was not introduced by us, implementation details can be found in the released code. We appreciate your suggestion, and, to avoid confusion, we have followed your advice by removing it. Thank you for your valuable input.

W2&W3&Q3.

Addressing the challenges posed by the abstract text prompt proves intricate due to the time-intensive processes involved in sample generation and questionnaire distribution, particularly for prompts requiring a crucial listening test. We may not be able to present a robust result at this point. Despite these constraints, we've included generated samples in the supplementary material (./SupplementaryMaterial/abstract_samples) for your reference. In internal testing, the majority of our team found the samples well-controlled with good musicality. However, team consensus varied, emphasizing the inherent subjectivity of such abstract text prompts. We eagerly await your feedback.

It's crucial to emphasize our primary contribution in tackling the pervasive low-resource challenge in the text-to-music generation task. Additionally, we aim to provide a valuable tool for musicians to enhance workflow efficiency. The experimental results and musician feedback in Section 4 affirm the method's beneficial impact. Acknowledging the inherent difficulty in addressing all challenges comprehensively or achieving perfection, even esteemed works [1,2] in audio generation may not consistently align with input texts.

Your suggestion to explore more obscure text descriptions in the future is notable. While extending our framework to diverse text descriptions, we recognize the importance of prompt engineering in constructing paired text-to-attribute data. Appendix D will incorporate this insight to guide further investigation. Once again, we appreciate your invaluable advice.

[1] Agostinelli, Andrea, et al. "Musiclm: Generating music from text." arXiv preprint arXiv:2301.11325 (2023).

[2] Huang, Qingqing, et al. "Noise2music: Text-conditioned music generation with diffusion models." arXiv preprint arXiv:2302.03917 (2023).

(3) It is never defined what AvgAttrCtrlAcc is in the main paper, how it is calculated, or how it is any different from ASA.

AvgAttrCtrlAcc serves as a key metric for evaluating control accuracy during the attribute-to-music generation stage. It specifically gauges the accuracy of predicting each attribute by calculating the proportion of correctly predicted instances across all testing samples. The resulting average accuracy for each specific attribute is presented in Table 11. It's important to distinguish AvgAttrCtrlAcc from ASA in two fundamental aspects: 1) AvgAttrCtrlAcc is computed by averaging accuracy over a particular attribute across the entire testing set, whereas ASA is an average calculated over samples containing multiple attributes within the testing set. 2) AvgAttrCtrlAcc is designed to evaluate the attribute-controlling performance during the attribute-to-music generation stage, focusing on the precision of individual attributes. In contrast, ASA is formulated to assess the overall alignment between generated samples and provided text descriptions.

Utilizing abbreviations may lead to a misperception that we are introducing an entirely novel metric. In reality, both metrics fundamentally measure accuracy in distinct aspects, which is not groundbreaking. To prevent any misunderstanding, we have decided to rename them as "Text-controlled Accuracy" and "Attribute-controlled Accuracy" in the updated version. Thanks for your advice.

(4) In Appendix B.2, it is mentioned that ASA values are calculated over different sample sizes and batch sizes for each method. While the costs for the GPT-4 baseline are understandable, I do not understand why the BART-base baseline could not have the exact same sample size as the proposed method.

Thank you for bringing this to our attention. The variance in sample size between MuseCoco and the BART-based baseline results from our listening test methodology. In the listening test, we typically sample three generated responses for each prompt. To manage time constraints, we opted to generate only five clips per prompt for the BART-based baseline. Your valuable feedback prompted us to reconsider, and in light of your input, we have increased the sample size for the BART-based baseline to 10 samples per prompt, aligning it with MuseCoco's sample size.

Consequently, we recalculated the ASA values, and the updated result for the BART-based baseline is now 32.47%, compared to the previous value of 31.98%. We will reflect this correction in the main paper. Your timely and thoughtful feedback has contributed significantly to the accuracy and completeness of our study.

W4. (1) ASA and AvgAttrCtrlAcc are metrics I have never seen before in prior work on controllable music generation, which makes it exceedingly hard to compare this work to prior works.

The metrics ASA (Attribute Specific Accuracy) and AvgAttrCtrlAcc (Average Attribute Control Accuracy) have been specifically introduced to gauge the control performance of the proposed system. Given the objective of guiding the music generation process through attributes specified in textual descriptions, it is imperative to assess whether the generated samples adhere to the attribute values outlined in the text.

ASA quantifies the correctness of predicted attributes within each sample and subsequently averages the accuracy across the entire test set. This metric is proposed for this text-to-symbolic music generation framework, which provides insight into the alignment between the generated sample and the textual descriptions, ensuring that the generated content aligns appropriately with the specified attributes. On the other hand, AvgAttrCtrlAcc measures the accuracy of each controlled attribute individually for the attribute-to-music generation stage. By evaluating the precision of attribute control for each specific parameter, this metric offers a nuanced understanding of the system's ability to accurately manipulate individual attributes.

In contrast to earlier methodologies [1,2] that employ end-to-end approaches for synthesizing symbolic music from textual descriptions, a significant challenge arises in assessing the alignment between the generated music and the specified attributes in the text. This difficulty arises because the prior methods do not explicitly identify these attributes. In contrast, MuseCoco takes a distinctive approach by explicitly defining objective attributes within the text descriptions. These defined attributes can be easily extracted from the generated music, facilitating a straightforward evaluation of alignment through direct accuracy calculations. This departure from traditional end-to-end methods not only enables a more precise assessment but also establishes a clear and robust relationship between textual descriptions and the resulting musical output.

[1] Shangda Wu and Maosong Sun. Exploring the efficacy of pre-trained checkpoints in text-to-music generation task. arXiv preprint arXiv:2211.11216, 2022.

[2] Yixiao Zhang, Ziyu Wang, Dingsu Wang, and Gus Xia. BUTTER: A representation learning framework for bi-directional music-sentence retrieval and generation. In Proceedings of the 1st Workshop on NLP for Music and Audio (NLP4MusA), pp. 54–58. Association for Computational Linguistics, 16 October 2020.

(2) How ASA is calculated.

ASA plays a crucial role in the Main Results section, specifically for comparing text-to-music generation against various baselines. Specifically, we produce samples for distinct baselines using identical text descriptions as inputs. These text descriptions are synthesized following the methods outlined in Section 3.3. Given that the text descriptions are generated based on the attributes defined in Table 8, we can easily assess the alignment of the generated music with the text descriptions. This evaluation involves extracting objective attributes from the generated music and comparing them with the attributes outlined in the text descriptions. For subjective attributes, we conducted a listening test to gauge control accuracy. Subsequently, we determine the proportion of accurately predicted attributes in each sample and compute the average accuracy across the entire test set.

To facilitate understanding, consider the following example:

“Text descriptions: 3/4 is the time signature of the music. This song is unmistakably classical in style. This music is low-tempo. The use of piano is vital to the music.”

Within these descriptions, three attributes—time signature, tempo, and instrumentation—are objective, while one, genre, is subjective. Objective attribute values are extracted from generated samples, and a listening test is conducted to categorize the subjective attribute (genre). Assume that correct predictions are made for time signature, tempo, and genre, while instrumentation predictions fall short. The resulting sample-wise accuracy is 75%. By aggregating these accuracies across the entire testing set, the averaged sample-wise accuracy (ASA) is determined.

We sincerely appreciate your valuable comments. In the following response to all the concerns and questions you have posted, we use W for weakness, Q for question, and L for limitation.

W1 & W2. The authors omit a large body of work in the generative music space that has dealt with attribute-conditioned generation, albeit in a more limited capacity.  It would be useful to contrast how the present work is sufficiently different from these previous papers, and to show whether the proposed method can beat prior works on the attributes they can condition on.

Thank you for your valuable suggestions. Firstly, MuseCoco significantly focuses on overcoming challenges in generating symbolic music from text descriptions. Consequently, a thorough comparison with methods specifically designed for this scenario is essential, and we have already undertaken this in Table 4.

Secondly, as attribute-to-music generation constitutes a crucial module in our system, it becomes imperative to conduct an ablation study to validate its effectiveness, as depicted in Table 5. In our analysis, we compare our control method labeled as "Prefix Token" with other widely used methods such as "Embedding" and "Layer Norm." The results reveal that in a multi-attribute control scenario, the "Prefix Token" method demonstrates significantly superior controllability, manifesting at least a 20% improvement.

The paper you referenced is indeed valuable for a thorough comparison to further validate our method. However, it's crucial to note that it falls within the category of methods we are already comparing. For instance, in [1], the conversion of segment-level conditions into an embedding vector, followed by concatenation with token embeddings at each time step, aligns with what we denoted as the "Embedding" method in Table 5.

Additionally, three other Works[2-4], namely "Compose & Embellish," "Multitrack Music Transformer," and "Compound Word Transformer," share similarities with the "Prefix Tokens" method employed in our approach. This similarity lies in the explicit representation of condition tokens in the input sequence instead of condensing them into a vector. Notably, this architecture is widely employed for decoder-only generation tasks, akin to Prefix language modeling [5].

We appreciate your observation regarding missing references and will promptly address this in Section 3.2 and Section 4.3.2 in the updated version. Thank you for bringing it to our attention.

[1] Wu, S. L., & Yang, Y. H. (2021). MuseMorphose: Full-Song and Fine-Grained Piano Music Style Transfer with One Transformer VAE.

[2] Wu, S. L., & Yang, Y. H. (2023). Compose & Embellish: Well-structured piano performance generation via a two-stage approach.

[3] Dong, H. W., Chen, K., Dubnov, S., McAuley, J., & Berg-Kirkpatrick, T. (2022). Multitrack Music Transformer.

[4] Hsiao, W.-Y., Liu, J.-Y., Yeh, Y.-C., & Yang, Y.-H. (2021). Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs.

[5] Wang, Thomas, et al. "What language model architecture and pretraining objective works best for zero-shot generalization?." International Conference on Machine Learning. PMLR, 2022.

Thank you for acknowledging our response.

Addressing the issues raised in the abstract text prompt presents challenges in providing concrete experimental results due to the time-intensive nature of sample generation and questionnaire distribution, particularly for prompts of this nature where a listening test is crucial. We may not be able to present a robust result at this point. Despite these constraints, we have included some generated samples in the supplementary material (./SupplementaryMaterial/abstract_samples) for your reference. During internal testing, the majority of our team members found that the samples were well-controlled and exhibited good musicality. However, consensus among the team varied, highlighting the inherent subjectivity associated with abstract text prompts of this nature. Your feedback on these samples is eagerly awaited.

It is important to underscore that our primary contribution lies in tackling the prevalent low-resource challenge in the text-to-music generation task. Additionally, we aim to offer a valuable tool for musicians to enhance their workflow efficiency. The experimental results and feedback from musicians in Section 4 demonstrate that our method has proven beneficial to some extent. It is essential to acknowledge the inherent difficulty in addressing all challenges comprehensively or achieving perfection in a single endeavor. Even notable works [1,2] in generating audio from text descriptions may not consistently align the generated music with the input texts.

Your suggestion regarding the consideration of more obscure text descriptions as a future avenue of exploration is noteworthy. In our efforts to extend our framework to encompass a broader range of text descriptions, we discovered the significance of prompt engineering in constructing paired text-to-attribute data. Experimenting with more meticulously designed prompts is imperative, and we will incorporate this insight into Appendix D to guide further investigation. Once again, we express our gratitude for your valuable advice.

[1] Agostinelli, Andrea, et al. "Musiclm: Generating music from text." arXiv preprint arXiv:2301.11325 (2023). [2] Huang, Qingqing, et al. "Noise2music: Text-conditioned music generation with diffusion models." arXiv preprint arXiv:2302.03917 (2023).

We sincerely appreciate your valuable comments. In the following response to all the concerns and questions you have posted, we use W for weakness, Q for question, and L for limitation.

W1. Evaluation details could be clearer - though the number of participants is stated in the appendix (19) the method by which they were sampled is not.

Thanks for asking. As we stated in Appendix B.1, “To ensure the reliability of the assessment, only individuals with at least music profession level 3 were selected, resulting in a total of 19 participants”. Since all the participants are above level 3, 19 is the exact number of participants without selecting by designed sample method.

W2. Claims that the model "closely resembles human performances in musicality" are not substantiated, since no comparison with human performance was done.

Thank you for bringing this out. Comparing the cost-effectiveness of our results with human compositions, where employing musicians to compose based on provided text descriptions incurs significant expenses, we have opted for a subjective listening test. This test is designed to evaluate musicality by gauging the similarity between the generated samples and human compositions. To measure this similarity, we have designed a scoring system ranging from 1 to 5 as illustrated in Section 4.1, with higher scores indicating a closer resemblance to human compositions, which is explicitly stated in questionnaire. To ensure the professionalism of the feedback, we only consider responses from participants with a proficiency level of at least 3, as defined in Table 9. This ensures that participants can provide expert evaluations on the gap between human compositions and the generated samples. MuseCoco attains a score of 4.06, significantly closer to 5 than the baselines. This scoring approach aligns with the methodology employed in numerous prior studies [1-4].

[1] Dong, Hao-Wen, et al. "Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.

[2] Yu, Botao, et al. "Museformer: Transformer with fine-and coarse-grained attention for music generation." Advances in Neural Information Processing Systems 35 (2022): 1376-1388.

[3] Schneider, Flavio, Zhijing Jin, and Bernhard Schölkopf. "Mo^ usai: Text-to-Music Generation with Long-Context Latent Diffusion." arXiv preprint arXiv:2301.11757 (2023).

[4] Shih, Yi-Jen, et al. "Theme transformer: Symbolic music generation with theme-conditioned transformer." IEEE Transactions on Multimedia (2022).

W3. Training was performed on private datasets so the result is not reproducible - it would be excellent if a comparison with publicly available data could be done.

We are committed to ensuring reproducibility, and therefore, we provide codes, and will release trained checkpoints later in the future, for transparency in our results. This will not only facilitate the replication of our findings but also enable the re-training of the model using publicly available data. While the released code supports re-training, it's worth noting that due to the limited availability of public data, utilizing the checkpoints directly is recommended for more robust reproducibility.

It's important to highlight that our past experiments indicate that larger datasets contribute to enhanced performance. To illustrate, we present the comparative results below:

In examining this table, it becomes evident that as both model size and data size increase, there is a corresponding improvement in musicality and control accuracy. Consequently, leveraging pretrained checkpoints becomes imperative to attain a relatively superior result.

We sincerely appreciate your valuable comments. In the following response to all the concerns and questions you have posted, we use W for weakness, Q for question, and L for limitation.

Q1. Perhaps because there are no examples in the body of the paper. Simply posting multiple sheets or piano rolls would probably improve the impression of the paper.

Thanks for your acknowledgement. Given that the generated music typically comprises multiple tracks and spans several bars, incorporating them into the main paper poses a challenge due to space constraints. Due to the page limit, we will put a link at the end of the demo page so that users can download the sheets for each sample in the demo page for future use.

Q2. And I realized after writing this that this is the baseline GPT-4 result to which you are comparing. That makes sense, but I would still like to see specific input examples and output results of the proposed method in the main body of the paper.

Yes, it is the baseline GPT-4 result that may not be controlled well. Given the limited page length, we showcase input examples and output results of our proposed method on the demo page, prioritizing the communication of the core idea within the main paper. We plan to release checkpoints in the future so that users can directly get the lead sheets from their given descriptions. Thanks for your attention.

Thank you for your response.

I understand that we can download generated data from the link, but my concern is that the GitHub demo page can be modified as much as you want without anyone except the most careful ones aware of it. Since Git basically allows log modification, it can be updated after the submission deadline, during peer review, or even after publication. This is not fair. The main contribution of the research results should be included either in the main text or in an appendix (supplementary materials). Appendices in particular have virtually no page limit, and it is not uncommon in this field for papers to have appendices that sometimes run into the dozens of pages. I don't think you can make the argument that the page limit is a reason not to include specific examples.

It seems very unnatural to me that the authors carefully include an example of the baseline method, but no specific information about the results obtained with the proposed method at all. A reader trained to read critically would suspect that the authors are withholding some inconvenient information.

I'm not asking you to list all the results, but I think it is normal for most papers in this field to include at least a few examples of piano rolls, if not the full score.

Thank you for your response.

When you suggested including the sheet in the "main body" of the paper, we opted to focus more on the general idea and results due to page limitations. However, if it's acceptable to add this information in the Appendix, we have done so in the updated version (i.e., Appendix E). We want to clarify that our argument is not an attempt to conceal inconvenient information; otherwise, we could have omitted the sheets from the demo page.

It's worth noting that not all papers of this nature necessarily include examples; it depends on the specific topics. For instance, works [1,2] also don't include examples, perhaps because it's challenging to assess effectiveness by examining only a few bars. In our case, explaining how the generated music aligns well with the text inputs is complex with just a few bars shown in the lead sheets. Some objective attributes need to be calculated over the entire sequence, and subjective attributes should be evaluated through listening experiences. We appreciate your proactive response and understanding, contributing to the enhancement and persuasiveness of our work.

[1] von Rütte, Dimitri, et al. "FIGARO: Controllable Music Generation using Learned and Expert Features." The Eleventh International Conference on Learning Representations. 2022.

[2] Wu, Shih-Lun, and Yi-Hsuan Yang. "MuseMorphose: Full-song and fine-grained piano music style transfer with one transformer VAE." IEEE/ACM Transactions on Audio, Speech, and Language Processing 31 (2023): 1953-1967.