Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models

Thank you for your review and we are glad to see the strengths of the paper is well-summarized. In this rebuttal, we mainly respond to the weakness and questions.

System complexity

We admit our system is indeed complex. In this paper, we believe the realization of compositional hierarchy (the main contribution of our work) is the “skeleton” of music composition; and on top of it we add autoregressive conditions and external conditions as “plugins”. these “plugins” are necessary and currently they are modeled in a simple way. In the future, we hope autoregressive condition can be extended to represent motif development and external condition can be developed into a hierarchical cross-modal control interface.

Ablation studies on model’s architecture

In our experiments, we ablated the assumption whether compositional hierarchy is necessary. We compared our proposed model with two types of baselines:

1. Models without compositional hierarchy at all. This corresponds to the baselines: Polyff. and TFxl(REMI).
2. Models with very simple compositional hierarchy (i.e., only phrase label as high-level language). This corresponds to the baselines: Polyff.+ph.l. and TFxl(REMI)+ph.l.

Both assumptions are implemented with diffusion models (more suitable to the current task) and Transformers (more conventional).

Moreover, we ablate “autoregressive condition” in our preliminary experiments. We trained a version with only compositional hierarchy but no autoregressive condition. The generated result is promising but only within 128 beats (the time scope of stage 2). For example, if two verse sections are far apart (e.g., 40 measures apart), the realization of the Counterpoint will be very different and so the melody lacks coherency. That’s why we incorporate autoregressive condition in our model architecture.

We see that there are other possible ways to define compositional hierarchy and it is important for this study to try more settings to ablate. Please let us know if there are other ablations you suggest and we can try to conduct the experiments as soon as possible.

Ablation studies on the efficacy of the control input

We train our model under both settings: 1) with external condition (under classifier-free guidance) and 2) without external condition. The performance of both models are similar when doing unconditional generation (without specifying latent codes of external conditions). The fact is also shown in the original classifier-free guidance paper [1].

The effectiveness of the external condition is studied and shown in generation example in section A.4 (Figure 5 (g)(h), Figure 6 (h)(i), and Figure 7(e)). In the Polyffusion paper [2], the authors already conducted systematic experiments to show the control efficacy in diffusion-based music generation models. If we have more time, we can do systematic experiments for whole-song generation.

Generalizability to other MIDI datasets

Currently, we use POP909 to study whole-song generation because our current focus is on pop songs and the dataset contains relatively accurate annotation. With the help of current MIR algorithms, it is possible to annotate larger MIDI datasets and train our model on these datasets, which is actually our ongoing project. We expect the performance will be better with more training data.

We admit there is a limitation for our current methods that our hierarchical language only works for pop songs (e.g., not for classical) and does not directly work for multi-track music. In our future work, we hope to define more general hierarchical languages to model those types of music.

[1] Ho J, Salimans T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598. 2022 Jul 26.

[2] Lejun Min, Junyan Jiang, Gus Xia, Jingwei Zhao: Polyffusion: A Diffusion Model for Polyphonic Score Generation with Internal and External Controls. ISMIR 2023.

Thank you for your careful review and acknowledging our contributions. In this rebuttal, we will respond to the weakness and questions: 1) plagiarism and overfitting issue and 2) dataset limitation. We also thank you for leaving some low-level comments which we will consider in paper revision.

Plagiarism issue

Thank you for pointing out this issue! Since this is really important and is also mentioned by other reviewers, we write a response to all reviewers to answer the question. Please kindly check the response to all reviewers for more detail. In short, the current results show that our model does not suffer from plagiarism problem.

Dataset limitation

Currently, we use POP909 to study whole-song generation because our current focus is on pop songs and the dataset contains relatively accurate annotation. With the help of current MIR algorithms, we can achieve not only phrase structure analysis, but also score quantization, melody extraction and so on. (Actually most of the annotation from the POP909 dataset is extracted using algorithms.) In our ongoing project, we are extracting annotations from larger MIDI datasets and train our model on these datasets. We expect the performance will be better with more training data.

We admit there is a limitation for our current methods that our hierarchical language only works for pop songs (e.g., not for classical) and does not directly work for multi-track music. In our future work, we hope to define more general hierarchical languages to model those types of music.

Thank you for your review and insight. In this rebuttal, we address the weaknesses in detail (in part one) and respond to the questions (in part two).

Weakness 1: non-standard diffusion model and unclear data representation

The diffusion model used in our method is the standard DDPM [1]. Polyffusion [2] has shown such method is an effective way to generate 8-measure music by regarding piano-roll as real-valued image data. (The training process is the same as DDPM. In inference the output of the generation will be quantized to discrete values and empirically such quantization operation does not yield syntax error [2].) In this study, we use the same method to represent music and its high-level languages. Data representation is discussed in section 3.1-3.2 and also in A.2. If there is other details about the model and data representation you think is missing, please let us know. We are happy to explain and make it clearer in the revised version.

Weakness 2: Limited contribution for ICLR community

Hierarchical music generation is the core problem and we propose to bring innovative methodology to solve it by 1) defining a music compositional hierarchy, and 2) training a corresponding cascaded generative model. The methodology is efficient in data usage and computational resource, and is more controllable and interpretable compared to other existing music generation methods. In this paper, we see Polyffusion as a tool to realize this goal. The Tonal Reduction Algorithm is a small but important design to make sure the hierarchical model is functioning.

For the ICLR community, hierarchical symbolic music generation, as a sequence generation problem, has a unique significance. As stated by Prof. LeCun [3], learning multiple levels of abstraction and hierarchical prediction is one of the main challenges to address for today’s AI research. Symbolic music has a rich and ambiguous structure, and the current research in symbolic music can shed light on learning hierarchical structure in natural language, image, as well as in art and real-world long-term planning in general.

Questions

Q1: The diffusion model used auto regressively

Autoregressively applying diffusion model is indeed not a standard way. The method is the same as image inpainting and is first introduced in Polyffusion [2] as a way to generate coherent sequences. Such method is necessary for whole-song generation because as we generate lower level languages, the sequence becomes longer.

The autoregressive generation process is introduced in Algorithm 1. Here we try to explain it more intuitively. For example, a diffusion model can generate a sequence of fixed length T, equal to the image width. In the autoregressive process, the model first generates [0, T]. Then the model shifts right by T // 2 and inpainst [T, T + T//2] conditioning on the generated part [T//2, T], and so on so forth.

Q2: What are the pretrained models for chord progression, rhythmic pattern, accompaniment texture?

For both latent codes of chord progression and accompaniment texture, we use the pre-trained encoders in [4]. For latent code of rhythmic pattern, we use the pre-trained encoder in [5]. These are introduced in A.4 and we will make it clearer in our revision.

Q3: "Conterpoint" term for the second stage is not appropriate as this stage describes the sketch of the melody and harmony t. "Draft" or "Sketch" stage may be more suitable?

Thank you for the suggestion. We realize that the “Counterpoint” might not be precise. We will consider to use “Draft” or “Sketch”. And please let us know if you have other idea since we want the name to be more specific.

In fact, “Counterpoint” is designed to emphasize the idea that the intermediate level music flow is represented by the interplay of melodic and harmonic development. From a Schenkerian perspective, we can regard melody and bass of the chord as a two-part counterpoint and the chord notes as “figured bass”. This is what we mean by “Counterpoint” here.

Q4: Leadsheet encoded as melody and chords? Are chords in string format?

The Lead sheet is encoded as melody and chords. The chords are represented as piano-roll as discussed in section 3.2. Section A.2 shows an example of such data representation in piano-roll format.

Q5: What is the sampling time for a whole song?

We train and inference our model on GeForce RTX 2080 Ti. It takes several minutes (5 - 20 minutes) to generate a song depending on the length. For example, it takes approximately 15 minutes to generate eight pieces (batch size=8) of 40-measure songs. On more powerful computing resources, we expect better performance.

References

[1] Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models. Advances in neural information processing systems. 2020;33:6840-51.

[2] Lejun Min, Junyan Jiang, Gus Xia, Jingwei Zhao: Polyffusion: A Diffusion Model for Polyphonic Score Generation with Internal and External Controls. ISMIR 2023.

[3] LeCun Y. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review. 2022 Jun 27;62.

[4] Ziyu Wang, Dingsu Wang, Yixiao Zhang, Gus Xia: Learning Interpretable Representation for Controllable Polyphonic Music Generation . ISMIR 2020: 662-669

[5] Ruihan Yang, Dingsu Wang, Ziyu Wang, Tianyao Chen, Junyan Jiang, Gus Xia: Deep Music Analogy Via Latent Representation Disentanglement. ISMIR 2019: 596-603

Thank you for your careful review and very detailed feedback. In this rebuttal, we first respond to the weaknesses (in part one) and then answer each of the questions (in part one and two).

Weakness 1: discussion of limitation is missing.

Thank you for pointing out our current limitation. We admit the current work is limited to pop music and we will put it in our revision.

On the other hand, high-level form does not need to be given. In the training of stage one, the samples of Form are zero-padded to 256 measures. So in inference, the model can generate Form containing 256 measures and we derive the song length M by finding the first time step that contains all-zero entries. In section 4, we show music piece of 40 measures under given form for better visualization and space-saving purpose.

Weakness 2: Evaluation of overfitting.

We indeed see the evaluation of overfitting is crucial. We are conducting a systematic evaluation on the issue. Please see the response to all reviewers for more detail. In short, the current results show that our model does not suffer from plagiarism problem.

Weakness 3: No evaluation method of whole-song generation.

Directly evaluating whole-song generation is hard because 1) the evaluation of longer segments is more subjective; and 2) long music segments take a longer time to listen to and is hard to be memorized. In this paper, we propose to effectively evaluate whole-song generation by decomposing the evaluation problem into structure and quality evaluation:

• Structure: a good structure usually implies clear phrase boundary, similarity between, e.g., two verses, and dissimilarity between, e.g., verse and chorus. These are evaluated by ILS in objective evaluation (Table 2) and “Boundary clarity” & “Phrase similarity” in subjective evaluation (Figure 3c&d).
• Quality: we consider music quality in both short-term (8 measures) and long-term (32 measures) using criteria “Naturalness”, “Creativity” and “Overall musicality” (see Figure 3).

Also, in the evaluation, whole songs are generated under several condensed music form such that the length of the piece is balanced: on one hand, the generated pieces won’t be too long (i.e., within 1’30’’); on the other hand, the music contains all necessary sections with repetition of verses or choruses.

Questions

• (Section 1) "and therefore we need to organize various music representations in a structured way." -> I cannot understand this sentence. Why do you mean by "organize representations"?

  The common sense of music hierarchies are ambiguous. There are a lot of possible music hierarchies (e.g., pitch contour, tonal structure or metrical structure), but these hierarchies are usually not compatible with one another and do not serve for generation purpose. In this paper, we rely on these concepts to define music compositional hierarchy that can directly used in music generation.

• (Section 3.2) "continuous" -> I'm not sure what "continuous" means here. Are you using binary or real-valued piano rolls?

  The data are represented as real-valued piano-rolls and the generated real-valued piano-rolls will be quantized. The process is the same as Polyffusion [1], where the authors show such quantization operation has almost zero syntax error in practice.

• (Section 3.2) "Both melody reduction and simplified chord progression ..." -> How were these achieved? A pointer to the Appendix would be helpful here.

  These are computed using Tonal Reduction Algorithm discussed in A. 1. We will add the pointer here.

• (Section 3.2) "13 times" -> Why 13 times? Isn't it 128/12 = 11 times?

  It should be 11 times. We will fix the typo in our revision.

• (Section 3.3) "We select Sk relevant music segments prior to t based on a defined similarity metric on X<k." -> Is this X^k or X^<k? The descriptions in this paragraph are somewhat confusing. From Figure 1, it seems like it's both X^k and X^<k. Please clarify this.

  Sorry for the confusion. The term $X^{<k}$ here is correct. We use $X^{<k}$ as the criteria to select segments but embed the corresponding $X^{\leq k}$ as condition. This is not clearly addressed in the submission and we will discuss the procedure in detail in our revision.

• (Algorithm 1) Isn't the song length M also an input?

  No, it’s not. In the quantization step discussed above, we quantize the generated real-valued piano-rolls. In the training of stage one, the samples of Form are zero-padded to 256 measures. So in inference, the model outputs Form containing 256 measures and length M is determined by finding the first time step that contains all-zero entries. We will make it clearer in the revision.

• (Section 4) "40 measures" -> How do you determine the number of measures to be generated?

  Here the first-level language Form is given and so the length of the piece is given. We select a given 40-measure Form for better illustration and space-saving purpose.

• (Section 5.1) "... and segment them into 8-bar musical segments with a 1-bar hopping size." -> What is this segmentation step for?

  We apologize for the unclarity. It should be “We segment pieces into 32-bar segments to train the second stage of the model, and segment pieces into 8-bar segments to train the third and forth stage of the model”. We will revise the paper accordingly.

• (Section 5.3) "Using pre-trained VAEs from Yang et al. (2019) and Wang et al. (2020)." -> Are these the models you used to extract autoregressive controls?

  No. Unlike external condition, for autoregressive condition, we use a trainable encoder consisting of convolutional layers. We will make these details clearer in the revision.

• (Section 5.3) How did you select the test inputs for the evaluation?

  All the samples are randomly selected. Samples that have obvious failure will be discarded for both proposed and baseline methods. Such failure happens rarely, e.g., blank for one measure). For 8-measure subjective evaluation, the first two measures are given, and for 32-measure subjective evaluation, the first 4 measures are given. This setting reduce the possible variance in unconditional generation evaluation.

  The reviewer also asks why not include ground truth samples in 32-measure subjective evaluation. The reason is that we regard 32-measure as a condensed pop song form (see response to Weakness 3). Such ground truth does not exist in the dataset. In the future, we should consider rewriting a ground truth piece into shorter piece for comparison.

We are thankful to the reviewer to leave other comments and suggestions regarding organization of the paper and detailed wording choice. We agree with the reviewer in general and will consider revising the paper accordingly. (Note that some of the comments are addressed in detail in the above paragraphs.)

Finally, as a response to the ethics flag, the arrangements in the POP909 dataset (the MIDIs and not the original audio) do not have copyright issue because it is made to the public by the authors of the POP909. Also, with our evaluation of plagiarism, we conclude that the plagiarism in our generation is unlikely, and if it happens, it can be detected.

References

[1] Lejun Min, Junyan Jiang, Gus Xia, Jingwei Zhao: Polyffusion: A Diffusion Model for Polyphonic Score Generation with Internal and External Controls. ISMIR 2023.