GETMusic: Generating Music Tracks with a Unified Representation and Diffusion Framework

• There is a potentially significant issue of missing references. The training objective and inference process for the proposed model GETDiff is quite similar in nature to those used in Huang et al. [1], in which the authors propose a discrete training objective, predicting missing notes from piano-rolls which have been randomly partially masked. At inference time, they use blocked Gibbs sampling, which is reminiscent of the inference procedure outlined in Section 3.2. Although the proposed approach is multi-track, and the framing of GETDiff as a discrete diffusion model changes the loss function, at the very least this work should be referenced and the similarities should be addressed in the related work. Some other relevant references are also missing, and are not present in ablation experiments, such as [2].

• If I understand correctly, the ablation experiment in Section 5 (L457-463) is not very well designed. By using 14 separate prediction heads and presumably sampling each column in the GETScore with a single forward pass, the training objective isn't accurately represented for GETDiff AR. As an example, when predicting the length of a note, it is impossible for the model to condition directly on the pitch of the note that it is predicting, and instead can only condition implicitly on the distribution of possible pitches predicted by the model. This introduces mathematical issues which may be responsible for the degraded performance. A much better ablation would be to compare against a transformer-decoder trained to predict the next token for a flattened version of GETScore. This should be technically possible as it would only require a context-length of 512*14=7168.

• There are some very minor issues about expressivity. According to our understanding, it is not possible to represent concurrent notes (e.g., chords) within a single track that have differing offsets.

[1] Huang, C.Z.A., Cooijmans, T., Roberts, A., Courville, A. and Eck, D., 2019. Counterpoint by convolution. arXiv preprint arXiv:1903.07227.

[2] Thickstun, J., Hall, D., Donahue, C. and Liang, P., 2023. Anticipatory music transformer. arXiv preprint arXiv:2306.08620.

The biggest weakness is the limited contribution, as diffusion models for symbolic music and conditional track generation have already been explored in previous work such as AccomMontage, SongDriver. The new representation method also lacks comparisons with alternative approaches.

The experiments are incomplete; each contribution requires validation. For instance, it’s unclear how the representation method outperforms others or how the diffusion model improves over baseline diffusion models. Additionally, more recent works should be included in task-level comparisons, as PopMAG was introduced four years ago.

This also suggests that the related work survey is incomplete, omitting recent studies on conditional generation in symbolic music.

These are not in order of importance.

1. There are already more symbolic music generation representations and models out there than I can keep track of, and they all sound pretty decent. I consider this problem basically "solved" since the release of OpenAI's MuseNet (which had no accompanying academic paper). It's not clear that this paper is a significant advance on what is already possible.

2. The CoCoNet model by Huang et al. (https://arxiv.org/abs/1903.07227) uses a setup that is very similar to this paper: multiple tracks are generated with arbitrary segments fixed as conditioning; instead of diffusion, the remaining portions are generated iteratively using Monte Carlo sampling.

3. The paper seems confused about the taxonomy of symbolic music representations, dividing the space into "image-based" and "sequence-based" representations. Here it would make sense to examine the pitch and time axes separately. Either axis can be treated in dense ("image-based") or sparse ("sequence-based") fashion.

   With time, the main reason one might use a sparse approach is to handle expressive timing; the dense resolution becomes extremely high. This paper does not model expressive timing and thus uses a dense approach, with exactly two tokens per time step. However, it's worth noting that the approach in the paper cannot easily be extended to handle not only expressive timing, but also things like triplets, without blowing up the time dimension.

   With pitch, the main reason to use a dense approach is to handle polyphony; for monophonic music the pitch axis can be collapsed into a single value at each time. However, for many polyphonic instruments e.g. piano, the space of possible pitches is quite large, making sparsity desirable. This paper handles polyphony in a somewhat unique way, flattening variable-length combinations of notes into single tokens (see next item).

4. The handling of polyphony is very unsatisfying. For example, all combinations of piano notes are compressed to a vocabulary of 1555 tokens. This isn't even enough to represent all pairs of piano keys! And the drum vocabulary is almost 3 times as large as the piano vocabulary; how did this end up happening?

   Here's a way polyphony could potentially have been handled that only minimally changes the setup. On the input side, instead of blowing up the vocabulary with combinations of pitches, sum (or average) the token embeddings of all active pitches. On the output side, instead of a softmax over pitch combination tokens, sample the binary presence/absence of each pitch independently (for a diffusion model, this independence should be okay since the other cells are sampled independently anyway) then sparsify. This shouldn't increase memory usage since you need to construct the softmax vector anyway.

   (It's entirely possible that you already tried the above suggestion and it ended up not working; if so please disregard.)

5. I am not especially knowledgable about diffusion modeling, even less so about discrete diffusion. But it's not clear whether the method in this paper goes beyond the standard approach. From what I can tell, the use of condition flags is new, but that raises the question of why previous discrete diffusion methods didn't need to use such flags, and the paper provides no discussion of this.