Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation

We deeply appreciate the reviewer's valuable comments. Please allow us to provide responses as follows:

1. Using this FGG is of course beneficial for most of the cases, but it "eliminates" the generation of such tension-imposed music. How can the authors justify this?

We agree that some composers intentionally put out-of-key notes to add “creativity”. However, it is observed that generative models often fail to create an accommodating context for such “creativity”. In other words, most of the out-of-key notes generated by models cannot add creativity, but instead disrupts harmony. We think the main reason is that out-of-key notes are relatively rare in most datasets, which makes it difficult for the model to learn how to “correctly” use out-of-key notes. Given such a limitation in data, the benefit of avoiding out-of-key notes could outweigh the loss of potential creativity.

2. I'm not sure if Proposition 2 is tight enough. If it's not tight, then the ground of Section 4 is largely weaken.

We also cannot theoretically verify whether it is tight. We would like to clarify the following: If Proposition 2 is not tight—meaning the error probability decays even slower than $O(n^{−1/(LH)})$—then the probability of generating "unresolved wrong notes" remains even higher. Therefore, the lack of tightness in Proposition 2 does not undermine the main argument in Section 4.

3. Could the authors provide more intuition of Eq. (8)?

Eq. (8) ensures that: if (l,h) is not an out-of-key position, predicted $X_0$ will be unchanged; if (l,h) is out-of-key and predicted $X_0$ exceeds 0.5, then predicted $X_0$ will be reset to 0.5. This acts as a projection method, enforcing that out-of-key notes cannot be present. The threshold of 0.5 is chosen because, in piano roll quantization, values <= 0.5 mean no note, while values >0.5 indicate a note. Instead of correcting only at the final step, we apply this constraint throughout the sampling process to maintain the flexibility of diffusion models while ensuring harmonic coherence.

4. Why is music treated as fundamentally different compared to image or video generation?

Music generation follows two main approaches: treating music as an image (e.g., piano roll representation) or as language (e.g., sequences of pitch, duration, and timing tokens). However, it has the following challenges:

• Precision Sensitivity – In image generation, slightly changing a single pixel usually has minimal impact, whereas one misplaced note in music can significantly disrupt harmony.

• Temporal Overlaps – Unlike text, where words follow a sequential order, musical notes often overlap, which leads to challenges when conducting music generation in the “next token predictor” manner.

• Genre-Specific Rules – Music follows complex, style-dependent structures, such as tonal harmony in classical music, or intricate rhythms in jazz.

These factors make it difficult to develop a unified, generalized model for music generation. For now, the fragmentation in music models is partially due to the complexity of musical structure and the precision required to generate high-quality music. But as generative AI matures, we should aim for models that can adapt dynamically to different musical styles rather than being strictly bound to a single one. In fact, our control method can also be applied in stylized music generation (see the next following paragraphs).

5. Does our method has a broader impact or scalability?

We think that our proposed method (especially the sampling control part) is generalizable. In our paper, we enforce a "no out-of-key notes" constraint within the sampling process. As shown in the demo page, this method can be applied to generate stylized music with special scales (e.g., Chinese pentatonic scale).

More broadly, our method can be adapted to various musical constraints. Many genres and styles can be defined through rules (e.g., harmonic, rhythmic, or structural constraints), which our sampling method can also incorporate. For example, rhythmic control could be implemented by ensuring that certain time positions must (or must not) contain notes. Then we can incorporate this requirement in the sampling process as follows: in each step, we edit $\epsilon$ to project predicted $X_0$ to the correct domain where the rhythmic constraint is satisfied. Such rhythmic control method could be used for generating jazz music which has special rhythmic requirements.

We deeply appreciate the reviewer's suggestions on revising! Due to this year's policy, it is not allowed to upload a revised paper, external links can only contain figures/tables, and rebuttal has a 5000 length requirement. Please allow us to describe a revision plan in the follows and promise to follow through in next phase.

Responses

1. To train a model that can handle both rhythm+chord and chord-only conditions, we must distinguish when rhythmic conditions are provided. We achieve this by using negative values to indicate the absence of rhythmic conditions, preventing the model from misinterpreting 0s and 1s as rhythmic constraints applied everywhere. This design choice was determined through empirical experimentation.

2. Chord progressions are harmonic structure of music, and the latent representation better accounts for the degrees of similarity between chords. For features like pitch, duration, and note density, we evaluate the overlapping area between their distributions in generated and ground truth segments, to evaluate how well the generative model captures the statistical properties inherent in the original compositions.​ We also added three metrics to directly measure generation quality: Direct Chord Accuracy, IoU of Chord, and IoU of Piano Roll. New results are shown in https://drive.google.com/file/d/1IAcAqK4qK4AiQVKWriFSJ91QNhrg5at-/view.

3. See "related literature" in the following.

Revisions in writing

1. Contribution:

Motivation: We theoretically and empirically characterize the challenge of precision in symbolic music generation

Methodology: We incorporate fine-grained harmonic and rhythmic guidance to symbolic music generation with diffusion models.

Functionality: The developed model is capable of generating music with high accuracy in pitch and consistent rhythmic patterns that align closely with the user’s intent.

Effectiveness: We provide both theoretical and empirical evidence supporting the effectiveness of our approach.

2. Related Work:

Thank you for your suggestions, we really appreciate the structure line of organizing this paragraph that you have provided us. We will integrate it as:

To leverage well-developed generative models for symbolic music, Huang et al. (2018) introduced a Transformer-based model with a novel relative attention mechanism designed for symbolic music generation. Subsequent works have enhanced the controllability of symbolic music generation by incorporating input conditions. For instance, Huang and Yang (2020) integrated metrical structures to enhance rhythmic coherence, Ren et al. (2020) conditioned on melody and chord progressions for harmonically guided compositions, and Choi et al. (2020) encoded musical style to achieve nuanced harmonic control. These advancements have contributed to more interpretable and user-directed music generation control.

To better capture spatio-temporal harmonic structures in music, researchers have adopted diffusion models with various control mechanisms. Min et al. (2023) incorporated control signals tailored to diffusion inputs, enabling control over melody, chords, and texture. Wang et al. (2024) extended this by integrating hierarchical control for full-song generation. To further enhance control, Zhang et al. (2023) and Huang et al. (2024) leveraged the gradual denoising process to refine sampling. Building on these approaches, our work addresses the remaining challenge of precise control in real-time generation.

3. Theory

We will make the theoretical proofs in appendix more detailed by (1) expanding more steps and (2) adding more recalls of formulations from the text directly to the appendix. We will also simplify proposition 2 by removing the conditional probability argument, the details of which are in our response to reviewer uD9t.

4. Sections 2 and 3.

We will add subtitles to section 2, namely “Data representation of the piano roll” and "Formulation of the diffusion model", at corresponding places.

We will delete the third paragraph “The FGG method improves…” of section 3.

We will revise the first paragraph of section 3.2 to:

We first provide a rough idea of the harmonic sampling control. To integrate harmonic constraints into our model, we employ temporary tonic key signatures to establish the tonal center. Our sampling control mechanism guides the gradual denoising process to ensure that the final generated notes remain within a specified set of pitch classes. This control mechanism removes or replaces harmonically conflicting notes, maintaining alignment with the temporary tonic key.

We will add subtitles or topic sentences “mathematical formulation of the harmonic sampling control”, "preliminaries", "Edit intermediate-step outputs of the sampling process" and “theoretical property of the sampling control” to corresponding places of section 3.2, and do further refinement.

We deeply appreciate the reviewer's suggestions on revising! Due to this year's policy, it is not allowed to upload a revised paper, external links can only contain figures/tables, and rebuttal has a 5000 length requirement. Please allow us to describe a revision plan in the follows and promise to follow through in next phase.

Responses

1. To enforce Dorian, we restrict pitch classes to the Dorian scale, and additionally use the Am-Em-C-D chord progression for shaping. The combination of them makes it different with G major.

2. We added a comparison with the inpainting baseline (see 4. Additional Experiments and demo page below). We did not initially frame our methodology from an inpainting perspective. Generating with minor correction looks more efficient to us, since a well-trained model aligns with chords, yielding only 2% incorrect notes. In contrast, inpainting adds complexity to the model, while much of the Roll[time,pitch]=0 in the input might be redundant.

3. We actually use [0,1] data for training.

4. "simple rule-based post-sample editing" is “training control edit after sampling”

Revisions

1. Additional related work. We will add two sections, one on precise control over symbolic music generation with other generative models, the other on image inpainting.

2. Modified claims.

• We will explain that the user-specified control refers specifically to user-designed control within the constrained piano roll format.
• We will remove the misleading word “out-of-sample”, and replace with the statement “our method can shape the output towards a specific tonal quality.”
• We will describe the demo page as “a demo page to showcase performances, which enables real-time generation”, removing "one of the first".

3. Theory. We decide to remove the argument regarding conditional probability $\widehat{P}(\bar{R}|O)$, which seems too intricate. Further, the introduction of proposition 2 (after the discussion of out-of-key notes and resolutions and before proposition 2) will be:

We provide an explanation using statistical reasoning. Consider a piano roll segment, represented as a random variable $M$. Suppose we are interested in whether this segment contains an out-of-key note (denoted as event $\{O\}$) and whether that note is eventually resolved within the segment (denoted as event $\{R,O\}$). In our training data, almost every out-of-key note is resolved, meaning the probability of unresolved out-of-key note is close to 0, i.e., $P(\bar{R},O)\approx 0$.

Now, we examine the probability in the generated music. The key question then is whether the generative model also learns to keep $\widehat{P}(\bar{R},O)$ small. The following proposition 2... (same as in manuscript).

4. Additional Experiments and Samples.

In numerical experiments, we have added a baseline named inpainting, in which we treat the pixels where there should not be a note as known (value should be 0), and let the model inpaint the remaining parts. To do this, we add a mask to model inputs, and trained an inpainting model. We also added rounding the out-of-key notes to the nearest allowable in our ablation study. We also added more interpretable metrics. Results are shown in https://drive.google.com/file/d/1IAcAqK4qK4AiQVKWriFSJ91QNhrg5at-/view.

Samples generated from ablation conditions are added to Section 3 of demo page. Across all ablations, we observed occasional occurrences of excessively high-pitched notes and overly dense note clusters.

5. Discussion of limitation:

The 16th-note quantization follows prior work (Wang et al., 2024), which admittedly reduces rhythmic flexibility, and cannot ingest data without beat information. A potential improvement is integrating our pitch class control method with (Huang et al., 2024), which adds a dynamic dimension, and uses 10ms time quantization for greater rhythmic flexibility. Another key limitation is the control format. Our method supports pitch class and rhythmic control in the piano roll representation, but does not accommodate more abstract forms or probabilistic control. Additionally, the evaluation methods and datasets present challenges in accurately assessing generated music quality. Since music evaluation is inherently detailed and partly subjective, the metrics used in this work has fundamental limitation in metrizing quality improvement.

6. More explanations of the demo page

In section 2 of demo page, the chord conditions are converted to condition matrices exactly following Figure 2 of the paper, the melody conditions are provided in an additional channel, also in the form of a piano roll, and we did not use rhythmic constraint in the generation. We will add a section in appendix to provide description of the matrix. Sample melodies are either randomly picked from the test set of POP909, or extracted by hand from some of our favorite existing pieces.

We deeply appreciate the reviewer’s recognition of our work's theory, experiments, and the practicality shown by the interactive demo.

Regarding OOD datasets, we unfortunately have not yet found a high-quality dataset other than the one we used. Alternatively, in the numerical experiments, the songs of POP909 dataset were split upfront into training and testing sets, to prevent data-leakage in the training process.

Meanwhile, although there is not an appropriate OOD dataset for us to run a thorough experiment, we tried to use our model to generate stylized music pieces in section 2 of our demo page. Those styles did not appear in the POP909 dataset but the generated results seem satisfying, which hopefully serves as an indication of our model’s generalization ability.

We would love to thank Reviewer 2Mem again for the comments and suggestions, which are very helpful to us.