Symbolic Music Generation with Fine-grained Interactive Textural Guidance

5. Results on Classifier-Free Guidance Implementation

Classifier-free guidance enables a single model to produce both conditional and unconditional outputs seamlessly by interpolating between guided and unguided sampling, offering flexibility in use cases without requiring multiple models. Since providing rhythmic condition specifies the rhythmic pattern of the model output, whereas no rhythmic condition allows the model to generate any kind of rhythmic pattern, one can choose to not insert rhythmic condition so that the model can generate diversified rhythms. That is what we meant by saying that classifier-free guidance enable the model to generate under varying levels of conditioning.

Thus, we do not intend to compare the results with and without using classifier-free guidance because that is not the focus of our proposed method. The main reason why we use classifier-free guidance is to obtain a unified model that can be both conditional on rhythmic patterns or unconditioned on rhythmic patters.

We admit that the previous sentence "While conditional generation enhances output stability, it comes at the cost of reduced sample variation, particularly when rhythmic conditions are applied." might be confusing, and we have removed it in the updated manuscript.

6. Request for a Conclusion Section

We thank the reviewer for the suggestion. We have added a conclusion section in our manuscript.

7. Comments on the evaluation metrics

Our evaluation metrics, including cosine similarity of chord embeddings and OA of features, have also been applied by other papers in the literature. [1, 2, 3, 4] We appreciate the reviewer’s suggestion of adding subjective evaluations. While having a comprehensive comparison study of subjective evaluations is challenging, we plan to explore ways of adding sensible subjective evaluations or other non-quantitive alternative analysis for music.

References

[1] Ziyu Wang, Lejun Min, and Gus Xia. Whole-song hierarchical generation of symbolic music using cascaded diffusion models. The Twelfth International Conference on Learning Representations, 2024.

[2] Ang Lv, Xu Tan, Peiling Lu, Wei Ye, Shikun Zhang, Jiang Bian, and Rui Yan. Getmusic: Generating any music tracks with a unified representation and diffusion framework. arXiv preprint arXiv:2305.10841, 2023.

[3] Dimitri von Rutte, Luca Biggio, Yannic Kilcher, and Thomas Hofmann. Figaro: Controllable music generation using learned and expert features. In The Eleventh International Conference on Learning Representations, 2023.

[4] Kristy Choi, Curtis Hawthorne, Ian Simon, Monica Dinculescu, and Jesse Engel. Encoding Musical Style with Transformer Autoencoders. In Proceedings of the 37th International Conference on Machine Learning, 2020.

We appreciate the reviewer’s valuable suggestions. We have modified our manuscript and provide the following response to the reviewer’s questions, followed by some other comments.

1. Direct Metrics for Rhythmic Coherence and “Wrong” Note Removal:

We provide an interface that can insert specific rhythmic control in part 3 (interactive real time generation) of our demo page (https://huggingface.co/spaces/interactive-symbolic-music/InteractiveSymbolicMusicDemo), under the “use customized chord” icon.

We thank the reviewer for pointing out the potential trade-off in eliminating the wrong notes in harming the model’s ability to provide “surprises”. We have the following remarks on this aspect.

(1) While the out-of-key notes are never necessarily “wrong”, it is observed that generative models often fail to create an accommodating context for such “surprises”. We have uploaded examples to part 1 of the demo page to demonstrate the phenomenon. Sometimes the generated accompaniment without control can be completely chaotic, completely mismatching the style of the melody (e.g., example 1 and 3 in part 1 of our demo page https://huggingface.co/spaces/interactive-symbolic-music/InteractiveSymbolicMusicDemo). We also aim to explain this phenomenon using section 3.1, which now has been revised to be more straightforward.

(2) Instead of eliminating a fixed note, a more important emphasis of our method is the precise control over the generation process, aiming to avoid producing any "unwanted notes." Since the sampling control can be designed by the user of FTG, the appearance of "wrong notes" can also be intentionally incorporated as part of the sampling guidance rules. This method may also be used to provide precise control to generate a class of pre-specified music types.

2. Clear Task Definition for Accompaniment versus Music Generation

The conditional guidance and sampling control proposed in our work can be integrated into existing diffusion-based frameworks for symbolic music generation, e.g., WholeSongGen[1]. Therefore, our methodology of applying precise control over generation is not limited to a specific music generation or accompaniment generation task. To avoid misunderstandings, we have changed the title of our work to “Efficient Fine-Grained Sampling Guidance for Diffusion-Based Symbolic Music Generation”, and we accordingly modified the structure of the content.

3. Intuitive Explanation of Theoretical Analysis

We have revised the section by removing unnecessary modeling on key signatures. Our theoretical analysis aims to answer the following question: Why is it hard for models to eliminate “wrong notes” all by themselves via training? Our theoretical analysis points out that the difficulty comes from the requirement of precisely generating every pixel of the piano roll canvas, combined with a high data dimension. Admittedly, providing rigorous theoretical analysis often requires complicated modeling and assumptions. In our new version, we removed unnecessary notations and assumptions. We also supplement the theoretical analysis with empirical samples generated by models, where the out-of-key notes result in significant dissonance, as well as a numerical experiment calculating the percentage of such notes, if no sampling control in applied.

4. Rationale for Excluding Triple Meter Pieces

Our method of fine-grained textural guidance can be applied to triple meter pieces as well as 4/4 pieces. The reason that we did not do experiments on triple meter pieces is lack of high-quality training data, as suggested in [1]. (“The capability of 3/4 song generation is limited, since the proportion of 3/4 songs in the dataset is pretty low.”)

References

[1] Ziyu Wang, Lejun Min, and Gus Xia. Whole-song hierarchical generation of symbolic music using cascaded diffusion models. The Twelfth International Conference on Learning Representations, 2024.

Thank you for the time and comments, and we appreciate your encouraging message on our response about providing user-defined control and intuitive explanation of the theory.

We have added an initial set of subjective evaluations in Section 5.1.4 and Appendix F, and some additional numerical experiments in 5.1.5 and Appendix H of the current manuscript. We uploaded more examples on the demo page. Regarding your feedback about remaining unresolved aspects, we hope to add the following comments:

Metrics and the “Wrong” Notes Debate

Thank you for the insightful discussion. We agree that developing a model that enjoys the ability to implicitly learn contextually appropriate "wrong notes" would be immensely valuable. We believe that this would be an important line of future work, and we really look forward to the progress that can be made in this area. We think that this line of exploration would greatly complement, rather than conflicts with, the objectives of our current work. In addition, we think it is also worthwhile to develop methods that enable the model to function as a well-controlled collaborative tool to aid human composers. We believe that both directions are compelling and hold great potential for advancing the field.

We have revised the descriptions and logic of our work to emphasize "control" rather than "correctness." Out-of-key notes are not inherently "wrong"; instead, the issue of degraded quality in generating out-of-key notes serve as an illustrative example of the challenges that current models face in accurately capturing distributional information in symbolic music data. Furthermore, the external sampling control is not limited to correcting "wrong notes", but is designed to provide a more efficient and reliable framework for aligning the model's output with the user's intent.

We further hope to make the following clarifications:

1. Our method is intrinsically different from “detecting wrong notes and regeneration” in terms of sampling efficiency. There is no guarantee that regeneration is capable of eliminating all undesirable notes within a controllable amount of time. The improvement of efficiency has a vital benefit for real-time situations such as improvisation.

2. Different from straightforwardly removing “wrong notes”, our method is not targeted on “deleting incorrect notes in the final sample”. As described in the paper, our method fundamentally guides the diffusion sampling process by adjustments in middle and intermediate steps. If we view the diffusion process as a Stochastic Differential Equation (SDE), our method adjusts the drift term while preserving the stochasticity of the volatility term. This ensures that we balance both accuracy and the creativity of the diffusion model.

Additionally, our method provides an effective controllable generation method with user-specified conditions of music. The method may be used to ensure certain inappropriate notes do not appear, and may also be used to maintain a pre-specified music type preferences. The efficient control by our method also enables the model to generate music of different styles and modes, even if such modes are not contained in its training set. We present such samples in Section 1 of our demo page.

We thank the reviewer for the valuable suggestions. We have modified our maniscript and have the following comments:

1. The trade-off between harmonicity and creativity

While the out-of-key notes are never necessarily “wrong”, it is observed that generative models often fail to create an accommodating context for such “creativity”. We have uploaded examples to part 1 of the demo page (https://huggingface.co/spaces/interactive-symbolic-music/InteractiveSymbolicMusicDemo) to demonstrate the phenomenon. Sometimes the generated results without control can be completely chaotic (e.g., example 1 and 3 in section 2 of our demo page).

Further, we have retrained our model (in the previous version we only run 10 training epochs, now we increase the number of epochs to 20), and now the OA score for duration and note density have become higher than WholeSongGen. Third, the broken chords are not a result of removing out-of-key notes, but also appear in the original training data.

Moreover, we additionally remark that our method is not targeted on “deleting incorrect notes in the final sample”. Instead, our method provides an effective controllable generation method with any user-specified conditions of music. The method may be used to ensure certain inappropriate notes do not appear, and may also be used to maintain a pre-specified music type preferences. As described in our paper, our method fundamentally guides the diffusion sampling process by adjustments in middle steps. If we view the diffusion process as a Stochastic Differential Equation (SDE), our method adjusts the drift term while preserving the stochasticity of the volatility term. This ensures that we balance both accuracy and the creativity of the diffusion model.

Effectively, during the diffusion steps, our approach does not just remove incorrect notes but also actively guides the model to generate more harmonically appropriate notes. This guidance process helps the model converge to musically valid outputs while still leveraging the generative freedom of the diffusion model.

To further illustrate this, in Figure 6 of our manuscript, we provide two concrete examples demonstrating that our sampling control not only removes incorrect notes but also encourages the generation of notes that are more harmonic and contextually appropriate. Additionally, Section 2 of our demo page (https://huggingface.co/spaces/interactive-symbolic-music/InteractiveSymbolicMusicDemo) provides samples showing how even completely chaotic results (e.g., Example 1) can be effectively guided back into musically valid outcomes (rather than removing everything) through our method.

2. “In practice, we could see longer pieces, and it is non-trivial to detect key variations within a piece. Given that key variations such as modal interchange are common in pop music, this could be a limitation which barriers the practical application of the FTG method.”

Unfortunately this is due to a misunderstanding of the reviewer. First, even in the 4-bar phrases we actually apply different keys, which are associated to chords. The “key signature” here does not refer to the key signature of a whole piece, but rather the key associated with the temporary tonic, as we have explained in section 3.1.

Considering longer generations, we can actually apply the inpainting methods of e.g., WholeSongGen[1] to create longer music out of 4-bar sections, but this is not the focus of this paper. Our main focus is to propose the precise controlling method for diffusion models, which can in fact be applied to other diffusion-based symbolic music generation methods.

References

[1]Ziyu Wang, Lejun Min, and Gus Xia. Whole-song hierarchical generation of symbolic music using cascaded diffusion models.

[2]Yujia Huang, Adishree Ghatare, Yuanzhe Liu, Ziniu Hu, Qinsheng Zhang, Chandramouli S Sastry, Siddharth Gururani, Sageev Oore, and Yisong Yue. Symbolic music generation with non-differentiable rule guided diffusion

3. About ablation study

We thank the reviewer’s suggestion of adding ablation studies. We have added an ablation study section (Section 5.1.5) to our manuscript. We run two additional experiments on the same accompaniment generation task to analyze the impact of the fine-grained conditioning during training and the fine-grained control in sampling. The first experiment involves the same model trained with fine-grained conditioning but without control during sampling, while the second is an unconditional model without any conditioning or control in both the training and sampling process. We evaluate the frequency of out-of-key notes and assess the overall performance using the same quantitative metrics as before. We find that conditioning in training and control in sampling both contribute to reducing out-of-key notes and improving overall performance.

Moreover, in Figure 6 of our manuscript, we provide two examples showing that our sampling control can not only eliminate incorrect notes, but also guide the model to replace them with more harmonic ones.

4. About comparison with other (non-diffusion) models and subjective evaluation

For comparing with other (non-diffusion) models, we did not conduct such experiments because our focus in this work is to provide a control methodology based on diffusion-based symbolic generation models, not to prove that diffusion models perform better than other models, which has been discussed in many works on diffusion-based models, such as [1] and [2]

As for subjective evaluation, we appreciate the reviewer’s suggestion of adding subjective evaluations. While having a comprehensive comparison study of subjective evaluations is challenging, we plan to explore ways of adding sensible subjective evaluations or other non-quantitive alternative analysis for music.

5. Can FTG be applied to WholeSongGen?

We first clarify that FTG not only contains ad hoc sampling control, but also contains pixel-wise conditioning for chord and rhythms in the training process. Therefore, it cannot be directly applied to WholeSongGen, but can be integrated into the framework of WholeSongGen with modifications and combinations. Generally, our idea of fine-grained control in the sampling process can be applied (with modifications) to other diffusion-based models for symbolic music generation.

6. Metric of chord accuracy in latent space

Designing a direct comparison metric for chord sequences—such as determining whether "D" or "E minor" is closer to "C" on the observational chord-wise basis—is inherently complex. Therefore, we prefer the cosine similarity of the latent chord feature because the distances of latent vectors are comparable. In fact, our metric of comparing the latent chord feature has already been applied in the literature, such as [1].

References

[1] Ziyu Wang, Lejun Min, and Gus Xia. Whole-song hierarchical generation of symbolic music using cascaded diffusion models.

[2] Yujia Huang, Adishree Ghatare, Yuanzhe Liu, Ziniu Hu, Qinsheng Zhang, Chandramouli S Sastry, Siddharth Gururani, Sageev Oore, and Yisong Yue. Symbolic music generation with non-differentiable rule guided diffusion

Thank you very much for encouraging and recognizing our effort.

Regarding the question about local key signatures, we would like to clarify that they are not "detected" but rather "designed" by the user, and provided to the model in the generation process.

To explain, two chromatic elements are included in the generation processes of our model: chord and local key regularization.

• Chord: During training, chords are inferred (either through standard chord recognition algorithms or human labeling) from the music, and provided as conditions. In the generation process, the user specifies a desired chord sequence, which the model uses as a condition to generate music based on it.

• Local Key Regularization: Unlike chords, the local key is not included in the training process. Instead, it serves as an additional user-defined input during generation, designed to better regularize the model's output. By specifying a local key, the user restricts the pitch classes that the model can use in its output, effectively shaping the generated music to align with a desired tonal structure.

For example, if the user wishes to generate a measure aligning with the chord $C$, they might choose the $C$ major scale as the "key regularization" for this measure. In this case, the user restricts the model from generating pitch classes $C\sharp$, $D\sharp$, $F\sharp$, $A\flat$, and $B\flat$ within that measure. Alternatively, the user could use the Chinese pentatonic scale as the "key regularization" for the measure, in which case the model would only output the pitch classes $C$, $D$, $E$, $G$, and $A$. This flexibility allows users to customize the tonal structure of the generated music. This feature has not yet been integrated into the "user interface" of our demo page—not because it is technically challenging, but simply due to time constraints.

While the term "local key" may initially seem misleading, it was introduced in earlier discussions about the challenges in precision in symbolic music generation. Specifically, when models generate notes outside the "local key", the resulting sample quality often degrades. Later, in the context of applying the FTG method, the design of every local "key regularization" is left entirely up to the user. A common choice is to use a scale that contains the pitch classes of the chord condition.

We hope that this clarifies the role of "local key" or "local key regularization" in our work. Again, thank you for the comments and discussion.

4. About “that noise correction can consistently limit off-key notes, that fine-grained control will not disrupt local patterns, and that early stopping time can be practically achieved.”

This appears to a misunderstanding. We would like to clarify that these are not assumptions for deriving the proposition, but instead the takaways indicated by the proposition. In our proposition 2, we have theoretically proved that the sampling control does not disrupt the distribution, and it consistently limit out-of-key notes by the construction (all out-of-key notes would be removed). This is also shown in the quantitative results and the examples on our demo page. Chord similarities increase after applying the sampling control, implying that harmonic patterns are not disrupted. As for early stopping time, this is commonly used and already well-stated in literature.[3,4] In fact, all practical diffusion models are a discretization of a SDE, and thus all diffusion methods are indeed using early stopping time because in practice one cannot sample for infinitely many time steps.

5. About “the study’s experiments are limited to 2-bar piano accompaniment generation”

First, splitting the samples into 2-measure slices is only applied when calculating the metrics. In the generation process, our model is based on 4-measure slices. Even that, we can actually apply the inpainting methods of e.g., WholeSongGen[1] to create longer music out of 4-bar sections, but this is not the focus of this paper.

Reply to the concerns in “questions”

1. We have revised our theoretical part by revising the mathematical modeling, formulation and removing as much assumptions as possible. The overall key intuition implied by the mathematical analysis is to characterize the challenge of symbolic music generation due to a combination of scarcity of data, high dimensionality and requirement for pixel-wise precision.

2. Scope of 4-bars: Again, we remark that we can actually apply the inpainting methods of e.g., WholeSongGen to create longer music out of 4-bar sections, but this is not the focus of this paper. We appreciate your suggestion of adding more in-depth musical analysis and subjective evaluations. While having a comprehensive comparison study of subjective evaluations is challenging, we plan to explore ways of adding sensible subjective evaluations or other non-quantitative alternative analysis for music.

References

[1] Ziyu Wang, Lejun Min, and Gus Xia. Whole-song hierarchical generation of symbolic music using cascaded diffusion models. The Twelfth International Conference on Learning Representations,2024.

[2] Yujia Huang, Adishree Ghatare, Yuanzhe Liu, Ziniu Hu, Qinsheng Zhang, Chandramouli S Sastry, Siddharth Gururani, Sageev Oore, and Yisong Yue. Symbolic music generation with non-differentiable rule guided diffusion. arXiv preprint arXiv:2402.14285, 2024.

[3]Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438–12448, 2020.

[4]Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, 2021.

We thank the reviewer for the valuable suggestions. We have modified our manuscript and have the following comments:

Reply to the concerns in “Summary”

1. About “data scarcity as a limitation”

We appreciate the reviewer’s observation regarding the focus on data limitations in the introduction. However, we respectfully clarify that the discussion of high-quality data scarcity is directly relevant to a central topic of our paper: enhancing control within symbolic music generation. Our point is that the unique challenges posed by data scarcity in symbolic music generation are inherently tied to the need for precise control mechanisms, as explained below:

High-quality music data are inherently more scarce than some other modalities. Unlike domains such as image or text generation with far more abundant data, in the music domain, assessing and creating high-quality symbolic music requires significant expertise and effort. For instance, while anyone with a camera can create countless photographs, crafting new, high-quality music compositions demands substantial musical experience and creativity. This disparity underscores why symbolic music datasets are inherently limited in scale and quality. Thus, even if we can scale up the data, the amount of available data will always be relatively much more scarce than other modalities such as text and image.

Furthermore, we described in our introduction that symbolic music generation requires an exceptional level of precision. Unlike images or text, where minor errors might be inconspicuous or tolerable, in tonal music, even a single incorrect note can be jarring and immediately noticeable, even to listeners with minimal musical training. This strict demand for accuracy amplifies the impact of limited training data, as errors such as "wrong notes" are far more significant in symbolic music.

Empirical evidence demonstrates the limitations of scaling up data. We added Table 2 in our manuscript, showing that a model without sampling control unavoidable generates wrong notes. This aligns with the theoretical insights presented in Section 3.1, Proposition 1, where we mathematically demonstrate that with finite data, the occurrence of wrong notes is unavoidable, and the decay rate of this error probability with respect to dataset size is slow ($O(n^{-1/KL})$). Thus, relying on scaling up data alone is inefficient and impractical in symbolic music generation, given the inherent scarcity of high-quality data and the slow decay rate of errors.

In contrast, our control mechanisms offer a more effective solution. Our method addresses this issue by introducing a fine-grained control mechanism that can precisely eliminate wrong notes, as demonstrated in our experiments. By guiding the generation process, we achieve harmonic consistency without needing to rely on a significant expansion of training data, which would be both resource-intensive and inefficient for this domain.

Thus, the discussion on data scarcity is indeed central to our argument. It provides essential context for why control mechanisms are necessary and more effective in this domain compared to other domains. We hope this clarification strengthens the alignment between the discussion of data limitations and the primary contribution of our work.

2. About “using diffusion model”

We admit that there are methods other than diffusion models for symbolic music generation. However, diffusion model is our main focus because our sampling control is plugged in the diffusion denoising process. Moreover, existing literature [1,2] has discussed and demonstrated the power of diffusion models in the related domain.

3. About “approach centers on adding fine-grained controls during training”

We appreciate the comment and believe this is a misunderstanding of our contribution. We argue that our approach is not simply adding fine-grained controls during training. There are two subsections in our Section 4, and fine-grained control in training is only one subsection. The other subsection is Fine-grained control in sampling process, which is the part we put more emphasize on. Such a sampling control is different from all established control techniques because all of them are in the training process and only obtains “soft control”.

References

[1] Ziyu Wang, Lejun Min, and Gus Xia. Whole-song hierarchical generation of symbolic music using cascaded diffusion models. The Twelfth International Conference on Learning Representations,2024.

[2] Yujia Huang, Adishree Ghatare, Yuanzhe Liu, Ziniu Hu, Qinsheng Zhang, Chandramouli S Sastry, Siddharth Gururani, Sageev Oore, and Yisong Yue. Symbolic music generation with non-differentiable rule guided diffusion. arXiv preprint arXiv:2402.14285, 2024.

We thank the reviewer for the valuable suggestions. We have modified our manuscript and have the following comments:

1. Objective evaluations and experiment details:

Experiments and ablation study: We thank the reviewer’s suggestion of adding ablation studies. We have added an ablation study section (Section 5.2.5) to our manuscript. We run two additional experiments on the same accompaniment generation task to analyze the impact of the fine-grained conditioning during training and the fine-grained control in sampling. The first experiment involves the same model trained with fine-grained conditioning but without control during sampling, while the second is an unconditional model without any conditioning or control in both the training and sampling process. We evaluate the frequency of out-of-key notes and assess the overall performance using the same quantitative metrics as before. The results indicate that conditioning in training and control in sampling both contribute to reducing out-of-key notes and improving overall performance.

Moreover, in Section 5.1 of our manuscript, we provide two examples showing that our sampling control can not only eliminate incorrect notes, but also guide the model to replace them with more harmonic ones.

2. About the comparison with GETMusic

As for the comparison with GETMusic, GETMusic in fact has an input channel for conditional chord sequences, see “(6) chord” on github repo https://github.com/microsoft/muzic/tree/main/getmusic. It also has a channel for melody. Therefore, in our evaluation section, all models (ours, WholeSongGen, GETMusic) are utilizing chord and melody conditions to generate accompaniments, which ensures that they are comparable.

3. About dense mathematical notations

We are sorry about the dense mathematical notations. We have revised our manuscript to be more straightforward, shortened the mathematical part, and added more empirical studies.

4. About comparison with each other and other methods

As mentioned in comment 1, we have added an ablation study section (Section 5.2.5) to our manuscript. As for the comparison with other established methods, we have already provided the comparison with WholeSongGen[1] and GETMusic[2] in Section 5.2.4 on the accompaniment generation task.

5. About "masking and repainting"

We did not consider repaint because it requires multiple times of resampling, which does not achieve the desired sampling efficiency as in this work. Further, there is no guarantee that repaint is capable of eliminating all undesirable notes within a controllable amount of time. In contrast, our sampling guidance guarantees that undesirable notes could be avoided in one sampling process, and the controls in middle sampling steps (rather than only in the last step) could retain the flexibility and creativity of diffusion models.

References

[1] Ziyu Wang, Lejun Min, and Gus Xia. Whole-song hierarchical generation of symbolic music using cascaded diffusion models. The Twelfth International Conference on Learning Representations,2024.

[2] Ang Lv, Xu Tan, Peiling Lu, Wei Ye, Shikun Zhang, Jiang Bian, and Rui Yan. Getmusic: Generating any music tracks with a unified representation and diffusion framework. arXiv preprint arXiv:2305.10841, 2023.

Thanks for your rebuttal.

I truly appreciate your revision to the manuscript and it does improve the paper quality. After discussion, I can feel that the main issue of this paper is its readability, so that we need a round of rebuttal to clarify some necessary details.

I admit that the theoretical part of this paper provides useful insights, which is a quite interesting section and is also coherent with previous intuitions. Then the last main issue would be sufficient subjective experiments. I totally understand that there is no time for experiment during rebuttal period, but it is the golden test for this kind of task.

Also, in my review I mentioned that it would be helpful if you can compare the performance with methods that also use control methods.

I am happy to raise the score to 8, assuming you will complete these experiments.

Thank you for taking the time to address my concerns.

The revised manuscript is much improved, and the additional ablation experiments are welcome.

Concurring with other reviewers, I still have doubts about the significance of this work and its evaluations. To properly evaluate this work, it's important to add additional experiments, particularly subjective comparisons with other approaches for conditioning diffusion models. As it stands, it's quite hard to evaluate whether this approach works better than rule-based filtering, or even just manually removing masked notes as a post-processing step, within the presented evaluations. I can't rely on the provided samples to evaluate the contribution of this work alone, as I don't know to what degree the samples were cherry-picked. Additionally, the output from the unconditioned model sounds much less musical than other approaches to accompaniment generation. This raises questions about whether training on the POP909 (which is relatively small) is really appropriate if the goal is to faithfully measure the impact of guidance on musical generation quality, which appears to be what the authors are trying to demonstrate with the demo page.

As I am unable to properly evaluate the paper's contributions, and am unsure if the proposed methods have any applications beyond efficiently conditioning data-scarce piano-roll diffusion models for accompaniment generation, I recommend rejecting the paper in its current form.

I must stress to the AC that as I am not an expert in diffusion models or piano-roll generation using diffusion models, I've selected a confidence of 2.

Hello,

I really appreciate the author's eagerness and efforts to revise the manuscript and respond to feedback; however, in my opinion, this work is below ICLR standards. I have some final remarks which I hope will help improve the manuscript further.

In Section 5.1.4, we added an initial group of subjective evaluations. We presented average ratings and confidence intervals for six different aspects (details described in Appendix F). The results suggested that our method outperforms the baselines in 6 different aspects.

The results displayed in Figure 3 appear to be quite marginal. Although no discussion of statistical significance is given, it seems very likely that these results are not statistically significant. This is below the scientific standards I would expect for a conference like ICLR.

Moreover, I have serious concerns about the design of the study. If I understand correctly, the results show that the ground truth (i.e., real accompaniment) was judged equally to the generated accompaniment in the subjective evaluation? This makes me incredibly suspicious of the experimental design, given what I have heard on the demo page.

The questions in the subjective evaluation are also not ideal. What does 'richness' mean in a musical sense? From what I can tell, the participants were not given sufficient information to answer these questions properly. The fact that every category follows the same distribution should raise serious questions about the experimental design in my opinion.

In section 5.1.5, we directly compared our method with an additional baseline method about "just manually removing masked notes as a post-processing step". Our sampling control method outperformed this baseline. In Section 1 of our demo page, we have added several examples showing the ability of our model in generating music of different styles and genres, even if such genres are not contained in its training set.

If I understand correctly, the differences between the 'training and sampling control' and 'training control with edit' are very marginal. As I mentioned in my previous response, in order to properly evaluate the proposed methods in this work, there needs to be better ablation tests. Table 2 could also benefit from having a more descriptive caption.

This ensures that we balance both accuracy and the creativity of the diffusion model, leveraging the structured gradual denoising process of diffusion models to preserve distributional properties of the original learned distribution.

I completely agree with this observation. The point I was raising is that the evaluations provided don't allow the reviewer to verify this claim.

We did not consider repaint because it requires multiple times of resampling, which does not achieve the desired sampling efficiency as in this work. Further, there is no guarantee that repaint is capable of eliminating all undesirable notes within a controllable amount of time. In contrast, our sampling guidance guarantees that undesirable notes could be avoided in one sampling process, and the controls in middle sampling steps (rather than only in the last step) could retain the flexibility and creativity of diffusion models.

I find this response unconvincing. From what I can tell, there is a massive amount of literature concerning in-painting with diffusion models. The problem proposed in this work seems to very closely resemble an in-painting task, where a mask is specified over the out-of-key notes, i.e., signifying that these notes are not present and shouldn't be generated. I don't think sampling efficiency is a sufficient reason to omit this very natural comparison. From what I can see, modern in-painting techniques seem to retain the flexibility and creativity of diffusion models while forcing the diffusion process to leave some areas of the image/piano-roll unaffected.

First, we respectfully ask the reviewer not to neglect, in addition to the sampling part of contribution, the other parts of our approach, for example the fine-grained training condition.

This is certainly a contribution in the context of piano-roll generations; however, I have doubts about the broader significance and novelty.

We appreciate the reviewer’s recognition of our previous round of responses, the improvement of manuscript and the additional experiments.

In addition to the previous round of responses and the revision made then, we have made the following changes to the latest version of our manuscript:

1. In Section 5.1.4, we added an initial group of subjective evaluations. We presented average ratings and confidence intervals for six different aspects (details described in Appendix F). The results suggested that our method outperforms the baselines in 6 different aspects.
2. In section 5.1.5, we directly compared our method with an additional baseline method about “just manually removing masked notes as a post-processing step”. Our sampling control method outperformed this baseline. In Section 1 of our demo page, we have added several examples showing the ability of our model in generate music of different styles and genres, even if such genres are not contained in its training set.

We have the following comments the recent response from reviewer emmS:

“As it stands, it's quite hard to evaluate whether this approach works better than rule-based filtering, or even just manually removing masked notes as a post-processing step, within the presented evaluations.”

Response: We further hope to make the following clarifications:

1. First, we respectfully ask the reviewer not to neglect, in additional to the sampling part of contribution, the other parts of our approach, for example the fine-grained training condition.

2. As for the sampling control, in our ablation study in Section 5.1.5, we added a comparison with the method of “post-sample rule-based editing”, where we numerically demonstrate that the method of inserting control in the sampling process is essentially different from rule-based post-sample editing. We additionally remark that our method is not targeted on “deleting incorrect notes in the final sample”. Instead, our method provides an effective controllable generation method with any user-specified conditions of music. The method may be used to ensure certain inappropriate notes do not appear, and may also be used to maintain a pre-specified music type preferences. As described in our paper, our method fundamentally guides the diffusion sampling process by adjustments in middle and intermediate steps. If we view the diffusion process as a Stochastic Differential Equation (SDE), our method adjusts the drift term while preserving the stochasticity of the volatility term. This ensures that we balance both accuracy and the creativity of the diffusion model, leveraging the structured gradual denoising process of diffusion models to preserve distributional properties of the original learned distribution. In contrast, rule-based editing, different from our approach, would employ a brute-force editing approach that disrupts the generated samples, disrupting melodic lines and rhythmic patterns. Both the theoretical guarantee provided by proposition 2 and the numerical results further validate this analysis.

3. As stated in our manuscript, our method is intrinsically different from “detecting wrong notes and regeneration” in terms of sampling efficiency. Also, there is no guarantee that regeneration is capable of eliminating all undesirable notes within a controllable amount of time. The improvement of efficiency has a vital benefit for situations such as improvisation.

4. Further, far different from straightforwardly removing “wrong notes” or "detecting wrong notes and regeneration", our method also enables the model to generate music of different styles and genres, even if such genres are not contained in its training set. We present such samples in Section 1 of our current demo page. Those examples demonstrate the power and flexibility of method.

“Additionally, the output from the unconditioned model sounds much less musical than other approaches to accompaniment generation. This raises questions about whether training on the POP909 (which is relatively small) is really appropriate if the goal is to faithfully measure the impact of guidance on musical generation quality, which appears to be what the authors are trying to demonstrate with the demo page.”

Response: As a clarification, what we are trying to demonstrate on the demo page is for the purpose of responding to the question “if eliminating 'wrong' notes entirely could risk making the output less interesting”. The samples on the demo page are not the general case, but specifically cases containing out-of-key notes, to support our statement: that there can be cases where model outputs are very unreliable, which provides motivation for our method providing strong control to improve the reliability of models.

We emphasize that the outputs from the uncontrolled model serves as an ablation study. The examples on the demo page shows that, even if a sample from the uncontrolled model is that unmusical, our sampling control can still guide it to a much more musical one. Such an improvement indeed emphasizes the power of our sampling control, and shows that the sampling control is an irreplaceable part of our method.

As for the question of using POP909, we note that this dataset was the only training dataset for literature like [1] “Whole-song hierarchical generation of symbolic music using cascaded diffusion models (ICLR 2024)”, one of the baseline methods. This partly suggest the appropriateness of using this dataset. The other part is that, there are very few symbolic music datasets that have the most structured data and clear labels. This is again, rooted in the facts that high-quality symbolic music data are genuinely scarce compared to some other data modalities, which in turn, further supports our work’s motivation and contribution to improve music generation quality via external control other than increasing data.

“I can't rely on the provided samples to evaluate the contribution of this work alone, as I don't know to what degree the samples were cherry-picked.”

Response: This comment can be made to all music generation papers to question whether the samples are “cherry-picked”. We do not think it is fair to raise concerns to our work about “cherry-picking” without discrediting effectively all the generation work in this domain.

In fact, to the best of our knowledge, we are the only work in symbolic music generation that provides a user interface on the demo page where users can self-generate directly from the model.

In our previous responses to reviewer emmS, we included two parts. First, we responded with concrete additional experiment results that demonstrated positive comparative performance of our method. Second, we pointed out objectively that the reviewer’s comment “I can't rely on the provided samples to evaluate the contribution of this work alone, as I don't know to what degree the samples were cherry-picked” is not fair by the comment itself, and even more unfair given that we are the only work in symbolic music generation that provides a user interface on the demo page where users can self-generate directly from the model.

Then the reviewer emmS chose to respond to our first part -- the positive additional experiments -- with very subjective and negative comments, using words like “quite marginal”, “incredibly suspicious”. The reviewer emmS chose to completely ignore the second part that provided evidence of the reviewer being unfair. Meanwhile, the reviewer reduced the rating to 3 in response to our positive additional experiments, which points out the fact that the reviewer is unfair.

The above facts and evidence suggest that the reviewer emmS is unfair and biased. It is meaningless for a fair audience to engage with reviewer emmS, before reviewer emmS honestly admits to the biased and unfair behavior in front of the evidence.

Hello Authors.

I'm very sorry if my comments were offensive or you found them to be unfair. That was not my intention. To add clarification based on the author's comments, my concerns about cherry-picking have been addressed. The inclusion of the user interface on the demo page is also certainly a good addition.

I lowered my score because of my concerns about the evaluation. I would recommend to the AC to take my review into context, given the low confidence score.

We thank the reviewer for the suggestions. We have made the following modifications to our manuscript:

1. We have added a section 5.1.5 of ablation studies. We run 3 additional experiments on the same accompaniment generation task to analyze the impact of the fine-grained conditioning during training and the fine-grained control in sampling. Tur first experiment involves the same model trained with fine-grained conditioning but only removes the out-of-key notes after the last sampling step; the second also incorporates fine-grained conditioning for training but without any control during sampling; the third is an unconditional model without any conditioning or control in both the training and sampling process. All experiments use the same model architecture and random seeds as the one with full control for comparability. We evaluate the frequency of out-of-key notes and assess the overall performance using the same quantitative metrics as before. We find that conditioning in training and control in sampling both contribute to reducing out-of-key notes and improving overall performance. Moreover, in Figure 6 of our manuscript, we provide two examples showing that our sampling control can not only eliminate incorrect notes, but also guide the model to replace them with more harmonic ones.

2. We refined the theoretical part to be significantly shorter, and added a conclusion section to summarize our work as well as discuss the limitations of our method.

3. For other fields of ML, in our future work we plan to analyze a more general framework discussing the joint effect of conditioning and sampling guidance in diffusion models. We will discuss the application in more domains such as AI safety.