MusicAOG: an Energy-Based Model for Learning and Sampling a Hierarchical Representation of Symbolic Music

It is a challenging task to put everything in one research paper including introduction of a formal approach, learning method, inference method and experiments. The approach introduced in this paper is intriguing, but still lacks some crucial material for readers to capture the main idea and the details.

1. The workflow of this study could be clearer? Is the goal of this study music analysis or music generation? Is the generation conditioned on the learned analysis? What music knowledge is assumed and what is not in analysis and generation? What are the major differences compared to existing generation methods, PCFG approach and GTTM?
2. The images in this paper could be clearer. Figure 1 is crucial for readers to understand the defined grammar. However, the defined edge types are not easily recognizable and lacks sufficient explanation about how is the graph achieved. It is even necessary to try to show the graph at multiple levels separately using multiple figures. Figure 2 is not readable.
3. Section 1 and 2 points the drawbacks of existing works. However, the highlight of this work and the workflow of the proposed method is missing.
4. From section 3 to 6, what are the intuition of the proposed method? Try to use intuitive ways to briefly introduce your results and shows the methods at certain level of abstraction. All details can be systematically summarized in the appendix.
5. For section 7, it is not effective to directly compare the methods with deep generative models. The proposed method is almost rule-based (with learnable parameter) and other methods are data-driven. With only three music samples, it is not convincing to say if the model can be generalized to the scenario where there are more data samples and by various composers in different styles. The attributes in Table 1 are not well-introduced and the scores might not be significant. It would be better if the study first present some generation result for analysis and show the learned concept or pattern. Then, evaluate the generative results objectively or subjectively on different learning settings to show generation quality and generalizability on different datasets.

• The paper highlights adopting Metropolis-Hastings sampling to enhance fine-grained control of music generation. However, the reviewer questions the extent of this control. The only controllable parameter, $T$, which encourages more varying and creative results, seems insufficient for fine-grained control, and the concrete way of variation seems not controllable.

• MusicAOG is a probabilistic graph model that can learn from a few annotated pieces and then leverage the learned probabilities to sample a new piece. Considering that different styles/genres of music may underlie varied probability distributions, it makes sense to train the MusicAOG model on a very specific style (like Bach chorales in this paper) and fit that style only. Yet, this may also imply a limited potential for a diverse generative capacity. Considering that Bach chorale is a relatively straightforward form of music, it is not surprising that MusicAOG outperforms those baselines with deeper architectures. On the other hand, Music AOG cannot be trained over large and diverse music corpus in a self- or unsupervised manner as those baselines can do. Hence it is questionable if it can cater to more diversified genres and more complex music forms.

• My biggest concern is the lack of generalizability of the present method. As the authors note that extensive human labeling is needed to create the MusicAOG for even one score, it is hard to imagine how usable the present method would be in large scale deployable settings.
• The evaluation section is relatively weak, as the focus on only a handful of example pieces calls into question how extrapolatable the authors' claims are. I think an overabundance of the paper is dedicated to explaining the framework for MusicAOG, which while is useful, could be traded for a more thorough and extensive evaluation.
• The clarity of the overall paper could be improved, as it is relatively hard to follow along the overall flow during the methods section.

• It appears to have fewer references than a typical ICLR empirical paper. For example, I think that there are various studies that should be included as references for GTTM, such as some of parsers (e.g. by Hamanaka) and some of applications, not only the original book. It may not be directly related to the discussion in this paper, but it would be easier to understand the research background if attention is paid to such peripheral discussions.
• The size of experiment is limited. Only three pieces by Bach were examined. I think the number could be increased more. The use of works by more diverse composers other than Bach and Schumann would also make this paper more compelling. Other polyphonic works including some of major composers like Mozart, Beethoven, Bruckner, Schoenberg, etc. and, many polyphonic choral works from the Renaissance periods, and some of academic composers in modern era. There are probably many composers worth considering.
• Although they say it can be used for music from other cultures, the only specific music shown is classical Chinese music, which is still unsatisfactory in quantity.
• It is not clear enough what is essentially different from the existing symbolic music models.