Structured Multi-Track Accompaniment Arrangement via Style Prior Modelling

Thank you for your review and constructive feedback! We acknowledge that some terms in our paper may not be sufficiently clear (W1). We also recognize our shortcomings in providing adequate motivation to justify our design choices (W3). Please allow us to first respond to the relevant points raised in W2 and Questions. We hope this could address your concerns. We will then comprehensively incorporate these clarified ideas into our manuscript to provide sufficiently clear introduction and well-justified motivation.

W2. Significance of Contribution

Our proposed style prior modelling is not built upon AccoMontage. It is an original methodology that effectively addresses long-term, structured conditional sequence generation. AccoMontage addresses piano arrangement only, while our method is more scalable to solve more challenging practical problems. Building upon style prior modelling, we introduce the first whole-song, multi-track accompaniment arrangement system, which supports variable music length ($T$) and controllable instrumental tracks ($K$).

Q1. Why concatenating style factors may contradict the structure of existing conditions

A: The general problem that we study is conditional sequence generation. The input is a conditional sequence $\mathbf{c}\_{1:T}$ and the output is an observational sequence $\mathbf{o}\_{1:T}$. We assume there underlies a style factor sequence $\mathbf{s}\_{1:T}$, which can render/arrange the content of $\mathbf{c}\_{1:T}$ to realize $\mathbf{o}\_{1:T}$. By style prior modelling of $\mathbf{s}\_{1:T}$, we aim to improve interpretability, controllability, and performance. We note that the existing condition $\mathbf{c}\_{1:T}$ often implies a long-term structure. The style factor $\mathbf{s}\_{1:T}$, if not structurally aligned with $\mathbf{c}\_{1:T}$, will lead to incoherent results.

Q2. Can diffusion models address this?

A: Diffusion models and Transformers can be distinct implementations of the same methodology. The context dependency of diffusion models differs from that of Transformers and we will explore this in our future research.

Q3. Relation to accompaniment arrangement

A: Accompaniment arrangement is a typical task of long-term conditional sequence generation. The input is a lead sheet that implies a verse-chorus structure of the whole song. The output is the accompaniment, and the style factor regards the sequential form of the accompaniment. In this paper, we consider multi-track, whole-song arrangement, where it is challenging to maintain both track cohesion and structural coherence.

Q4. Prompt of the formulation in lines 32-36

A: Continuing from Q1 & Q3, our general idea is to model $\mathbf{s}$ conditional on $\mathbf{c}$. We thus come to the formulation of $p(\mathbf{s}\_{1:T}^{1:K} | \mathbf{c}\_{1:T})$. Superscript $k=1, 2, \cdots K$ denotes track indices of multi-track arrangement. Subscript $t = 1, 2, \cdots, T$ denotes component segments of the whole song. We consider each segment as a 2-bar music snippet, which is a proper scale for learning content/style factor representations.

Q5. Piano reduction vs 'piano track'

A: We use piano reduction to denote the overall content from a multi-track piece. We will unify all instances of piano reduction into italic text. On the other hand, a ‘piano track’ simply refers to a general track played by piano.

Q6. Rationale for VQ-VAE and VAE

A: We use VAE to learn the nuanced content from the piano reduction because existing works have shown VAE’s effectiveness for music representation learning. We further choose VQ-VAE for the orchestral function because common patterns of orchestral tracks (like syncopation, arpeggio, etc.) can naturally be categorized as discrete variables. Moreover, VQ-VAE learns a discrete latent space that facilitates a prior model to mount on.

Q7. Motivations for Multi-Track Orchestral Function Prior

A: By constructing Multi-Track Orchestral Function Prior, we recover the underlying style factor sequence that can further render/arrange the input content into multi-track accompaniment. This design is more interpretable and controllable while also adhering to music composition practice. Experiments also demonstrate superior performance against non-prior baselines.

Q8. Model inputs in Figure 4

A: In Figure 4, the model receives two inputs: a lead sheet shown by the “Mel” staff, and a set of instruments (user control) shown by the rest staff labels. The output is the accompaniment in the rest staves. The length of lead sheet determines $T$ and the number of instruments determines $K$.

Q9. ABC notation

A: ABC is essentially a score notation for lead sheet, which can be converted into MIDI and the representation in Section 3.1.

Q10. Computational Efficiency of GETMusic

A: GETMusic is a diffusion model with only 100 diffusion steps, thus achieving notable computational efficiency.

Q11. How the style factors are controlled and tested

A: The control over style factors is covered in lines 173-176. Firstly, user can customize the instruments to steer the Multi-Track Orchestral Function Prior. Moreover, a starting prompt can optionally be provided. In our experiment, these control choices are randomly sampled from Slakh2100 test set. The effectiveness of $\mathbf{s}\_{1:T}$ is manifested from the quality of the resulting $\mathbf{o}_{1:T}$. Experiments show that our method achieves the top chord, structure, and DOA scores for long-term arrangement, which demonstrates the effectiveness of style factors and the superiority of style prior modelling.

Thank you for your review and constructive feedback! Please allow us to first respond to the points raised in Limitations and Questions. We hope this could address your concerns. We will then comprehensively incorporate these clarified ideas into our manuscript to provide sufficiently clear organization (W1) and well-justified motivation (W2).

L1. Elaboration on the model architecture

A: In this paper, we introduce a two-stage system to address the challenging task of multi-track accompaniment arrangement. Stage 1 arranges lead sheet into piano, and Stage 2 arranges piano into multi-track full sheet. Fig. 3 in the manuscript demonstrates the two-stage pipeline, which sequentially arranges piano/full sheets using respective style factors. The technical method of this paper mainly focuses on Stage 2, where we use the autoencoder (Fig. 1) to disentangle piano reduction $\mathrm{pn}[\mathbf{x}]$ (content factor) and orchestral function $\mathrm{fn}[\mathbf{x}]$ (style factor) from full sheet $\mathbf{x}$. In Stage 2, $\mathrm{pn}$ is given as condition and we propose the prior model (Fig. 2) to infer $\mathrm{fn}$.

L2. User control on instrument designation

A: Our system does rely on user designation of instrumental tracks, and we see it as an intuitive and handy control. In practice, users can try out preset instrument ensembles (e.g., pop band with piano, guitars, and bass) without added burden. In our experiment, without loss of generality, this control choice is randomly sampled from Slakh2100 test set (mentioned in line 222).

Q1. Input/output of autoencoder and training detail

A: The autoencoder takes two inputs: the piano reduction $\mathrm{pn}[\mathbf{x}]$ and the orchestral function $\mathrm{fn}[\mathbf{x}]$. The output is the full sheet $\mathbf{x}$. Note that both $\mathrm{pn}[\mathbf{x}]$ and $\mathrm{fn}[\mathbf{x}]$ are deterministic transforms from $\mathbf{x}$. This is an inductive bias for content/style disentanglement. The ultimate input is $\mathbf{x}$ and the training is based on a self-supervised reconstruction objective. We can see similar autoencoder designs in other disentanglement works [1] as well.

Q2. How the model generalizes to the test-time scenario

A: At test time, the autoencoder takes the piano arrangement from Stage 1 as its first input.

Q3. The rationale behind segment scale

A: Yes. We consider 1 segment = 8 beats. This is a proper scale (i.e., neither too short nor too long) to capture composition structures. Existing studies on music representation learning have also applied the 8-beat scale in their work [2,3].

Q4. The rationale for Gaussian noise against discrete ones

A: We note that the piano reduction encoder learns continuous content representations instead of discrete ones (vector quantization is not applied here). Music content can be nuanced and thus better described with continuous variables. We hence see Gaussian noise as a natural choice for the domain shift regularization.

Q5. How the $\mathbf{s}$ embeddings are interpretable

A: The $\mathbf{s}$ embeddings are encoded from orchestral function $\mathrm{fn}[\mathbf{x}]$, which essentially describes the form, or layout, of multi-track music $\mathbf{x}$. It contains rhythmic intensity information, telling the model where to put more notes and where to keep silent. When learning $\mathbf{s}$ embedding from $\mathrm{fn}[\mathbf{x}]$, we apply vector quantization because common rhythmic patterns (like syncopation, arpeggio, etc.) can naturally be categorized as discrete variables. Moreover, VQ-VAE learns a discrete latent space that facilitates a prior model to mount on.

[1] Z. Wang, et al. Audio-to-symbolic arrangement via cross-modal music representation learning. ICASSP 2022.

[2] A. Robert, et al. A hierarchical latent vector model for learning long-term structure in music. ICML 2018

[3] R. Yang, et al. Deep music analogy via latent representation disentanglement. ISMIR 2019.

Thank you for your review and thoughtful feedback! We hope the following will address your concerns:

Weakness: Discussion on the impact of the piano reduction quality

A: We introduce piano reduction as an intermediate representation from the input lead sheet to the final orchestra arrangement. Intuitively, piano reduction is a hierarchical and more abstract planning of the final orchestra. The orchestrator at Stage 2 may add notes, but it still falls within the scope implied by the piano reduction.

To formally investigate the impact of piano reduction, we conduct an ablation study by replacing the original piano arranger with the Whole-Song-Gen model [1], which, to our knowledge, is the only existing alternative that can handle a whole-song structure. The ablation study is conducted in the same setting as Section 5.3. We report objective evaluation results regarding the final orchestra’s chord accuracy, structure awareness, and degree of arrangement (DOA) as follows:

We can observe that Whole-Song-Gen at Stage 1 generally deteriorates the quality of the final orchestra. To see why this happens, we further compare Whole-Song-Gen with our original piano arranger exclusively on the piano arrangement stage. We report objective evaluation results regarding the piano’s chord accuracy and structure awareness as follows:

By comparing the two tables, we can see that a higher-quality piano reduction generally encourages a more musical and creative final orchestra result. Particularly, piano reduction lays the groundwork for (at least) chord progression and phrase structure, both of which are important for capturing the long-term structure in whole-song arrangement.

Overall, we confirm that the piano reduction is an abstract planning of the final orchestra result. Thus its quality is positively correlated to the orchestra quality. Meanwhile, we see that our current piano arranger significantly outperforms existing alternatives and guarantees decent piano quality, thus being the best choice for our model. Additionally, both the piano arranger at Stage 1 and the orchestrator at Stage 2 can be applied as independent modules to address respective subtasks.

[1] Z. Wang, et al. Whole-song hierarchical generation of symbolic music using cascaded diffusion models. ICLR 2024.

Q1. What information is encoded in $\mathbf{s}$

A: The $\mathbf{s}$ embeddings are encoded from orchestral function $\mathrm{fn}[\mathbf{x}]$, which essentially describes the form, or layout, of orchestra sheet $\mathbf{x}$. It contains rhythmic intensity information, telling the model where to put more notes and where to keep silent.

Q2. What kind of correlation to be expected along with $\mathbf{s}\_{1:T}$

A: $\mathbf{s}\_{1:T}$ encodes the sequential form of orchestra sheet. As $t$ goes from $1$ to $T$, with the development of music, we may see new rhythm patterns being introduced and new instrumental tracks being activated (e.g., in Figure 4 of the manuscript, the piano track is activated in the second half of the piece to mark increased atmosphere). More importantly, the lead sheet (and piano reduction) $\mathbf{c}\_{1:T}$ implies a verse-chorus structure of the whole song. Our prior model infers $\mathbf{s}\_{1:T}$ conditional on $\mathbf{c}\_{1:T}$, thus guaranteeing the structural alignment between $\mathbf{s}\_{1:T}$ and $\mathbf{c}\_{1:T}$.

Thank you for your review and constructive feedback! We value your thorough and insightful comments and will revise our manuscript based on these suggestions to improve the presentation. We also hope the following could address your existing concerns:

Weakness 1. Clarification of music concepts

A: Yes, we see the importance of clarifying music-related terms to make our paper more comprehensible. Certain terms (like 'groove') may have slightly varied definitions in music and AI research. We will use them in the research context and make the best effort to explain them well. We provide an explanation to the raised points as follows:

(Count each note onset position as 1) - In this paper, we represent MIDI tracks using the modified pianoroll representation [1]. Instead of spreading out the full duration of a note, each non-zero $\mathbf{x}_t^k =1, 2, \cdots, 32$ denotes the duration of a note at timestep $t$ and track $k$. Hence the indicator function $\mathbf{x}_t^k>0$ recovers the note onset positions.

(Orchestral function, rhythm, and groove) - Our intuition with orchestral function is inspired by the "grooving pattern" introduced in [2], the essence of which is to use note onset densities (in our case, intensities) to describe rhythms. Groove admittedly has a subjective nature, but it is generally associated with the rhythm. In our paper, we mention "groove" in the same context as [2] and we will clarify this in our future revision.

[1] Z. Wang, et al. Learning interpretable representation for controllable polyphonic music generation. ISMIR 2020.

[2] S.-L. Wu and Y.-H. Yang. The jazz transformer on the front line: Exploring the shortcomings of ai-composed music through quantitative measures. ISMIR 2020.

Weakness 2. Clarification with other terms

A: Yes, we see the benefits of interpreting the essence of specific terms that may otherwise be too abstract to comprehend. Your interpretation of the ILS metric is well aligned with our intention. We will follow this example to elaborate on the other terms.

Weakness 3. Comparability to baseline

A: We will clarify our model’s comparability to each baseline in Section 5.2.

Weakness 4. (Q1.) Interleaving Layers

We will improve the presentation of layer interleaving in our future revision. Regarding model dimensions, we have tensors $(B, K, T, D)$, each being batch size, track number, time frames (downsampled), and feature dimension. The inter-track encoder operates on tensor $(B \times T, K, D)$, and the autoregressive decoder on $(B \times K, T, D)$. We note that the inter-track encoder is not autoregressive, but a standard residual Transformer Encoder layer. This gives the user full control of initializing the number ($K$) and instruments of tracks.

We will supplement the missing details in the VQ-VAE framework and revise the ambiguous terms. Thank you again for your constructive advice on the paper presentation!