Presto! Distilling Steps and Layers for Accelerating Music Generation

We thank the reviewer for their review, and sincerely appreciate their detailed comments. Below, we address overall comments and concerns brought up in the review:

”The explanation of Presto-L seems to assume prior knowledge of ASE…”

Regarding the noted weakness of assuming prior knowledge of ASE, we acknowledge and agree that our readability and self-containment can be improved. To address this, we have included pseudo-code for the Presto-L algorithm and a longer form description in Appendix A.6.

”The contribution to the research community in terms of reproducibility is limited…I strongly recommend either releasing the codebase or providing sufficient implementation details“

We acknowledge this shortcoming and agree that releasing code (model checkpoints and data) would greatly help reproducibility as well as alleviate issues w.r.t. details and clarity. However, we do not plan to do so due to constraints outside of our control. Thus, to ameliorate this important issue and improve reproducibility of our work, we have followed your suggestion and included considerable additions in the paper as we note in our general rebuttal. To expand more here, we have

1. Added pseudo-code and long-form explanation for our layer distillation method, Presto-L (Appendix A.6)
2. Added detailed torch-style pseudo-code walkthrough for Presto-S (Appendix A.4) to complement our existing pseudo-code (Appendix A.3 Algorithm 1)
3. Added a new tutorial figure that expands upon figure 1 to provide a careful and in-depth system overview to better describe the high-level method design (Appendix A.5). Please also see our main rebuttal.

”Are there plans to conduct text-to-audio generation experiments during the rebuttal period?”

We appreciate the reviewer catching this error on our part, and have since removed the mention of TTA, as we only focus on the subset task of TTM and do not plan to run general TTA experiments during this period.

”How exactly is the CFG-augmented real score, represented by $\mu_{\text{real}}^w$ in Eq (5), formulated?”

The reviewer is exactly correct, as $\mu_{\text{real}}^w$ is calculated as:

$\tilde{\mu}^w_{\theta}(x, \sigma, e) = \mu_{\theta }(x, \sigma,\emptyset) + w (\mu_{\theta}(x, \sigma, e) - \mu_{\theta }(x, \sigma, \emptyset))$

Where $e$ is the condition embedding. This is noted as an inline equation on line 147.

”how robust is this to different model training setups, architectures, and datasets? For example, if the number “5” were changed to “4” or “6,” how significantly would it affect training stability and generation quality in those sense?”

Though we followed the recipe of DMD2 with 5 fake score / discriminator updates per generator update, initial experiments showed that varying the update rate in the [4, 6] range did not impact performance much. When we reduce the update rate $\le3$, the performance noticeably worsened, and in particular led to poor estimation of the fake score distribution. We have since added clarifying information in Appendix A.1 to discuss this fact.

”which specific part of the DiT block do the “intermediate feature activations” refer to? Also, how is the architecture of the discriminator head $D_\psi$ actually configured?”

We recognize the reviewer's concern, and have since updated the appendix to highlight more details on the Discriminator design. Specifically, our discriminator design follows the design of the original DMD2 work, where our discriminator is a series of 4 convolution blocks interleaved with groupnorm and SiLU activations, with a final linear layer to collapse to a single channel, using the following modifications:

• As our data representation are 1D sequences, all 2D convolutions are replaced with 1D convolutions
• Since we use the LSGAN formulation, we do not downsample our representation all the way down to a single true / false value. Instead, the output of the discriminator is a heavily downsampled version of the input representation, specifically at around 2.8Hz.
• Since our model is DiT based rather than a UNet as in DMD2, the “intermediate feature activations” refer to the outputs of the 14th DiT block of the model (i.e. the halfway point of our 28 block DiT), as DiTs lack a real “bottleneck” layer (that UNets have). We have thus added this information into Appendix A.1 to actively highlight this.

”What values do the authors set as $\nu_1, \nu_2$ ? Are both $g_1, g_2$ optimized with Adam at a learning rate of 1e-4? Also, do the authors use learning rate schedulers?”

We thank the reviewer for catching this lack of information on our part, and have thus modified Appendix A.1 to provide full details. Specifically, for both $g_1, g_2$ we use Adam with a learning rate of 5e-7. We set $\nu_1=0.01$ and $\nu_2=0.005$. Following DMD2, we do not use any learning rate schedulers.

I sincerely appreciate the time and effort you put into thoroughly addressing my concerns. I now feel that all of them have been addressed very effectively. Thank you for presenting such an interesting paper—it was truly a pleasure to review your work.

We thank the reviewer for their detailed review, and are glad to see the recommendation of acceptance. Beyond our main rebuttal comments, please see our response to your individual comments below.

”Although the paper is well-written, some readers would struggle to reproduce the results without their code public. (I understand some institutes/companies do not make their code publicly available due to their policy.)”

We acknowledge this shortcoming and agree that releasing code (model checkpoints and data) would greatly help reproducibility as well as alleviate issues w.r.t. details and clarity. Unfortunately, however, we do not plan to do so due to constraints outside of our control. To ameliorate the important issue and improve reproducibility of our work, we have gone to great lengths to address as we note in our general rebuttal. To expand more here, we have

1. Added pseudo-code for our layer distillation method, Presto-L
2. Added a detailed torch-style pseudo-code walkthrough for Presto-S to complement our existing pseudo-code (Appendix A.3 Algorithm 1)
3. Added a new walkthrough figure that expands upon figure 1 to provide a careful and in-depth system overview to better describe the high-level method design. Please also see our main rebuttal.

”Since the proposed training framework is general, I thought I would like to see experimental results in the text-to-audio generation task evaluated on AudioCaps. Furthermore, if we have a DiT-based baseline model for text-to-image or text-to-video generation, we can apply the proposed method to them.”

Thank you for the important comment and desire to apply our proposed method on other domains such as general-purpose audio generation, image generation, and video generation. While we would be very motivated to include such work, we believe expanding the scope to this level would be relatively difficult to contain within our manuscript here and still manage to focus on our goal of music generation.

”The explanation of baselines in Section 4.1 is a little unclear to me. My understanding is that the authors trained the teacher model and applied the proposed technique and previous ones... For MusicGen and Stable Audio Open, they just downloaded and evaluated the official distributed models.”

We recognize the reviewer's concern, and the reviewer is exactly right in their idea of what Section 4.1 implies, with the acceleration algorithms being used on our base model and the external baselines just being used as is from the official repositories. To make this fact more clear, we have modified Section 4.1 slightly in order to highlight this difference and elaborated much more in Appendix A.2 to highlight this fact explicitly.

We thank the reviewer for their insightful review. Below we address comments brought up in the review:

”I am interested in how well the framework could be when applied to other VAEs or generator backbones, especially if no resource is open. Is it possible to open the implementation without sharing pretrained model weights? Or open a part of key tricks or components in the work?”

We fully recognize the reviewer’s concern about the complexity and how it relates to open source code. To address this issue, please kindly see our main rebuttal. To expand here – we have taken several steps to improve the reproducibility and understanding of our work including adding pseudo-code for Presto-L as well as a more detailed torch-style pseudo code walkthrough for Presto-S to complement our existing algorithm. We would also like to note the original DMD2 paper [Yin 2024] does have open source code, so although the algorithm is different and applied to discrete-time diffusion models and not continuous-time models, it is closely related and can be used together with our torch-style pseudo code. It is our best intention that this will aid researchers in reproducing our method.

Regarding other VAEs or generator backbones, in initial experiments we tried our method on variants of the final VAE and generator setup (such as changes in the compression rate of the VAE or the size of the DiT hidden state) and found similar results.

We thank the reviewer for the detailed review and are grateful for your critical feedback. Below, we address overall comments and concerns:

”Figure 1 is dense and, despite a detailed caption, remains challenging to interpret. The figure also uses various notations that lack prior explanation, making it difficult for readers to follow the intended process.”

We recognize the density of figure 1, and have sought to address your feedback by improving readability. Specifically, we have added a further detailed diagram in Appendix A.5 (now Figures 8 and 9) that goes through the Presto-S process step-by-step. We also introduce multiple additional pseudo-code algorithms to alleviate any issues with respect to detail.

”Additionally, the models are trained on internally licensed data, so the results cannot be easily referenced in future work. With no indication that the code will be released, reproducing this complex setup—including multiple models, distributions, and training phases—would be exceptionally difficult.”

Thank you for your concern and comments and acknowledge the difficulty in reproducing work, particularly when trained on licensed data to do our best to avoid copyright issues. To help address this, we have added multiple additional algorithm text blocks including a torch-style pseudo-code walkthrough for Presto-S to complement our existing pseudo-code and pseudo-code for Presto-L with corresponding textual explanation. For more discussion on the topic, please also see our main rebuttal as well as our ethics and reproducibility statement where we outlined our constraints w.r.t. open source code.

”In line 35, does 5-20 seconds latency refer to 32 seconds audio? If yes, please clarify because it's mentioned in the abstract but not in the introduction.”

We thank the reviewer for catching this, and have since modified the introduction accordingly.