Thank you for the constructive feedback! Below, we address all comments and outline planned improvements.

---

1. Presentation Improvement

We agree the current version can be clearer and will revise structure, improve diagrams, and expand explanation in the final version.

If you have specific instructions on the organization, please let us know!

Relation between AudioNTP and AudioMNTP. Why show both AudioNTP and AudioMNTP? Are they both necessary? What is the final pipeline?

AudioNTP introduces continuous-valued tokens, and AudioMNTP extends it with MNTP training. Both are our novel contributions, so introducing both is necessary. Table 1 shows both are key to achieving SOTA audio generation with LMs. AudioMNTP—integrating both—is our final proposal. For clarity, we illustrate the full AudioMNTP pipeline below, covering training and inference.

Training

Please see Figure A for the training pipeline.

Waveforms are encoded into latents, masking is applied, and remaining tokens form a new sequence. A Transformer decoder performs next-token prediction on this masked sequence, guided by target positional embeddings. The LM outputs ($z$) go into a small MLP diffusion head, trained with diffusion loss to predict the next unmasked token.

See our response to Reviewer v7YS for more on target positional embeddings.

Inference

See Figure B and Figure C for step-by-step inference.

Given the BOS token and text embedding, the model generates latent token $z^0$ using one Transformer pass. Conditioning on $z^0$, the audio token $x^0$ is sampled from noise via diffusion. Then $x^0$ is used to generate $x^1$ autoregressively. After generating a chunk of tokens, they are decoded into waveform using a VAE decoder and HiFi-GAN vocoder.

Are all the tokens general noise in the beginning?

Yes — each token is initialized with Gaussian noise and refined via the MLP diffusion head.

Why use MLP for diffusion?

A simple MLP keeps latency low and enables real-time generation, which current TTA diffusion models don’t support. If the diffusion head is too slow, denoising becomes a bottleneck. See our response to Reviewer v7YS for latency discussion.

Does the model mainly replace LDMs with a Transformer?

No. Our Transformer runs once per token to produce a conditioning input for the small MLP, which does iterative diffusion. LDMs run the full model at each diffusion step, causing high latency.

2. Does not compare to Diffusion-Based Audio LM Baselines

To our knowledge, this is the first use of continuous-valued tokens in LMs for TTA. If we missed related work, we’d appreciate pointers. The most relevant work cascade a discrete audio LM and a diffusion model, not a single diffusion-based audio LM. Also, it's not open-sourced.

We propose AudioNTP (the first diffusion-based Audio LM) and improve it with AudioMNTP. Table 1 compares our methods to LDMs and discrete LMs, supporting two key claims:

1. Continuous audio LMs outperform discrete ones.
2. MNTP improves training over next-token prediction.

3. Demos

Thanks for the suggestion! Please see our demo page. Our method outperforms AudioGen Large and AudioLDM2-Full-Large and matches Tango 2. Due to internal download limits, we can’t include more samples for the ablation. But Tables 3 & 4 support our claims.

4. Why not scale up the datasets?

Our method already achieves SOTA on standard datasets. We are scaling up and will include results in the final version if accepted (see reply to Reviewer 8Ca7). Thanks for the suggestion!

5. Why split into speech vs. non-speech?

Speech is harder to generate and not a common case in TTA. Most TTA systems — including ours — can’t generate intelligible speech. Since ~50% of AudioCaps involves speech, it can obscure performance on sound. So, we split evaluation into pure sound and sound + speech (see Appendix G). We agree current naming can be confusing and will revise them. The rationale will also be added to the main text.

6. Missing experimental details?

Most details are in Appendix F and G due to page limits. Let us know if anything should be moved to the main text or is missing.

7. AudioBox and Re-AudioLDM

These models are not public, so direct comparison is difficult. We will include discussion in the final version.

8. Inference time

Latency details will be added in the final version. Please see our response to Reviewer v7YS for results and discussion.

---

We appreciate your thoughtful feedback! Your comments helped us improve clarity and depth. If you feel your concerns are addressed, we’d be grateful if you could consider raising your score. Let us know if anything can be further improved. Thank you!

1. Inference Latency Comparison with Diffusion Models

We appreciate the reviewer’s comment and will include a latency discussion in the final version. We present the latency comparison in Table A.

Table A. Latency comparison of TTA models. Measured with batch size = 1 on a single NVIDIA A100 using 10-second clips. RTF = inference time / 10 sec. Bold: best; Italic: second best. AudioGen Base has no reported latency.

AudioMNTP v.s. Diffusion Models

AudioMNTP Base and Large achieve much lower latency and better quality than diffusion models. This is due to using a small MLP for diffusion steps, instead of repeatedly applying the full model. Over 90% of the parameters run only once per token.

The token sequence is 256 long—much shorter than the typical 1000+ denoising steps used in diffusion models.

AudioMNTP v.s. AudioGen

Compared to AudioGen Large, AudioMNTP gives much better quality with only a small latency increase. Both use a single-pass Transformer, but AudioMNTP replaces the single-pass categorical head with a multi-pass diffusion MLP. The added diffusion steps greatly boost generation quality with a slightly higher latency.

Real-Time Applications

Only AudioGen and AudioMNTP reach near real-time generation (RTF ≈ 1). On an H100 GPU, real-time is easily achievable. Diffusion models remain too slow for such use cases.

2. Discussion on MNTP Schedule

Masking v.s. Dropping

Table 3 (rows C–E) shows similar results for zero masking, Gaussian masking, and dropping:

We chose dropping due to its lower compute cost, which improves efficiency and scalability.

Max Mask Ratio

We sample mask ratios from $0, 1$ . When the ratio hits 1, we retain one random token and use the BOS token and positional embedding to predict it. We’ll clarify this—thank you for the observation.

Positional Encoding

We use absolute positional embeddings, as both training and inference operate on 10s clips. Extending this to longer sequences using RoPE is a promising direction for future work.

Can you give more explanation about the design of target positional embedding?

Certainly! Traditional prediction always targets the immediate next token. MNTP, in contrast, predicts tokens at variable distances. For instance, if the sequence is 0, 3, 6, 9, token 3 predicts 6, not 4. Since the offset varies, the model can't guess the target from context alone.

To help, we add a target positional embedding that tells the model which position to predict.

In Figure 3, each token predicts the next unmasked token at a variable offset. The embedding $p_t$ encodes this offset (e.g., $x^0 + p_t^3$ → $x^3$, $x^3 + p_t^5$ → $x^5$). We use absolute embeddings for $p_t$.

Thanks again—we agree the paper’s explanation was brief and will expand it in the revision.

3. Demo

We didn’t include the demo in the original submission. It’s now live at https://audiomntp.github.io/. Please have a look!

---

We truly appreciate your feedback. It helped improve the paper in concrete ways. If you feel your concerns are resolved, we’d be grateful if you could consider a higher score. Let us know if there's anything else we can improve. Thank you again!

We are sincerely grateful for your kind words and thoughful comments. Regarding our weaknesses, we address them in the following:

1. Larger scale in both model size and dataset

We fully agree that exploring the scaling behavior of our proposed methods would provide valuable insights. In this current submission, our primary focus is on introducing continuous-valued tokens and the MNTP method. With smaller-scale models and datasets, we achieve SOTA performance, justifying the effectiveness of our proposal. Nonetheless, we acknowledge the importance of examining whether these methods maintain their effectiveness at larger scales.

We have preliminary evidence of scaling effectiveness, where increasing model parameters from Base to Large by approximately 140% led to notable improvements in overall performance metrics. This positive trend is noteworthy, especially given that such improvements are not universally observed, as exemplified by the comparative performance of AudioLDM-full vs. AudioLDM-full Large and Magnet small vs. Magnet large.

We agree that scaling analyses are broadly relevant in the context of large language models. We would be eager to further investigate scalability; however, the substantial increase in convergence time and resource requirements associated with training larger models and datasets means we cannot provide these results during the rebuttal period. Should our submission be accepted, we commit to including detailed scalability analyses in the final manuscript.

Note. As a practical consideration, training the Large model on AudioSet, similar to the approach adopted by AudioGen, requires approximately one month of computational resources utilizing around 100 V100 GPUs. Even scaling down to the Base model demands around two weeks of training. These estimates do not account for the additional resources necessary to explore even larger model configurations.

2. Generalizability of MNTP to discrete tokens

Thanks again for the helpful comments! In this work, our primary claim emphasizes that both continuous-valued tokens and MNTP are crucial for achieving SOTA audio generation using language models, supporting MNTP's efficacy specifically for continuous data. However, extending MNTP methodologies to discrete tokens is indeed a compelling new direction and will be part of our future research efforts. We appreciate your suggestion!

3. Typo

We will fix the typo. Thanks for the comment!

4. Demo

We omitted the demo from our initial submission. Here is our demo page. Please take a look!

---

We appreciate your valuable comments! If you feel we have adequately addressed your concerns, please consider increasing the score. If you have any further suggestions for improving our paper, please do not hesitate to let us know. Thank you!