ETTA: Elucidating the Design Space of Text-to-Audio Models

Thank you for your supportive review. We address your questions as follows:

Q: according to Table 6, the 2.08B 36-layer model seems to produce better results. So why is the balance achieved by the 1.44B model optimal?

A: We observed improvements for the deeper model (2.08B), but the gap narrows with CFG (Table 16 vs. 17). The subjective quality has been similar from the preliminary study. For this work we consider 1.44B a good balance between quality and speed, but we also expect that larger models will become better as we scale up the synthetic dataset.

Q: CFG could suffer FD penalties via other mechanisms, such as distortion caused by CFG's extrapolation.

A: This is a very good point. We also notice oversmoothing and/or distortion when CFG is set too high. This can be another source of its impact on FD. We will update the description accordingly to reflect this.

Q: I suggest letting human raters compare the captions in AF-Synthetics and those in the source dataset + An alternative and potentially more straightforward metric would be Frechet embedding distance.

A: Thank you for your suggestion to evaluate the subjective quality of the synthetic caption from AF-Synthetic. We also agree CLAP does provide a good proxy that quantifies the caption quality, as evidenced by ETTA’s generation results by using AF-Synthetic as the training dataset.

We are happy to conduct human evaluation of AF-Synthetic captions. Since it is a very large scale, we will randomly sample a subset of captions and report human results in the final version.

The purpose of the max-sim metric is to assess to which extent we generate novel captions in AF-Synthetic. A max-sim close to 1.0 indicates that generated captions are copied from the source captions, which is not desired. We show in Figure 2 that max-sim is not close to 1.0, suggesting the generated captions are novel and diverse (see Table 9 and 10 for some qualitative examples).

Q: It would also be nice to include AudioCaps, WavCaps, and Laion-630K.

A: Nice suggestion. We will include the full results in the paper.

Q: The authors make several architectural changes to the DiT model (AdaLN, GELU, etc.) How were these modifications decided? Were they added all at once, or one by one? + AdaLN didn't work with vanilla implementation but worked for ETTA? Why?

A: ETTA-DiT modifications have been made entirely, inspired by recent best practices. We conjecture adding AdaLN to both self-attention & cross-attention input with unbounded gating provides stronger conditioning signals.

We provide training loss curves between Stable Audio-DiT vs. ETTA-DiT (Figure-Rebuttal-1), link: https://anonymous.4open.science/r/etta_demo-778A/index.md . ETTA-DiT provides better convergence whereas Stable Audio Open-DiT plateaus early.

Q: To my knowledge, score-prediction diffusion models are also very popular and are known to produce high performance. Has it been considered to compare with them?

We did not consider the score (epsilon) prediction diffusion but used velocity (v) prediction in this work, as v-prediction has been the default choice of our baseline (Stable Audio Open). To the best of the author's knowledge, v-prediction has been favored by practitioners due to its better stability during training. We also find that OT-CFM provides even better stability.

Q: Make-An-Audio 2, TANGO-AF&AC-FT-AC, TANGO2, and GenAU-L all seem strong. Are these models comparable with "ETTA (AudioCaps only)"? Were they trained on additional data?

A: Baselines models are from their own official results and/or the public checkpoints trained with different datasets. ETTA (AudioCaps only) in Table 2 is also to illustrate improved results by fine-tuning ETTA on AudioCaps resuming from a pre-trained ETTA with AF-Synthetic. We’ll add the extra training data column to the main tables.

Q: Is there any chance that the AF-Synthetic dataset can be open-sourced?

A: We will release the model code and data preparation methods to reproduce the result of ETTA models.

Q: I found it interesting that FLAN-T5-Large is not universally better than T5-Base.

A: Thank you for mentioning this. While we are not drawing a conclusive claim, it suggests that an optimal choice may also depend on other factors (i.e. “quirks”). We also speculate that another possible reason is the difference in text embedding variance between the encoders [1].

[1] Xie, Enze, et al. "Sana: Efficient high-resolution image synthesis with linear diffusion transformers." arXiv preprint arXiv:2410.10629 (2024).

Q: What does "Finally, we also sub-sample long audio segments except for music and speech" mean in Section 3.1?

A: For long sound (non-music or speech) data, we found most of them to be homogenized (e.g. an hour of continuous engine sound). For these samples, we sub-sample a few ten-second segments instead of adding the entire 360 segments to AF-Synthetic.

Thank you for your supportive review. We address your concerns as follows:

Q: The paper lacks sufficient experiments directly comparing the dataset's contribution, such as showing how other SoTA models, such like Tango and AudioLDM, perform when trained with AF-Synthetic. What is the performance of other text-to-audio systems that train on AF-Synthetic? Such as AudioLDM, Tango, Make an Audio.

A: We acknowledge your concern; current main results (Table 2 and 3) are structured to represent prior works in their original form by using the official results and/or their public checkpoints.

Stable Audio Open is our most modern and direct baseline, and we did perform the ablation study in AF-Synthetic dataset’s direct impact while keeping the baseline model identical in Table 4 (+AF-Synthetic). Compared to the public Stable Audio Open checkpoint, AF-Synthetic provides significant improvements from the training dataset alone.

In our preliminary study, we also did our best effort to train previous models (other than Stable Audio Open) using AF-Synthetic, but except for Stable Audio Open we have faced difficulties using their original training recipe (e.g. model not converging, loss instabilities, etc.). We speculate these models would need non-trivial optimization in the training recipe to accommodate significantly larger-scale data such as AF-Synthetic.

Q: The main weakness of the paper is that using audio-language model, such as Audio Flamingo to generate the caption making the result more like a conversation, not just normal captions. Most of the captions will begin with ”the audio consists”, “the audio features”,”There is a”. Which is actually different to the real-world cases. + Did the audio-language format of the caption lead to some problems? Is there any way to avoid such structure and re-caption into a more formal structure?

A: Thank you for asking this important question. While AF-Synthetic captions mostly start with a conversational prefix from the audio language model (Audio Flamingo), we empirically find that the model is robust in generating good quality samples using plain captions, as evidenced by our main benchmarks that use the original captions (Tables 2, 3, and 22) along with the OOD imaginative prompts in the demo webpage.

We also tried using an external LLM (Nemotron-340B [1]) to rephrase and shorten the AF-Synthetic captions to be more caption style (like AudioCaps). We found ETTA has similar quality when trained on this rephrased dataset.

[1] Adler, Bo, et al. "Nemotron-4 340b technical report." arXiv preprint arXiv:2406.11704 (2024).

Q: What is the difference between ETTA and Stable-Audio(Despite the DIT module)?

A: Other than the improved DiT backbone, ETTA uses an improved VAE, a better training objective (OT-CFM w/ logit-normal t-sampling), and is trained on AF-Synthetic.

Q: Why choose the current CLAP threshold, will the dataset be improved if raises the score?

A: The CLAP threshold >= 0.45 is set following a suggestion in existing work [2]. Improving the CLAP threshold will retain higher quality subset at a cost of reduction of the dataset size.

[2] Kong, Zhifeng, et al. "Improving text-to-audio models with synthetic captions." arXiv preprint arXiv:2406.15487 (2024).

Q: Why the ETTA does not outperform previous models in some MOS metrics?

A: For music, Stable Audio Open is trained on high quality music from Freesound and Free Music Archive (FMA) datasets, but many of these samples are not available as of today. In contrast, ETTA is not trained on any studio-quality music datasets (not even MusicCaps). As a result, ETTA has a slightly worse OVL (which only measures audio quality and does not consider text) on MusicCaps and SongDescriber.

While only trained on synthetic captions, ETTA outperforms Stable Audio Open on the REL (audio-text relevance) score on AudioCaps and MusicCaps (Table 2-3), and matches it on an OOD testset SongDescriber (Table 22).

Q: Could you also provide some demos on music?

A: We have added several MusicCaps benchmark samples to the updated demo webpage. Link: https://anonymous.4open.science/r/etta_demo-778A/index.md

Thank you for your review. We address your concerns as follows:

Q: Table 5 indicates that performance with AF-AudioSet is comparable or even superior… It remains challenging to conclude that AF-Synthetic is crucial for model improvement… This observation questions the necessity of AF-Synthetic compared to AF-AudioSet.

A: We showed that a scaled-up dataset with AF-Synthetic shows much better overall results for out-of-distribution data (Table 23). Note that AudioCaps and MusicCaps are considered in-distribution as both are derived from AudioSet. This means that using larger-scale training data (AF-Synthetic) provides better OOD generalizability.

AF-Synthetic brings other practical benefits during training as well: we found that training ETTA using AF-AudioSet diverged after 250k training steps, whereas AF-Synthetic provides stable training up to 1M steps without instabilities. This means AF-Synthetic helps ETTA converge better from its scale.

We added the training loss curves of AF-AudioSet vs. AF-Synthetic (Figure-Rebuttal-3) along with the generated samples using the OOD challenging captions to the demo webpage, to better showcase the necessity of AF-Synthetic for generalization. Link: https://anonymous.4open.science/r/etta_demo-778A/index.md

Q: The rationale behind adding each component is unclear… Additional analyses—such as human perception studies, loss curves, or convergence times—would better illustrate each component's impact.

A: To address this concern, we added loss curves to the updated demo webpage to better illustrate the design choices: https://anonymous.4open.science/r/etta_demo-778A/index.md

Figure-Rebuttal-1: Stable Audio-DiT vs. ETTA-DiT, under the same v-diffusion objective and AF-Synthetic training dataset for both. We find that Stable Audio-DiT plateaus around 300K steps and starts to diverge early. ETTA-DiT continues to improve its quality with better loss for the same training steps. This shows clear benefits of the ETTA-DiT architecture.

Figure-Rebuttal-2: v-diffusion vs. OT-CFM training objective, under the same ETTA-DiT architecture and AF-Synthetic dataset for both. Note that the loss scale of OT-CFM is different to v-diffusion. For prolonged training (e.g. over 500K steps), v-diffusion starts to be unstable, whereas OT-CFM provides better stability up to 1M steps. This shows practical advantages of OT-CFM over v-diffusion and is a motivation to use it for ETTA.

Figure-Rebuttal-3: AF-AudioSet vs. AF-Synthetic training dataset, under the same OT-CFM objective and ETTA-DiT architecture. AF-AudioSet quickly diverges around 250K steps and is unable to continue its training, whereas AF-Synthetic provides better convergence with continued improvements up to 1M steps.

We believe these additional loss plots better illustrate the rationale behind each of the design choices of ETTA.

Q: Why are the results of Stable Audio Open different between Table 3 and Table 4? Are these different models?

A: Table 4 turned off the classifier free guidance (CFG) for all configs as an ablation study, including the original Stable Audio Open, so the difference compared to Table 3 (which uses a default CFG scale of the baselines) is expected.

Q: Table 2 reports human scores for both ETTA and ETTA-FT-AC-100k, while Table 3 only includes ETTA.

A: Thank you for mentioning this, we indeed measured ETTA-FT-AC-100k on musiccaps as well: OVL: 3.30 ± 0.10, REL: 3.44 ± 0.12. This shows that fine-tuning ETTA on AudioCaps (general sound) degrades the music generation quality in subjective evaluation, as expected. We will improve the writing to avoid confusion.

Q: In Section 3.1, when generating synthetic captions, if all generated captions have CLAP similarity below the threshold, is the audio discarded?

A: Yes, the audio is discarded in that case.

Thank you for your supportive review and appreciating our large-scale extensive study on the design choices of TTA to reach state-of-the-art results.

Q: AutoCap has open-sourced a dataset of size 40+ millions.

A: Thank you for mentioning the status of this concurrent work. We will discuss this concurrent work in the final version of the paper.

Q: Subjective evaluation is close or underperform baselines (e.g Table 3, Table 22).

A: For music, Stable Audio Open is trained on high quality music from Freesound and Free Music Archive (FMA) datasets, but many of these samples are not available as of today. In contrast, ETTA is not trained on any studio-quality music datasets (not even MusicCaps). As a result, ETTA has a slightly worse OVL (which only measures audio quality and does not consider text) on MusicCaps and SongDescriber.

While only trained on synthetic captions, ETTA outperforms Stable Audio Open on the REL (audio-text relevance) score on AudioCaps and MusicCaps (Table 2-3), and matches it on an OOD testset SongDescriber (Table 22).

Q: The authors have not promised the release of their checkpoints or code.

A: We will release the model code and data preparation methods to reproduce the result of ETTA models.

Q: The paper lacks a clear discussion on the difference between the dataset collection pipeline of AF-Audioset and AF-Synthetic.

A: We scaled-up the data collection method proposed in AF-AudioSet [1], and added several filtering methods (Section 3.1 and Appendix C.1).

[1] Kong, Zhifeng, et al. "Improving text-to-audio models with synthetic captions." arXiv preprint arXiv:2406.15487 (2024).

Q: How much of the initial dataset is filtered with CLAP and others?

A: The total filtering rate is about 73.5% (acceptance rate 26.5%) under a CLAP threshold of 0.45 and other filtering methods.

Q: AF-audioset performance is comparable with AF-synthetic in Table. 5, despite having a smaller size. While AF-synthetic achieves better performance in Tab. 23.

A: This is a great question. First of all, we find a scaled-up dataset with AF-Synthetic shows much better results for out-of-distribution data (Table 23) (Note that AudioCaps and MusicCaps are considered in-distribution, derived from AudioSet).

AF-Synthetic brings other practical benefits during training as well: we found that training ETTA using AF-AudioSet diverged after 250k training steps, whereas AF-Synthetic provides stable training up to 1M steps without instabilities.

We added the training loss curves of AF-AudioSet vs. AF-Synthetic along with the generated samples using the OOD challenging captions to the demo webpage, to better showcase the necessity of AF-Synthetic for generalization. Link: https://anonymous.4open.science/r/etta_demo-778A/index.md