Elucidating the Design Space of Text-to-Audio Models

Dear reviewer 5Eah,

We appreciate your supportive review and constructive feedback. We clarify your questions as below:

Q: “the paper lacks sufficient experiments directly comparing the dataset's contribution, such as showing how models like Tango or AudioLDM perform when trained with AF-Synthetic”

This is a good point. We found some training instability issues when re-training these two models with AF-Synthetic, and therefore we could not finish the experiments by the rebuttal deadline. We will do our best to identify the issues and include their results into our paper.

We do directly measure the dataset’s contribution with the Stable Audio Open architecture. The results are shown in Table 4 & 5 on page 8 (1st vs 2nd row). To further remove the effect of classifier free guidance on the evaluation results, we turn off classifier free guidance and report additional results in Table 26 & 27 on page 27 (1st vs 2nd row). The results validate the improvements from AF-Synthetic.

Specially, we note that AF-Synthetic contains no studio-grade music recording samples with annotations from musicians, but Table 5 indicates Stable Audio Open trained with AF-Synthetic has better objective metrics and music-text relevance than their original model trained with large scale professional music data (FMA-large).

Q: “The paper lacks experiments where the proposed model is trained on the same dataset (e.g., AudioCaps) to clearly illustrate the improvements contributed solely by the system itself.” … “Train ETTA on other datasets like AudioCaps or TangoPromptBank and compare performance to when trained on AF-Synthetic.”

Table 6 & 7 on page 8 include results of ETTA trained on AudioCaps and TangoPromptBank. For example, Table 6 (1st row) indicates ETTA trained only on AudioCaps achieves better $FD_O$, $FD_P$, $KL_S$, $CLAP_L$, and $CLAP_M$ than non-proprietary models in Table 2. We will add this row to Table 2 for better presentation.

To remove the effect of classifier free guidance on evaluation results, we additionally run evaluations with no classifier free guidance in Table 28 & 29 on page 27. The improvements of each component of ETTA are more consistent and obvious.

Q: “including subjective evaluations like the Mean Opinion Score (MOS) would be beneficial, particularly for comparing models that achieve top performance across some metrics.” … “Develop human evaluations on a subset of generated samples, comparing ETTA to top baseline models, such as Mean Opinion Score (MOS) or other equivalent subjective ratings on audio quality, text relevance and so on.”

This is a very good point. Per your request we added MOS evaluation in our general response (part 1): Subjective evaluations for benchmark datasets. The results are consistent with the objective metrics overall.

Q: “More clearly articulate the novel aspects of their model architecture.” …”What distinguishes the AdaLN layer in the proposed ETTA-DiT structure from the one used in the original DiT model mentioned in a previous section?”

We will describe the differences more clearly in the paper. In summary, (1) ETTA-DiT applies AdaLN-zero to the cross attention input with unbounded gating; (2) ETTA-DiT’s AdaLN operates in full numerical precision (FP32) for both training & inference; and (3) ETTA-DiT uses the final projection layer with zero-initalization along with AdaLN with identity initialization. (4) we use GELU activation with tanh approximation. (5) ETTA-DiT’s RoPE embeddings operates in FP32.

Q. AF-AudioSet and Sound-VECaps have around 161K samples remaining after filtering with a CLAP score of 0.45. Since both datasets are primarily developed from AudioSet, is there a significant overlap in the subset that achieves a higher CLAP score? If so, could this raise concerns regarding the reliability of using the CLAP score as a filtering metric?

This is a very interesting question. We computed the overlap and found there are only 14,133 overlapping samples between AF-AudioSet and Sound-VECaps, which is less than 10% of their sizes (both around 160K). This is likely because their captioning methods are very different.

Dear reviewer 5Eah,

Thank you for reviewing our manuscript. We have addressed your concerns and conducted the additional requested experiments.

As the rebuttal window nears its end, we kindly request your review of our responses and welcome any further feedback.

We look forward to hearing from you.

Authors

Dear reviewer rAqM,

Q: Since the dataset is curated using CLAP scores, I find the CLAP score not a reliable indicator, while FAD and KL use old models to extract features.

We appreciate that you raised this important point. We address your concern below.

• First, It is a common practice to use CLIP to filter and curate training data in text-to-image models, for example, [1] [2]. We follow this approach and apply it to the audio generation domain.

• We used $CLAP_L$ (CLAP trained by Laion) to curate our training data. In evaluations, we additionally report $CLAP_M$, a completely separate CLAP model trained on different datasets by Microsoft. We believe this metric is not biased in our setting.

• We report FAD and KL in order to conduct thorough comparison with baselines. Many baseline models, especially those not opensourced, report these metrics in their papers. While we agree that VGGish used in FAD is old, PANNs and PaSST used in KL are among the most representative audio classifier backbones currently available.

• We also report newer metrics, especially $FD_O$ using the OpenL3 feature extractor (which could evaluate 44.1kHz stereo audio), and find our model significantly outperforms baselines on this metric.

• Our intention was to report as many well-established evaluation metrics as possible to provide the most comprehensive evaluation in the text-to-audio community. However, we do see your point and we will improve our paper to include detailed analysis of each metric including their limitations.

[1] Schuhmann, Christoph, et al. "Laion-400m: Open dataset of clip-filtered 400 million image-text pairs." arXiv 2021.

[2] Hong, Rachel, et al. "Who's in and who's out? A case study of multimodal CLIP-filtering in DataComp." ACM EAAMO 2024.

Q: There are no subjective scores in the results except for Table 9 which compares the final model with others.

We address your question in our general response (part 1): Subjective evaluations for benchmark datasets.

Q: The improvements in ETTA-DiT, OT-CFM and t-sampling are not consistent across metrics.

We appreciate your mentioning of the metric disagreement.

• Indeed, we find these metrics disagree with each other often. For example, optimizing FAD, especially to extreme, often leads to worse $FD_P$ and vice versa. This happens to not only ETTA but also most baseline models (Tango-series, AudioLDM-series, etc.). However, we would like to be transparent about our results and decide to report all of them, and hope to call the community’s attention to study this contradiction in depth.

• We also think that classifier free guidance potentially caused the inconsistency. We additionally turn off classifier free guidance and report evaluation results in Table 26 & 27 on page 27. The improvements of each component are clear and consistent in most metrics including FD, KL, and CLAP. The only exceptions are FAD (which is old and not preferred as mentioned) and IS (which measures sharpness and is most affected by classifier free guidance).

Q: Using AF-AudioSet and AF-Synthetic yields similar results in AudioCaps and MusicCaps despite much larger size (Table 6-7)

First, we find that AF-Synthetic leads to more stable training than AF-AudioSet towards longer (>1M) training steps.

We also hypothesize that the reason why AudioCaps and MusicCaps results are similar is that both are subsets of AudioSet, and so AF-AudioSet already covers the distribution. To validate this hypothesis, we additionally evaluate on SongDescriber, a high-quality music dataset (44.1kHz stereo) and an OOD test set in our setting. In contrast, MusicCaps has varied sample rates and channels.

The main results are in Table 36 on page 28, and the comparison between AF-AudioSet vs AF-Synthetic are in Table 37 on page 28. Results indicate pretraining with AF-Synthetic outperforms AF-AudioSet in most metrics, showing our AF-Synthetic pretraining set offers stronger generalization ability to the model.

Q: the contribution is lack of novelty since it does not propose new techniques or reveal surprising findings.

We summarize our novelty and contribution in our general response (part 2): Novelty and Contribution.

Q: “What can be the reason behind mode-collapsed models consistently produce good scores across multiple metrics, while the backbone for these metrics are different?”

We think the reason is related to small, homogeneous fine-tuning and test sets. We include further analysis in our general response (part 3): Clarification on potential mode-collapsing issue of ETTA-FT-MC.

Dear reviewer rAqM,

Thank you for reviewing our manuscript. We have addressed your concerns and conducted the additional requested experiments.

As the rebuttal window nears its end, we kindly request your review of our responses and welcome any further feedback.

We look forward to hearing from you.

Authors

Thank you for your response.

I increased the score to 5 due to the extensive and transparent results reported, but some of my points remain a concern, so I am still inclined to recommend rejection.

Since objective scores are not reliable and maybe biased (e.g., for the case of CLAP), I find them suitable only for development, but any significant claims made require subjective eval scores. The authors provided the scores for the final model, but not for any analysis in the paper. Since the important contribution of the paper is to compare between design choices, I recommend reporting those for at least Table 4 and 5. My apology for not pointing this out earlier to allow enough time for eval (so my score has already partially exclude this weakness). Since subjective scores are the most important metrics, we can rely on that to compare and evaluate the significance, rather than having to look at many inconsistent metrics (btw, they look quite consistent in Table 8 as expected, raising the question of how to evaluate when there's less agreement). I must also say this is also specific to the text-to-audio task since even the data used to train those models for objective scores may also be noisy; audio-caption data is less reliable than for example speech-transcript or audio-audio similarity. The authors have pointed out problems of the metrics; however that doesn't change the fact that the results are not enough to show the improvement. Looking at Table 26 and 27, I can't tell if the last two rows are better than ETTA-DiT.

"Scalability with Model Size" and "Choice of Sampler and its Impact on Metrics" in Page 9 do not provide any new discovery. The results should improve with larger model sizes and both samplers show similar results, also FD_V is a less reliable metric as authors pointed out (another metric reported in the appendix can be used here). Also if the goal is to compare between solvers, there are more candidates (e.g. DDIM) and this is also not something new with no evidently winning candidate.

Overall, I think the biggest contribution of the paper is the dataset and engineering efforts to train a good model on the dataset. However, the technical contributions are limited, with some unconvincing results and discoveries. The paper may still be valuable for its many results and observations.

Dear Reviewer gSe7,

Thank you for your thoughtful review and valuable feedback on our paper. We are pleased that you found our contributions of the AF-Synthetic dataset, practical guidance for design choices of TTA, and good presentation to be strengths of our work. We address your questions below.

Q: “As the authors themselves mentioned, the paper doesn’t propose a novel method. Moreover, as a paper based on extensive experiments, while it offers valuable conclusions, it lacks an innovative insight or discovery that stands out. Therefore, I believe this paper might be a better fit for the Dataset & Benchmark track.”

We address your concern in our general response (part 2): Novelty and Contribution.

Q: “Could the authors further analyze the ‘potentially mode-collapsed mode’ part? In theory, the FD metric should be able to measure sample diversity.”

We answer your question in our general response (part 3): Clarification on potential mode-collapsing issue of ETTA-FT-MC.

Generally FD could measure quality and diversity for unconditional generation. However, it may not be able to measure diversity perfectly in text-conditional generation, especially when the test set is small and homogenous. The diversity in the latter case refers to generating different audio with a slightly different caption or a different random seed with the same caption. We ran additional human evaluation for this model and got the following results: OVL=$3.77\pm0.10$ where GT=$3.88\pm0.10$, REL=$3.73\pm0.09$ where GT=$3.90\pm0.10$, which also show very good quality. Therefore, we would like to leave our finding as an open problem and call the research community’s attention.

Q: “Since the authors have already tried logit-normal t-sampling, could they also test the Min-SNR Weighting Strategy?” …“Regarding the Auto-guidance section, it might be worth experimenting with removing CFG (Classifier-Free Guidance) for the first 40% of steps and then adding CFG for the remaining 60% of steps to see if this improves the FD score.”

These are really good points. We explored these training (Min-SNR weighting [1]) and sampling strategies (CFG on limited interval [2]) per your suggestion. The results are below.

• Min-SNR: We use $\gamma=5$ per convention, and trained ETTA-DiT for 250k steps with the v-diffusion objective. Results can be found in Table 32 & 33 on page 28 (+Min-SNR-$\gamma$ ($\gamma=5$)). Most metrics became worse, but $FD_O$ and $IS_P$ are slightly better in music generation.

• CFG on a limited interval: We remove CFG for the initial 40% of diffusion steps and then apply CFG ($w_{cfg}=3$) for the remaining 60% steps. We inference and evaluate both with and without AutoGuidance. Results can be found in Table 34 & 35 on page 28 (+ CFG @ [0, 0.6]). Most metrics became worse, but $FD_P$ is slightly better in audio generation using the small (XS) model.

[1] Hang, Tiankai, et al. "Efficient diffusion training via min-snr weighting strategy." ICCV 2023.

[2] Kynkäänniemi, Tuomas, et al. "Applying guidance in a limited interval improves sample and distribution quality in diffusion models." arXiv 2024.

Remark

We conduct additional evaluations on the SongDescriber dataset. See our general response (part 4): Additional music evaluation on SongDescriber dataset.

Dear gSe7,

Thank you very much for acknowledging our response. We are glad that our response has been comprehensive and appreciated. We are happy to discuss any remaining questions you may have.

Authors

Dear Reviewer dkey,

Thank you for your thoughtful review and valuable feedback on our paper. We appreciate your recognition of our contributions, including the creation of a large-scale synthetic audio-caption dataset and the implementation of comprehensive evaluation metrics. We address your concerns and questions below.

Q: “It would be helpful to explain what each evaluation metrics focus on. For example, there are three types of Frechet Distance - what are the differences?”

Thanks for your suggestion. We will add a detailed explanation to each metric, in addition to the high level descriptions presented in Section 4.3. The three types of Frechet Distances are different in their feature extraction networks, which are from different classifiers (VGGish, PANNs, and OpenL3). VGGish is the oldest and the least accurate, and therefore $FD_V$ is less preferred (as pointed out by AudioLDM and Stable Audio). PANNs and OpenL3 are newer and have better quality, and so $FD_P$ and $FD_O$ are more preferred. We report all possible evaluation metrics used by the community in order to compare with baselines thoroughly.

Q: “It is not clear how ETTA perform without pretraining on the large-scale datasets. For example, what would the result look like if the ETTA model is trained on AudioCaps from scratch (I assume this is a common setup)?”

The first row of Table 6 includes results of ETTA only trained with AudioCaps. This model achieves better $FD_O$, $FD_P$, $KL_S$, $CLAP_L$, and $CLAP_M$ than non-proprietary models in Table 2. We will add this row to Table 2 for better presentation.

Q: “Out of the seven main conclusions in the paper, only one is backed by the subjective evaluation. This can make the conclusion less convincing.”

Per your suggestion, we report MOS results in our general response (part 1): Subjective evaluations for benchmark datasets. The results are consistent with objective evaluations overall. The results indicate ETTA has high quality and relevance to captions compared to baselines.

We kindly disagree with the statement that our conclusions are less convincing due to lack of subjective evaluations. We included all standard objective metrics in the literature. Similar to metrics in image generation (FID, bits/dim, CLIP, …), while we think a single metric is not comprehensive enough, these combined are very informative about the model’s quality.

Q: “The architectural improvement seems a bit marginal. The improvement of ETTA seems to come from the new synthetic dataset mostly.”

We disagree that the architectural improvements are marginal. However, we highly appreciate that you raise this issue from our existing results. To better understand the improvements from each component, we turn off classifier free guidance (CFG) and re-run all evaluations in Table 4 & 5. The results without CFG are shown in Table 26 & 27, page 27 in the updated pdf. The results show that the ETTA-DiT architecture leads to significant and consistent improvements on most metrics.

Q: ”‘We initialize the final projection layer of DiT to output zeros.’ Do you mean zero-initialize all the weight and bias in the projection layer? Or do you use gated operation? More explanations are welcome.”

We use both. We zero-initialize the weights and biases of the final projection layer. We also employ AdaLN with identity initialization with the gating mechanism.

Q: “Please consider adding the result of Make-an-Audio and AudioLDM in Table 2 and Table 3, as they are also very important baselines to compare with.”

Thank you for this suggestion. We have updated Table 2 & 3 to include these results. We also note that we have already included their successive work (Make-an-Audio 2 & AudioLDM2), which have better quality.

Q: “In Table 6, the ETTA trained on AudioCaps shows 3.00 FAD on the audiocaps evaluation set. This is significantly lower than the current state-of-the-art. It would be helpful if the author could explain this result.”

FAD (i.e. $FD_V$) uses VGGish network to extract audio features. It is older and less accurate than PANNs ($FD_P$) and OpenL3 ($FD_O$), as pointed out by the authors of AudioLDM and Stable Audio. Therefore, $FD_P$ and $FD_O$ are preferred over FAD. In addition, $FD_V$ is constrained with 16kHz mono audio whereas $FD_O$ measures up to 48kHz stereo, compatible with our model’s audio quality (44.1kHz stereo). We will include a more detailed analysis of these metrics in the paper.

We also find that FAD often contradicts with $FD_P$. By this we mean optimizing FAD, especially to extreme, often leads to worse $FD_P$ and vice versa. This happens to not only ETTA but also most baseline models (Tango-series, AudioLDM-series, etc.). However, we would like to be transparent about our results and decide to report all of them, and hope to call the community’s attention to study this contradiction in depth.

Dear Reviewer dkey,

Thank you for reviewing our manuscript. We have addressed your concerns and conducted the additional requested experiments.

As the rebuttal window nears its end, we kindly request your review of our responses and welcome any further feedback.

We look forward to hearing from you.

Authors

(4) Additional music evaluation on SongDescriber dataset

As several reviewers have questions regarding music evaluation, we additionally conduct evaluations on SongDescriber, a high-quality music dataset (44.1kHz stereo). In contrast, MusicCaps has varied sample rates and channels. The results are in Table 36 on page 28.

ETTA outperforms baselines in most metrics, especially $FD_O$ which could measure quality of 44.1kHz stereo outputs. We are also preparing subjective human evaluation (OVL/REL) on this dataset as well.