Taming Data and Transformers for Audio Generation

Q1: What does the rejection rate in line 282 refer to? Is the rejection related to downloads?

Rejection rate refers to the percentage of videos that are rejected when filtering the dataset from speech and music. In other words, it represents the percentage of videos that do not contain ambient sounds and rather speech and music from randomly selected videos.

Q2: In Section 3.2, the authors mention that they use the captions of each video from the video-captioning model in Panda-70M. Does this imply that the "metadata" is essentially raw captions generated by another captioning system?

The metadata used for training are a caption for the visual modality and the title of the input clip, as specified in L386-388 under training details. We emphasize that the captions used for the metadata are specifically for the visual modality and do not incorporate information from the audio input.

Q3: Can the authors provide more details about the collected video datasets? Specifically, what is the average length of each video sample?

Appendix Sec. A includes statistics for the collected dataset. Fig. 9 shows the distribution of the clip lengths in AutoRecap-XL, with over 40% of the videos being at least 10 seconds long.

Q4: How does GenAu perform if only one text encoder (either FLAN-T5 or CLAP) is used?

We have not conducted such an ablation, as it is well-established in the audio generation literature [1, 2] that a dual text encoder strategy offers improvements. The same observation has been validated in the image generation [3] and video generation [4] domains Since the use of dual text encoders is not a primary contribution of our work, we have not run an ablation on this design.

Q5: What is the performance of GenAu when trained on the proposed AudioReCap-XL dataset?

Please refer to our response to Weakness 5.

Q6: Could the authors provide some demo examples of GenAu?

We would like to respectfully draw the reviewer’s attention to the attached website in the supplementary material, which contains a large quantity of qualitative results of GenAU, as well as dataset samples, and qualitative comparisons on both the generation and captioning tasks.

Final message: We sincerely thank the reviewer for their comments and useful feedback. We believe we have addressed all the questions and concerns raised. If the reviewer has any additional questions or finds any of our responses unsatisfactory, we would be more than happy to revisit and address them further.

References

1- Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation, 2023

2- AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining, CVSSP 2023

3- Scaling Rectified Flow Transformers for High-Resolution Image Synthesis, 2024

4- Movie Gen: A Cast of Media Foundation Models, 2024

We appreciate the reviewer's feedback. Below, we address the reviewer’s main concerns. Due to the character limit, we split our response into two messages.

W1: The paper lacks a detailed pipeline explaining how captions are analyzed and how speech or music-related content is filtered out. The authors should provide more clarification on this method

In Fig. 2 and Sec. 3.2, we give a high-level overview of the dataset collection pipeline. We include in Appendix Sec. A.1 the initial data selection strategy, Sec. A.2 describes the initial phase for filtering music and speech, and Sec A.3 describes our post-processing strategy for filtering speech and music. In summary, we assume that YouTube audio segments from a given video with no transcriptions are more likely to contain ambient sounds (no speech or music). After selecting such segments and captioning them with AutoCap, we further filter audio segments with captions containing keywords related to music and speech (such as talking, singing, etc).

W2: The proposed audio captioning system appears to require external "metadata" to generate captions. However, the definition of this "metadata" is ambiguous throughout the paper. The authors should offer more details on what this metadata consists of and how it is used. If this metadata is not readily accessible from raw audio, it limits the applicability of the system to real-world scenarios. W3: According to Table 1, the comparison indicates that the proposed AutoCap does not achieve the best performance on several metrics, especially when only audio is used as input data.

We build a captioning method specifically targeted at producing a large-scale audio generation dataset instead of aiming for raw audio captioning. We specify the metadata used for our final training in L386-388 under training details, which are video captions and the title of the video/audio clip. Such metadata is available for a significantly large set of 57M clips, showing the applicability of the system in real-world scenarios. Lastly, AutoCap without metadata (trained with AC + CL + WC) in Tab. 1 still outperform baselines in most metrics, except BLUE4 (achieving 28.1 compared with 28.9), and CIDEr (80.0 compared with 80.6), showing that the method is applicable even to specific use cases where only raw-audio is present.

W4: For the audio generation system, the proposed model largely follows the architecture from Huang et al. (2023), incorporating the 1D-VAE and LDM modules. This feels more like an engineering effort, where the existing system is applied and scaled to larger datasets. Moreover, the model uses both FLAN-T5 and CLAP for conditional input, whereas previous models generally employ only one text encoder to achieve satisfactory results. The authors should explain why both encoders are necessary, along with experimental comparisons showing if using only one encoder leads to a drop in performance.

In contrast to Huang et al. (2023) which uses a DiT, our audio generator uses an FiT architecture. We show, for the first time, the superiority of this architecture in the audio domain in Tab.5 where GenAU-S (493M params) trained without recaptioning is preferred over a bigger model Make-an-audio 2 (937M params) trained under similar data settings. We followed previous work [1, 2] in employing both FLAN-T5 and CLAP embeddings, which have been found to bring improvements in the literature as cited in L460-462. We do not claim novelty in the usage of dual text encoders and we refer to prior studies for analyses on the text encoder.

W5: The paper lacks an evaluation of the effectiveness of the proposed AutoReCap-XL dataset.

We provided AutoReCap-XL as an additional contribution to the paper, complementing our contributions in audio captioning, generation, and dataset collection. However, due to the significant size of the dataset (57M clips) and our limited resources, we are not able to train GenAU on the full dataset. Instead, we provide numerous evaluations that validate the usage of synthetic captions using a dataset size comparable to previous work:

1- In Fig. 3 (right), we show that the model scales well with the use of more data with synthetic captions. This strongly suggests that AutoReCap-XL can bring compounding improvements.

2- In Tab. 3, we show that GenAU-S training with synthetic captions outperforms significantly GenAU-S without recaptioning in out-of-distribution settings, improving IS by 60.4% and confirming the importance of data recaptioning.

Thanks for the reply from the author. I am happy to increase the score to 6.

We sincerely thank the reviewer for engaging in the discussion. Below we address the reviewer's remaining concerns.

GenAU contribution

We demonstrated the feasibility of adapting the  FiT architecture to latent audio diffusion models and demonstrated the scalability and superiority of the FiT architecture in the audio generation task. This is significantly important since in contrast to other domains (e.g. image and video) which benefit significantly from scaling, previous and concurrent UNet-based and DiT-based methods [1, 2] have reported poor scalability. For instance, AudioLDM-2 [1] reported that their 712M model (AudioLDM 2-Full-Large), achieve worse results than their 346M model in all metrics, except REL (3.80 compared with 3.77). Similarly, a concurrent DiT-based method, EzAudio [2] reported a worse CLAP score, and only marginal improvement of 0.26% in IS when comparing their largest to the smaller model. This is in contrast to our proposed FiT architecture which shows its superior scalability (see Fig. 2, left). 

While we do not claim novelty on the FiT architecture itself, we would like to argue that introducing an existing architecture to a new task, providing insights into its optimal configuration, and validating its effectiveness and scalability is an important contribution which has has been deemed novel and influential in many prior work [3, 4].

AudioReCap-XL

We have contributed AutoReCap, a significant dataset of 761.1k text-audio pairs, and have verified its effectiveness in scaling audio generation (Fig. 3, right, Tab. 3, and Tab. 5). Our analysis highlights that even this dataset alone brings substantial value to the audio generation community. 

In an effort to provide the community with an even more significant dataset, we scaled the dataset collection approach to provide AutoReCap-XL, which follows the same collection strategy of AutoReCap, and we have provided several analyses on the dataset distribution in Sec. A. 

We agree with the reviewer and have acknowledged in the limitation section L1111-1112 that further validation of AutoReCap-XL would be desirable, but we also believe it is valuable to release such data to the community as it currently suffers from a lack of large-scale dataset, a point raised by other reviewers (4Kh5, DQFx), and it is reasonable to suggest that conclusions on a subset of the dataset (AutoReCap) would likely transfer to the XL version (AutoReCap-XL) as the same collection strategy is followed. While it is not feasible to train GenAU with AutoReCap-XL within the discussion period deadline, if the paper gets accepted, we will include this experiment in the camera-ready version for completeness.

We would like to respectfully reiterate our core contributions in the paper:

1- AutoCap: a novel audio captioning architecture that achieves state-of-the-art performance.

2- GenAU: an adaptation of the FiT architecture to the audio generation task. We demonstrate its effectiveness and scalability in this new setting while achieving state-of-the-art performance. 

3- A scalable data collection pipeline for ambient audio that we used to collect AutoReCap, of which we extensively evaluated the benefits in the audio generation task.

Finally, we offer AutoReCap-XL as an additional contribution, providing the community with access to a large-scale dataset of 57M audio-text pairs. We believe this is valuable and decided to provide it as a supplement alongside the extensively evaluated AutoReCap dataset, despite its more limited evaluation.

We sincerely thank the reviewer for their engagement in the discussion and hope that our responses have adequately addressed their concerns. Should the reviewer have any additional questions or concerns, we would be glad to revisit them.

Reference

1- AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining, TASLP 2024.

2- EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion Transformer, 2024.

3- Scalable Diffusion Models with Transformers, ICCV 2023. 

4- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR 2021

Thank you for the author's response. While the author has addressed my concerns to some extent, the absence of an evaluation for the proposed AudioReCap-XL dataset remains a significant issue. Despite its large scale and the challenges associated with training or evaluation, the dataset should be tested to ensure its reliability and address any potential issues prior to publication. Otherwise, the dataset cannot be claimed as a contribution and should not be used by the public. Additionally, the author states that the proposed system utilizes a FiT structure, which differs from the Make-an-Audio2 system. However, it is unclear whether any novel improvements have been made to the original FiT structure introduced in the prior CV paper. As it stands, the approach seems to involve incorporating various "new" techniques into the existing system to achieve better performance, rather than presenting a fundamentally innovative framework. I look forward to your further clarification on these points.

Dear Reviewer bns3, We would like to renew our availability to further address any remaining concerns that may be present in these last days of discussion. We are looking forward to receiving your feedback.

We appreciate the reviewer's feedback. Please find below our detailed responses to the reviewer’s comments. Due to the character limit, we split our response into two messages.

W1: Fig. 1 says CLAP Encoder has one token. CLAP uses HTSAT as the audio encoder which also has intermediate representations. This only means that the authors used the CLS token (or some pooled representations) which is not specified. The authors should also clearly mention "CLAP audio encoder".

We use the CLAP [1] audio encoder to extract audio embedding (a single token) following the instructions in their Github page [1]. We have clarified this in the updated paper.

W2: The caption says "We then compact this representation into 4x fewer tokens using a Q-Former (Li et al., 2023a) module.". The figure shows only HTSAT encoder representation is passed to QFormer. The authors should rewrite the caption.

Only the HSAT embedding (1024 tokens) is passed to the Q-Former. Since the CLAP audio embeddings (a single token) and BART metadata embeddings (64 tokens) are limited, we pass them directly to the BART encoder as the figure shows. We have refined the caption to indicate that Q-Former only receives the HTSAT representation.

W3: The claim "4x fewer tokens" just because QFormer was employed in one part is not justified. The authors are also using BART embeddings, etc. Do you mean "4x fewer tokens" compared to your own baseline which does not use a QFormer?

We compare the input of Q-Former (1024 tokens) with the output (256 tokens) when reporting the 4x reduction. In other words, if Q-Former is not employed, 1024 tokens would be passed to the BART encoder, similar to previous work [4, 5]. We additionally pass metadata embeddings (64 tokens) and CLAP audio embeddings (1 token) directly to the BART encoder. We have refined the caption to make it clear that 4x fewer tokens refer to the HTSAT representation.

W4: QFormer has been used for audio earlier [3]. Additionally, it does not seem like you are pre-training the QFormer? Does this mean you are training it E2E? An E2E model as big as QFormer trained on such small datasets is not very sound.

We do not pretrain Q-Former, and AutoCap is trained E2E. Our Q-Former is lightweight, containing only 4.7M parameters. We have updated Appendix Sec D.1 with details on the hyperparameters that we use for the Q-Former module. We would like to note that GAMA [2] appeared on arxiv only three months before the ICLR deadline and we consider it as concurrent work.

W5: The authors say fewer tokens but do not talk about the increase in parameter count. I would like to see ore discussion on this please.

AutoCap introduces only a total of 6.2M parameters, with 4.7M for Q-Former, 0.9M for embedding layers, and 0.6 for projection layers. We have updated Appendix Sec D.1 with details on the parameter count.

W7: I am concerned why only audio clips with "No subtitles" were uses. Is the synchronization of time and subtitle correct? Also, human speech is an important sound and heavily present in audiocaps. If the authors believe not, then I think the paper should refocus on "environmental sounds" and not "audio".

As mentioned in the abstract and frequently throughout the text, our work focuses on “ambient sounds” or, in other words, environmental sounds, as the reviewer suggests. Data for this task is more difficult to obtain since most online videos are dominated by speech and music. Additionally, although YouTube automatic transcriptions give accurately synchronized subtitles, the subtitle content is sometimes inaccurate. To address this, we employ an additional keyword-based filtering step to filter speech and music clips from the dataset as mentioned in Fig. 2 and Sec A.3.

W8: (Not a big weakness) I feel its important that synthetic data generation pipelines have some verification strategy (human or automatic).

We have obtained these captions using our state-of-the-art captioner (AutoCap). We state in the limitations section our agreement with the reviewer that more validation would be desirable and we acknowledge that the captions are not perfect, leaving room for improvement. We plan on releasing the dataset, providing the opportunity to improve the captions in future work.

Thank You for your response.

We respectfully note that HTSAT encodes a 10-second audio clip sampled at 32kHz into 1024 tokens, each with a dimension of 527, as mentioned in the training details of HTSAT [1], “The shape of the output feature map is (1024, 527) (C=527).”

I would like to respectfully clarify a misunderstanding on the authors end. For (1024, 527) -- 527 is the number of audioset classes and 1024 is the dimension of the embedding. Thus, the authors have probably used the latent output of the classification layer for their experimental setup which is not sound. No paper does it, neither for audio or vision. The representations in this latent will have very less useful information and is steered towards multi-label classification as is a collapsed embedding passed through classification layers.

What should have been used is the latent representation before the classification layer, which is 64 x 1024.

Dear Reviewer TUnZ and 4Kh5, we sincerely thank you for providing your feedback and engaging in the discussion. We are happy to clarify the point regarding HTS-AT embeddings extraction and usage.

What is the dimensionality of the HTS-AT latent representation?

We apologize for the confusing reference in our previous response. With reference to Sec. 2.1.2 in [1], HTS-AT encodes a 10s audio signal into a latent representation before the classification layers (referred to as latent tokens in Fig. 1 in [1]) of shape (T/(8P), F/(8P), 8D) = (32, 2, 768), where T=1024, F=64, D=96, P=4 (See Sec 3.1.1 in [1]).

Which HTS-AT latent space do we use?

We confirm that we use the customary latent representation before the classification layer. We do not use the classification layer output.

We closely follow [2] codebase in extracting HTS-AT embeddings. Although not explicitly mentioned in [2] manuscript, [2] first encodes the audio signal using HTS-AT, producing a latent representation before the classification layer of shape (32, 2, 768), referred to as “latent tokens” in Fig. 1 of [2]. It then performs averaging over the second dimension (corresponding to the frequency dimension), obtaining a representation of shape (32, 1, 768), i.e. 32 tokens with 768 channels. As a final step, [2] repeats each token 32 times, obtaining a representation of shape (32, 32, 768) that is flattened into (1024, 768), i.e. 1024 tokens with 768 channels. This representation is referred to as “fine_grained_embeddings” in the codebase of [2] and several other codebases such as CLAP [8]. Several other works employ this “fine_grained_embeddings” representation [3, 4, 5, 6, 7]. We perform further studies of the HTS-AT representation of [2] to show the beneficial effects of using the fine-grained representation.

Is the token replication operation of [2] necessary?

We perform a series of ablations on HTSAT-BART [2] employing different variants of the procedure of [2] for the extraction of HTS-AT embeddings. We start with HTS-AT output tokens after the averaging operation of shape (32, 1, 768), and apply different token repetition factors to produce embeddings with 32 tokens (no token repetition), 256 tokens (8x token repetition), and 1024 tokens (32x token repetition following [2]). For completeness, we perform the same ablation on our model, using as input to the Q-Former 32 tokens (no token repetition) and 1024 tokens (32x token repetition). We followed [2] training procedure and we report evaluation results on AudioCaps test split on the last checkpoint in the following table. Please note that training hyperparameters of AutoCap are modified to match HTSAT-BART [2] for the purpose of the ablation.

Please refer to Fig. 13 in the updated manuscript for a better understanding of the effect of token replication, where we reported the performance across various epochs in the pretraining and finetuning phases. As the ablation shows, the token replication operation of [2] consistently improves model performance. We believe this finding can be attributed to the increased computation in the downstream model caused by it. Although the improvements with fine-grained representation are less pronounced in our model, we observed that the fine-grained representation consistently outperforms the 32 tokens representation (see Fig. 13). Consequently, we adopt the best performing 32x token replication embedding extraction procedure of [2] throughout our work. We also remark that with the chosen Q-Former design and the HTS-AT representation of [2], our audio captioning model provides state-of-the-art performance (see Table 1).

We sincerely thank Reviewer TUnZ and 4Kh5 for giving us the chance to discuss the embedding extraction procedure of [2] and to enrich the manuscript with deeper insights and ablations related to it. We updated the manuscript with the discussion and ablation results provided in this answer (see Appx. B.1, Appx. G.3). We hope this answer clarifies all of the Reviewer’s remaining concerns and we remain available if any further concern is present.

We thank the reviewer for their feedback.  Below, we address the reviewer’s comments.

W1: In scenarios where metadata lacks detail, audio captioning may struggle to disambiguate sounds accurately. The model also tends to falter in capturing the temporal relationships between sounds and differentiating foreground from background noises.

In Tab. 1, we show state-of-the-art audio captioning performance. Without metadata (trained with AC + CL + WC) our method still outperforms baselines in most metrics, except BLUE4 (achieving 28.1 compared with 28.9), and CIDEr (80.0 compared with 80.6). Additionally, in the attached Website, we show that our method has better temporal understanding than the previous approaches. We agree with the reviewer that no perfect audio captioner exists. For this reason, we highlight this observation in our Limitations section.

W2: Fine-tuned on AudioCaps, which contains a limited vocabulary of 4,892 unique words. The limited vocabulary of the paired texts, even though extensive, hampers the model’s ability to accurately generate audio for long and detailed prompts.

Audio captioning performance is limited by the available datasets with ground truth captions, with AudioCaps being the largest. This is a key challenge faced by most previous work. Using the visual modality as an additional input helps improve the diversity of the generated words, producing words unseen in the training dataset. However, as we discussed in the limitation section B.2, we acknowledge that there is still room for improvement.

W3: The proposed dataset, AutoReCap-XL, is substantial in size but features a constrained vocabulary of only 4,461 unique words. Furthermore, despite its potential as a significant contribution, this dataset has not yet been extensively analyzed for caption accuracy or performance in downstream tasks.

We introduce AutoReCap-XL as an additional contribution after contributing a state-of-the-art audio captioner, a state-of-the-art audio generation model, and AutoReCap, for which we evaluated its benefits in audio generation. As an additional contribution, we introduce AutoReCap-XL, which is significant in size, featuring 57M audio-text pairs. We expect the data scaling trends shown in Fig. 3 to transfer to this dataset. While we have conducted limited analysis of this dataset in Sec. A, as noted in the conclusion (Sec. 5), we leave more extensive analyses for future work.

Q1: Didn't train baselines on the new dataset to show the proposed architecture is actually superior. I would have liked to see the baselines trained on the 57M example dataset. This would clearly show if the better performance of the proposed method is due to architecture or just the scaling of the dataset.

We conducted evaluations to show that our architecture is better regardless of the data. In Tab. 5, where we show that GenAu-S without Recap outperformed the strongest available baseline, Make-an-audio-2, when trained at a similar data setting (without audio recaptioning), confirming the superiority of our architecture. This evaluation setting is common in text-to-audio evaluation. Running several large-scale baselines on a large-scale dataset would be an unnecessarily expensive way of evaluating the same architectural advantage.

Q2: Is there a way to verify the quality of the dataset in terms of captioning? assuming the community adopts the use of this new data how are you to ensure the data is of high quality? maybe some human evaluation would be in place.

We verify the quality of AutoReCap by means of the downstream task of text-to-audio generation. Our out-of-distribution evaluation (Tab. 3) confirms that recaptioning with AutoCap significantly improves the quality of the generated audio, achieving 60.4% better IS. We have verified the effect of scaling audio generation models with this synthetic dataset in Fig 3 (right).

Q3: Please move the limitations section into the main body of the paper.

We have included a summarized discussion of the method’s limitations in the updated main manuscript according to the reviewer’s suggestion, and we will keep a detailed limitation section in the Appendix.

Final Message: We sincerely thank the reviewer for their feedback. We believe we have addressed all concerns but are happy to revisit any additional questions if needed.

We appreciate the reviewer's feedback. Please find below our detailed responses to the reviewer’s comments. Due to the character limit, we split our response into two messages.

W1: The authors shared the audio samples from Stable-Audio 1.0[7] but the comparison is missing in results table.

Since Stable-Audio Open [1] generates at a different setting than all other baselines (stereo, 44khz), we decided not to include it in the table of compassion. We would also like to note that Stable-Audio open is an arxiv preprint that was published two and half months before the ICLR deadline. As requested by the reviewer, we included a comparison of our work with Stable-Audio Open in Tab. 4, showing that GenAU-L outperforms Stable-Audio open in all metrics.

W2: The performance improvement when compared to increase in number of parameters in GenAu is marginal.

We would like to clarify the subject of the question with the reviewer. If the question concerns a comparison between our method and baselines with respect to the number of parameters, we refer the reviewer to our subsequent answer to W12.

If the question instead concerns scalability properties of the proposed architecture with respect to model size, We respectfully disagree with the statement. We discuss model scaling in L509-L513 and we have verified the effectiveness of model scaling in three different evaluation settings:

1- Scaling trend (Fig. 3): Fig. 3 (left) in the paper shows a clear trend of improvements in both FD and IS when increasing the parameter count. Notably, the largest model (1.25B) outperformed the smallest (123M) by 56.3% in IS and 36.6% in FD under identical settings.

2- Out-of-Distribution Evaluation (Tab. 3): In out-of-distribution evaluation in Table. 3, the 1.25B model has 20.4% improvement in IS and 5.5% in CLAP score over the 493M model.

3- User Study (Tab. 5): In our user study in Table. 5, the 1.25B model achieved 61.20% preference in realism, and prompt alignment compared to the 493M model.

We disagree that such improvements are marginal.

W4: Recently, Large Audio Language models [1,2,3,4] are being employed for audio captioning task, but the authors don't compare AutoCap to these baselines which in my opinion should be an important comparison.

We followed recent AAC methods [2, 3] in our evaluation protocol to compare with baselines trained in similar data settings. Large Audio Language models perform captioning in zero-shot settings. While promising, they still underperform current SOTA AAC methods. Moreover, comparing these models using automatic evaluation metrics is not entirely fair, as such metrics are heavily influenced by the captioning style used in the ground truth. We evaluated GAMA—the state-of-the-art Audio LLM—using their most capable model (GAMA-IT) on the AudioCaps test dataset with their default prompt, 'Describe the audio.' Our results show that GAMA-IT reports significantly lower scores across all metrics compared to our model.

Please note that GAMA only reported a SPICE score of 18.5 using 500 random samples from AudioCaps, a different evaluation setting than all previous AAC methods which uses AudioCaps test dataset. We have included in Sec. E.2 a discussion on Large Audio Language models.

W6: Why do authors not show generation results of GenAu without Recap dataset or baseline + Recap in normal settings? It should be an important ablation to show where the performance gain is coming from.

We do! We discussed data recaptioning in L501-L506. We verified our methods in two axes:

• Architecture: as shown in Tab. 5, GenAu-S without Recap dataset outperformed Make-an-audio-2 when trained at similar data setting (without audio recaptioning), confirming the superiority of our architecture.
• Data quality: In Tab. 3, we show that GenAU-S greatly outperforms GenAu-S without Recap, improving IS by 60.4% and confirming the importance of data recaptioning. We have also included in the website in the supplementary a qualitative comparison between GenAu-S without Recap and GenAU-S, where the improvements in the quality and prompt following are obvious. For example, in the “Motorcycle starting and taking off” sample, the baseline without recaptioning does not follow the prompt well.