RiTTA: Modeling Event Relations in Text-to-Audio Generation

We appreciate the reviewer for spending time reviewing our work and further providing useful comments and advices for us to coninue improving our work.

Q1: The corpus only contains 25 categories, which seems to not easily generate to other categories with relations. Previous training materials like AudioCaps and Clotho have over 100 classes.

A1: We thank the reviewer for their careful examination of the corpus. Below, we address the concerns regarding the category number and its potential for generalization.

1. Corpus Design and Intent: The primary goal of RiTTA is not only try to cover a wide range of audio events but also guarantee that the introduced audio events are suitable for relation modeling. Therefore, the 25 categories are well-defined and meticulously constructed with a focus on ensuring high-quality, diverse and authentic (Sec. 3.3, L209-238) audio events that can effectively demonstrate relation modeling as detailed in section 3.1.

Datasets like AudioCaps and Clotho with over 100 audio classes, serve broader purposes of general audio captioning and classification, they contain non-event audio (e.g., music audio, ambient environment sound) and audio without explicit eventness indication (e.g., ocean waves) that can not be used to model audio events relations. However, we agree that introducing more audio event categories will improve the benchmark robustness, we leave it as the next step endeavour.

2. Adaptability and Expansion: RiTTA framework is designed to scale, allowing for the inclusion of additional categories and relation types as more comprehensive data is curated. Moreover, we decouple events relation from event category, allowing us much flexibility to scale up the data from either relation or event category separately and independently. This adaptability ensures that the dataset can evolve to include more diverse categories akin to larger training materials like AudioCaps and Clotho.

3. All Existing TTA Models Perform Poorly on Our Benchmark. The detailed experiments show that all existing TTA models perform poorly on audio events relation we proposed in this paper, they failed to generate audio exhibiting even simple relations (Table 1, Figure 1). Therefore, our proposed benchmark is useful in terms of difficulty level and complex to benefit the TTA community to develop more advanced text-to-audio relation-aware models.

4. Comparison with Prior Work. The most relevant work CompoA [1], it simply discussed temporal relation and contributed a benchmark with 400/200 test instances. our introduced benchmark is much more diverse in terms of relation classes and test instances (we have 8k test instances, and can be easily scaled up further).

[1] Sreyan Ghosh et al., CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models. ICLR 2024.

Q2: The paper targets on mono-channel audio, but in this case, the spatial distance makes little scene. How can you accurately measure whether the event is from 1m or 7m? I think multi-channel audio would also be interesting, e.g. a car driving from left to right.

A2: We thank the reviewer for the in-depth examination of spatial distance indication from audio. It is true that mono-channel audio can not precisely identify the absolute position of audio events, which instead requires multi-channel audio as audio events' spatial position usually lies in inter-channel phase difference. We focus on mono-channel audio generation because 1) all existing text-to-audio (TTA) methods focus on mono-audio generation. 2) multi-channel audio generation require much more complex text description specifying each single audio event's spatial position, which remains as another future research topic and goes beyond the scope of this work.

To make the spatial distance measurable, we propose to roughly measure the relative spatial distance between audio events, which is reflected in audio event's loudness that can be measured in mono-channel audio. This is also echoed by sub-relations we used in Spatial Distance relation: close-fist, far-fist, and equal distant (in Table 2) that can be measured by loudness in mono-channel audio. More details are given in section 3.1 L170-173, section 5.2 L415-427).

In summary, precise spatial distance relation does require multi-channel audio. We reconcile this dilemma by approximating spatial distance by loudness variation, and we show the feasibility of our proposed spatial distance relation measurement in experiment Table 8.

Dear Reviewer, thank you for your further comments. We would like to provide further feedback regarding your new comments.

1. Comments: Additionally, Tango and Tango 2 used a PANN model rather than a VGGish model to calculate the FAD and KL metrics, contrary to what you mentioned.

Feedback: We double-checked Tango paper (link: https://arxiv.org/pdf/2304.13731). In page 6, it writes: in addition to FAD, we also used Frechet Distance (FD) as an objective metric. FD is similar to FAD, but it replaces the VGGish classifier with PANN. So for the FAD score it does use VGGish model. We followed the same setting to use VGGish model to calculte FAD and KL score.

2. Comments: I still cannot find any explanation or experiments in your method demonstrating how to generate new categories.

Feedback: We are currently running experiments testing how the model performs when generating new categories, will report the result promptly. Thanks for your patience!

Thanks you for spending time reading and commenting our feedback. We hope it discuss more with you if there is a need.

We thank the reviewer for the constructive feedback and suggestions, which are really helpful for us to continue to improve our work. Regarding your concerns, we provide one-by-one feedback.

Q1: I am unclear as to why RiTTA was submitted to the generative model track, as the primary contribution clearly stems from the new dataset and evaluation metrics related to the benchmark, rather than from advances in generative modeling?

A1: Thank you for highlighting your concern regarding the appropriate track for our submission. We agree that RiTTA has introduced significant contributions audio events relation modeling in TTA task, including a new benchmark, novel evaluation metric and a fine-tuning strategy. These aspects indeed align well with the scope of new datasets and evaluation metrics.

However, we would like to emphasize that we have conducted in-depth and systematic research audio events relation problem that shows potential not only merely in TTA task, but also in a much broad spectrum involving audio events relation analysis (e.g., holistic acoustic scene understanding, reasoning). We submit it to generative model because existing TTA models perform poorly in audio events relation modeling (Table 1 and Fig. 1).

Moreover, we find some other ICLR accepted papers that also contributed benchmark and corresponding neural network framework, such as [1] and [2], We hope this clarifies our rationale for submitting to the generative model track and underscores the broader impact of our contributions on the field of text-to-audio generation.

[1] Sreyan Ghosh et al., CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models. ICLR 2024.

[2] Garrett Tanzer et al., A Benchmark for Learning to Translate a New Language from One Grammar Book. ICLR 2024.

Q2: In terms of innovation, the authors are not the first to explore audio event relationship modeling. RiTTA extends temporal relationships to four types of relationships, but the overall process appears to be data-driven. However, merely using the TTA model to model complex relationships is insufficient. As demonstrated in Figure 7, there is a performance decline with "not" relationships. I was expecting a novel TTA framework capable of jointly modeling relationships between different audio events. Unfortunately, RiTTA only fine-tunes the TTA model, which limits its novelty.

A2: We acknowledge the concerns raised and would like to address them with clarifications based on our methodology and results:

1. Relation to prior work: Although prior works have touched upon one relation: temporal relation, RiTTA provides a systematic study that sets it apart by establishing a comprehensive benchmark specifically for modeling audio event relations. We scale up the temporal relation to 4 different relations, accompanied by new audio events, new evaluation metric, new benchmark. That is, RiTTA isn't an incremental work based on prior temporal relation work.

2. Fine-Tuning Strategy: The critique regarding the novelty of fine-tuning is understood. However, we emphasize that the RiTTA framework was not designed to merely apply existing models but to adapt and expand upon them with targeted fine-tuning. Our method fine-tunes the latent diffusion model component of Tango, enabling it to better model audio events relations (Section 4). This nuanced strategy yielded significant improvements across key evaluation metrics, showcasing our approach’s practical impact (Table 8).

3. On "Not" Relation: The reviewer’s observation regarding "not" relationships is valid. We conducted further analyses that indicated this outcome was tied to the silent audio data representation paired with <Not> prompts, leading to potential model ambiguities (Section 5.4). Moreover, it is worth noting that not being able to model "Not" relation is an intrinsic challenge for all TTA models, because the text just tells what to not generate, no guidance what to generate at all. Future work could explore how to handle this situation.

We hope these points illustrate the innovative aspects and address concerns regarding RiTTA’s contributions to TTA models. We are committed to further refining the framework to enhance its robustness and clarity.

Q3: Dataset shortcomings. First, the dataset construction process does not seem to require significant effort. Second, the dataset itself feels somewhat toy. While the authors attempt to enhance its randomness, expanding each example to five events via ChatGPT is far from sufficient (only 5?); 500 events might be a more appropriate target. Additionally, the use of only one-channel audio for spatial relationships could limit the dataset’s diversity.

A3: We sincerely thank the reviewer for their detailed feedback and constructive points regarding the dataset. (continue in Feedback 2).

We sincerely thank the reviewer for the useful comments and suggestion for future expansion.

Q1: The paper benchmarks seven recent TTA models (such as AudioLDM and Tango) but focuses primarily on single-strategy models, and without testing in more complex scenarios like audio events in virtual reality or interactive environments. This limits the framework’s applicability across different model types and environments. A wider variety of model types, such as multimodal generation models or models optimized for extended audio sequences, and more diverse application datasets, like multi-event VR audio environments are recommended.

A1: We appreciate the reviewer’s insightful suggestion.

1. Focus on Single-Strategy TTA Models: We agree that our current benchmarking focuses on recent TTA models (e.g., AudioLDM, Tango, Tango 2) that utilize a single-strategy approach. Our goal in this initial study was to establish a robust foundation by systematically evaluating models designed specifically for text-to-audio generation. By benchmarking these models with consistent metrics, we aimed to create a controlled environment for meaningful comparisons (Sec. 5.3). It is worth noting that our focus in this work is to benchmark audio events relation modeling in TTA task, and we found all existing TTA models perform poorly on all proposed relations. So our introduced benchmark makes significant contribution to benefit TTA community to develop more advanced audio events relation aware frameworks.

2. Complex Scenarios and Extended Environments: We acknowledge that testing in more complex scenarios such as virtual reality (VR) or interactive environments could provide valuable insights into a framework's robustness. To accommodate multi-event VR audio or extended interaction-based tests, we can base on current relation and audio events corpus to involve more complex AR/VR context information for text description generation, and further design more complex techniques to generate the AR/VR aware audio. For example,

• Generate a dog barking audio, followed by cat meowing audio in a 3D spacious and empty room with pedestrian walking around.

• Create a vegetable chopping audio and toilet flushing audio simultaneously and each audio resembles well with the indoor environment's kitchen and restroom positions.

In conclusion, we appreciate the reviewer’s thoughtful suggestions and acknowledge that expanding our benchmarking to more complex, multimodal, and interactive environments will strengthen the framework. We view this as an ongoing project that will evolve to address these richer applications and extend the study’s reach to more diverse model types and use cases.

Q2: The description of the fine-tuning strategy is too general, lacking detailed information on training settings, parameter tuning strategies, and specific experimental procedures. The lack of details such as specific hyperparameter settings, number of iterations, and choice of loss function also weakens persuasiveness.

A2: We thank the reviewer for point out the lack of details in fine-tuning strategy. We use Adam optimizer with a learning rate of $3\times10^{-5}$, batch size of $16$, SNR gamma value of $5$. The training was performed for $40$ epochs on $4$ A$100$ GPUs. The rest of the hyperparameters were kept the same as base model. We will include these specific details in the revised version of the paper.