WavJourney: Compositional Audio Creation with Large Language Models

We would first like to thank the reviewer for the insightful review. Please find our responses to the questions in terms of system-level ablation studies below.

”How much does the system rely on the capabilities of LLMs? How would it perform if LLM is not GPT-4, but rather those open-source alternatives like Llama.”

We evaluated the system using both GPT-4 and Llama2-70B-Chat to investigate this dependency for the audio script writing. We prompted Llama2-70B-Chat to generate audio scripts from the storytelling benchmark. The generated audio scripts were fed into the script compiler, which validated their accuracy. We compared the Compilation Success Rate (CSR) of Llama2-70B-Chat with that of GPT-4 used in WavJourney. Detailed experimental setup are described in the Appendix A.7.1 in the revision.

In short, Llama2-70B-Chat initially had a low CSR of 10%, which then improved to 54% after the removal of hallucinated content, GPT-4 achieved a high CSR of 94%. These results indicated GPT-4's good performance in audio script writing and suggested that potential enhancements in open-source LLMs may be required to match GPT4’s capabilities in complex tasks such as audio script writing. We also show and discuss several errors made by Llama2-70B-Chat and GPT-4 in the revision.

”The script writer acts like an agent. There are numerous works and existing methods working on connecting LLM to perform various tasks, e.g., AutoGPT. But the authors didn’t compare against these agent-based system design.”

We appreciate the reviewer's insightful comments. Existing works, such as AutoGPT, focus on using LLMs as agents to perform various tasks, these typically involve using LLMs as deterministic task solvers (e.g., calling APIs, question answering). To the best of our knowledge, the use of LLM agents as creators of engaging, realistic media content remains largely unexplored. WavJourney specifically addresses this gap. WavJourney is designed for compositional audio creation, engaging LLMs in a specific and complex process. As WavJourney asks LLMs to create scripts with structures (creative) that serve for Python code generation (deterministic), which makes direct comparisons between WavJourney and AutoGPT challenging.

However, in WavJourney, we propose and design a hand-crafted script compiler to generate Python code from audio scripts. This approach is similar to previous work using LLMs to generate code for complex vision-language deterministic tasks via simple in-context prompting [1, 2]. To further validate our design, we conducted additional comparative experiments between a GPT-4-based script compiler and our proposed hand-crafted script compiler (see Appendix A.7.2 in the revision). The results demonstrate that our approach mitigates instability in script compilation and significantly improves computational efficiency. We believe this experiment provides additional insights into system-level design ablation studies.

Additionally, we acknowledge the importance of comparing our approach with several very recent LLM-based video creation agent designs, such as [3, 4]. We plan to explore this in our future research.

[1] Surís, Dídac, Sachit Menon, and Carl Vondrick. "Vipergpt: Visual inference via python execution for reasoning." arXiv preprint arXiv:2303.08128 (2023).

[2] Gupta, Tanmay, and Aniruddha Kembhavi. "Visual programming: Compositional visual reasoning without training." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[3] Lin, Han, et al. "Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning." arXiv preprint arXiv:2309.15091 (2023).

[4] Lian, Long, et al. "LLM-grounded Video Diffusion Models." arXiv preprint arXiv:2309.17444 (2023).

We sincerely thank you for your insightful feedback! Please see the following responses to the questions.

”what the basis of the metrics proposed in Audio Storytelling benchmark is”

The proposed metrics is inspired by expert-driven metrics design process [1]. Specifically, we develop these metrics with input from audio and product experts. These five metrics (creativity, relevance, emotional resonance, engaging, pace & tempo) allow for an assessment of audio storytelling that goes beyond traditional evaluations (e.g., Mean Opinion Score) to consider narrative complexity, music and sound design, and vocal delivery in a coherent and comprehensive manner. We have added these texts in the Section 4 in the revision.

[1] Shah, J.J., Smith, S.M. and Vargas-Hernandez, N., 2003. Metrics for measuring ideation effectiveness. Design studies, 24(2), pp.111-134.

”It’s not clear what the preference test is focusing on: the overall quality, or the adherence to the caption.”

The preference test is designed to evaluate the overall impression of the generated audio. This evaluation is supported by evidence from A/B testing [2, 3] a widely accepted method for comparing two versions of a variable to determine which performs better. In our case, the A/B tests consistently showed that listeners preferred audio that not only had high-quality sound but also closely matched the provided caption.

[2] Siroker, D., & Koomen, P. (2013). A/B testing: the most powerful way to turn clicks into customers. John Wiley & Sons.

[3] A/B Testing: https://en.wikipedia.org/wiki/A/B_testing

”I wonder if the authors ran the GT audio through the same audio quality enhancement model for evaluation.”

We didn’t run VoiceFixer for GT audios as it will suppress all non-speech sounds.

”In table 3, the 6th column seems to have the wrong entry bolded. GT audio has a higher REL score than WavJourney audio.”

Thanks for your suggestion! We have fixed it in the revision.

“The fact that the results are so one-sided for Clotho dataset gives the impression that the result is not too signification (even though the statement is true), or that there is something wrong in one of the two datasets.”

We would like to highlight that the improvements achieved by WavJourney over AudioGen are considerable (WavJourney uses AudioGen), as evidenced by both subjective and objective results on the Clotho dataset, and subjective results on the AudioCaps dataset, including the additional subjective tests (ablation studies, Turing test) we conducted, as reported in the Appendix A.5 and A.6 in the revision.

WavJourney achieved negative objective results on AudioCaps dataset compared with AudioGen. However, we argue that objective evaluation metrics, such as FAD, may not always effectively capture the performance of text-to-audio generation systems. This viewpoint aligns with conclusions drawn in previous work [4], which suggests that systems with good subjective evaluation results may not always achieve high FAD scores.

[4] Choi, Keunwoo, et al. "Foley sound synthesis at the dcase 2023 challenge." arXiv preprint arXiv:2304.12521 (2023).

”I would also be interested in seeing a Turing-test type question asking users if they can tell whether the audio is generated or not.”

Thanks for your constructive suggestions! It’s interesting to see if WavJourney can generate audio that is indistinguishable from real-world audio in human perception, considering its good results achieved on text-to-audio generation benchmarks. To do so, we conduct both Turing test on AudioCaps and Clotho dataset. To mitigate any potential bias that might skew listeners’ perceptions, we provide human participants with audio clips only (without texts). The experimental setup and results are detailed in the Appendix A.6 in the revision. The conclusion we made is, WavJourney outperforms AudioGen and AudioLDM systems in Turing test score, but can not pass the Turing test, indicating that there is a gap between WavJourney generated synthetic and real audio as judged by human listeners. The discrepancy between the results of the Turing test and other subjective metrics (e.g., OVL, REL, preference-based test) on AudioCaps may be attributed to text-to-speech generated speech. Although this generated speech can positively influence subjective metrics, it may be easily to distinguish due to human's sensitive perception of speech.

”Based on the samples shared, I felt all the GT audio to be more natural sounding than WavJourney. I would encourage the authors to also showcase examples where WavJourney was rated higher than GT by the listeners.”

Thanks for your suggestions! We have uploaded more results to the project page (https://anonymous.4open.science/w/WavJourney_Anonymous/).

We appreciate the reviewer's constructive and clear feedback! Please see the following responses to the concerns & questions raised in the review.

“The author points out that WavJourney outperforms previous state of the art methods in both subjective and objective evaluations. However, on the objective indicators of the AudioCaps dataset, WavJourney's performance is lower than AudioGen's.”

As we described in Section 5.2, objective evaluation metrics (e.g., FAD) may not always be an effective way to evaluate the performance of text-to-audio generation systems. This is consistent with conclusions drawn from previous work, which suggests that some systems with good subjective evaluation results may have poor FAD scores [1]. We believe that the subjective evaluation results detailed in Section 5.2 (OVL, REL, and Preference-based test), as well as the additional results provided in Appendix A.5 and A.6 of the revision (ablation studies and Turing tests), are sufficient to demonstrate that WavJourney outperforms AudioGen on the AudioCaps dataset. Additionally, more examples can be found on the project page at https://anonymous.4open.science/w/WavJourney_Anonymous/.

[1] Choi, Keunwoo, et al. "Foley sound synthesis at the dcase 2023 challenge." arXiv preprint arXiv:2304.12521 (2023).

”Considering that WavJourney uses AudioGen for text-to-audio synthesis, the phenomenon of superiority over AudioGen on objective metrics of the clotho dataset requires further analysis. Is this due to the introduction of a text-to-speech synthesis model by WavJourney, or is it due to WavJourney's use of LLM to improve its understanding of long captions?”

We would like to thank the reviewer for pointing out the importance of conducting ablation studies to explain the superiority of WavJourney over AudioGen. We have conducted ablation studies on both the AudioCaps and Clotho datasets. We manually categorized the test audio clips into two splits: 1) Speech-Inclusive, which includes speech and sounds, and 2) Non-Speech Exclusive, which consists only of non-speech elements. The results from the Non-Speech Exclusive evaluation can tell us if the improvements come from the LLM, while the results from the Speech-Inclusive category can show the combined effectiveness of the LLM and TTS models.

Detailed results and analysis are provided in Appendix A.5 of the revision. In conclusion, WavJourney consistently outperforms AudioGen in OVL and REL metrics for both Speech-Inclusive and Non-Speech Exclusive categories on the AudioCaps and Clotho datasets. More significant enhancements were observed in the Speech-Inclusive category. These results suggest that WavJourney's advancements are derived from both the LLM and TTS models.

“The experimental part of the article is rather inadequate, the analysis of composite audio generation capabilities of the method is limited to the scene of storytelling”

We would like to clarify that both the AudioCaps and Clotho datasets consist of all speech, music, and sound effects elements. Comparative experiments with state-of-the-art end-to-end generative models such as AudioGen and AudioLDM (as well as ground truth audios) on text-to-audio generation tasks also provide valuable insights into the advancement of our proposed compositional approach. We have also included additional results (e.g., Turing tests on text-to-audio and system-level ablation studies) in revision. We hope that these additional results and analysis will address your concerns regarding the adequacy of our experiments.

“More examples of speech content control should be provided. Whether users can control the speech content in the story? It’s important if you want to involve text-to-speech function”

We have provided an additional study on a radio-play-based storytelling case focusing on voice content control, as shown in Appendix A.9 in revision. In the example, we can control the content of the speech by "topic" or "specific sentences" and the type of language in the multi-turn conversation.

The analysis of interactive audio creation in multi-round dialogues is also oversimplified, making an inadequate demonstration of the core capabilities of the model.”

We agree that interactive audio creation in multi-round dialogues is important. However, the focus of this work is more on the compositional generation of audio with sound effects, speech, and music using LLMs. While existing end-to-end methods often failed in providing a solution for this task - which is important and useful in real-world audio production applications. In future work, we will explore the interactive capability of WavJourney further.

Thank you for the useful feedback and all these valuable questions! Please see the following responses to the concerns & questions below.

”The evaluations are limited to audio captioning tasks. The evaluation of speech and music qualities is missing. How well does the WavJourney perform in music and speech benchmarks?”

We would like to clarify our focus. Our target is general compositional audio creation task, which differs from traditional audio generation with task-specific conditions (e.g., text-to-speech). We focus on the creation of storytelling audio from text, encompassing speech, music, and sound effects, rather than the generation of individual components such as speech and music. As acknowledged by Reviewer jF76: this task opens up a huge space for AI-powered general audio creation.

For evaluation of text-to-audio generation, we have conducted comprehensive experiments on both the AudioCaps and Clotho datasets. Both the AudioCaps and Clotho datasets consist of all speech, music, and sound effects elements. These experiments include objective evaluations (e.g., FAD, KL, IS) and subjective evaluations (e.g., REL, OVL, preference-based test, Turing-test). Furthermore, we have constructed a storytelling benchmark with diverse genres and conducted comparative experiments with the AudioGen and AudioLDM systems (as updated in the revised Table 4). We believe these extensive evaluations adequately demonstrate the effectiveness of WavJourney in a variety of scenarios. We appreciate your suggestion to consider additional benchmarks in future work.

”As an AI audio creation tool, it would be nice to enable the editing of existing audio content. It happens a lot in movie or TV creation. In this case, the framework needs to take the audio as an input as well. However, it seems like the current framework cannot support that.”

We appreciate your suggestion to enable the editing of existing audio content, a feature that would indeed be valuable in applications such as movie or TV creation. This work focuses on the compositional audio creation from text, which is a novel and relatively unexplored task to the best of our knowledge. We will study the compositional audio editing task in the future.

“What are some failure cases”?

Some failures are that sometimes audio generative models (e.g., text-to-audio) produce unexpected results.

”What is the exact time cost to produce an audio sample?”

To analyze the time cost, we randomly selected 20 text captions from the AudioCaps test set and calculated the average inference time required by WavJourney and AudioGen to generate 10-second audio clips. For WavJourney, we report the time costs associated with script writing and audio generation. The testing was conducted on a machine equipped with a single NVIDIA GeForce RTX 2080 Ti GPU. The results are shown in Table below and added to the revision. In practice, by carefully optimizing the use of computational resources (e.g., parallel inference) for WavJourney, it is possible to achieve high-quality synthetic audio generation while minimizing the time required for inference, which we leave as future work.

We thank the reviewer for the constructive and clear feedback, which will undoubtedly enhance our work. In response to the feedback, we have included new results and analysis & insights in the revision:

• A detailed description of the script compiler implementation can be found in Appendix A.4.
• We have conducted experiments and comparative analysis of GP4-based script compiler and proposed hand-crafted script compiler, as shown in Appendix A.7.2. The results demonstrate that our approach effectively mitigates instability in script compilation and greatly improves computational efficiency.
• We have included quantitative experiments, analysis, and a study of error cases on the use of open-source LLMs such as Llama-70B, as described in Appendix A.7.1.
• Ablation studies and Turing tests on text-to-audio generation can be found in Appendices A.5 and A.6, respectively.
• We have also further explored interactive audio creation in multi-round dialogues, as shown in Appendix A.9.

Regarding the storytelling creation task:

• Our motivation for introducing this task is to address real-world audio production scenarios. In practice, people often begin with an abstract or simple concept rather than a detailed idea (description) for each component (e.g., speech, music, audio). This is a common yet challenging scenario that existing audio generative models struggle to handle effectively. With recent advances in LLMs, we now have the potential to automate this task. Our aim is to establish the task with appropriate evaluation metrics and benchmark baseline performance for future research.
• We compared WavJourney with AudioLDM and AudioGen systems and found that WavJourney outperforms them in all subjective metrics. To ensure consistency during subjective evaluation, we discarded the results in the submission version and re-evaluated the results. The updated experimental results can be found in Table 4 of the revision. We have also uploaded the generated audios of these baselines to the project page for listening (https://anonymous.4open.science/w/WavJourney_Anonymous/).
• We appreciate your concern about the source of the impressions in our evaluation. Our proposed five metrics, inspired by the expert-driven metrics design process outlined in Shah et al. [1], are designed to capture the holistic impression of the generated audio content, which includes music, sound effects, and speech. Specifically, our subjective evaluation protocol is developed with input from audio and product experts, considering narrative complexity, music and sound design, and vocal delivery in a coherent and comprehensive manner. Therefore, the impression captured by each metric are derived from the overall audio experience, rather than any single component. We have updated the descriptions in the Section 4 in the revision.

[1] Shah, J.J., Smith, S.M. and Vargas-Hernandez, N., 2003. Metrics for measuring ideation effectiveness. Design studies, 24(2), pp.111-134.