Listen, Think, and Understand

Dear Reviewer wB1w,

Thank you for the positive feedback and all these very valuable and constructive questions/suggestions! Let us respond to your questions point by point.

Question 1 - Additional Audio-Conditioned Experiments

For the open-ended questions, this work seems to focus mainly on solely LLMs assisted question answer generation. It is not intuitive to understand to what extent the model relies on the input audio versus on the common sense knowledge that is already encoded in the LLMs. It would be great to define and identify beyond current close-ended tasks with new lower level tasks which really require using the audio, such as counting sound events, ordering of events, etc. These type of questions might already exist in the proposed dataset, it would provide more insights to dive deeper into those.

We totally agree with the reviewer on this point! And this is the reason why we rigorously evaluate the model performance on closed-ended tasks while the main novelty on open-ended tasks. In the paper, Appendix K, we conduct a probe experiment to check if LTU can really understand the sound concepts rather than associates the acoustic concept with a sound class. In our training data, samples of the ambulance siren class are always described as high pitched. To check if LTU can disentangle the concept of a high-pitch and sound class of ambulance sirens. We manually lower the pitch (with librosa.pitch_shift) of 53 evaluation audios of the ambulance siren class and check LTU's output to the question What is the pitch? on these audios. As shown in Figure 2 of the paper, LTU's prediction aligns well with the actual pitch, indicating it indeed learns the concept of pitch rather than just associating it with a specific sound class.

Question 2 - Temporal Analysis Experiments

It would be great to define and identify beyond current close-ended tasks with new lower level tasks which really require using the audio, such as counting sound events, ordering of events, etc.

In the temporal analysis paragraph in 3.1, how are the understanding of order of sounds evaluated?

In 5.2.1, LTU can answer follow-up questions about the details, there is an example where LTU shows that the bell rings three times, are there other examples showing the counting capabilities? Is it possible to further quantify this?

We sincerely appreciate the reviewer's suggestion to explore the temporal analysis capability of the model, and in response, we are proceeding with the following new experiments. Specifically, we conduct experiments on the audio event order recognition task (seen in the closed-ended training stage) and audio event counting task (unseen in the closed-ended training stage) with ESC-50, a dataset not being used in training. For both experiments, we only include the cases in which LTU follows the instruction.

1. Recognizing the Order of Sound Events

We create a test set where each test sample consists of two audio samples from different categories of the ESC50 dataset, with no temporal overlapping. We then ask LTU to predict the order of the sound events with the prompt Which sound begins and ends first?. We use regular expressions and cosine similarity (based on gpt-text-embedding-ada) to interpret the raw LTU output. For example, for the LTU raw output the applause starts first, while the footsteps follow in a rhythmic pattern afterward., we first extract applause and footsteps follow in a rhythmic pattern using regular expressions. Then we compare the cosine similarity between the text embedding of the ground truth clapping with the text embeddings of applause and footsteps follow in a rhythmic pattern, respectively. Since the cosine similarity between the text embedding of the ground truth clapping and the LTU prediction applause is higher, we count this sample as correct.

We evaluate LTU on 550 such samples, obtaining 390 correct and 160 incorrect answers, achieving an accuracy of 70.9%. This shows that LTU can recognizes the order of sound events with a reasonable accuracy.

Dear Reviewer oFtB,

Thank you so much for taking the time to read our paper and providing very valuable comments and suggestions regarding the model architecture and comparing LTU with Pengi. Please see the following for our point-by-point response. Please also note that we have added substantially more materials in the revised paper (please check the updated PDF file).

Question 1. Audio-Text Alignment

There seems to be a lack of thought about model structure and loss. It is just a combination of the strong pretrained LLM and the existing audio encoder, AST. Utilizing strong pretrained LLM with multimodal inputs has an alignment issue. Thus, while concatenating the audio feature and the text feature can introduce desired performance, there could be some advancements not just combining pretrained audio model and LLM. For example, BLIP-2 [1] solves misalignment between text and image using 3 losses: 1) Image-Text matching, 2) Image-Grounded Text generation, and 3) Image Text Contrastive Learning. The simple combination of the audio model and LLM does not seem to be novel.

We understand the reviewer’s concern about the architecture and are grateful for the highly constructive suggestion of adopting BLIP-2-like alignment training. Just as the reviewer points out, we connect the audio encoder with the large language model with only a linear layer and before LTU training, this audio encoder has not been trained for any audio-text alignment tasks.

In our design, the AST's output serves as soft prompts for the large language model. According to [1,2], these (multimodal) soft prompts do not necessarily need to be in the text embedding space. In practice, some multi-modal large language models in the vision community such as PaLM-E, LLaVA, and Kosmos use a similar architecture and achieve good performance. One reason for not allowing us to directly use Q-Former in BLIP-2 is that the learnable queries are not guaranteed to be temporally aligned, which is important for audio tasks.

We sincerely appreciate the reviewer's suggestion to explore an additional training stage for audio-text alignment, and in response, we are proceeding with the following new experiments.

[1] Lester, Brian, et al. The Power of Scale for Parameter-Efficient Prompt Tuning. EMNLP 2021.

[2] Driess, Danny, et al. PaLM-E: An Embodied Multimodal Language Model. 2023.

In the new experiment, we add an additional audio-text alignment training stage before LTU training. We test two types of alignment modules: one is the linear layer used in the original LTU, and the other is a Transformer layer, which allows for more model capacity in the alignment. We train the alignment module on 1.6M audio-text pairs (comprising audio labels or captions) for 10 epochs, using a batch size of 1536 distributed over 4 GPUs, with 384 samples on each GPU. We start with an initial learning rate of 1e-3 and halve it after each epoch. During this stage, we keep the audio and LLaMA text encoders frozen and only train the alignment module. We use the following loss:

where $\lambda = 10$ is used to balance the scale between MSE loss and contrastive loss, $e^t$ and $e^a$ is the text and audio embeddings, respectively, $N = 384$ is the batch size, $s_{i,j}$ is the similarity score calculated as the dot product of normalized text embedding $e^t_i$ and audio embedding $e^a_j$, and $\tau = 0.05$ is the temperature. This loss aligns both the scale and direction of the audio and text embeddings.

Question 1. Value of OpenAQA (cont.)

In our paper, we conduct experiments to compare models trained with closed-ended data only (akin to Pengi) with those trained with both closed-ended and open-ended data (default LTU). We show results in Table 5 right and Table 14 of the paper. These can be summarized as the following table.

These results demonstrate that including the open-ended section of OpenAQA leads to improvements in performance for both closed-ended and open-ended tasks. Specifically:

• For closed-ended tasks, the inclusion of open-ended training results in enhanced performance (from 47.3 to 50.3), indicating that learning to understand can bolster perception ability.
• For open-ended tasks, the model trained only with closed-ended QAs shows a markedly lower instruction following rate compared to the one trained with both types of data (22.5% vs 96.9).

This evidence justifies the importance of utilizing GPT to generate open-ended QAs in our training regimen. We hope our explanation and experiments could address the reviewer's question.

Question 2. Compare with Pengi

We thank the reviewer for the comment and for pointing us to the pretrained Pengi checkpoint. We were able to download the checkpoint and run experiments based on it. We agree that comparing LTU with concurrent work would better place LTU in the context. The reviewer is correct that Pengi is better on some closed-ended benchmarks (e.g., ESC-50), but as we will show below, on average, LTU is slightly better than Pengi on closed-ended tasks and significantly better for open-ended tasks.

We compare LTU with Pengi in three aspects.

1. Technical Similarity and Difference

Similarity:

• Architecture-wise, both Pengi and LTU connects an audio encoder with an autoregressive language model. Both models can generate text from the given audio and text prompt.
• Performance-wise, both Pengi and LTU can do zero-shot predictions on unseen datasets. On closed-ended tasks, Pengi and LTU perform similarly.

Difference

• Motivation: Pengi focuses on transfer learning, i.e., using a single model for 8 audio perception tasks, while LTU focuses on unifying audio perception with understanding. Unlike LTU, Pengi is not designed for audio understanding and answering free-form open-ended questions from users.
• Language Model Scale: Pengi employs GPT2-Base (124M parameters) as its language model, whereas LTU uses LLaMA (7B parameters), which is over 50x larger than GPT2-Base. Additionally, LLaMA is trained with significantly more data than GPT2. Scaling up the language model substantially enhances LTU's understanding capabilities.
• Training Data: Pengi is trained on multiple audio datasets, and directly uses the original text-form labels as answers for AQA. This approach is similar to the closed-ended part of our OpenAQA dataset. However, OpenAQA also includes another 3.7M open-ended AQAs generated with the proposed audio instruction generation method (not generated from a template). The open-ended portion of our training data is pivotal in enabling LTU to answer any free-form open-ended questions.
• Training Curriculum: To better unifying audio perception and generation, LTU adopts a unique perception-to-understanding training curriculum.
• Open-Ended Evaluation: LTU undergoes evaluation for both closed-ended and open-ended tasks, incorporating both subjective and objective evaluations. Conversely, Pengi is evaluated solely with closed-ended tasks. Please note that the definition of an open-ended task differs between the Pengi paper and this paper; Pengi considers audio captioning as an open-ended task, while we categorize them as closed-ended tasks.

Dear Reviewer 2qgj,

We sincerely thank you for your very positive feedback and are gratified to recognize from your insightful comments a deep comprehension of our paper's content and objectives. Let us respond to your questions point by point.

Question 1. Value of OpenAQA

It is not clear to me what the value of OpenAQA dataset is on top of of the textual metadata available with most of these datasets. It would have been fascinating if the authors were to do an ablation study to train their model in the format of (Audio, Text) --> Text format - something similar to what PENGI does. Use the text data that comes with each dataset as is and compare this with : a) Using OpenAQA (current setting) b) Augmenting the original audio text pairs with OpenAQA.

We thank the reviewer for the question. As the reviewer pointed out in the summary, the OpenAQA consists of two parts: the closed-ended part (the question is paraphased by GPT but the answer is generated with a rule based algorithm based on ground truth labels), and the open-ended part.

The closed-ended part of OpenAQA is actually very similar to the Pengi training data:

• Both the OpenAQA closed-ended dataset and the Pengi dataset use a template to formulate answers. For instance, in Pengi, the format for audio event classification is {event a}, {event b},... (e.g., Ambulance, Traffic noise, Accelerating), while in OpenAQA, it is Label: {event a}, {event b} (e.g., Label: Ambulance, Traffic noise, Accelerating).
• The differences between the OpenAQA closed-ended dataset and the Pengi dataset are twofold:
  • Pengi has only one question (input prompt) for each task, such as this is a sound of for audio event classification. In contrast, the OpenAQA closed-ended dataset paraphrases the question multiple times (about 100 times) using GPT-3.5-Turbo. Example questions for the audio event classification task include Generate audio labels, Produce audible tags, and Generate audio annotations. This approach enhances LTU's robustness to question perturbations. The Pengi paper (Appendix C and Table 17) reports varying model performance with different input prompts. Our design aims to mitigate this issue.
  • OpenAQA uses GPT-3.5-Turbo to generate acoustic feature descriptions, which are then utilized to train the LTU model. This enhances its ability to recognize acoustic features alongside sound classes, allowing LTU to generalize predictions for unseen sound classes based on their acoustic features. An example is high-pitched, rapidly changing frequency for the ambulance class, leading to an answer like high-pitched, rapidly changing frequency → Ambulance. Pengi does not have this design.

However, these differences are minor. Essentially, the closed-ended part of OpenAQA and the Pengi training data are generated using very similar methods.

The major difference between LTU and Pengi is the open-ended part of OpenAQA. Here, both the question and answer are generated by GPT, as opposed to a rule-based algorithm or template. We present examples in Table 2 of the paper:

Question: What can be inferred from the fact that traffic noise fades while the ambulance siren echoes?

Answer: It can be inferred that the ambulance is approaching or near the location where the audio clip was recorded, and that the traffic is yielding to the emergency vehicle.

Question: What mood or atmosphere does the audio clip convey?

Answer: The audio clip conveys a sense of urgency, due to the sound of the ambulance siren.

The answers to open-ended questions are not taken directly from the text-form labels of the original dataset, but are instead derived by GPT through background knowledge and reasoning, based on the ground truth labels. As shown in Figure 3 of the paper, over 95% of the questions in the 3.7M dataset are unique, covering a large portion of the question space. This enables LTU to answer any question, not just ones it has previously seen. In contrast, when Pengi encounters a new question, it struggles to respond appropriately, for example:

Question: What can be inferred from the combination of mechanical, male speech, and light engine sound events?

Pengi: two

LTU: It is likely that there are multiple machines or vehicles in operation nearby.

In summary, while the closed-ended part of OpenAQA is very similar to Pengi's training data, with only minor differences, the key distinction between OpenAQA and Pengi lies in OpenAQA's open-ended section, where answers are not based on original labels. It is this open-ended aspect that enables LTU to effectively address open-ended questions.

Dear Reviewer 1UHZ,

Thank you for the positive feedback and all these very valuable and constructive questions/suggestions! Let us respond to your questions point by point.

Question 1 - Impact of model size

The performance of this model is closely related to both the AST encoding frontend and the LLaMA model's performance. Ablation studies on the varying parameter counts of these two components would be valuable, if possible.

We thank the reviewer for this very constructive suggestion. We conduct the following new experiments to study the impact of model size.

1. Impact of the size of the large language model

Throughout the original paper, we use LLaMA model consisting of 7 billion parameters (LLaMA-7B) as the LTU language model. In this revision, we conduct experiments on LTU based on LLaMA with 13 billion parameters (LLaMA-13B).

As indicated in the table above, we observe a slight decrease in audio classification performance with the increasing size of the LLaMA model while the captioning performance improves slightly with larger model sizes. There is no significant difference in the instruction following rate for pure language and audio open-ended questions. However, we note that the 13B model tends to provide more rational answers due to its more extensive knowledge base, as exemplified in the following sample:

Question: How would the audio clip be different if the power tool was being used in a larger or open space?

LTU-7B: The sound of the power tool would likely be louder and more noticeable in an open space. (wrong)

LTU-13B: If the power tool was being used in a larger or open space, the sound of the power tool would likely be more diffuse and less intense than it is in this clip. (correct)

2. Impact of the size of the audio projection module

Since large language model training is expensive, we can only train a limited number of models with our computational resources. Thus, we started with the strongest audio encoder we could find when we prepared this work (AST with CAV-MAE pretraining and finetuning). A smaller audio encoder could lead to similar performance, but unlikely to outperform the larger audio encoder. In addition, the number of parameters of our audio encoder is just about 1.3% of the LLaMA language model.

The original LTU utilizes a linear layer to project audio embeddings to the large language model input, which might be overly simple. Therefore, in this revision, we consider to use a Transformer layer instead. To ensure a fair comparison, both the linear and Transformer layers have been pretrained using audio-text contrastive learning, as detailed in Appendix F.

As shown in the table, expanding the layer capacity results in decreased performance, which can be attributed to the introduction of a larger number of randomly initialized parameters that complicates the training process. We thus conclude that the original configuration of a linear layer is a good design choice.

We thank the reviewer again for the insightful comment, we have added these discussions to Appendix E and F of the paper.