Acoustic Prompt Tuning: Empowering Large Language Models with Audition Capabilities

Thank you for your valuable feedback! We address your comments one by one as follows:

[Maximum input sequence length]

Although AudioMAE ingests fixed-length waveforms, the APT-LLM supports audio clips of variable length as input. Specifically, we will truncate or pad audio to 10 seconds length and generate acoustic tokens for each 10-second audio clip. After that, we will concatenate the acoustic tokens together so that the APT-LLM can digest varying-length audio clips.

Regarding “ without imposing any constraints on input sequences”, we replace it with “impose zero constraints on input format” to avoid potential misunderstanding.

[Benefits of combining multitask training with in-context learning]

Thanks for your valuable feedback. We have updated the contribution part for clarity.

In terms of the benefits of the combination, as shown in the Table 3, fewer audio-text pairs and parameters are introduced to the overall network compared to the existing approaches while APT still yielded a comparable performance compared to other audio-language models.

[Performance on audio tagging and captioning]

We notice that compared to the common classifiers, there is a performance drop due to the difficulty for the open-set issue and text matching. For example, when CLAP, a open-set classifier pretrained with contrastive loss, is used for evaluation, it calculates the similarity with 527 classes whereas APT predicts an appropriate token out of 32k tokens. Another issue is how to match the predicted answers with the ground-truth labels. Suppose APT outputs “rat” and we get the class of “mouse” in the ground truth. In that case, a proper text matching approach is required for a fair comparison. During evaluation on audio classification tasks, we used the APT’s checkpoint from audio-text-alignment pretraining as the text encoder. However, we did find that using different text encoders to evaluate the same checkpoint impacts the classification values a lot. We are planning to apply some discriminative text encoders, such as CLIP text encoder and ChatGPT Ada model, for the evaluation, and we will add the discussion on text matching to the Appendix.

Regarding the audio captioning task, training on multiple datasets does not improve the performance on a specific dataset because the model cannot fit the dataset-specific vocabulary. In addition to the general-purpose audio [1], we found the same phenomena in other domains [2,3]. Following [3,4], we have conducted additional experiments to finetune APT on both Clotho and AudioCaps datasets for two epochs only. We compared the finetuned APT with expert models trained on the same datasets. As shown in Table 4, the APT yielded the best scores on both tasks.

[Why LLM?]

It is very tempting to build up foundational models by prompting LLMs: a) LLM can understand diverse texts as input such that we don’t have to design a special token for each task, e.g., [5]; b) LLMs have the ability to reasoning, facilitating us to explore more audio tasks, such as natural language audio reasoning; c) since some works [6-7] support that GPT/transformer can be a good in-context learner, LLMs can help the model figure out the correlations among input features in the few-shot settings. Given LLMs can bring the above advantages, it would be tempting to explore how to adapt LLMs in the audio domain.

[Architecture design]

Regarding “Generating a fixed number of tokens is not so much for audio input sequences due to their variable input lengths”, as we mentioned in “[Maximum input sequence length]”, it is actually 10-second audio clip corresponds to a fixed number of tokens. We have updated the manuscript for a better understanding.

Regarding the triplet losses in the audio-text alignment pretraining, we did a few preliminary studies on the pretraining and decided to use the combination for a better alignment.

[Size of Vicuna]

We used Vicuna 7B throughout the work. We have clarified this in the paper.

[Strength of APT than other approaches]

APT highlights three features compared to the other approaches: a) Training efficiency: as shown in Table 1, APT uses fewer training data and roughly 100M learnable parameters while yielding a competitive performance than other approaches; b) Audio understanding and reasoning across recordings: APT improves the framework of audio language model by juxtaposing acoustic and text embeddings as inputs. This facilitates APT digesting multiple audio clips, enabling it to tackle more audio tasks; c) As a audio adapter, APT extends LLM/VLM without optimizing the parameters of language models and losing their original abilities as an LLM/VLM. Our experiments show that APT can work with VLM without any audio-visual learning.

[1] PREFIX TUNING FOR AUTOMATED AUDIO CAPTIONING

[2] LLARK : A MULTIMODAL FOUNDATION MODEL FOR MUSIC

[3] SIMVLM: SIMPLE VISUAL LANGUAGE MODEL PRETRAINING WITH WEAK SUPERVISION

[4] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

[5] AudioPaLM: A Large Language Model That Can Speak and Listen

[6] Why Can GPT Learn In-Context? Language Models Secretly Perform Gradient Descent as Meta-Optimizers

[7] Transformers learn in-context by gradient descent

P.s. Sorry, we have to make the reference a new comment due to the limited characters.

Thank you for your valuable feedback! It helps us to improve the quality of the paper. We address your comments one by one as follows:

[Experiment on audio tagging and captioning]

For the audio classification, we found that there is a performance drop due to the difficulty for the open-set problem and text matching. For example, when CLAP, a open-set classifier pretrained with contrastive loss, is used for evaluation, it calculates the similarity with 527 classes whereas APT predicts an appropriate token out of 32k tokens. It is harder for generative models to carry out multi-label classification where the threshold is essential to predict the presence of a sound event.

Another issue is how to match the predicted answers with the ground-truth labels. Suppose APT outputs “rat” and we get the class of “mouse” in the ground truth. In that case, a proper text matching approach is required for a fair comparison. During evaluation on audio classification tasks, we used the APT’s checkpoint from audio-text-alignment pretraining as the text encoder. However, we did find that using different text encoders to evaluate the same checkpoint impacts the classification values a lot. We are planning to apply some discriminative text encoders, such as CLIP text encoder and ChatGPT Ada model, for the evaluation, and we will add the discussion on text matching to the Appendix.

Regarding the audio captioning task, training on multiple datasets does not improve the performance on a specific dataset because the model cannot fit the dataset-specific vocabulary. In addition to the general-purpose audio [1], we found the same phenomena in other domains [2,3]. Following [3,4], we have conducted additional experiments to finetune APT on both Clotho and AudioCaps datasets for two epochs only. We compared the finetuned APT with expert models trained on the same datasets. As shown in Table 4, the APT yielded the best scores on both tasks.

[Background of the audio-visual learning task]

Thanks for your feedback. To improve the readability, we have added the introduction on both BLIP-2 structure and the audio-visual task to Section 4.4.

[Experiment on few-shot learning]

Due to the limited computational resource, we didn’t extend the contextual windows to meet the few-shot settings. We are now re-training the models with a longer max length of sequence and will update the paper accordingly.

[missing tokenizer]

Thanks for pointing that out. e have now updatedFigure 1.

[Answer to Q2]

Actually, the LLM’s embedding layer was not removed in APT-LLM. Instead, we truncated the input sequence as per positions of audio indicators and concatenated them again once acoustic/text tokens have been encoded (separately).

Regarding the reason why we froze the LLM, the motivation of this work is to extend the existing LLMs/VLMs to the audio domain and prevent it from losing its original abilities as an LLM/VLM. To this end, instead of adjusting the LLM, we learn an audio aligner to generate soft prompts, conditioned on audio-text embeddings, for the language model. While finetuning LLMs with PEFT will boost the overall performance, we argue that it may jeopardise the LM performance on original tasks, even causing catastrophic forgetting. By tuning audio aligner only, we build up an audio language model with fewer parameters and fewer training samples while yielding a comparable performance on each understanding task compared to other audio foundational models.

[the NLAR task]

Thank you for pointing that out. We will add a more detailed introduction to the Appendix A.3.

[Missing legend in Fig. 3]

Thank you for your comment. We will update the revised one in the latest version of the manuscript.

[1] PREFIX TUNING FOR AUTOMATED AUDIO CAPTIONING

[2] LLARK : A MULTIMODAL FOUNDATION MODEL FOR MUSIC

[3] SIMVLM: SIMPLE VISUAL LANGUAGE MODEL PRETRAINING WITH WEAK SUPERVISION

[4] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

Thank you for your valuable feedback! It is very helpful to improve the quality of the paper. We address your comments one by one as follows:

[Experiments on common audio tasks]

We notice that compared to the common classifiers, there is a performance drop due to the difficulty for the open-set issue and text matching. For example, when CLAP, a open-set classifier pretrained with contrastive loss, is used for evaluation, it calculates the similarity with 527 classes whereas APT predicts an appropriate token out of 32k tokens. Another issue is how to match the predicted answers with the ground-truth labels. Suppose APT outputs “rat” and we get the class of “mouse” in the ground truth. In that case, a proper text matching approach is required for a fair comparison. During evaluation on audio classification tasks, we used the APT’s checkpoint from audio-text-alignment pretraining as the text encoder. However, we did find that using different text encoders to evaluate the same checkpoint impacts the classification values a lot. We are planning to apply some discriminative text encoders, such as CLIP text encoder and ChatGPT Ada model, for the evaluation, and we will add the discussion on text matching to the Appendix.

Regarding the audio captioning task, training on multiple datasets does not improve the performance on a specific dataset because the model cannot fit the dataset-specific vocabulary. In addition to the general-purpose audio [1], we found the same phenomena in other domains [2,3]. Following [3,4], we have conducted additional experiments to finetune APT on both Clotho and AudioCaps datasets for two epochs only. We compared the finetuned APT with expert models trained on the same datasets. As shown in Table 4, the APT yielded the best scores on both tasks.

[Advantages of foundational models]

Audio foundational models highlight in two-fold:

Fast adaptation: Audio foundational models fit a new dataset-specific scenario promptly by leveraging task homogeneity. For instance, rather than learning from scratch, a foundational model has already done a very good job on feature extraction and thus learns to understand the abstract concepts only. This can be supported by our experiments: when learning from the natural language audio reasoning task, it only requires roughly 1-hour audio data to train APT as it has already learned to capture acoustic patterns from a series of understanding tasks.

Improved performance: Task diversity is important for audio foundational models to learn better audio feature representation compared to other models. As shown in Table 4, the APT-LLM yields the best results on both datasets, outperforming the state-of-the-art models.

[Comparison between AudioPALM and SpeechLM]

AudioPALM and SpeechLM are great works bridging speech and text modalities. They expanded textual vocabulary to the speech-text one as both modalities share a basic unit - phonemes. They then mix the multi-modal tokens together and thus succeed in cross-modality transition. However, general-purpose audio language models feature the challenge that the basic units, namely sound events, are unstructured, sparsely distributed and often overlapping, sharing very little common place with phoneme-level text units. Moreover, the tasks solved in this work need more discriminative information compared to speech-text tasks, calling for a robust audio encoder and text-audio alignment.

[Regarding data curation]

Thanks for pointing that out. We actually manually cleaned the data after curating them automatically. We will add this content to the Appendix.

We appreciate your valuable feedback. We will update the content we have discussed above to the manuscript.

[1] PREFIX TUNING FOR AUTOMATED AUDIO CAPTIONING

[2] LLARK : A MULTIMODAL FOUNDATION MODEL FOR MUSIC

[3] SIMVLM: SIMPLE VISUAL LANGUAGE MODEL PRETRAINING WITH WEAK SUPERVISION

[4] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

Thank you for your enlightening feedback! We address your comments one by one as follows:

[Performance on certain audio tasks]

We notice that compared to the common classifiers, there is a performance gap due to the difficulty for the open-set problem and text matching. For example, when CLAP, a open-set classifier pretrained with contrastive loss, is used for evaluation, it calculates the similarity with 527 classes whereas APT predicts an appropriate token out of 32k tokens. Another issue is how to match the predicted answers with the ground-truth labels. Suppose APT outputs “rat” and we get the class of “mouse” in the ground truth. In that case, a proper text matching approach is required for a fair comparison. During evaluation on audio classification tasks, we used APT’s checkpoint from audio-text-alignment pretraining as the text encoder. However, we did find that using different text encoders to evaluate the same checkpoint impacts the classification values a lot. We are planning to apply some discriminative text encoders, such as CLIP text encoder and ChatGPT Ada model, for the evaluation, and we will add the discussion on text matching to the Appendix.

Regarding the audio captioning task, training on multiple datasets does not improve the performance on a specific dataset because the model cannot fit the dataset-specific vocabulary. In addition to the general-purpose audio [1], we found the same phenomena in other domains [2,3]. Following [3,4], we have conducted additional experiments to finetune APT on both Clotho and AudioCaps datasets for two epochs only. We compared the finetuned APT with expert models trained on the same datasets. As shown in Table 4, the APT yielded the best scores on both tasks.

[Error handling] We can roughly divide the errors into two types: a) label error (noisy label): It is actually really interesting to notice in our preliminary study that if we replaced the name of sound event with an abstract but distinguishable one, e.g., “dog” -> “class A”, APT will still figure out the most similar one in terms of acoustic characters and return with “class A”. b) audio discrepancy: this is really a good question. To be honest, we didn’t set up experiments for that so far. Yet, it is very interesting for future work. We have added our discussion on error handling for the future work at the end of the manuscript.

[Handling different audio signals] The instruction-aware aligner processes audio signals by extracting the relevant audio features according to input text. Therefore, in addition to the audio characteristics, it relies on prompts as well. Regarding more specific types of audio, we are planning to extend APT to speech and music as well. Since we use AudioMAE, a generative model trained with reconstruction loss, as the audio encoder, APT has the potential to generalise to recognise speech and music.

[1] PREFIX TUNING FOR AUTOMATED AUDIO CAPTIONING

[2] LLARK : A MULTIMODAL FOUNDATION MODEL FOR MUSIC

[3] SIMVLM: SIMPLE VISUAL LANGUAGE MODEL PRETRAINING WITH WEAK SUPERVISION

[4] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

Thank you for your valuable feedback! We address your comments one by one as follows:

[Motivation]

To clarify, proposing a new network architecture for LLMs is not the aim of this work. Instead, as stated in the introduction, our goal is to extend LLMs and VLMs to the audio domain and further explore a more general generative audio-language framework that ingests as inputs multiple audio clips in the inference while maintaining their ability as LLMs or VLMs. Our experiments verified that the proposed generative framework is a promising direction and we hope it may be of interest for future multi-modal LM research. We will update our manuscript to better present our motivation and methodology.

[APT v.s. BLIP-2]

Motivated by BLIP-2’s idea on Qformer, this work applies a BERT-ish transformer to bridge the modality gap. After the preliminary study, we found the combination of contrastive loss, grounded loss, and language generative loss, as depicted in BLIP-2, performed the best to pretrain the transformer. We also adjusted the number of transformer blocks and dimension of hidden states for a better performance of audio alignment. While BLIP-2 pretrained this transformer with the captioning task only, this work combines captioning and tagging tasks to mitigate the lack of high-quality data in the audio domain.

The distinction of this work is to learn a more versatile framework that is capable of learning from more diverse tasks and can easily adapt LLMs/VLMs to the audio modality. As shown in Fig. 1, this work highlights a trainable audio identification token added to each group of acoustic tokens from the audio aligner and the juxtaposition between grouped acoustic tokens and text embeddings. This enables the proposed framework to address multiple diverse tasks, such as few-shot learning and complex audio understanding, in both training and inference stages compared to BLIP-2. Moreover, to evaluate more rigorously how the proposed model benefits from LLMs beyond common audio tasks, we proposed a new task, namely natural language audio reasoning, that tests the logic capacity of models with a series of comparison and summarization tasks.

[APT v.s. SpeechGPT]

SpeechGPT is a great work that integrates speech recognition and generation within one architecture. However, the focus of SpeechGPT is speech recognition/generation which is quite different from this work where an adapter is studied to extend LLMs/VLMs to general-purpose audio understanding tasks.

We acknowledge that speech processing is essential to AI development, but perceiving the general audio is also challenging. For example, different from speech with well-known phoneme structure, basic units of environmental audio, namely sound events, are distributed more sparsely without a clear structure, and they are more prone to overlap with each other.

However, inspired by the idea of SpeechGPT, this work can be extended to explore foundational models for joint understanding and generation of general audio in the future.