MATS: An Audio Language Model under Text-only Supervision

We sincerely appreciate your time and effort in reviewing our manuscript. Your positive evaluation is highly encouraging. Thank you for your valuable feedback.

Q1: The proposed Santa does not directly use the disc_L1 metric.

1. In our design, we only have access to text-only data during training, making it impractical to directly use the disc_L1 metric to reduce the modality gap. Instead, as shown in Figure 3 of main paper, our Santa achieves a similar effect in effectively reducing the distance between audio and text embeddings.
2. Further, Table 1 presents relevant statistical information. Specifically, we randomly select 350 samples from AudioAIA dataset and generate their language embeddings $Z_T$ and audio embeddings $Z_A$ using the CLAP encoder. Santa is then applied to the audio embeddings, denoted as $f_{\mathrm{Santa}}(Z_A)$. Next, we calculate the L1 distance between the prototype of audio embeddings $Z_A/f_{\mathrm{Santa}}(Z_A)$ and language embeddings $Z_T$. As shown in Table 1, Santa effectively reduces the distance between audio and text embeddings, validating its effectiveness in bridging the two spaces closer.

Table 1: The statistics on the modality gap within CLAP. Note: We randomly select 350 samples to calculate the L1 distance.

Q2: Lines 165-177 on the left column of page 4 seem to be a repetition of stuff on page 3 line 264: identity -> identify

Thanks for your reminder. We will update it in the next version.

Q3: Why do you need 5M text samples? Is the description space that large?

1. Using 5M text samples aims to improve the generalization of MATS. With approximately 7B parameters, MATS-LLaMA requires a substantial amount of training data to effectively scale its capacity. As shown in Table 2, most existing LALMs of similar size (around 7B parameters) are trained on over 5M audio-text pairs.

2. Furthermore, we conducted an ablation study to assess the performance of MATS-LLaMA with different training dataset sizes. As shown in Table 3, the model's performance improves as the dataset size increases, especially in open-ended scenarios. This result validates the necessity of using 5M text samples.

Table 2: The number of train dataset from current LALMs

Table 3: Performance of MATS-LLaMA under different training data ratios

W1: Training an ablated system that replaces Santa with mechanisms of prior works (PromptAAC and DRCap).

Following your suggestion, we replace Santa with the modality-gap reduction mechanism of PromptAAC and DRCap, referred to MATS-PromptAAC and MATS-DRCap. As shown in Table 1, Santa achieves the best performance on both closed-ended and open-ended tasks, which validate that Santa outperforms previous text-only methods.

• DRCap enhances audio captioning performance by leveraging a benchmark-specific memory bank, fully mapping the audio embedding to weighted language embeddings. But DRCap discards the original audio embedding, making its performance heavily dependent on the relevance between the memory bank and the test benchmark. However, in multi-task setting, the memory bank is no longer tailored to a single benchmark but instead aggregates information from multiple benchmarks. As a result, the mapping process may introduce unintended noise, projecting the audio embedding into a less relevant textual embedding space, leading to performance drop.

• PromptAAC adopts an augmentation-based approach that involves injecting noise and substituting similar language inputs. It retrieves audio events by matching audio embeddings with language embeddings derived from 527 predefined audio labels in AudioSet. However, the limited variety of audio events restricts the diversity of the retrieved information, resulting in inferior performance compared to Santa.

Table 1: Comparison results on CLS, CAP, and AQA benchmarks.

W2: What is the difference between the mechanism in the first column of Table 4 and DRCap?

• DRCap introduces the Retrieval-Augmented Generation (RAG) and Projection-Based Decoding (PD) strategies. In Table 4 of main paper, we only use PD strategy (denoted as Memory-based).
• We further report the performance of DRCap (RAG+PD), where we replace Santa with DRCap in our framework. As shown in Table 2, it still underperforms the performance of DRCap (reported in original paper) in single-task setting .
• This is because DRcap fully discards original audio embedding during inference, making its performance heavily dependent on the relevance between the memory bank and test benchmark. However, in multi-task setting, the memory bank is no longer tailored to a specific benchmark but instead integrates information from multiple benchmarks. This broader integration can introduce unintended noise during the mapping process, projecting the audio embedding into a less relevant textual embedding space. As a result, MATS-DRCap, trained in a multi-task setting, experiences a performance drop compared to DRCap.

Table 2: Ablation Study on AudioCaps.

W3: Improve Figure 2.

Thanks for your suggestion. We will update the figure in next version to better illustrate the "Modal-Transfer module" and its connection to noise injection/Santa.

Q1: Could you elaborate or provide evidence explaining why individual CLAP text embeddings have limited representational power? What factors lead to insufficient semantic relevance?

1. To validate it, we perform a retrieval task between CLAP audio embeddings and CLAP text embeddings on Clotho test set. Specifically, we compare the error rates of two strategies: top-K retrieval; K-means clustering followed by top-K retrieval. As shown in Table 3, the K-means method achieves a lower error rate in capturing semantically relevant captions, effectively mitigating the impact of irrelevant textual information caused by the limited representational capacity of individual language embeddings.

2. It may be attributed to the CLAP text encoder compressing textual information into a 1024-dimensional embedding space. Such aggressive dimensionality reduction leads to a loss of fine-grained semantic details, resulting in insufficient representational capacity of individual text embedding.

Table 3: Retrieving Error Rate@5 on the Clotho Test Set

Q2: How exactly is noise variance determined?

The variance is treated as a hyperparameter. As suggested by [1], the optimal value roughly aligns with the used strategy (calculating infinity norm between audio and text embeddings over 30 randomly selected samples), also shown in Figure 4 of the main paper.

[1] Training audio captioning models without audio.