ADIFF: Explaining audio difference using natural language

We thank the reviewer for recognizing our contributions and providing constructive feedback. We address every question and hope that our response resolves your concerns. Any follow-up questions are welcome.

The choices of captioning metrics seem insufficient ... other signal processing techniques?
The reviewer brings up a valid point on the limitation of metrics. We make similar observations in Section 5.1. Comparing to the link pointed by reviewer, we have all the metrics listed there except the SPIDER-FL and FENSE metric. However, even the SPIDEr-FL and FENSE metric fails to take into account the audio and signal information that the reviewer suggests. Therefore, to overcome the limitation of objective metrics, we decided to conduct human evaluations whose setup is explained in Appendix H.2 and results in Section 4 and Table 3. For the ablation studies, we rely on objective metrics, as human evaluation for each ablation study would be too costly given the size of the test set.

Showing the complete captioning metrics in all the tables (2, 4, 5, 6) ... referred to in the text.
We thank the reviewer for this suggestion. We address this in the global comment under question 2. We hope this improves the readability of the numbers and clarity of the paper.

In table 1, is the number of examples from the number of audio in each dataset? ... these are not clearly explained in the main paper nor in the appendix.
We thank the reviewer for bringing this up. The reviewer is right and the dataset indeed consists of {a_i, a_j, e_i}. We have updated this text in the paper and apologize for any confusion caused. For dataset statistics, the Table 1 is correct and indicates the triplets {a_i, a_j, e_i} in each dataset and split respectively. For example, the Train set of CLD consists of 19195 triplets for each Tier. For generating data, we perform random sampling where the sampling is done per audio, this removes the edge case of multiple captions. This is done in three steps. First, we flattent the .csvs to be audio file and single caption. For Clotho which has 5 captions, this results in 5 entries for each audio. Second, we perform sampling. For example, let's say the dataset contains N audios. We will loop through the N audios, for any ith audio we will randomly pick the second audio (jth) not including i to i+4th audio. This is implemented as a constructing index list excluding i, i+1,..i+4 index and sampling from this index list. Third, the selected ith and jth audio are then used to look up the caption for the ith and randomly picked jth audio. This caption then becomes the seed for LLM to generation difference e_i. We added this description on sampling in Appendix E. We again apologize about the text error and hope this clarifies the question!

What are the types of differences described in different tiers? ... analysis from LLM generated captions.
We appreciate the reviewer's question and understand the importance of exploring the LLM-generated captions across the three Tiers. We can perform the analysis, by splitting text into words, converting them to lowercase, and focusing on words related to audio, we can identify words that appear in Tier 2 but not in Tier 1, showing unique audio characteristics used for the difference explanation in Tier 2. Similarly, we can use the same method to examine the distinctions between Tier 2 and Tier 3, where the differences often lie in more detailed and immersive sound characteristic descriptions. The results of this analysis are shown in Figure 7 with the description in Section F. Overall, we see the audio characteristics used in Tier 3 over Tier 2 (Figure 7, right subplot) and Tier 2 over Tier 1 (Figure 7, left subplot) are different, and focus on the different aspect and details between two audios.

For table 3, is the human evaluation also based on the difference between two ... not included in the human evaluation?
The reviewer brings up a valid question. We specifically choose to source audio files from different datasets instead of ACD and CLD for human evaluation for two reasons. First, we wanted to compare against other Audio-Language Models like Qwen-Audio which supports two audios. By training naive baseline and ADIFF on the train set of ACD and CLD, we will be inherently matching the distribution of the test dataset both in terms of audio and language used. This is unfair to Qwen-Audio which might not have seen similar type of audio recordings. Therefore, to have a fair comparison, we didn't utilize audios from ACD and CLD test set in the human evaluation. Second, we wanted to see how performance changes across difference types of audios. For example, different acoustic scenes, same audio events, different domains, etc. By sourcing audios from Studio, FSD50K, GTZAN we were able to get the desired level of control and simulate the scenarios explained in Section 4. We hope the above answer helps.

We thank the reviewer for recognizing our contributions. We want to highlight that our main contribution is the creation of the two difference datasets which are human-verified, then the second contribution is the baseline and the proposed ADIFF model, and lastly the various presented findings. We address every concern point by point and hope that our response resolves your concerns.

The three-tier captioning process seems arbitary; no ablation study is conducted for this (i.e. if trained only on tier-3, can the models have good performance on tier-1? If they are purely hierarchical, then training on tier-3 should in theory yield good results on lower tiers.)
The three-tier captioning process is designed for evaluation purposes and to benchmark the comparative reasoning abilities of Audio-Language Models. The motivation for creating a three-tier dataset is detailed in Section 2. This motivation is rooted in auditory psychoacoustics, particularly sound perception studies, which examine how humans perceive sound, including how the body receives sound and how the brain interprets it. Human explanations draw from a diverse array of information sources, including acoustic details, human perception, and linguistic nuances. This diversity allows us to classify audio difference explanations into three tiers, with each tier providing more information than the previous one. Tier-3 explanations offer greater granularity in descriptions compared to Tier-1 explanations; however, this does not imply that Tier-1 descriptions are not as good as Tier-3 descriptions.

Both AudioCaps and Clotho's captioning quality is questionable;
AudioCaps and Clotho are the only two human-annotated datasets available in audio-language literature. Therefore, we use their audio and captions as the source for creating our dataset. Additionally, we employ human annotators to verify the descriptions in our test set, ensuring the removal of any questionable captions.

WavCaps and other larger caption datasets contain more diversity, yet no experiment is conducted on them.
We appreciate the reviewer’s suggestion. In current audio-language literature, AudioCaps and Clotho are used for evaluation. WavCaps, being LLM-generated and lacking human supervision, has never been used for evaluation as its correctness cannot be guaranteed. Additionally, AudioCaps and Clotho are considered diverse enough for evaluation purposes. However, as an alternative, it makes sense to use WavCaps to generate additional training data for differences and then test on the human-verified ACD and CLD test sets. We added this experiment, and the results are presented in Section P, Tables 20 and 21. We observe that adding WavCaps improves vocabulary and coverage of sound sources in produced audio difference explanations, though it might not be directly reflected in the linguistic metrics. The analysis is available in Section P and we will release both the newly created WCD dataset and the model checkpoint. We hope this addresses the reviewer's question.

The SOTA model, Qwen-Audio, seems to not be finetuned for this task as well. It prompts the question that finetuning a stronger model with out ADIFF may solve the problem entirely just through the proposed dataset.
We understand the reviewer's point and address this in the global comment question 1. We have added two more baselines in the objective and subjective evaluation section: First, Qwen-AC with LoRA fine-tuned on the training set of ACD and CLD, and second, Qwen-AC with full fine-tuning (no frozen parameters) on the training set of ACD and CLD. The results are shown in Table 2 and Table 3, with analysis in Section 4.2. We hope this addresses any concerns on the baseline.

We sincerely thank the reviewer for the suggestion and the help provided in improving the manuscript.

We add an analysis from an information theory perspective to Appendix Section Q, specifically Appendix Section Q2 and Table 24. The analysis focuses on Entropy for each tier of audio difference explanation. The perplexity would simply be 2 raised to the entropy value.

The results are shown in Table 24 and we provide a summary here. A higher entropy at the character or word level suggests greater diversity or unpredictability in the corpus. A higher character entropy would mean the text has a wide variety of characters with a relatively even distribution (e.g., many unique letters, numbers, punctuation marks, etc). Similarly, a higher word entropy would mean the corpus has a rich vocabulary with words appearing in a more evenly distributed manner. We see a higher word-level entropy for Tier 3 signifying a more complex and varied vocabulary, whereas lower entropy in Tier-1 indicates a more uniform and constrained language, typical of shorter texts or structured responses. We also computed the numbers per data source and found the CLD datasets tend to have slightly higher word-level entropy compared to the ACD datasets, which reflects differences in the diversity of text content as ACD is sourced from AudioSet which mainly contains speech. From a model development perspective, higher word-level entropy datasets (e.g., Tier-3) will be more challenging for models, requiring greater capacity to handle vocabulary diversity. The full explanation can be found in Appendix Section Q2.

We sincerely thank the reviewer for recognizing our contribution and novelty! We hope that our response resolves your concerns. Any follow-up questions are welcome.

The presentation of the experimental section requires further refinement. The experimental tables are somewhat difficult to follow, as the model names are not consistently listed in most of them. Additionally, highlighting or bolding the best values for each metric would enhance clarity and emphasize the differences more effectively.
We apologize for the difficulty in reading the experimental section. We address this in the global comment under question 2. We have updated the PDF and retained only the BLUE_4, METEOR, and SPIDEr metric in the main paper, while moving the full tables to the Appendix. Moreover, we have bolded the best-performing number for each metric in the Table. We hope this improves the readability of the numbers and clarity of the paper.

Regarding the structure of the sequence sent to the cross-projection layer in Figure 3, what is the length of the audio prefixes (i.e., [prefix 1] and [prefix 2])? From my understanding, HTS-AT provides only one audio embedding for the entire segment of audio samples. Do [prefix 1] and [prefix 2] consist of just a single token representing the latent embedding from HTS-AT? I suggest including a more detailed explanation of this aspect, rather than solely focusing on the right side of Figure 3.
We thank the reviewer for the question. The length of prefix_1 and prefix_2 is 40 each. The output from HTS-AT is a single embedding (dim=768), which is then converted into [seq_len, d] using audio projection. The audio projection (right side of Fig 3) first expands the hidden dimension to a larger size dimension k, which is then split to form [seq_len, d] where k = seq_len * d. This is followed by concatenating with a learnable constant, resulting in [seq_len + c, d]. This output is then passed to the transformer, followed by clipping of the learnable constant output c. The resulting output of audio projection is of shape [seq_len, d]. This architecture of audio projection, specifically expanding a single embedding to seq_len, is inspired by previous Audio-Language Model papers published in NeurIPS 2023 and ICASSP 2024. We added this explanation in Section 3.1 under "Audio Projection" in the main paper. We hope this clarifies the reviewer's question.

We sincerely thank the reviewer for the detailed comments. We carefully address each of the reviewer’s concerns below and hope that our response resolves your concerns. Any follow-up questions are welcome.

Absence of a concrete baseline - the baseline the ... components of the model.
We appreciate the reviewer's suggestion and have addressed it in the global comment question 1. We use the existing literature on prefix-tuning as the baseline for the audio difference explanation benchmark and systematically improve upon it, showing consistent performance improvements. Finally, the ADIFF model significantly outperforms the naive baseline. Nevertheless, we understand the reviewer's point and have added two more baselines in the objective and subjective evaluation section: First, Qwen-AC with LoRA fine-tuned on the training set of ACD and CLD, and second, Qwen-AC with full fine-tuning (no frozen parameters) on the training set of ACD and CLD. The results are shown in Table 2 and Table 3, with analysis in Section 4.2. We hope this addresses the concerns on the baseline.

The authors motivate that cross projection layer ... that cross posting is useful.
The reviewers bring up a valid point. We would like to address it by dividing the question into two parts. (1) How useful is the cross-projection layer? The utility of cross-projection is highlighted in Table 4, where the middle section represents the baseline model with a frozen LLM, and the last section shows the baseline model with the addition of the cross-projection layer and sep token. These results indicate that cross-projection is a useful addition in prefix-tuning architecture where the LLM is frozen. We also performed a qualitative study that shows how this helps the model utilize text prefixes to store difference information, as detailed in Appendix I. (2) Now given that the LM is also fine-tuned, does the cross-projection layer retain its utility? This is a valid question, and we thank the reviewer for bringing this up. To verify this, we conducted an ablation study suggested by the reviewer, comparing the proposed final ADIFF model (three-stage training with cross-projection) against ADIFF (three-stage training but without cross-projection). The results and analysis are added in Appendix N. We see that the performance improvement from cross-projection is reduced when the LM is also fine-tuned, but ADIFF with cross-projection still outperforms ADIFF without it on all three tiers of ACD and CLD. We hypothesize that the cross-projection acts as block attention over the two audio and text prefixes, which has been shown to improve performance in the vision domain [1]. We hope this clarifies the reviewer's question.

Authors use GPT2 as the LLM model which ranges ... ablation might not yield any difference there.
We apologize for any confusion. We use the GPT2-base (128M) in all experiments, including cross-projection. Only in the scaling section (Section 5.3) do we vary the LM size from 128M to 1.5B. While we do not disagree that a larger LLM with fine-tuning might learn the task of cross-projection, we did not scale to 8B pairs and thus cannot conclusively comment on this. We did our experiments from the perspective of building small audio-language models in the future, as audio tasks are generally performed on-device (embedded) where running large LLMs is not feasible. Nonetheless, we reinforce that the contribution of cross-projection to prefix tuning and even fine-tuning remains valid from the experiments conducted in the paper and the new ablation study added in Appendix N.

The paper’s structure and presentation can be improved ... to interpret the results accurately.
We apologize for any confusion and space limitations. We have addressed this in the global comment question 2. We combined the results in Tables 2 and 4 for ease of comparison. The Table 3 scores and their meaning were moved to Appendix H2 for lack of space. We will ensure that pointers to the appendix are explicitly specified in the main text.

The paper inaccurately equates compute across ... their scaling comparisons in this section.
We thank the reviewer for bringing this up. We explicitly state in Section 5.3 - "Each model is trained with the same compute budget, approximately equivalent to 30 epochs". The only conclusion we draw in Section 5.3 is the last two statements "This suggests that aligning and guiding larger models with prefix-tuning requires a greater number of epochs. Given the computation budget and performance, we choose to use GPT2-base for the subsequent experiments." We agree with the reviewer that higher per-epoch computational costs result in increased compute requirements for larger LMs. We have updated the Section 5.3 and added this discussion. Our conclusion (in quotes) for the section still holds and becomes even stronger as small LMs require even less compute.

[1] https://arxiv.org/abs/2407.07726

Its unclear from the paper why Qwen-Audio Chat is a valid ... do audio difference explanation task?
The reviewer brings up a valid point and we have addressed this in global comment question 1. We trained Qwen-Audio Chat on the ACD and CLD datasets to create a stronger baseline. We used both LoRA fine-tuning and full fine-tuning and added the results to Table 2 and Table 3. We hope this addresses the concern regarding Qwen-Audio Chat not being designed for the task of audio difference explanation.

Authors mention "ADIFF sees the largest improvement in ... could be more readable.
The Table 3 scores and their meaning are described in Appendix H2. We hope this clarifies the numbers, and we will add better pointers to Appendix Section H2.

Its unclear why the datasets used for Table 3 different from ACD and CLD datasets?
The reviewer raises a valid question. We specifically chose to source audio files from different datasets instead of ACD and CLD for human evaluation for two reasons. First, we wanted to compare against other Audio-Language Models like Qwen-Audio, which supports two audios. Training the naive-baseline and ADIFF on the training set of ACD and CLD would inherently match the distribution of the test dataset in terms of both audio and language used, which is unfair to Qwen-Audio, which might not have seen similar audio recordings. Therefore, to ensure a fair comparison, we excluded audios from the ACD and CLD test set in the human evaluation. Second, we aimed to observe how performance changes across different types of audios, such as different acoustic scenes, same audio events, different domains, etc. By sourcing audios from Studio, FSD50K, and GTZAN, we were able to achieve the desired level of control and simulate the scenarios explained in Section 4. We hope this helps!

Its unclear how the LLM is finetuned. Is a full ... larger models?
The LM has been fully fine-tuned. We did not explore using PEFT on larger LLMs. We hope that the Qwen-Audio-Chat with PEFT (LoRA) fine-tuning provided above covers this baseline. If not, we are willing to conduct additional experiments that are within are compute budget.