ACUS: Audio Captioning with Unbiased Sliced Wasserstein Kernel

We appreciate the valuable time and feedback of the reviewer to improve your work. We have revised our manuscript and would like to address your concerns outlined below.

Weakness: The motivation of the paper does not sound convincing. Exposure bias is a general problem to auto-regressive networks. It is not a critical problem for audio captioning. In general, exposure bias can be mitigated by a better training of model. I believe using a larger decoder will ease the problem a lot. Specifically, I do not see in the paper how much the exposure bias problem is harming the audio captioning performance. Even if we regard this as a serious problem, reinforcement learning (RL) should be a popular way to solve it as RL trains the model according to its inference output. The paper doesn't discuss about RL and address the audio captioning problem in a very narrow perspective.

Response: Exposure bias is a critical issue for autoregressive models trained by the maximum likelihood technique, and a lot of studies mentioned clearly how exposure bias harms pretrained model’s performance[1, 2, 3]. Audio captioning models are autoregressive models and are trained by maximum likelihood procedure, thus, they indeed suffer from the exposure bias issue. You mentioned that larger models can solve exposure bias, however, this claim only holds when you have a large amount of training data. There have not been large-scale datasets to train foundation models for audio captioning, thus, it is still crucial to investigate new training and inference techniques for audio captioning.

RL could mitigate the exposure bias issue for autoregressive models, however, it is unstable for training audio captioning. The instability comes from the policy gradient training method, which requires pretraining audio captioning models for a certain number of steps. Thus, the RL method is an ad-hoc method for the audio captioning task, and each model has a unique training process. On the other hand, our method seamlessly utilizes a wide range of encoder-decoder audio captioning models.

Reference: [1] Generalization in generation: A closer look at exposure bias. [2] Relating Neural Text Degeneration to Exposure Bias. [3] Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation.

Question: As proposing the unbiased kernel is the core technical contribution in the paper, how much gain is there from a biased kernel to a unbiased kernel?

I would like to provide a comparison between biased and unbiased kernel performance in our framework

As shown in the above table, our unbiased kernel outperforms the biased kernel significantly in five quantitative evaluation metrics, thus, it demonstrates advantages of the unbiased kernel against the biased kernel in our framework.

Dear Reviewer 4cxP,

We appreciate that you are likely to review multiple other papers, however, as we approach the end of the discussion period. We would greatly appreciate your feedback on our rebuttals. We are happy to address any remaining concerns or questions to improve the paper further. Thank you so much!

Kind regards,

Authors

Weakness 4: The study does not deeply explore the sensitivity of the framework to key hyper-parameters

We first clarify that the coefficient $\alpha$ in equation 14 is the hyper-parameter for the inference stage. The training objective is equation 13, in which there is no coefficient $\alpha$. During training, the coefficient’s ratio between likelihood and USW-RBF terms equals 1, thereby, the coefficient $\alpha$ is set to 0.5 to be compatible with the training stage. Although tuning the coefficient $\alpha$ during inference can increase slightly the performance, it is trivial to conduct hyper-parameter tuning for coefficient $\alpha$. We provided the ablation study for the number of Mote Carlo samples $L$ in Table.8 in Appendix A.3.

Weakness 5: The paper dedicates considerable space to explaining the USW-RBF kernel but provides a limited description of how it integrates with the model itself.

Thank the reviewer for helping us improve the clarity of our submission. We use the text embedding from the final hidden layer of the text decoder to measure the cross-modal similarity between audio and captions. We will update the manuscript according to your comments to explain our method more clearly. We detailed the training and inference stages of our proposed framework in section 3.2. We also demonstrated how the proposed kernel is integrated with the encoder-decoder architecture for training and inference with audio captioning models in Figure. 1.

Weakness 6 and question 1: Although the paper separates AAC models into encoder-decoder and prefix-tuning architectures, with experiments performed only on the encoder-decoder type. Could the proposed method be adapted for application within prefix-tuning structures?

Thank the reviewer for helping point out the typo. Although both encoder-decoder and prefix-tuning architectures have an audio encoder and text decoder, they are different backbones for the audio captioning task. Both the encoder and the decoder of encoder-decoder architecture are finetuned during training, while only the encoder of prefix-tunning architecture is finetuned during training. Furthermore, the generation at the inference stage is different between the two architectures. Regarding the encoder-decoder architecture, the encoder first extracts semantic representation from a given audio and then sequentially generates an audio caption based on the semantic representation. Regarding prefix-tuning architecture, the encoder maps a given audio to a prefix on the text embedding space of the fixed-pretrained text decoder. Then, the text decoder sequentially generates an audio caption based on the prefix.

We did experiments on the prefix-tuning backbones for our method, but our method does not work on prefix-tuning backbones. The explanation is that the encoder of prefix-tuning aims to learn the mapping from audio features to text embedding, but the encoder is not able to learn meaningful representation for acoustic features. Thus, it is challenging for the USW-RBF to measure the similarity across acoustic and linguistic modalities precisely. As a result, we emphasized in our manuscript that our framework is merely developed for encoder-decoder architecture.

Question 2: What is the increase in computational cost introduced by the framework?

We understand the reviewer's concern regarding the trade-off between efficiency and effectiveness. Thus, we would like to provide an experiment regarding using stochastic decoding methods to generate $B$ candidate captions in our framework. The inference time is measured by using the real-time-factor (RTF) metric, which measures how many seconds a method needs to generate a caption.

As illustrated in the above table, our framework can generate audio captions in real-time, which makes it suitable for real-world applications.

Question 3: Could the proposed approach be adapted to work alongside LLMs to achieve higher-quality captions?

I do believe that if we leverage advanced LLMs for AAC using our framework, there will be a significant improvement in the AAC task. However, LLMs should be integrated for the AAC task in the encoder-decoder scheme to be compatible with our framework rather than the prefix-tuning scheme.

We appreciate the valuable time and feedback of the reviewer to improve your work. We have revised our manuscript and would like to address your concerns outlined below

Weakness 1: Combining the USW-RBF kernel with stochastic decoding strategies may lead to high computational costs

We understand the reviewer's concern regarding the trade-off between efficiency and effectiveness. Thus, we would like to provide a comparison of inference time between our method and the traditional approaches. The inference time is measured using the real-time-factor (RTF) metric, which measures how many seconds a method needs to generate a caption.

As illustrated in the above table, our framework is able to generate audio captions real time, therefore, it can be utilized for real-world applications. The inference time of our method increases twice compared with the baseline method, however, our method is able to generate more diverse and high-quality captions.

We also understand the reviewer's concern about the computational cost of the Monte Carlo sampling during training and inference. However, sampling projecting directions for sliced Wasserstein is very computationally efficient i.e., it is equivalent to sampling from a standard multivariate Gaussian. Moreover, projecting directions do not need to be resampled multiple times i.e., we can reuse the projecting directions for some iterations before resampling. Finally, from Proposition 3, the approximation rate is $\mathcal{O}(L^{-1/2})$ which suggests that we might not need a very large number of Monte Carlo samples to obtain a good approximation. According to empirical experiments, we choose the number of Monte Carlo samples $L=50$ for both training and inference in our framework. Furthermore, We conducted an ablation study on the trade-off between computational cost and the quality of generated captions at the inference stage in the Table. 8.

Weakness 2: The performance increase with the proposed approach is relatively minor.

We tried to reproduce the experimental results reported in the EnCLAP backbone by using author’s public codebase, however, the reproduced results are slightly lower than the original results. We realized that the mismatch in terms of results was caused by the number of GPUs used to train the EnCLAP model. The authors used four A100 GPUs to achieve the reported results, while we used two A100 GPUs to reproduce the results. Therefore, we reported the reproduced results in our manuscript for a fair comparison. With the same training setups, the improvement of our method is substantially significant compared with the baseline EnCLAP-large with the MLE method. The SPIDEr score improves from 0.48 to 0.5 for the AudioCaps dataset and from 0.273 to 0.275 for the Clotho dataset.

We realized that the quantitative evaluation metrics are not good enough to evaluate the performance of the text generation task, audio caption generation. Thus, we carried out comprehensive qualitative experiments. Both subjective and objective metrics are used to evaluate our method and baselines in the section 3.2. According to our qualitative experiments, our method is able to enhance caption length, lexical diversity, and descriptiveness of audio captions.

Weakness 3: The application scope of this work appears limited.

We develop the ACUS framework to deal with exposure bias and local temporal distortion when measuring cross-modal alignment of the audio caption tasks. The audio-text retrieval and speech recognition tasks are out of the scope of our study. Furthermore, the speech recognition task is not an ideal application for our proposed framework. The speech recognition task requires measuring monotonic alignment between acoustic and linguistic features, and our method is not better than traditional methods such as Dynamic Time Wrapping. Regarding audio-text retrieval, we do believe that our framework can enhance audio-text retrieval accuracy, however, it is challenging to compare with the pretrained CLAP model. The reason is that we can not acquire a large number of GPUs to pretrain the CLAP backbone using our proposed framework on a large amount of datasets. Thus, we are not able to conduct a fair comparison between our framework and the contrastive learning for the CLAP backbone on audio-text retrieval.

We appreciate the valuable time and feedback of reviewer to improve your work. We have revised our manuscript and would like to address your concerns outlined below

Weakness 1: which contrastive method and how does it ignore "the temporal information when measuring similarity across acoustic and linguistic modalities"?

We would like to thank the reviewer for helping us improve the quality of the submission. We will definitely rephrase these sentences to make our abstract more clear . We mention the classical contrastive learning (CL) method for multimodal learning in this work. The relevant CL works for audio captioning are cited in our introduction for clarity. The main drawback of CL method is performing an average-pooling operation before measuring the cosine similarity across acoustic and linguistic modalities. Thus, the temporal information in acoustic and linguistic features are discarded due to the average-pooling operation. Our method avoids this drawback by directly measuring the discrepancy between two sequences.

Weakness 2: The Monte Carlo sampling and stochastic gradient optimization may increase computational costs.

Thank you for your insightful comments. We understand the reviewer's concern about the computational cost of the Monte Carlo sampling. However, sampling projecting directions for sliced Wasserstein is very computationally efficient i.e., it is equivalent to sampling from a standard multivariate Gaussian. Moreover, projecting directions do not need to be resampled multiple times i.e., we can reuse the projecting directions for some iterations before resampling. Finally, from Proposition 3, the approximation rate is $\mathcal{O}(L^{-1/2})$ which suggests that we might not need a very large number of Monte Carlo samples to obtain a good approximation.

Weakness 3: missing prefix-tuning baselines for the audio captioning task

Thank you for pointing out the relevant prefix-tuning baselines for audio captioning. We will update the experiment section to include all relevant baselines based on your suggestion. Also, we would like to emphasize that our proposed method still significantly outperforms all prefix-tuning baselines for audio captioning.

Citations

[1] https://ieeexplore.ieee.org/abstract/document/10448030.

[2] https://ieeexplore.ieee.org/abstract/document/10096877.

Question 1: How does the performance of USW-RBF compare with other non-Wasserstein-based kernels for audio captioning tasks?

We compare with soft-DTW and Dynamic Time Wrapping (DTW) methods which are equivalent to non-Wasserstein kernels as mentioned in [1,2]. The experimental results are illustrated in the Table. 5. As discussed thoroughly in our work, DTW and soft-DTW are incapable of handling local temporal distortion for the audio captioning task, while our kernel is able to deal with the distortion. Thus, our proposed kernel substantially outperforms DTW and soft-DTW for measuring similarity across acoustic and linguistic modalities for the audio captioning task.

References:

[1] A kernel for time series based on global alignments

[2] Fast Global Alignment Kernels

Question 2: What are the potential trade-offs between the accuracy improvements and the computational costs introduced by Monte Carlo sampling and stochastic gradient optimization in ACUS?

We conducted an ablation study regarding trade-offs between efficiency and effectiveness of our method regarding Monte Carlo sampling in the Table. 8 in the Appendix.3. According to proposition 3, the approximation rate of USW-RBF kernel is $\mathcal{O}(L^{-1/2})$ which suggests that we might not need a very large number of Monte Carlo samples to obtain a good approximation. As shown in the Table. 8, our proposed kernel can improve the performance of the baseline method significantly with a decent number of Monte Carlo samples $L=50$. I would like to illustrate the experimental results below for clarity:

Question 3:

We choose rotary positional embedding based on the ablation study in the table. 6. There are a lot of the same observations[1, 2] aligned with our ablation study, which is the rotary PE outperforming absolute PE in encoding positional information for sequential data. We believe that if we have a better positional encoding(PE) technique, the PE technique can enhance our proposed method.

References:

[1] The llama 3 herd of models

[2] Palm: Scaling language modeling with pathways

Question 4: Why audio captioning? Can the method be useful to other tasks? Audio understanding? Speech Recognition?

We chose the audio captioning task to show the effectiveness of our proposed framework for two reasons. The first reason is that the audio caption task requires cross-modal alignment, in which temporal information from each modality is crucial to consider when measuring cross-modal similarity between acoustic and linguistic features to generate high-quality captions. The second reason is that the cross-modal alignment for the audio captioning task is not a monotonic alignment. Thus, the traditional method, like Dynamic Time Wrapping (DTW), can not handle non-monotonic alignments since local temporal distortion exists.

The audio understanding and audio question-answering tasks are similar to the audio captioning task. All of them require measuring cross-modal alignment to better generate either a caption or answer for a given audio, therefore, our method is well-suited for these tasks. I do not see any benefits in applying our method to the speech recognition task. The speech recognition task requires measuring monotonic alignment between acoustic and linguistic features, and our method is not better than traditional methods, such as DTW, for handling monotonic alignments.

As the rebuttal period is coming to an end, we hope to have clarified the novelty of our contribution and addressed the reviewer's concerns. We truly appreciate the reviewer's time on our paper and are looking forward to your responses and/or any other potential questions. Your feedback would be very helpful for us to identify any ambiguities in the paper for further improvement.

We appreciate the valuable time and feedback of reviewer to improve your work. We have revised our manuscript and would like to address your concerns outlined below

Before addressing the review's concerns, we would like to emphasize the main contribution of our work is that we develop a framework method to address the exposure bias of current training frameworks, maximum likelihood and contrastive learning, for audio captioning models. Our contribution is a new framework rather than a new model for the audio captioning task. We dedicated a section 3.2 and the Figure. 1 to detail the training and inference procedure of our proposed framework. The Figure.1 demonstrates how the proposed USW-RBF kernel is integrated into encoder-decoder architectures for training and inference stages. Furthermore, we would like to argue that audio and speech are similar to some extent, but they are different. Speech tasks focus on understanding the linguistic and speaker information of audio. On the other hand, audio tasks focus on understanding acoustic information and sound events of audio.

Weakness 1: Evaluate our method on speech benchmarks: SUPERB, Dynamic-SUPERB, and SpeechCaps

All those mentioned benchmarks are designed to evaluate the transferable capability of pretrained speech models which is totally irrelevant to our work, our work mainly focuses on solving the exposure bias for audio captioning. Even though speech and audio are similar to some extent, they are two different. I would like to articulate the reasons why these benchmarks are not suitable for evaluating our proposed method as follows. Both SUPERB and Dynamic-SUPERB are used to evaluate the transferable capability of universal pretrained speech models, which is irrelevant to the main research question of our work, solving the exposure bias issue for audio captioning models. The second reason is that we propose a new framework to deal with the exposure bias of audio captioning tasks rather than a new model or architecture for transfer learning, therefore, there is no connection between our work and SUPERB and Dynamic-SUPERB benchmarks. The SpeechCaps benchmark is designed to evaluate the understanding capabilities of speech models in prosodic information. Thus, it is not appropriate to use for benchmarking our method.

weakness 2: lack of sufficient details on how this kernel is integrated within the overall model architecture

We detailed the training and inference stages of our proposed framework in the section 3.2. We also demonstrated how the proposed kernel is integrated with the encoder-decoder architecture for training and inference with audio captioning models in the Figure. 1.

Question 1: There are some minor errors, like in line 187, it should be ( \nu = \frac{1}{N} \sum_{j=0}^N \delta_{z_y^j} ), not ( \nu = \frac{1}{M} \sum_{j=0}^N \delta_{z_y^j} ), to make two empirical distributions have the same number of supports.

We would like to thank the reviewer for pointing out these typos to improve our submission. We will update our manuscript accordingly based on your suggestion.

Question 2: Could you provide more analysis on how the proposed method performs in more challenging scenarios (e.g., multi-speaker or noisy environments) to better highlight its strengths? If not, do you believe that temporal information is important in audio captioning task?

Those mentioned benchmarks and settings are irrelevant to our work, therefore, we do not think our proposed method should be evaluated on the suggested benchmarks. The explanations are articulated in my aforementioned response. We conducted experiments to compare our proposed method against the contrastive learning and baseline methods, which are incapable of considering temporal information. Our method remarkably outperforms baseline methods in terms of quantitative and qualitative metrics for the audio captioning task in the section 5.1 and 5.2, respectively. We also provide qualitative examples in Appendix. A4 to demonstrate that our method is capable of generating more high-quality captions against baseline methods. According to experimental results, the temporal information is crucial for the audio captioning task, therefore, it should be taken into account when measuring the similarity between acoustic and linguistic modalities for encoder-decoder audio captioning models.

Question 3: The application area of this work seems limited.

I think that applying our method for multilingual audio captioning would be great. If there is a standard dataset for multilingual audio captioning, we would like to evaluate our method on that dataset to show its effectiveness. There is no benefit to utilizing our method for the automatic speech recognition (ASR) task. We developed the ACUS framework to deal with the local temporal distortion issue for text-audio alignment, and the ASR task does not suffer from this issue. The reason is that ASR task requires a monotonic alignment across acoustic features and ground-truth transcript, therefore there is no benefit of leveraging our method for ASR.

Our proposed kernel method is a non-parametric method and works on the output space of the audio encoder and text decoder. Thus, our method can be seamlessly utilized with the multilingual audio captioning tasks for encoder-decoder models without any modification in model's architectures.

Question 4: Since you use stochastic decoding strategies in the inference stage, which may lead to high computational costs, and the reported score in your results is not very decent, we might not need such a high-cost method to get a minor improvement. Thus, could you provide more details on the differences in diversity and quality of captions generated by your approach?

We understand the reviewer's concern regarding the trade-off between efficiency and effectiveness. Thus, we would like to provide a comparison of inference time between our method and the traditional approaches. The inference time is measured by using real-time-factor (RTF) metric, which measures how many seconds a method needs to generate a caption.

As illustrated in the above table, our framework is able to generate audio captions real time, therefore, it can be utilized for real-world applications. The inference time of our method increases twice compared with the baseline method, however, our method is able to generate more diverse and high-quality captions. We conduct extensive qualitative experiments in the Section 5.2(both subjective and objective experiments) to demonstrate the high-quality of generated captions by leveraging our proposed framework against traditional methods. Our method can improve caption length, lexical diversity, and semantic of generated captions, as shown in the Table. 3, and the ACUS framework substantially enhances the descriptiveness and correctness of generated caption, evaluated by human evaluation, in the Table. 4. Furthermore, we also provide qualitative examples in the Appendix. A4.

Dear Reviewer 2JoP,

We appreciate that you are likely to review multiple other papers, however, as we approach the end of the discussion period. We would greatly appreciate your feedback on our rebuttals. We are happy to address any remaining concerns or questions to improve the paper further. Thank you so much!

Kind regards,

Authors

Dear Reviewer 2JoP,

We appreciate that you are likely to review multiple other papers, however, as we approach the end of the discussion period. We would greatly appreciate your feedback on our rebuttals. We are happy to address any remaining concerns or questions to improve the paper further. Thank you so much!

Kind regards,

Authors

Dear Reviewer 2JoP,

We appreciate that you are likely to review multiple other papers, however, as we approach the end of the discussion period. We would greatly appreciate your feedback on our rebuttals. We are happy to address any remaining concerns or questions to improve the paper further. Thank you so much!

Kind regards,

Authors

We appreciate the valuable time and feedback of the reviewer to improve your work. We have revised our manuscript and would like to address your concerns outlined below.

Weakness 1: No analysis of computational overhead from the USW-RBF kernel

We understand the reviewer's concern regarding the computational overhead of our kernel. Our kernel is approximated using the Monte Carlo method, however, sampling projecting directions for sliced Wasserstein is very computationally efficient i.e., it is equivalent to sampling from a standard multivariate Gaussian. Moreover, projecting directions do not need to be resampled multiple times i.e., we can reuse the projecting directions for some iterations before resampling. Finally, from Proposition 3, the approximation rate is $\mathcal{O}(L^{-1/2})$ which suggests that we might not need a very large number of Monte Carlo samples to obtain a good approximation. We conducted the trade-off between efficiency and effectiveness of our framework in Table 8 in Appendix A.3.

Weakness 2: Unclear how the method performs on longer audio sequences

Our proposed framework does not distinguish between handling short and long audio. Technically, our method measures similarity across acoustic and linguistic modalities similarly, regardless of audio duration.

Weakness 3: While ablation studies are included, there's limited discussion of how sensitive the method is to various hyperparameters. Could benefit from more guidance on hyperparameter selection for new datasets.

We conducted the hyperparameter selection for our method on the number of Monte Carlo samples $L$ and the bandwidth parameter $\gamma$ in Table.7 and Table.8 in Appendix A3, respectively.

Weakness 4: Lacks detailed analysis of failure cases

We would like to thank the reviewer for this insightful feedback. We will incorporate an analysis of failure cases in the Appendix.

Question 1: How does the computational complexity scale with audio length and batch size compared to baseline methods?

The baseline method, contrastive learning method, has a computational complexity of $\mathcal{O}(n)$ for a minibatch of $n$ audio-caption pairs, where each audio and caption has a length of $d$. While our proposed method has a computational complexity of $\mathcal{O}(n.d.\log{d})$

Question 2: How robust is the method to different audio qualities or noise levels? Was this tested?

Since the robustness of our method for noisy audio is not the primary objective of our work, we have not tested it in noisy audio.

Question 3: What is the impact of different positional embedding choices on the final performance? While rotary embeddings performed best, is there a theoretical justification for this?

We conducted the ablation study on the impact of positional embedding (PE) techniques on our framewok in Table. 6. The rotary PE technique achieves the highest performance based on the empirical experiment. We hypothesize that the rotary PE technique is well-suited for our framework because it effectively encodes the positional information for a long sequence of audio features.