Revisiting Deep Audio-Text Retrieval Through the Lens of Transportation

We appreciate the valuable time and feedback of reviewer to improve your work. We have revised our manuscript and would like to address your concerns outlined below

Q1: Apart from the performance, it would be good also to show the acquired network architecture and run-time efficiency for cross-modal inference.

A1: The focus of our work is to learn an effective joint embedding space for audio and text retrieval. We acknowledge the importance of run-time efficiency and leave that to our future work.

Q2: The reference of "noise" is not clear and potentially confusing, even with clear references ...

A2: We totally agree with your suggestion and have added a sentence in the third paragraph of the introduction to articulate noisy data in the audio-text matching task.

Q3: Some minor issues in language usage

A3: We have proofread the manuscript and corrected all language usage.

Q4: Do you think if it is possible to open-source the models?

A4: We will definitely publish our source code and pretrained models to the community.

Q5: Do you think your models will be benefitted by further fine-tuning the network model or pre-trained encoders? Or you think basically that's it (which is totally fine)?

A5: We leverage pre-trained encoders for our work(also used by prior works) to improve performance and help models converge quickly. Therefore, good pre-trained encoders could help to improve the performance of audio-text retrieval tasks. Regarding further fine-tuning, we think that if we finetune our pre-trained models on a diverse and high-quality dataset, it could enhance the model’s performance like most deep learning models.

Q6: Do you think your model can be adopted to other sound-related tasks, with rather little amount of data for domain adaptation? Or maybe collection of experts is needed?

A6: Our model could be used for other downstream tasks by finetuning the pretrained model, for example, text-to-audio generation. However, we need an appropriate decoder/adapter model for decoding embedding space to data space.

Q7: Would you classify your approach as essentially an "unsupervised" or "supervised" contrastive loss?

A7: We would like to classify our approach as a supervised method since the m-LTM needs to use aligned audio-text pairs for training.

Q8: Do you see any possibility on developing a "parameter-free" variant of your framework?

A8: We assume that you are asking about replacing the Mahalanobis distance with a parameter-free one, like kernel distance. We thought about that but left it for future work.

We appreciate the valuable time and feedback of reviewer to improve your work. We have revised our manuscript and would like to address your concerns outlined below

Q1: Retrieval applications have an expectation of scaling. Ideally a single query would be used to retrieve one or more corresponding examples from an extremely large source. However, in this paper the datasets (particularly the test sets) have a fairly small source to retrieve from (a few thousand examples typically). It would strengthen the work substantially to demonstrate the capabilities of the algorithm to scale to instances where there are orders of magnitude more examples to retrieve from than queries.

A1: We agree that scalability is important for retrieval applications. An accurate similarity function for ranking is also crucial. In this work, we focus on measuring the similarity of examples in a rich and expressive joint embedding space designed for audio-text retrieval problems. We leave the investigation of the scalability issue to future work.

Q2: There is a claim that the correspondence in Figure 1 is obviously better in figure 1b than figure 1a. I think this is not as obvious as claimed. I don’t think this image adds particularly value to the understanding of the work.

A2: Figure. 1 demonstrates the joint embedding space for zero-shot sound event detection setting. We would like to show that the joint embedding space learned from our framework is more expressive than contrastive learning. All sound event labels are described in plain text by the template “this is a sound of {sound event}.” Each text embedding in Figure. 1 is equivalent to a sound event label. The sound event detection is now converted to an audio-text retrieval task, therefore, a compact and clustering embedding space is beneficial to the performance. Figure. 1b shows that the m-LTM framework encourages the compact clustering of audio and event texts in the embedding space compared with contrastive loss shown in Figure. 1a. The claim is supported by the performance of zero-shot sound event detection experiment shown in Table. 3.

Q3: Partial Optimal Transport appears to be performed by noising some percentage of the data prior to learning. When this noise is applied with a batch size of 2, there is a substantial likelihood that both batch elements will contain incorrect (corrupted) examples. Is there any risk in this from an algorithmic perspective?

A3: We agree that there is a risk in this when the batch size is very small. The underlying assumption of our algorithm is that there is at least a proportion of matched examples in a batch. If a batch is too small, either there are no matched examples, or all are matched. Hence, small batches are not the focus of this algorithm. The intuition of utilizing POT for noise correspondence data is to discard as many noisy pairs as possible. In the case of a small batch size, there is a risk of a whole minibatch of training data being noisy, POT regularization fails to handle that circumstance since it always imposes a percentage of matching(the mass parameter for POT) between two sources. The risk of a corrupted minibatch is critical for all existing methods, including my proposed approach. However, with a large enough minibatch, POT regularization is able to mitigate the harm of misaligned training data.

Q4: How does this approach deal with instances where no match is available?

A4: The POT approach deals with misaligned instances within a minibatch by a parameter, which forces only to match a percentage of audio-text pairs, the total transportation mass between two sources $0 \leq s \leq 1$. The total transportation mass of POT acts as a regularizer to discard the less certain matches, therefore, it is able to mitigate the harmfulness of mismatched training data.

Q5: The authors describe one of the positive attributes of this work as identifying a joint embedding space for both speech and text through which to perform retrieval. Has this embedding space been used for other downstream tasks? This would strengthen the argument that this is a good joint space rather than solely useful for retrieval.

A5: We conducted a zero-shot setting for a downstream task, sound event detection, on the ESC-50 test set in the Table. 3 to illustrate the expressiveness of the learned join embedding space.

Q6: How effectively does the learned representation work across corpus? For example, when training on Audiocaps, how effective is retrieval of Clotho test sets and vice versa? This would be a more effective measure of the robustness of the algorithm than the intra-corpus analyses.

A6: To answer your question, we compare our method with the baselines in the cross-corpus setting. As shown in the table below, the m-LTM is more robust across audio-text retrieval datasets than triplet and contrastive loss. Regarding training on the AudioCaps and testing on the Clotho dataset, our proposed method acquires the highest performance, 15.01%, in terms of R@1 for the text-to-audio retrieval task, compared with 10.41% and 12.31% for triplet and contrastive loss respectively. The same observation is seen for the setting training on Clotho and then testing on AudioCaps dataset.

Q7: The delta_gap described in table 4 is the difference between the means of the embeddings. First, is this gap an absolute value? it’s remarkable that mean(f(x_i)) is always greater than mean(g(y_i)). Second, is the corpus mean of text and audio embeddings a reasonable measure of performance? it seems like this measure could be easily gamed by normalizing the embedding spaces to have a 0-mean, but this wouldn’t add anything to the transferability of the embedding. Also distance in the embedding space isn’t well defined. It’s not clear that a unit of distance in the triplet loss space can reasonably be compared to a unit in the m-LTM space.

A7: The gap value is the length of the discrepancy vector $\vec{\Delta_{gap}} = \frac{1}{n}\sum_{i=1}^n f(x_i) - \frac{1}{n}\sum_{i=1}^n g(y_i)$ which is subtracting vector between the mean of embedding vector of two modalities, we described it in the second paragraph of section 5.1. Therefore, the gap is always positive. The discrepancy between the two modalities proved that it is an effective metric for studying the transferability of a jointed embedding space[1]. We use Euclidean distance on the embedding space to measure the modality metric. To achieve a fair comparison, all embedding is normalized to unit vectors and then used to compute the modality gap.

[1] Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning.

We appreciate the valuable time and feedback of reviewer to improve your work. We have revised our manuscript and would like to address your concerns outlined below

Q1: A comparison with the entropy regularized contrastive loss baseline

A1: Below are the results of the baseline (CL with entropy regularized) suggested by Reviewer efev, in comparison with our method and the one without entropy regularization (CL w/o). More specifically, the baseline is trained by using contrastive loss and entropy regularized on the matching matrix, $L(\theta,\phi) = L_{CL} + \epsilon*L_{ent}(R)$, where $\mathcal{L}_{CL}$ is contrastive loss described in the Equation. 1, R is the ranking matrix between two sets of audio and caption and $\epsilon$ is the coefficient of entropy regularized term. We have reported the experiment in Table. 7 in Appendix A.4. The experimental results demonstrate that entropy regulation can boost audio-text retrieval performance slightly, however, there is a significant gap in performance between the m-LTM and contrastive loss with entropy regularization.

Q2: The m-LTM method presented is somewhat lower in novelty as a variation on LTM that investigates the use of minibatch training and deep networks, but 'revisiting' is explicitly called out in the title, and this seems a worthy exploration.

A2: We agree that adapting the learning-to-match framework for deep learning is a valuable exploration. However, we acknowledge that the title and abstract can not be modified during the discussion stage. We will definitely change our title regarding your comment when submitting the camera-ready version.

Q3: There are a few grammatical errors throughout the paper, but the paper is, in general well structured and adequately well-written. A glaring exception is the abstract, which is in really poor shape. Authors, please resolve this.

A3: We have corrected all the grammatical errors and will definitely restructure the abstract when submitting the camera-ready version. Here is our new abstract:

The Learning-to-match(LTM) framework proves to be an effective inverse optimal transport approach for learning the underlying ground metric between two sources of data, facilitating subsequent matching. However, the conventional LTM framework faces scalability challenges, necessitating the use of the entire dataset each time updating parameters of ground metric. In adapting LTM to the deep learning context, we introduce the mini-batch Learning-to-match (m-LTM) framework for audio-text retrieval problems. This framework leverages mini-batch subsampling and Mahalanobis-enhanced family of ground metrics. Moreover, to cope with misaligned training data in practice, we propose a variant using partial optimal transport to mitigate the harm of misaligned data pairs in training data. We conduct extensive experiments on audio-text matching problems using three datasets: AudioCaps, Clotho, and ESC-50. Results demonstrate that our proposed method is capable of learning rich and expressive joint embedding space, which achieves SOTA performance. Beyond this, the proposed m-LTM framework is able to close the modality gap across audio and text embedding, which surpasses both triplet and contrastive loss in the zero-shot sound event detection task on the ESC-50 dataset. Notably, our strategy of employing partial optimal transport with m-LTM demonstrates greater noise tolerance than contrastive loss, especially under varying noise ratios in training data on the AudioCaps dataset.