An eye for an ear: zero-shot audio description leveraging an image captioner with audio-visual token distribution matching

Thank you for your review. We’re glad to hear that you found the exploration of our loss functions as novel directions for alignments between different modalities and the timely application scenarios of audio-visual LLM valuable. Below, please find a point-by-point response to your feedback.

The motivation is unclear. The author should specify practical scenarios where zero-shot audio captioning is needed.

We posit that Zero-shot audio captioning is particularly valuable in scenarios where annotated audio data is scarce or unavailable. Current audio captioners like Audio Flamingo and Pengi, trained on nearly all publicly available audio-text pairs, seem to have reached the limits of the supervised approach. In contrast, the abundance of videos makes unsupervised audiovisual training a viable way to scale further. For example, domain-specific topics (like wildlife sounds) that lack supervised datasets but non-annotated videos are available, which is realistic given that there exist only two "real" academic audio captioning datasets.

The author randomly selects one frame from a video without analyzing the relevance of the image to the audio.

We indeed chose one random frame from the video to match its distribution with the audio and analyzed the image-audio relatedness. The paragraph Datasets of Section 4 as well as the full section C of the Appendix details this process: (line 579-581) “As AudioSet (...) features audio events that may be unrelated to the visual event depicted (...) we filtered out audiovisual discrepancies”. Pairs were chosen “by computing the similarity between the noisy audio labels and an image caption generated by BLIP-2” (line 269-270). This filtering considers the entire video,averaging similarity scores across frames sampled at one-second intervals. To address how often the image reflects the audio, we retained the 500k cleanest pairs, which best retain related pairs (line 269: “we train our models on a subset of 500k videos from AudioSet”). Additionally, since the image and audio can contain partial, non-common information, we used attentive optimal transport to learn only the common parts of the distributions (Section 3.2, paragraph 3).

The author states that they used image-text pairs for p-tuning, which differs from "zero-shot learning"

We acknowledge your concern about potentially breaking the “zero shot” set-up with our handling of data. To clarify this concern, as we explained in line 53 of the submission, and consistent with Salewski et al., we define zero-shot audio captioning as the model's ability to generate audio captions without training with manually assembled audio-caption pairs, as they also used textual-only descriptions (AudioSet tags from a "keyword bank") as opposed to actual audio files and their corresponding captions.

While we did use captions for prefix tuning, we emphasize that we never used audio-caption pairs for the targeted audio captioning task, which would indeed compromise the experimental setup. Instead, we used a few image-caption pairs without accessing the corresponding audio files. This use was solely to retask the LLM to create audio-centric captions. We maintain that our approach effectively addresses the zero-shot audio captioning task, allowing direct comparison with Salewski et al. and other methods.

Is the comparison with ImageBind fair? Data used for ImageBind p-tuning isn't specified. Why is ImageBind absent from Table 2?

Firstly, we would like to clarify that our work does not compare directly to ImageBind, as it lacks the capability to perform audio captioning. Instead, it is a model that maps multiple modalities to a shared space. Hence, comparing our method directly to ImageBind is not feasible.

We instead compare with Shaharabany et al., which, as described in Lines 108-114. This work is the closest to ours in terms of backbone training setup, as it has been trained using audio-image and image-text pairs but not audio-text pairs.

Second, Shaharabany et al. did not perform p-tuning, relying instead on an “audibility score” (part of the training loss, core of their work). In our case, we used p-tuning for the same purpose (using only image-text pairs). Since neither work used audio-text pairs, comparison is feasible and fair.

We acknowledge that, as ImageBind also contains an image encoder, prefix tuning can be applied. Unfortunately, as Shaharabany et al.’s code is not available, we couldn’t test his interesting idea.

Finally, the work of Shaharabany et al. is not included in Table 2 as they did not report their results on Clotho. We will clarify this by extending Table 2’s caption to ”Shaharabany et al. did not report results on Clotho.”

Does adding audio to the audio-visual description improves performance ? What was the purpose of this task and what additional value did it provide?

We would like to recall, that our method is designed to handle multiple modalities, such as image, audio AND audio-visual for the zero-shot audio captioning task. This is achieved by substituting or concatenating modality-specific tokens. As demonstrated in Table 1, our method performs better when both image and audio are used together (6th row, labeled DALI_OT^att+Image) compared to using the image alone (1st row). This highlights the usefulness of including audio, offering details about sound sources occluded or out of view in the image, e.g., a ringing phone inside a bag.

Our results show that using both modalities leads to more informative audio captions. We also show that using audiovisual captions as pseudo-captions to supervise an audio-only model in a second stage (audiovisual distillation), yields a more robust audio captioner. One core motivation was to leverage a powerful image captioner to enhance audio captioning, creating a virtuous cycle where both modalities mutually benefit. Thus, including audio proves significant value beyond what visual data alone can offer.

Thanks for the last answer and those new remarks. However, we do not agree with your points:

1-Concerning the difference between image only and audio only, it's important to notice that our method targets AUDIO captioning, which consists in generating a caption while having access to an AUDIO and not an image. Therefore, there is no point in trying to compare audio and image performance.

2-The prefix tuning does not introduce any bias or unbalance between methods as the prefixes are ALSO used with the image-only method, because as previously mentioned, the visual-LLM off-the-shelf is not able to produce audio-centric captions.

We would encourage the reviewer to perform a more careful reading of the paper as he/she seems to have missed a lot of important details which are very clearly stated.

Thank you for your insightful review. We’re happy to hear that you found our paper well written and easy to follow, appreciated the helpful diagrams, and recognized the value in our approach to addressing audio-visual correspondence and leveraging visual captioning systems. Below we address your concerns.

Using too many metrics muddies the results.

We understand that using many metrics complicates the reading. Here's a simplified analysis: CIDEr emphasizes the precision of n-grams (targeting key words), while SPICE focuses more on the semantic content, making it complementary to CIDEr. SPIDEr, an average of both, is therefore used as the primary metric.

On AudioCaps, all distribution alignment methods outperform the contrastive baseline (SPIDEr: 0.0871) after the first stage. While MMD (0.1385) performs better than standard OT (0.1183), attentive OT improves even further (0.1443). After distillation, all methods improve except for MMD (0.1360), with attentive OT remaining the best (0.1592) and the contrastive baseline improving significantly (0.1524). These results confirm the intuition that MMD learns a visual bias and therefore does not benefit from the distillation. Note that other metrics and especially in ROUGE_l, the contrastive approach remains less effective than other methods (0.2914 against 0.3025 and 0.3106 for MMD alignment and attentive optimal transport respectively).

On Clotho, MMD’s visual bias is beneficial, outperforming other methods after the first stage (0.0640 vs. 0.0377, 0.0299, and 0.0355 for contrastive, OT, and attentive OT, respectively). Post-distillation, MMD does not improve (0.0655), while other methods do, with attentive OT (0.0625) and MMD remaining slightly ahead of the contrastive approach (0.0620). ROUGE_l shows similar trends to AudioCaps.

This discussion aligns with the paper’s conclusions. We'll revise the manuscript accordingly.

Line 96 suggests audio data lacks richness. However, this is a data selection and task issue, not an audio modality shortcoming.

We'll revise Line 96 for clarity: "Current audio captioning datasets often fall short in capturing the richness and context compared to what the vision datasets offer. This limitation arises from both the selection of data for captioning and the nature of the questions posed to captioners, leading to descriptions that are less precise and more subjective."

No mention is made of the size of the various models or how much data they were trained on.

We agree mentioning this information is important. We will revise as follows:

• Lines 104-107: “ZerAuCaps uses CLAP (an 80.8M-parameter model trained on 3.4M audio-text pairs) (...) to prompt an LLM (OPT 1.3B)”.
• Lines 109-111: “The authors adopted ImageBind (using an audio encoder of 86M parameters trained on AudioSet) (...) They fine-tuned a GPT-2 model (117M parameters) (...)”

The reported are low compared to SOTA. What is the value of including multiple such approaches.

We want to clarify that our approach extends image captioners to audio without compromising their original image performance. While better audio performance might be achievable by sacrificing some image performance (typically by unfreezing the LLM), our focus is on learning from videos for scalability due to their widespread availability. To our knowledge, this is the first attempt at this problem. This is a first step towards large-scale training, and a model trained on 500k videos cannot be compared with others trained on millions of audio-text pairs. Future work could scale this method further to compete with supervised approaches. For now, exploring alternative distribution alignment methods is valuable, but given the minor differences between methods, future work might focus on a single alignment method.

What was done in the ablation, what was taken away?

Tables 1 and 2 present ablations for different training stages. The rows show model performance after the first stage, which includes only distribution alignment and prefix tuning. What was taken away is the role of the second stage: audiovisual distillation (as written in the Table captions: “Ablation of audiovisual distillation”). We will clarify this further in the text.

Why wasn't the attention weighting used with MMD in addition to OT?

In attentive OT, cross-attention is applied in the token space to weight each sample of the empirical distributions. MMD, however, computes the expectation of the distribution in the kernel space, so weighting tokens before averaging would significantly alter the distance formulation, which is why we did not initially explore this. We acknowledge this is an interesting experiment and hace since tested it. The results, shown in Table B reveal that while performing well across all metrics,it underperforms in CIDE_r. The model generates overly detailed captions, which affects CIDE_r (measuring n_grams) but not in SPICE.

Does the audio-visual alignment treat each instance as a "bag of tokens"? Would there be value in considering the sequence of tokens ?

Audio-visual alignment does treat each instance as a "bag of tokens". However, our preliminary experiments revealed that token order does impact llava's performance in audio captioning with images (on AudioCaps). To support this claim, we shuffled image tokens before feeding them to the LLM and found that the results were similar to those without shuffling (see Table A). This indicates that the sequential order of image tokens is not crucial for this task.

Were any categories held out of training to be evaluated in zero-shot category generalization?

We only focused on zero-shot modality generalization, but exploring other aspects is worth pursuing in future work.

Was the filtering applied to both training and test data ?

The filtering was applied only to the training set to keep test performance comparable with other methods. We will clarify this in Line 271.

Thank you for your review. We’re pleased to hear that you found our paper well-written and easy to follow and appreciated the novelty in our use of MMD/OT for multi-modal alignment in the audio-visual space. In the following, we'll address your comments in detail.

There is a lack of insights for showing why the OT and MMD can outperform, more explanations are needed.

First, regarding the performance gap between our method and the contrastive baseline, we explain the need for distribution alignment to perform this task instead of contrastive learning in Section 3.2, lines 200-206: “Llava for instance, makes use of the full token distribution as input, naturally creating the need for a full token distribution alignment, to allow for swapping the pretrained encoder with a new one targeting a new modality. Moreover, contrastive learning faces the so-called “modality gap” problem where each modality is encoded in distinct sub-regions of the embedding space (see Figure 4). For all these reasons, replacing image tokens with audio tokens obtained by alignment through a contrastive learning approach may yield undesirable responses from the language model.”

Concerning the modality gap that does not appear in our method, it is important to note that contrastive learning brings associated (positive) samples close together while pushing away unpaired (negative) samples. The cited paper [A] showed that by pushing away the negative samples in a high-dimensional space, combined with multiple false positive pairs, multimodal contrastive learning encodes each modality in a separate subspace, creating a modality gap. A more recent paper [B] goes further and shows that this phenomenon is purely due to the behavior of the contrastive loss itself in high dimensions and is not related to multimodal learning. Now, optimal transport and Maximum Mean Discrepancy (MMD) do not suffer from this issue, as they do not rely on negative samples but focus on explicit distribution alignment instead. However, their application in audio-visual alignment has been limited due to the common practice of averaging the final representation into a single token, which does not allow for the computation of distribution distances. Employing these losses for their ability to avoid the modality gap is a core motivation behind our work.

We will modify the beginning of the Discussion paragraph in Section 5 to make clearer our motivations and why our methods do not suffer from such a modality gap. Specifically, we’ll complement it by adding the following paragraph: "Unlike standard contrastive methods, we do not average tokens, which allows us to employ sample-level distribution alignment methods that do not depend on negative samples. Indeed, both Optimal Transport (OT) and Maximum Mean Discrepancy (MMD) focus on aligning the distributions of the embeddings directly. OT minimizes the cost of transporting one distribution to another, effectively aligning the distributions in a way that does not rely on negative samples. This approach avoids the issue of pushing different modalities into separate subspaces. Similarly, MMD measures the distance between the kernel-projected mean embeddings of the distributions, ensuring that the overall distributions are aligned without the need for negative sampling. By leveraging these methods, we can achieve a more cohesive embedding space where different modalities are aligned more closely, thus avoiding the modality gap problem inherent in contrastive learning approaches."

Why would Contrastive outperform other methods in this Clotho without audiovisual distillation?

First, on Clotho, before the audio-visual distillation, the alignment through MMD, which is also part of our method, yields significantly better results (sometimes twice as good) than the contrastive baseline (0.0659, 0.0621, 0.0640 against 0.0460, 0.0293, 0.0377 for CIDEr, SPICE and SPIDEr, respectively). As explained in the paper, the good performance of MMD is likely to be due to the learned image bias: line 343-344: “the image bias inherent in DALI_MMD becomes beneficial when confronted with out-of-distribution data.” It is important to note that Clotho is known to be a more challenging dataset than AudioCaps. Moreover, it contains new audio concepts not seen in AudioSet, representing an out-of-domain scenario for our models (trained only on AudioSet), hence the observed drop in performance, compared to AudioCaps. CIDEr and SPICE are indeed important metrics for captioning, and these metrics are roughly equivalent between contrastive and attentive OT (0.0302 0.0355 for attentive OT against 0.0293 0.0377 for contrastive in CIDEr and SPICE respectively). As both scores are fairly low, trying to interpret such small differences would be unreliable. However, those models exhibit significantly different performances when focusing on other metrics. In particular, the ROUGE_l, which measures the longest common subsequence, and takes the order of the words into account (interpreted as the “coherence” of the generated sentence), is significantly higher in our method (0.2101 for attentive OT against 0.177 for the contrastive). A tentative interpretation could be that, while both methods failed to generate accurate enough captions, the attentive OT still encodes the audio in a space that can be "comprehended" by the LLM as it stays in the same region as the image tokens, ending up in a more coherent caption, while the contrastive approach encodes something that cannot be interpreted by the LLM, leading to hallucinations or incoherent captions.

[A] Weixin Liang and Yuhui Zhang and Yongchan Kwon and Serena Yeung and James Zou (2022) Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning [B] Abrar Fahim and Alex Murphy and Alona Fyshe (2024) It's Not a Modality Gap: Characterizing and Addressing the Contrastive Gap

Thank you for your thorough review. We’re glad to hear that you found our paper’s goal, contributions, and execution clear and appreciated the good set of experiments and visualizations, along with the improvement in alignment demonstrated by our application of MMD or OT learning. We are now addressing your comments in detail.

Section 3.1 is rather confusing. It seems Figure 2 was revamped and the references to the figure in section 3.1 was not completely updated. Figure 2 should also include a description of steps 2d and 2e in the descriptive caption (the description of 1c needs updating too I believe. The reference on line 177 for 2-c should probably be 2-b; line 190 2-c should be 2-d; and on line 196 2-d changed to 2-e. Overall, it would be clearer to first indicate the stage and then explain it, e.g. "1-a) prefix tuning the prefix tokens" rather than the other way around. Also each of the bolded titles in section 3.1 should correspond with a stage in Figure 2 and include the stage number for better clarity.

We thank the reviewer for pointing out these corrections. We will ensure to make the following changes:

• Figure 2 caption: Full training pipeline: a prefix tuning is performed using a few image-captions pairs (1-a). In the meantime, the audio backbone is aligned with the image backbone (1-b) through distribution alignment. Audio captioning can then be performed by switching the image backbone with the audio backbone while adding the prefix tokens (1-c). Visually-informed audio captions are then generated using both audio, image, and prefix tokens. The MLP that maps the audio encoder to the language model is fine-tuned using these pseudo captions (2-d). The final inference is performed by forwarding the output of the aligned audio backbone to the trained MLP to obtain the LLM input (2-e).

• References to Figure 2: line 177 “We align the distributions (Figure 2:1-b)”; line 190 “This procedure is illustrated in Figure 2: 2-d”; line 196 “The resulting audio captioner is shown in Figure 2: 2-e”

In section 3.2, it would be helpful to the reader to provide concrete equations on how to compute the quantities and eqns 1-5. Having these in an appendix would be fine. While its understood that the code will eventually be released, having the equations that the code is intended to implement is always useful for clarity.

We agree with those remarks. Since the details of the $\alpha$ and $\beta$ computation are already given in Appendix F, and since the OT does not have a closed-form solution, the title of Appendix F will be changed to “Implementation details” and the following precision will be added to ease the understanding of the practical computations of MMD:

The MMD distance between the audio token distribution ($X$) and the image token distribution ($Y$) is computed as follows:

$\text{MMD}(X, Y) = \frac{1}{m^2} \sum_{i=1}^{m} \sum_{j=1}^{m} k(x_i, x_j) + \frac{1}{n^2} \sum_{i=1}^{n} \sum_{j=1}^{n} k(y_i, y_j) - \frac{2}{mn} \sum_{i=1}^{m} \sum_{j=1}^{n} k(x_i, y_j)$ where $k$ is the Gaussian kernel.

Question: Has there been any analysis done in comparison to other large, strong multimodal LLMs such as GPT-4o, Gemini 2 or Llama 3?

It is important to keep in mind that the purpose of our work is to extend vision captioners so they can also perform general audio captioning (i.e., describe an audio event—possibly happening outside of the field of view—, such as ‘a dog barking in the street' or ‘a siren heard at a distance'). Large Multimodal models like the suggested ones are trained to perform various tasks, including video captioning. However, they largely rely on the visual modality for that task, and the use of the audio modality is restricted to the speech present in the video.

Therefore, no fair comparison can be made with them as they are not trained to perform the same task as us (general audio captioning).