Questions:

Q. Regarding the data curated for training, there are CompA-661k, and AudioSet-compA, another 110k audio-text pairs for complex compositional audios? How are these different subsets used in different training scenario? It might worth adding a table to show clearly what combination of datasets, training strategies, whether there is hard negative and modular contrastive utilized, and link them directly to your abbreviation in the result table (CLAP, CompA-CLAP, CLAP-CompA-661k, etc.) Currently it is not easy to associate from the narratives to exactly how each variation in the result tables comes from.

We apologize for the confusion. Here is a table to make things clearer. We have also added this table to our Appendix as Table 7.

Table 1: List of various datasets used in our experiments, their sources, and the specific experiments they are used in.

Note: As also mentioned in the paper, no audio used to create our CompA benchmark was included in the training set at any stage. For modular contrastive learning, we only take individual sliced acoustic events from the AudioSet Strong (not the entire audio), to generate our own synthetic data.

Q. In Table 1, what is the difference between CLAP (ours) and CLAP-CompA-661k (ours)?

Ans: We apologize for the mistake and thank you for pointing this out. CLAP (ours) and CLAP-CompA-661k (ours) are the same models, and CLAP-CompA-661k (ours) in Table 1 (right) is a typo. Since we trained our version of CLAP on CompA-661k, we had initially named it CLAP-CompA-661k but later changed it to CLAP (ours).

Q. For the common mistakes mentioned in the discussion, is it possible to perform some error analysis? Or confusion matrices to show these more quantitatively?

Ans: Thank You for the suggestion. We have added both quantitative and qualitative comparisons in Appendix C.4 in Tables 11, 12 and 13. In terms of qualitative analysis, we have highlighted specific acoustic compositions in the CompA-order and CompA-attribute benchmarks where we observed significant improvement in CompA-CLAP compared to the other baselines.

Table 2: Some examples of acoustic compositions in the CompA-order and CompA-attribute benchmarks where we observed significant improvement in CompA-CLAP compared to the other baselines

References:

[1] Hershey, Shawn, et al. "The benefit of temporally-strong labels in audio event classification." ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021.

[2] Gemmeke, Jort F., et al. "Audio set: An ontology and human-labeled dataset for audio events." 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017.

[3] Liu, Xubo, et al. "Wavjourney: Compositional audio creation with large language models." arXiv preprint arXiv:2307.14335 (2023).

We thank you for your thorough review and constructive feedback. We have tried to address each of your concerns point by point.

Weaknesses:

Q. It is a bit difficult to get an entire picture of number of different datasets curated and how each of them are curated, a lot of the information are buried in the appendix. Consider in the main paper including a table describing different datasets curated, how many of them for each subset used for training, and evaluation, their source, and methods (leverage ChatGPT, replace the sounding object, swap the orders, etc.), and then refer to the table in the narratives.

Ans: Thank You for your suggestion. Due to space limitations, despite our best efforts, we could not include such a table in our revised version due to space constraints. However, we have made the following changes to make it clearer:

• We have added Table 7 in the Appendix of our paper. This Table describes all the data used in our experiments, including their source and the stage of training in which they were used. We also show the Table in the Questions section.
• We have made several writing changes in all sections to make it clearer which data is being used in which stage of training. The changes can be seen in the revised version of our paper.

Q. The data curation is one of the main contributions of this work. However, currently, a lot of details are left in the appendix. Consider integrating more information, such as the source datasets and some manipulated examples to the main narratives, this can help the reader to understand this work early on.

Ans: Thank You for your suggestion. We have made the following changes to incorporate more details about data curation both in the main paper and appendix:

1. We expanded the details about the annotation in Section 2 and included more details about the exact annotation process.
2. In Appendix B3, we provide more details about annotator demographics.
3. In Appendix B3, we list some rules adhered to during the annotation process.

We would also like to point out that both our proposed contrastive learning approaches are novel and also serve as the main contributions of the paper. Compa-CLAP serves as the first step toward solving compositional reasoning in audio-language models. Thus, to accommodate everything within the 9-page limit, we made a division by allocating 3 pages to the dataset description and 4 to Compa-CLAP.

Q. A minor suggestion: The figures and tables are far from the actual text in the appendix, consider reorganize them to make it easier to read.

Ans: Thank You for your suggestion. We have reorganized the Figures and Tables in the appendix of the revised version of our paper to make it easier to read.

Q. A minor comment: A lot of the texts in the tables and the figures are very small and difficult to read.

Ans: Thank You for your suggestion. We have revised the text in our figures in the revised version of our paper to make them clearer.

Dear reviewer Cw3v,

Thank you for the time you spent reviewing our paper. We have submitted our response to your concerns, including a revised version of our paper. Please let us know your comments and if you have any more concerns. Thank You again!

We appreciate your acknowledgment of CompA's importance, clarity, novelty, effectiveness, and consistent performance gains. We're grateful for your recommendation to accept our paper. We have tried to address each of your concerns point by point.

Weaknesses:

Q. As previously stated, my major (and almost only) criticism comes from the fact that the concept of « compositionality » presented on this paper is rather largely focused on « temporal » compositionality. Although I understand that it is mandatory to make choices as this is still a very young research topic, I think the paper would gain in strength if a more clear definition of various types of compositionality would be presented.

Ans: Thank you for your valuable suggestion. We have added a subsection in the Appendix discussing different types of compositionality observed in audio. The same can be found in Appendix A.

Questions:

Q. With regards to generating examples, although I understand that using prompts and GPT can perform a large array of tasks, your temporal task could be quite easily simulated (collating examples together, such as done with the Mixup augmentations). Could you quantify gains and differences with those more straightforward approaches ?

Ans: Thank You for the question. Below, we have compared the results with traditional text augmentation methods, such as switching nouns and verbs using the Spacy library. We find that GPT-generated negatives outperform traditional text augmentation techniques. We hypothesize that this is due to the lower-quality and incoherent negatives generated by traditional NLP methods. We compare negatives from various LLMs and traditional NLP methods in Table 8 in the Appendix. Additionally, we provide code in the Supplementary material.

Table 1: Comparison of CompA-CLAP with negatives generated by GPT-4 and Traditional NLP methods.

Q. Why do the authors fine-tune two separate times, instead of performing a single phase of joint fine-tuning?

Ans: We implement a two-stage training strategy to address specific challenges observed in the fine-tuning process as pointed out by Goyal et al. (2022)[1], Wortsman et al. (2022)[2], and Kumar et al. (2022). The first stage involves fine-tuning the model while sampling a single positive and negative per audio. This initial phase focuses on handling hard negatives in a batch. The second stage follows, fine-tuning the model with multiple positive and negative audi- text pairs, addressing the complexity of having multiple positive and hard negative sentences simultaneously. This two-stage learning approach is designed to allow the model to first learn to handle hard negatives individually before tackling the more intricate task of managing multiple positive and hard negative instances concurrently. Empirically, we find 2 stage training to perform better than single stage training.

Q. Why does the fine-tuning occur (hard negatives (3.3) --> synthetic pairs (3.4)) instead of the reverse ordering?

Ans: The core rationale behind performing hard negative training first and modular contrastive training next is a Coarse-to-Fine Contrastive Learning approach. Hard negative training with an entire caption as a positive or a negative in the batch is an easy task and teaches CLAP to distinguish between entirely different contexts and concepts. On the other hand, modular contrastive training, a bit more difficult task, makes CLAP focus on specific compositional relationships of the entire scene, thereby refining its ability to discern subtle distinctions and nuances within similar contexts. Table 3 provides results when the order is changed. Empirical results prove that the order matters for training.

Table 3: Comparison of CompA-CLAP results with the order of compositionally aware hard negatives and modular contrastive learning changed.

Q. All of these decisions could be validated empirically, but I do not see these results in the paper.

• Both examples in Figure 4 both seem to be flawed training example. (1- Left) The negatives both appear to be correct. All of these examples (the true caption, and the negatives) describe three sound events occurring simultaneously. (2 - right) "Tiger roars amidst thunker" and "Tiger roar followed by" cannot both be positive labels for the same sample - only one can be correct. Both of these issues seem to raise concerns about the quality of the data used here; particularly since the "highly custom prompt" described does not appear to be shared in the paper.

Ans: We apologize for any confusion and would like to provide a more detailed clarification regarding Figure 6. The figure illustrates three distinct acoustic events: "Tiger roar," "Thunder," and "Human conversation." The ultimate objective is to combine these acoustic events to construct a coherent acoustic scene, specifically "Tiger roar followed by human conversation amidst thunder." As thunder serves as a background element, "Tiger roars amidst thunder" is considered a valid positive scenario. Furthermore, within the acoustic scene, the tiger roar precedes the human conversation, making "Tiger roar followed by human conversation" another valid positive sample. You can find more examples in Appendix, Table 9.

• The paper makes several additional changes to the CLAP model. Ans: Yes we only make a single change to the original CLAP model proposed by Wu et al. (2023). We replace the RoBERTa encoder with an instruction-tuned Flan-T5-large encoder (Chung et al., 2022) as this has shown to further improve performance. This is detailed in Section 3.2 Methodology paragraph.

We thank you for your thorough review and constructive feedback. We have tried to address each of your concerns point by point.

Major Comments:

Q. The tone of the paper often leans too strongly toward "sales pitch" for my taste. I would recommend to tone down this language, and focus on making quantifiable and verifiable claims based on the data and methods in the paper. Just a sampling of the phrases the authors use to dismiss other work, minimize problems, or promote theirs:

• "only a word changed" (a few sentences after the claim "the audios have the exact same words")

Ans: The two claims refer to two different types of captions. The captions for the audios with two pairs have the exact same words but in a different order. For audios with three pairs, only a single word that defines the preposition between the events is changed. We have incorporated this change in the revised version of the paper.

• "it is trivial for ALMs to perform well on these benchmarks"

Ans: We have replaced this instance of the word “trivial” with 'easy' to convey the intended meaning. We have made the necessary changes in the revised version of our paper.

• "CLAP undergoes only minor degradation in retrieval performance"

Ans: We have changed the wording of the caption and also added metrics to it in the revised version of our paper (Figure 1).

-"25% synthetic...but highly curated by expers" (what is highly curated?)

Ans: The term "highly curated" implied a strong level of involvement and careful selection by experts in the generation process and validation process. We have addressed this by using clearer language, specifying that the audios are "carefully validated by experts who also supervise the generation process." This revision provides a more detailed and precise description of the expert's involvement in both the validation and supervision of the synthetic audio generation.

• "a trivial but ineffective solution"

Ans: We have replaced this instance of the word ‘trivial’ with the word “simple” to convey the intended meaning. We have made the necessary changes in the revised version of our paper.

• "a highly custom prompt"

Ans: The term "highly" was used to emphasize the thorough and thoughtful nature of the prompt customization process. However, to provide a clearer and more concise description and tone down the language, we have now simplified it to "custom prompt," which still conveys the idea that the prompt was carefully tailored for the task without explicitly using the word "highly." (Section 2.3)

• "Our version outperforms ... by significant margins" (no measures of significance used)

Ans: Our version of CLAP outperforms [1] on all existing retrieval benchmarks for both text-to-audio and audio-to-text retrieval by 0.15%-4.67%, and CompA-order and CompA-attribute by 11.85%-23.8%. We have incorporated this change in the revised version of the paper.

Q. Since the authors are not only introducing a novel model, but actually proposing their dataset to serve as a benchmark, more justification and detail are needed. For example:

• There is no mention in the paper of where the benchmark can be accessed, how it can be used by other works, what format it is provided in, etc. This is essential information for a public benchmark (it is ok to maintain anonymity by describing where the benchmark will be available upon acceptance, if the description is sufficiently clear).

Ans: Post acceptance, we plan to release the entire dataset on our GitHub with CC-BY-NC 4.0 License. Together, with the benchmark, we also plan to release all evaluation codes and training codes for benchmark.

To ensure uniformity in scores, we plan to release a website like the SUPERB Benchmark for spoken language processing tasks (https://superbbenchmark.org/). Researchers will be able upload their models for automatic evaluation and see how their models performs on private and public leaderboards. Since evaluation can be done on a CPU, this would not be a great overhead. The leaderboard will host scores by averaging across 3 runs and report standard deviation across scores.

Note: We are not authors of the SUPERB benchmark.

For now, we have provided a sample of our dataset in a presentation (.pptx file) in the supplementary materials and have also included an anonymous link to access the complete dataset in the supplementary readme file.

References:

[1] Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5. IEEE, 2023.

Dear reviewer Xast,

Thank you for the time you spent reviewing our paper. We have submitted our response to your concerns, including a revised version of our paper. Please let us know your comments and if you have any more concerns. Thank You again!

We thank you for your thorough review and constructive feedback. We have tried to address each of your concerns point by point.

Weaknesses:

Q. The core technical contribution of submission boils down to - "selecting appropriate negatives for the task at hand", an established idea in contrastive learning, thus raising the question of novelty in terms of technical contribution.

Ans: We agree that hard-negative-based contrastive learning is a well-established idea in contrastive learning. However, as also stated in our paper, we would like to argue that just adding hard negatives is not our primary technical contribution. Our primary contributions are as follows:

1. (Technical) Section 3.3: We modify the formulation of naive contrastive learning with hard negatives such that hard negative captions for a particular audio are ignored by other audios in the batch.
2. (Technical) Section 3.4: We propose an entirely new algorithm called “modular contrastive learning.” Modular Contrastive Learning has two primary innovations:
   • We propose a novel and simple methodology of data generation that overcomes the data scarcity issue of compositional audios: Template-based synthetic creation of audio-caption pairs.
   • Our proposed method promotes fine-grained composition learning. Each audio in the batch has multiple positives that describe compositional relationships between certain acoustic events in the batch. Each audio in the batch also has multiple negatives for each positive. An example of this can be seen in Figure 4 (Figure 6 in revised version).
   • We modify the contrastive learning formulation such that each audio in the batch ignores the positives and negatives for all other audio.
3. (Non-Technical) Section 2: We propose two benchmarks, CompA-order and CompA-attribute, to evaluate compositional reasoning in Audio-Language Models (ALMs). The expert-annotated benchmarks, which are also the first of their kind, are complex to curate and annotate and annotate and represent a significant advancement in an understudied field. By focusing on this niche, our benchmark paves the way for a more nuanced understanding and development of ALMs.

Q. The structuring of manuscript could be better. e.g.,

• Transition from section 3.3 to 3.4 is not clear in the first reading.
• Every section reads like a mini-paper with its own background, methodology. May be add a paragraph at the beginning of section to prep the reader of upcoming sections and their differences.

Ans.: Thank you for bringing these concerns to our attention! In response, we've incorporated a few introductory lines at the beginning of each section to concisely outline the main theme and facilitate smoother transitions between sections, all while considering space constraints.

Questions:

Q. Have the author's considered rich transcription approach? Since authors are using strongly annotated data, can the model be trained to output a sequence like .. <event1_start> <event2_start><event2_end><event1_end> .. here event2 occurs in the midst of event1. This would avoid the process of carefully designing negatives, and the model would still learn temporal relationships among audio events. Attributes can also be modeled in a similar fashion. I would imagine this format could be more flexible in terms of attribute modeling. (e.g., personA-talking vs personA-singing and so on.. ). An LLM should have no problem taking such an output sequence and convert it to human-readable format (either implicitly or explicitly).

Ans: Yes, we have. Indeed, this is the exact methodology we employ to generate AudioSet-CompA, a novel dataset contribution of our paper (described in Section 3.3). As the Section specifies, we generate AudioSet-CompA by prompting GPT-4 for time-aligned compositional captions for audios in the AudioSet strong dataset. The AudioSet strong dataset has time-aligned labels, i.e., the start and the end time for each acoustic event in the 10-second audio is specified. An example is:” [('Male speech, man speaking', 2.532, 3.207),('Cluck', 3.087, 3.388),('Generic impact sounds', 3.181, 3.388)]” Thus, we prompt GPT-4 with these time-aligned labels and ask it to generate a coherent caption from this, and indeed, GPT-4 performs accurately at this task. The prompt is provided in Appendix B.5.

For compositionally aware hard harmful generation, detailed in Section 3.3, we do not employ this as we already have captions available for each audio. Thus, we directly prompt GPT-4 to generate hard negatives from this caption. The prompt is provided in Appendix B.5.

Q. Can the author's clarify if the order of hard negative training and modular contrastive training matter. (section 4: 2nd paragraph).

Ans: The core rationale behind performing hard negative training first and modular contrastive training next is a Coarse-to-Fine Contrastive Learning approach. Hard negative training with an entire caption as a positive or a negative in the batch is an easy task and teaches CLAP to distinguish between entirely different contexts and concepts. On the other hand, modular contrastive training, a bit more difficult task, makes CLAP focus on specific compositional relationships of the entire scene, thereby refining its ability to discern subtle distinctions and nuances within similar contexts.

Table 1 provides results when the order is changed. Empirical results prove that the order matters for training.

Table 1: Comparison of CompA-CLAP results with the order of compositionally aware hard negatives and modular contrastive learning changed.

Dear reviewer x6Uk,

Thank You for your time in reviewing our paper. We have submitted our response to your concerns, including a revised version of our paper. Please let us know your comments and if you have any more concerns. Thank You again!

Thank You for the question. We are grateful for your time and glad we could address your concerns regarding the paper. We hope that addressing all your concerns can impact our score positively.

We acknowledge that the suggested rich transcription methodology can be used for training CLAP. However, below, we would like to answer why we did not try the rich transcription methodology in our experiments. However, we assure you that we will add this experiment to the final version of our paper as requested.

1. CLAP-like models (like the one employed in experiments in our paper) are trained to understand natural language. Understanding natural language helps in free-form text-to-audio retrieval and vice-versa, which is used in downstream applications. Outputting such a sequence and training the model on such a sequence using contrastive learning affects natural language understanding of the model and might make the model act as a “bag of words”. Let's take an example:

   Rich Transcription: <human> <bird></bird></human>

   Two compositionally different captions:

   • A person speaking in a serene outdoor setting, intermittently interrupted by the chirping of birds."
   • Birdsong dominates the soundscape with faint, distant human voices in the background.

   Rich transcriptions like the one provided can translate to compositionally different audios by highlighting the unique combination and interaction of various sound elements within the scene. Thus, this would hinder learning compositionality and specific nuances in audio.

2. Training using rich transcriptions avoids the model learning variabilities in free-form natural language. This hinders real-world downstream applications.

3. Rich transcriptions, though beneficial in learning temporal ordering, do not provide a straightforward way to capture noun attributes. Natural language captions often describe important nuances in audios through attributes binded with nouns, such as “sad” in “sad song” or “baby” in “baby tiger”. Thus, training with hard negative natural captions makes CLAP understand various audio cues, and training with rich transcriptions might make CLAP overfit to only focusing on temporal ordering.

4. Training this model would need an extra step in inference time to convert natural language into the rich transcription format since the model was trained to understand only rich transcriptions.

5. Elaborating on our motivation for Modular Contrastive Learning (Section 3.4) - Real-world audio is often complex and unpredictable with multiple acoustic events. This hinders the model from learning fine-grained attribute-binding and order information with captions or rich transcriptions. Thus, though rich transcriptions can be an alternative to compositionally-aware hard negative training (Section 3.3), Modular Contrastive Learning still solves this significant issue that rich transcription methodology still faces.

We would also like to mention that our proposed approach for hard-negative-based contrastive learning, which employs compositionally different hard negative captions as negatives, has also been implemented by several prior works in the vision-language space [1,2,3]. All these works have also been cited in our paper. However, our exact formulation for loss calculation is different and innovative and is elaborated in the rebuttal - Response to Reviewer x6Uk (1/2).

[1] Paiss, Roni, et al. "Teaching clip to count to ten." arXiv preprint arXiv:2302.12066 (2023).

[2] Yuksekgonul, Mert, et al. "When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It?." The Eleventh International Conference on Learning Representations. 2022.

[3] Momeni, Liliane, et al. "Verbs in action: Improving verb understanding in video-language models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.