Weakly-supervised Audio Separation via Bi-modal Semantic Similarity

We appreciate the reviewer’s positive review of our work and the interesting questions raised. In the following, we will address these questions.

1. "Although it might not be a weakness, it is a bit questionable how the model performance will improve in large-dataset settings."

Our response: in our experiments, the VGGSound and AudioCaps datasets contain nearly 200K and 50K samples, respectively. In addition to extensive experimentation on each dataset (Table 1, 8, 10), we present large-scale semi-supervised training across datasets in Table 2. Therefore, in that sense, we can confidently claim our proposed framework yields significant improvement under decent size data regimes. However, if the reviewer’s question is to whether the gain that we obtain through our proposed weakly-supervised framework starts diminishing as the capacity of the model and the training data size becomes very large, that is an interesting research question. Intuitively speaking, that should happen when the generalization error starts converging toward its minimal value (i.e. the inherent noise/uncertainty level for the problem) as the model capacity and training data size become larger and larger. Answering this question experimentally, however, is beyond the scope of this work. Nonetheless, we believe this should not be considered as a weakness of our work since this issue can be equally raised against any contribution in the field aiming at reducing the generalization error of a Machine Learning model. And while the diminishing return phenomenon might indeed occur in massive-scale training regimes, we believe it is still valuable to significantly improve the model generalization in medium-scale training settings, which are, by the way, more accessible to the majority of researchers and practitioners in the field.

2. “Does the proposed method necessitate knowledge of all components present in the mixture? Furthermore, if the data contains background noises not mentioned in the text prompt, could this impact the model's performance?”

Our response: having the knowledge of all components in the mixtures would be ideal; however, realistically speaking, that’s not always possible; in such scenarios, the absent sources from the textual captions effectively act as “noise” during training. This type of noise particularly misleads the Consistency Reconstruction Loss (CRL) component which aims at reconstructing the original mixture from the predicted single-source components. But the good news is that depending on the noise level of our training dataset we can contain its negative impact by adjusting the relative weight of the CRL component. In fact, we have conducted ablation studies to find the optimal weight ratios for different loss components using heldout data from each dataset. The results are added in Table 7 and Appendix G.2 in the revised version. Interestingly, we observe that the optimal relative weight of CRL drops more than 50% from MUSIC dataset to VGGSound dataset, and 50% from VGGSound dataset to AudioCaps dataset. This trend happens to be the same order as the noise-level order for these datasets, which suggests that using this procedure, we can adapt the training dynamics of our framework based on the noise-level of the training dataset to minimize the negative effects of noise in training data.

3. “… if the prompt mentions people talking and birds chirping, but the recording was made in a city environment with background motor sounds, how would the model handle such discrepancies?”

Our response: Such discrepancies are partly handled by the CNT loss component which incorporates the frozen CLAP model pretrained on the large-scale text-audio pairs. The implicit assumption here is that the existence of extraneous and noise components in the model’s prediction would negatively impact the CLAP similarity score, which is reasonable due to large-scale pre-training of CLAP. As mentioned earlier, the other way we address this issue is by adjusting the relative weight of the CRL loss component based on the dataset noise level by conducting a hyper-parameter search procedure before training (see Appendix G.2). By doing so, we effectively minimize the negative impacts of the extraneous and noise components in the training data.

For further demonstration, we invite the reviewer to check out the new qualitative demo website uploaded on the anonymous Google Drive link below, containing samples with background noise and/or extraneous source components and their separation results using our model and the CLIPSep baseline.

Qualitative Demo: https://drive.google.com/file/d/1tR7o8cXoHqS9Ns8riHB_TD6Hfenoz0zH/view

We appreciate the reviewer’s diligent review and helpful suggestions. We have followed the reviewer’s suggestions to address their concerns, as we will explain below:

1. "The use of CLAP [...] is not as solid as it appears. This is because CLAP is also trained with the same noisy, weakly-labeled data. [...] This raises questions about whether the constraint effectively operates as intended,.."

Our response: we completely agree with the reviewer that the CLAP model does not provide sufficient supervision for the audio separation task on its own, not just because it’s trained on noisy data but also because the bi-modal semantic alignment itself is a different task than audio separation. This is exactly why we use the CLAP loss in combination with the mix-and-separate and consistency losses (see Table 6). Nevertheless, since CLAP has been trained on large-scale language and audio pairs, it provides an informative regularization signal for the separation task while at the same time enabling the introduction of single-source prompts via weak supervision. By incorporating it within our framework, we effectively transfer that knowledge to the audio source separation domain. To further show the importance of the CLAP loss, we have conducted a new set of ablation studies where we add the CLAP loss to the purely supervised learning; the results are added under the new appendix section G.6 and Table 11 in the revised version. As these results show, the addition of CLAP loss significantly improves the supervised learning performance across all cases. In other words, the CLAP loss provides valuable regularization, even in the case of purely supervised learning where we have full access to single-source samples during training.

2. "... why not use a timbre classier to verify it? [...]. Or a more direct question is: did you try this as an ablation study to show that using CLAP to align the semantic similarity is better than training a timbre classifier ..."

Our response: we have repeated our experiments with our proposed CLAP-based loss replaced by a timbre classifier, where the classifier shared the exact same architecture as that of the CLAP audio encoder, but the contrastive loss is replaced by the cross-entropy loss. We trained this loss using the same training data for the original separation problem under two scenarios: (I) concurrently with the separation model, and (II) pretrained beforehand. See the added results in Table 1 and the discussion in Section 4.1 of the revised version for more details. In short, we have observed that the concurrent version performs worse than some of the baselines, and while the pretrained version does better than the baselines, its performance is still significantly lower than our proposed CLAP loss, not to mention its restricted applicability due to the fixed number of classes. We hypothesize the superiority of the CLAP-based supervision comes from the large-scale pretraining of CLAP which enables us to transfer that knowledge to source separation. In other words, in the limit for large-scale training data and gigantic number of classes, the classification approach should perform as well as the CLAP-based loss, but at that point, we might as well use CLAP.

3. "... However, the experimental results on the MUSIC dataset, achieving an SDR performance of 7.9 and 9.9 under the semi-supervised learning paradigm, do not significantly outperform the results in the recent LASSNet paper [1] ...."

Our response: please note that the cited work and ours report two different metrics (SI-SDR and SDRi vs. SDR) which are not directly comparable. But even if we both used the same metrics, these two results would not be still comparable. Note that, here our primary contribution is a novel weak-supervision training strategy to reduce the gap between the training and test data distributions in the LASS problem. Our experiments are also mainly designed to evaluate this hypothesis while controlling for other factors. This is why unlike many other existing works (including AudioSep [1]), we've insisted on re-implementing the competitor methods using the same architecture and training data as much as possible. With that in mind, an AudioSep model trained using the same datasets and architecture backbone as in our experiments boils down to the “Supervised” row in our Table 2. This is because an AudioSep model is trained using the vanilla supervised strategy, for which we have shown the addition of our weak supervision through the CLAP loss does indeed improve the SDR significantly from 8.1 to 9.5. Of course if, like AudioSep, we had also trained our model on a much bigger multitude of datasets, the SDR could have been even higher, but that was not our primary objective in this work.

4. "any listenable demo for this paper? ..."

Our response: we have prepared a new qualitative demo website uploaded here: https://drive.google.com/file/d/1tR7o8cXoHqS9Ns8riHB_TD6Hfenoz0zH/view

We appreciate the reviewer’s meticulous feedback and suggestions. We have done our best to meet the reviewer’s suggestions. Below, we discuss these efforts point by point and address the concerns raised by the reviewer.

1. "The authors should make clear the distinction of when the proposed method is trained using only weak supervision and when it is semi-supervised trained..."

Our response: we apologize for the lack of clarity. All the results in Table 1 including our proposed framework row are actually obtained under unsupervised (multi-source) training regime. The supervised (single-source) column in Table 1 is merely provided as an upper-bound for the competitor models’ performances. On the other hand, the results for semi-supervised (i.e., mix of single-source and multi-source training samples) training scenario are provided in Table 2. We appreciate the reviewer’s useful feedback about the terminology clarification; following that, we have added a paragraph in Section 4 in the revised version, describing our terminology as the reviewer suggested. We have also clarified the terminology through our tables and captions.

2. "... I think that when one is using these large pre-trained networks on single-source data like CLAP, the underlying method becomes supervised in a sense, or to put it more specifically supervised with unpaired data..."

Our response: we note that CLAP is NOT pretrained on single source samples for the most part to begin with. The training data for CLAP consists of large corpora of uncurated audio-text pairs, many of which contain more than one sounding source mixed with various background noise components while accompanied by arbitrary text captions. Therefore, based on our definition of supervised, CLAP is not a supervised model either. The only advantage of CLAP over our framework is that it has been trained on massive-scale training data, which has enabled it to discern language semantics despite the noise. Even after such massive pre-training, CLAP by itself alone is not sufficient for audio source separation and needs our other loss components (see Appendix G.2 and Table 6). Only after combining it with unsupervised reconstruction and consistency losses, we can introduce the benefit of weak supervision on the top of the classical mix-and-separate approach, but that weak supervision itself comes from large-scale, self-supervised pretraining rather than supervised learning. In that sense, our proposed framework cannot be considered a supervised technique.

3. "I think the authors should include stronger text-based sound separation baselines like the model and ideally the training method that uses heterogeneous conditions to train the separation model in [A] ..."

Our response: we would like to emphasize that our primary contribution in this work is a novel weakly-supervised training strategy which is orthogonal to other aspects one may employ to boost the performance of source separation models including using heterogeneous conditions. To further show this, we have originally conducted experiments using learnable text conditions (as opposed to template-based conditions) to show that the SDR for our framework can be further improved by employing learnable prompts (see Appendix G.8). Nevertheless, per the reviewer’s question, we also conducted new ablation experiments using the OCT and OCT++ methodologies proposed in Tzinis et al. 2023. The results and further discussions are added to Appendix G.8 in the revised version. In short, we observe both OCT and OCT++ improve our framework’s performance compared to using template-based prompts, and while OCT gives us an improvement on par with the learnable prompt approach, OCT++ goes even further. But then again, these additions are completely orthogonal to what we have primarily proposed and evaluated in this work, and they can improve both the baselines and our proposed framework.

4. "Would it be possible to add some human evaluations for the best baseline and the proposed method?"

Our response: we have performed a subjective evaluation involving 20 human evaluators comparing our framework with CLIPSep-Text (Dong et al. 2022) which was the second best method overall under both unsupervised and semi-supervised training regimes. We have followed the methodology presented in Zhao et al., 2018 and Dong et al., 2022 for subjective evaluation. The results are added under a new appendix (Appendix H) in the revised version. As these results show, our framework significantly outperforms the baseline under both unsupervised and semi-supervised training setups according to the human evaluation. More importantly, these results are consistent with our quantitative evaluation results, which further corroborate our conclusions.

Finally, for further demonstration, we invite the reviewer to check out the new qualitative demo website uploaded here: https://drive.google.com/file/d/1tR7o8cXoHqS9Ns8riHB_TD6Hfenoz0zH/view