Supervised Contrastive Learning from Weakly-Labeled Audio Segments for Musical Version Matching

We thank a lot the Reviewer for his/her assessment and constructive feedback. We answer to and discuss on the questions posed by the Reviewer below. As suggested by the Reviewer, we are considering adding a figure to describe the CLEWS model structure and pipeline, especially along with the shared code to facilitate understanding.

1. Computational cost ---- This is a good suggestion. Given track lengths of $N$ and $M$, the computational cost of $R_{\text{bpwr-r}}$ is $O(NM+N+M)$. Essentially, like the cost of the other reductions considered in the paper, it is quadratic (considering $N\approx M$). We will add the cost of all reductions to the Appendix. It should be noted that, with non-overlapping 20-second segments (the configuration we use for training), $N$ and $M$ are relatively small. For instance, for a 5-minute song we get $N=15$ segments, which is orders of magnitude smaller than the typical length we usually see in other quadratic and sequential operations like the ones performed, for instance, in Transformer's cross-/self-attention. Regarding computational resources used during training, we used two NVIDIA H100 80GB GPUs. We will add this information to the manuscript. Regarding retrieval runtime, we want to clarify that all approaches, when operating at a segment level, incur a natural increase as compared to track-based retrieval. Thanks to the Reviewers' suggestion, we collected these numbers and will report them in the Appendix. We paste them below for convenience. In summary, we do not see any big issue that would severely limit the utilization of the proposed approach in front of the other evaluated segment-based alternatives.

2. Different kinds of musical material --- In contrast to the SHS data set, which essentially features pop and rock songs, the DVI data set contains a wide variety of musical genres and material (from jazz standards to electronic versions, including musicals, rock, reggae, etc.). The test set of DVI is very large compared to existing evaluation standards, including over 100k different tracks. We believe that DVI-Test is representative enough to assess the generalizability of the approach with regard to western contemporary music.

3. Fine-grained variations in music versions --- We believe the DVI data set is representative enough of the real world, including cases such as the ones outlined by the Reviewer (for instance the one in which lyrics remain consistent while the musical form varies significantly). Our approach, like the vast majority of version matching approaches, focuses on the audio content, and it is true that it may underperform in cases where only the lyrics, but not the melody, are preserved. For those cases, which are not the most common ones for musical versions (according to what we see in the available data sets and sources), including a lyrics transcription system and combining with CLEWS could be a fruitful approach.

4. Song plagiarism --- This is an interesting aspect. Song plagiarism is a controversial and multifaceted topic in music copyright infringement. Often times, court decisions on music plagiarism cases are not only rooted on musical concepts and similarity, but go beyond that by considering external factors such as the possibility of access to the source by the offender, the amount of money involved, the resources employed or the collaborating artists, etc. Focusing on similarity alone, we believe the current data sets, and especially DVI, have the potential to start supporting comprehensive similarity searches, at least for western contemporary music (with the exception perhaps of some niche genres like experimental music).

Table: Training and inference runtimes (informal) using a single NVIDIA H100 GPU. Training runtime is measured using a batch construction as specified in the main text (2.5 min audio blocks, 25 anchors, 3 positives per anchor, total of 100 audio blocks). Inference runtime is measured following the segment-based evaluation protocol (5 s hop size, $\tau=20$ s, $R=R_\text{min}$). Retrieval time corresponds to a database with 2000 candidates, and grows linearly with candidates.

Approach             Parameters   Training [s/batch]                  Inference
                                                       Embedding [ms/song]   Retrieval [ms/query]
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
CoverHunter-Coarse   28M          0.68                 22.5                  2.9
CQTNet               35M          0.48                 57.6                  2.0
DVINet+              11M          0.38                 60.9                  2.2
ByteCover3           969M         0.71                 27.4                  5.1
ByteCover1/2         202M         0.50                 62.2                  5.1
CLEWS                199M         1.19                 40.3                  3.5

We are grateful to the Reviewer for his/her assessment and comments. We now answer and discuss about the main questions raised in the Review.

1. Additional data sets ---- This is a relevant suggestion that is worth discussing. We originally computed the results for covers80 and we paste them below. However, we finally decided to not include them in the manuscript due to the multiple issues and the low representativity of covers80. Covers80 is a very small data set, consisting of only 80 tracks (as a reference, DVI-Test contains around 100k tracks). The small size of covers80, together with the fact that different existing approaches train with different existing datasets in order to report results on covers80, makes the data set problematic and non-reliable as a reference for performance measurement (this can also be seen with the high confidence intervals reported for covers80). The Da-TACOS data set comes with pre-extracted features, which difficults a fair comparison with audio-input methods due to different feature extraction pipelines and data augmentation procedures. Like covers80, Da-TACOS does not provide a train set, which additionally creates issues for fair comparison as models are typically not trained with the same data. Finally, and related to the latter, our internal analysis shows that around 90% of Da-TACOS is contained in the DVI data set. Thus, we consider that using DVI is preferable (and enough) for comparing approaches.

2. Analysis comparing runtimes ---- Thanks to the Reviewers' suggestion we performed both a complexity and a runtime analysis, and we will include them in the Appendix of the manuscript. For convenience, we reproduce the runtime results in the answer to Q1 from Reviewer ySSi, who asked a similar question, and we refer to it for further details due to space limitation. In summary, we do not see any big issue that would severely limit the utilization of the proposed approach in front of the other segment-based evaluated approaches.

Table: Track-level evaluation on the Covers80 test set. We distinguish between models trained on DVI and models trained on SHS. The +- symbol denotes 95% confidence intervals.

Approach                        Trained on DVI               Trained on SHS
                                NAR         MAP              NAR         MAP
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 
CoverHunter-Coarse (our impl)   7.85+-2.37  0.558+-0.068     5.80+-2.52  0.735+-0.061
CQTNet (reported)               n/a         n/a              n/a         0.840
CQTNet (our impl)               3.33+-2.05  0.869+-0.048     2.10+-1.14  0.861+-0.050
DVINet+ (our impl)              0.88+-0.51  0.897+-0.043     1.95+-1.35  0.889+-0.045
ByteCover3 (our impl)           1.13+-0.88  0.900+-0.043     0.96+-0.75  0.916+-0.038
ByteCover3 (reported)           n/a         n/a              n/a         0.927
ByteCover2 (reported)           n/a         n/a              n/a         0.928
ByteCover1/2 (our impl)         1.46+-0.93  0.905+-0.042     1.67+-1.18  0.929+-0.038
CLEWS (proposed)                0.72+-0.63  0.955+-0.030     1.31+-1.11  0.944+-0.034

We thank the Reviewer for his/her assessment of our work and the constructive feedback. We first answer to the questions posed by the Reviewer and later discuss on a few further aspects that appear in the main review.

1. Label inconsistencies ---- Like the big majority of machine learning models, our model assumes a low (or nonexistent) number of noisy labels. Hence, it does not explicitly handle song label inconsistencies. In the weak label assignment, most of our reduction strategies try to assign the song label to the best (lowest distance) segments in a pairwise comparison, while discarding the rest of the comparisons. According to the obtained results, we believe that the model ends up assigning the weak labels to the correct segments pairs (otherwise we would not be obtaining good performance). Nonetheless, it is true that perhaps one could implement further mechanisms to explicitly handle possible inconsistencies. We plan to actively consider this situation in the next version of our approach.

2. Contrastive loss modifications ---- We think our modifications to the contrastive loss could be beneficial for other retrieval tasks, and we think the examples provided by the Reviewer are perhaps the ones that have more potential, as they also correspond to domains where sequence information is important. As mentioned to Reviewer oJzQ, the extension of the proposed approach to other domains is out of the scope of the current manuscript, but nonetheless a very interesting direction for future work.

3. Self-supervised learning ---- We did consider the use of self-supervised learning techniques in addition to weak supervision. However, we did not have a clear idea on which self-supervised/proxy task to include. Predicting future segments as an aid to weak labeling was one task we played with during our experimentation, but we were not able to obtain any meaningful performance increase. We also studied additional data augmentations to create different views of the segments (we share them in the code), but again found no benefit when evaluating with the largest data set we consider (DVI). We hypothesize that such data set is large enough to not benefit from trivial data augmentations, and that the variability it naturally includes is more informative than the artificial augmentations we introduced.

4. Analysis of failure cases ---- This is a good suggestion, and we are studying which is the best way to include such analysis in the manuscript or the Appendix. Unfortunately, we have not developed a demo of the system yet. Informally, we can comment that it is hard to find a clear denominator for failure cases, as most of the time we do not see a clear or systematic failure that we could address in a further development. For instance, some true positives show a spectacular robustness towards genre variations, while still we find similar genre variations of the original song among the false negatives.

5. Ablation of loss modifications ---- We agree with the Reviewer. It was our original intention to include ablations for every loss modification. However, we finally did not do it due to two main reasons. On the one hand, the decoupling modification introduces just a marginal improvement while, in the other hand, we cannot isolate the other modifications as each one is conceptually and practically related to the other. For instance, the introduction of the smooth threshold $\varepsilon$ is needed because the sum of exponential terms inside the logarithm can become zero, and the latter is due to switching from cosine to Euclidean distance, which can grow indefinitely and cause a numerical error. Therefore, we cannot study the effect of switching from cosine to Euclidean without introducing the soft smoothing threshold $\varepsilon$. We can however study the effect of different values of the latter, which we do in Sec. 5.

6. Segment-level boundary effects ---- We think we do not fully understand the question, but we will nonetheless try to address it the better we can. In the way we perform the segment-based evaluation, segments are extracted before concatenating all candidate songs, and segment embeddings (computed independently) are then stacked together in the candidate database. At the retrieval stage, we know which is the song identifier for each segment embedding, and can thus avoid mixing different songs. Since segment embeddings are computed independently, we also avoid boundary effects with other segments and songs.

Regarding confidence intervals, we want to mention that we do report 95% confidence intervals in all our results (both tables and figures). We will add further clarification in the Table captions to avoid such confusion. Regarding additional citations to self-supervised literature in music information retrieval, we are totally open to include them. If the Reviewer has specific examples in mind, we will consider them in the next revision of the manuscript.

We thank the Reviewer for his/her comments and insight. We want to clarify two misunderstandings by the Reviewer, and comment on the further analyses part. We think that the Reviewer may increase his/her score once these two misunderstandings are clarified, as they are the only two strong criticisms we could find in the review.

1. The Reviewer mentions that the method regards all the segments in the songs as similar ("then they regarded all the segments in the songs are similar") ---- We want to stress that it is actually the opposite. As mentioned in the manuscript, considering all compared segments between two songs as similar corresponds to the $R_{\text{mean}}$ strategy. We clearly motivate why this is unrealistic for music (Sec. 3.1) and further corroborate that models do not learn well with this reduction (Sec. 5). In addition, we have to note that all reduction strategies developed after $R_{\text{mean}}$ go precisely in the direction of avoiding the situation where "all segments in the songs are similar". In particular, the proposed $R_{\text{bpwr-r}}$ reduction explicitly avoids that by construction.

2. The Reviewer mentions that the proposed approach lacks novelty, "as the authors noted that, this can be regarded as one of a triplet sampling method, or it can also be viewed as one of the multiple instance learning method." ---- We believe that this is not true, and we want to state that we do not explicitly mention that. On the one hand, none of the proposed techniques corresponds to triplet sampling, nor we use any triplet loss (except for showing better results with the proposed loss). On the other hand, the fact that we try to conceptually relate some of the reduction techniques we propose (e.g., $R_{\text{min}}$) to concepts employed in other multiple-instance learning methods (e.g., hard mining of examples) does not mean that we use a similar technique or implementation. Actually, we propose $R_{\text{bpwr-r}}$, which we do not find a clear parallelism with hard, semi-hard, or random instance mining. Besides, the development of the proposed loss is novel and does not employ triplet sampling or existing multiple instance learning methods.

Regarding further analyses with different modalities, we want to recall that our main focus is on the audio retrieval field, and that the manuscript is submitted to the "Applications" track, using the "Application-Driven Machine Learning" tag. We believe a number of the techniques we employ could be transferred to other domains as they do not include audio-specific processing. However, demonstrating it in a rigorous way would require much more experimentation and results which are beyond the scope of the current manuscript. Regarding the analysis of some specific examples and on how two audio recordings are compared, we are studying to possibility to add an additional figure to the manuscript, as this was also mentioned by another reviewer. We thank the Reviewer for this suggestion.