MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training

We are grateful for your thorough review and insightful suggestions.

To address your Concerns:

1. Novelty of Our Method Compared to HuBERT: We understand your concern regarding the apparent similarity of our mask language modelling approach to HuBERT. However, we believe MERT significantly contributes to the field as it is the first model to apply masked language modelling specifically for music understanding. Our approach incorporates RVQ-VAE discrete labels for general semantic information and CQT reconstruction loss for pitch learning, tailored to the unique requirements of music modelling. The exploration of different combinations of acoustic features and cluster numbers for K-Means (as detailed in Table 3) is particularly noteworthy. While there are similarities with HuBERT in MLM tasks, MERT's distinct training logic and additional features, such as CQT reconstruction targets and transformer architecture optimizations for training stability (Section 4.3, paragraph 2), substantiate its novelty.

2. Claim of Being Lightweight: We apologise for any ambiguity in our earlier statement. Our comparison was intended against the performance of probing Jukebox5B (Castellon et al., 2021, Dhariwal et al., 2020), not HuBERT. We highlight MERT's efficiency by noting its significantly fewer parameters (1.9% and 6.6% compared to Jukebox5B) in Section 5.1, Paragraph 3.

3. Necessity of Using All Codebooks: Your question regarding the use of all eight layers of codecs in RVQ-VAE is well-founded. Specifically, we have tested:

    a. Predicting the most informative codebook at layer 0=> score 60.5
   
    b. Predicting the most informative codebook at layer 7 => score 57.6
   
    c. Predicting one random layer of the codebook each time => score 65.8
   
    d. Predicting all layers of codebook in parallel => score 66.9
   

   Our investigation (referred in Table 3) shows that utilising all layers concurrently is optimal. However, we speculate that selecting a random layer for prediction and extending training could potentially yield similar results. Detailed analysis is presented in the last paragraph of Section 5.2.

4. Effectiveness of EnCodec Representation: Thank you for this valuable suggestion. We have conducted additional experiments to address this, which are now included in the revised paper. As Reviewer-7KYT also raised this question, we invite you to refer to point 2 in our reply to Reviewer-7KYT to avoid redundancy in our rebuttal.

5. Averaging Metrics: While averaging may not always be the optimal approach, it is a common practice in evaluating pre-trained models across established benchmarks (JukeMIR, GLUE, OpenCompass). We agree that more robust aggregation metrics could be developed for a more nuanced evaluation of pre-trained models.

6. Downstream Training Set Usage for Evaluation: The diverse training set sizes (from hundreds to 300K) across the 14 downstream tasks demonstrate the robustness of our evaluation method and generalisability of the model. We are also conducting few-shot setting evaluations, to further address your concerns. The results are anticipated in the next 2-3 days.

7. Performance Gap Between Local and Global Level Tasks: As discussed in Section 5.1, MERT is trained with 5-second contexts to balance performance and computational resources. Longer contexts might benefit certain tasks, but this involves a trade-off with batch size. Our decision aligns with recent findings in the field (Ma et al. 2023, On the Effectiveness of Speech Self-supervised Learning for Music, ISMIR), which is described at Training Settings paragraph in section 4.3.

We hope these responses adequately address your concerns and clarify our approach.

Hello, reviewer. Please review the author's response to see whether it addresses your concerns.

Addressing the Concerns about Novelty:

We are grateful for your remarks on the novelty of our approach. While the outcomes in Tables 1 and 2 may seem predictable, the true innovation of our work lies in the adaptation and integration of music-specific techniques, such as CQT reconstruction loss and the exploration of optimal pseudo targets for Masked Language Modeling (MLM) tasks, into the acoustic pre-training. This approach goes beyond mere incremental additions; it signifies a substantial advancement in adapting existing methods to the complexities of music understanding, a field that has been less explored than general audio processing.

Furthermore, music audios contain a rich tapestry of elements, demanding more extensive training data compared to speech models. We have delved into maintaining stable pre-training on large-scale datasets, with specific architectural modifications and analysis detailed in Section 4.3 and Appendix B.3. This exploration is a testament to our commitment to advancing music audio modelling.

At first glance, the performance improvements in MERT might seem attributable solely to the EnCodec model. However, we contend that these gains stem primarily from our refined pre-training stage. This is evidenced by:

• The superior performance of MERT^{K-Means} in non-pitch-oriented tasks, such as instrument detection (Tables 1 & 2), compared to HuBERT trained on similar music data. This indicates the efficacy of our pre-training stage enhancements.
• Additional verification showing that using only continuous embeddings from EnCodec results in a significant performance gap compared to both MERT^{K-Means} and MERT^{RVQ-VAE} (cf. Table R2 in the reply to Reviewer 7YKT). This underscores that MERT's learning extends beyond just the pseudo labels and encompasses a broader spectrum of information during the self-supervised pre-training phase.

Responses to Questions:

1. Downstream Tasks Metrics: In Section 4.1, we've outlined metrics for downstream tasks by type. We agree with the reviewer that including a wider range of metrics would enhance the comprehensiveness of our evaluation. Our metric selection aligns with prior research to facilitate reproducibility and fair comparison. Comprehensive details of the training, validation, and test splits are provided in Appendix B.1.
2. Batch Size Terminology: The use of “1.5 and 5.5 hours” to describe audio contained in a training batch follows precedents set by works like HuBERT (Hsu et al. 2021) and WavLM (Chen et al. 2022). This approach allows readers to easily compare the scale of training batches in terms of audio content, complemented by the provided GPU numbers and clip length information.

Closing Remarks

We are dedicated to contributing significantly to the field of music audio modelling. The amendments made in response to your insights are aimed at enriching the content and understanding of our research. We are hopeful that these enhancements will elevate our paper to meet the acceptance criteria.

Hello, reviewer. Please review the author's response. You are concerned about the novelty of the paper. Does the author's response address your concern?

We are grateful for your positive evaluation of our work and appreciate the opportunity to discuss aspects of our methodology further.

Regarding Short Training Contexts: We acknowledge your point about the potential benefits of longer segments in pre-training. Our current use of a 5-second context is a starting point, and we agree that extending this to capture more global information in music audio would be advantageous. Recent advancements in efficient transformer architectures offer promising avenues for achieving this with improved memory efficiency. Additionally, we are considering alternatives to the Conv1D encoder, which currently poses challenges in terms of stability and memory consumption, to facilitate longer context windows.

Responses to Review Questions:

Audio Collection and Quality: We sourced audio from open-access streaming services and open-source datasets. The audio quality varies, with sampling rates of 16KHz, 24KHz, and 48KHz, necessitating upsampling or downsampling for different training settings. We ensured quality through random sampling and conducting hearing tests from the dataset. Related ethical considerations are added in Appendix D.

Reference Formatting: Thank you for pointing out the issue with reference formatting. We have added the missing conference information and corrected the numbering in the bibliography.

Terminology Clarifications: We have reintroduced the full names of ROC and R2 in the experimental section for clarity.

Describing “Previous SOTAs”: We appreciate your observation on the use of "Previous SOTAs". To avoid potential confusion, we have revised Table 2 to read “(Previous) SOTA”.

General Formatting and Language Consistency: Based on your suggestions, we have:

• Unified the language to British English.
• Modified the phrase on page 4 from “speech and language dataset” to “sequential dataset”.
• Changed “advanced SOTA” to “current SOTA”.
• Corrected punctuation and added missing commas.
• Improved figure accessibility by adopting markers and a new colour palette suitable for colour-blind users.

We are thankful for your detailed formatting suggestions, which have been incorporated into the latest version of our paper. We hope that the challenges and solutions we have shared will be valuable for future research in music audio modelling.

We sincerely appreciate the reviewer's recognition of our method's effectiveness.

For addressing the reviewer's concerns:

1. On Potentially Unfair Comparison between Pre-Training Targets: We acknowledge your observation regarding the comparison between RVQ-VAE and K-Means targets. While it may appear unfair at first glance due to the variance in training-data size, it's important to note that K-Means models benefit from handcrafted acoustic features, providing a distinct inductive bias. This aspect makes a direct comparison with self-supervised RVQ-VAE not entirely straightforward. However, we concur with your insight that RVQ-VAE models, given their unsupervised nature, potentially have higher capacity limits, especially in learning from large-scale datasets.
2. Performance of Vanilla RVQ-VAE Representation: We have conducted additional experiments to evaluate the audio embeddings from the RVQ-VAE encoder, which are the continuous vectors used for quantization to discrete codecs. These supplementary results, now included in Table R1, indicate that the codec continuous representations alone are insufficient for a robust music understanding baseline. We are grateful for your suggestion of this complementary experiment, which substantiates the effectiveness of the self-supervised learning phase. Please note, these findings and discussions have also been added to Appendix A.2 in the revised manuscript for the benefit of future readers.

Table R1: Evaluating the EnCodec RVQ-VAE Embeddings

3. Clarifications on Table 2 Presentation: Regarding the average score calculation in Table 2, we chose this approach as all metrics positively correlate with task performance. We apologise for the missing bibliography numbers; this was an oversight, and we intended to use numbered hyperlinks for easier reading. This error has been rectified in the revised manuscript.

We trust that these clarifications and additions adequately address your concerns and enhance the paper's contribution. Your feedback has been invaluable in improving our work, and we are grateful for your thorough and constructive review.