Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Prediction

Dear Reviewer nPyR,

Thank you for your supportive feedback on our work. We are pleased to see the alignment in our perspectives on the potential future trajectories of our research. In terms of writing and presentation, we are diligently integrating the collective advice from all reviewers, including notation linkage between figure and content, rectifying formulation errors, fixing missed descriptions, and elucidating experimental setups. Should you have any particular recommendations for further improvement, we would be eager to hear them.

Addressing your queries:

• We have not yet evaluated our system with real conversational data as it was beyond the scope of our initial focus. Indeed, compared to WavLM, which benefits from additional and noisy data augmentation, our system might not currently perform as well. We recognize the significance of this direction and intend to explore it in subsequent research.
• Concerning your question about multiple resolutions, we direct you to Appendix B.2. Here we assessed the impact of incorporating three resolutions. The breadth of the investigation necessary to thoroughly examine this, including the consideration of more aggressive utterance-level embeddings and finer 10ms resolutions, would be vast. Given this, we have earmarked the exploration of diverse resolutions as a future direction, as noted in Appendix F.

We are grateful for your thoughtful review and hope that our responses shed light on the aspects you've inquired about. We remain committed to advancing our manuscript with the insights gained from your input.

Dear Reviewer vPyr,

We are grateful for your positive evaluation and the attention to detail you have demonstrated in reviewing the appendix results of our paper.

Regarding the issues you have raised:

• Concerning ASR performance, we acknowledge that MR-HuBERT does not unequivocally surpass the original HuBERT in every test case, although it does in the majority. As we mentioned in the abstract, MR-HuBERT "exhibits superior or comparable performance to the original HuBERT."

  • Specifically, with the Librispeech experiments, we have observed a marginal degradation in the dev_other set compared to the publicly available pre-trained HuBERT, yet MR-HuBERT shows better performance across the remaining sets. It is essential to recognize the inherent randomness in HuBERT training, especially from K-means clustering, which can lead to variations in performance. For a fair comparison, we have also pre-trained a HuBERT-large* using identical K-means clusters as MR-HuBERT. The results in Table 1 indicate that our mono-large model outperforms HuBERT-large* in all aspects except for a negligible difference in the dev_clean set.

  • In the SUPERB benchmark, we have noticed the ASR performance is not as robust compared to the original HuBERT models. We have elaborated on potential reasons for this in Appendix C.3 (ASR bullet), and we regard this as an avenue for future improvements to MR-HuBERT. The comprehensive explanation provided in the appendix should serve to clarify this point further.

• We are thankful for your scrutiny of our detailed ablation analysis. While there is some divergence in model performance across the base setting in Librispeech experiments, the flexible up/downsampling module demonstrates more consistent results in large-scale experiments and the SUPERB benchmark. We encourage you to refer to Appendix B.8 (Simplified sampling is not recommended bullet) and the results depicted in Table 18 for models (B.8)-i and (B.8)-j. Based on these observations, we maintain that our proposed flexible up/downsampling module is the more reliable choice over its simplified counterpart.

We appreciate your thorough examination of our ablations and insightful feedback. We trust that the explanations provided will satisfactorily address your concerns and contribute to a better understanding of our work.

Dear Reviewer hKsL,

Thank you for the encouraging remarks on our work and for your constructive feedback.

In response to your points:

• We concur that the core concept behind our approach is straightforward. Nevertheless, we are thrilled about the introduction of MR-HuBERT to the research community. We believe that it can unlock numerous novel pathways in the field of speech representation learning, as thoroughly discussed in Appendix F of our paper.

• With regards to the performance comparison with WavLM, we acknowledge that since we have not replicated the WavLM setups precisely, including data augmentation and the use of additional datasets such as Gigaspeech&Voxpopuli, a direct comparison is not feasible at this stage. We aim to explore these enhancements in our future research.

• We appreciate your attention to detail concerning the notation in our formulation. We will make the necessary corrections to our figures to improve clarity, including the reversal of $f_1^q$ and $f_2^q$. We also apologize for the oversight of not including the definition of $\phi$ in the manuscript. This will be corrected in our revised edition. Regarding the high-resolution encoders depicted in Figure 1, we will clarify that different $f_1^q$ and $f_3^q$ functions represent them, which should resolve any confusion.

• Regarding the usage of MACs, your observation is astute. Our empirical findings did indicate that the actual execution time (measured by tokens_per_second in fairseq) aligns with the MACs calculations. However, due to the inherent variability in real-time measurements, we opted to present MACs as they provide a theoretical and more consistent metric. Taking your suggestion into account, we will include the real inference time from our Librispeech fine-tuning experiments in an appendix. For your immediate reference, here is a preliminary overview of the speed improvements on Librispeech dev-clean inference:

• Finally, we are grateful for your suggestions regarding writing improvements. These will be addressed in our forthcoming submission.

Your feedback has been instrumental in enhancing the quality of our manuscript and thank you again for your effort in reviewing this paper.

Dear Reviewer MxEg,

Thank you for your effort and for your positive feedback on our work.

Regarding your observations:

• We acknowledge the reversal error between $f_1^q$ and $f_2^q$ that you have identified. We will fix this mistake in our forthcoming revised manuscript.
• Your second point has brought to our attention the need for greater clarity in describing the use cases of pre-trained MR-HuBERT. To address this:
  • For the fine-tuning setting, we indeed utilize the hidden states from the last layer for downstream models, followed with a shallow linear CTC decoder. It is important to note that the lower-resolution layers are not directly engaged in this process. The intention is to demonstrate the effectiveness of the pre-trained MR-HuBERT as a comprehensive encoder. We will enhance the manuscript by including an additional sentence to elucidate this for the benefit of our readers.
  • Regarding the frozen setting (SUPERB and ML-SUPERB), we employ a weighted summation of hidden representations from all layers. To synchronize the varying resolutions, we repeat-upsample the time axis of the lower-resolution representations, ensuring alignment across all layers. This approach is in line with established practices found in prior work, which can be reviewed at https://arxiv.org/abs/2306.01084.

We hope that these clarifications satisfactorily address your comments and enhance the understanding of our proposed methodology.

Thanks for the response. I think the paper can be accepted after the two issues are solved.