SyllableLM: Learning Coarse Semantic Units for Speech Language Models

We would like to start by thanking kvPb for their clear and significant commitment to understanding our paper in detail. In our rebuttal below, we try to demonstrate that the presentation clarity errors pointed out only need minor changes to be fixed.

Regarding "Confusing equation":

• The unnumbered equation between line 126 and line 127 defines the similarity matrix from MLM probabilities. It makes reference to $Y_t$ without specifying which $t \in M$ is used to define $C_{r,c}$. As a result, after having read the paper 6 times over, I still do not know how to compute $C_{r,c}$.

There is a typo in the equation for $C_{r,c}$: The letter $t$ should be replaced with the letter $c$. We apologize and hope that correcting this typo eliminates any confusion about LossPred.

Regarding "References to items not yet introduced":

• Lines 153-154, the phrase "our loss" make it sound like a referrence to the masked language model loss discussed in the previous sub-section, whereas in fact it is referring to Equation (3), a yet-to-be-introduced loss for SylBoost.

We will replace “our loss” with “our loss (Equation 3).”

• Lines 159-162 give a very vague description of the "similarity matrix" and the "cut algorithm" which can only be known if the reader has already seen the subsequent Section 3.3.

We apologize that the sentence starting on line 159 is written awkwardly. It should instead read “This results in a matrix of pairwise frame distances where element $i,j$ represents the $L_2$ distance between the features at frame $i$ and frame $j$.”

• Starting at line 188, Section 4.2 makes repeated references to "mHuBERT". "mHuBERT" appears to be name given to the acoustic tokens in (Hassid et al., 2023) by this paper (line 241). (Hassid et al., 2023) itself does not use this name, so an ordinary reader would not be able to tell what an "mHuBERT" model is when they work through Section 4.2.

Agreed, and we will fix this confusion by replacing “mHuBERT” with “TWIST Tokenizer.” The name mHuBERT was taken from the github repository published by Hassid et al.

Regarding "Confusing terminology":

• "pretraining": This paper makes a liberal use of the term "pretraining" to the point it's very difficult to tell which is the model being "pretrained". For example,
  • Line 113 mentions a "pretrained HuBERT teacher model", then line 119 says "during pretraining, the student model ...". The teacher and the student are presumably not trained at the same time, yet the use of "pretraining" in this context make it appear that the contrary is happening.
  • Line 225 says "for all pretraining experiments". A reader will have to look really closely to see this means "training of the speech LM", not "pretraining HuBERT, etc".

We will alter Line 225 to say “for all language model pretraining experiments” and in line 119 replace “pretraining” with “training.” We believe that lines 113-114: “We consider the setting of having a pretrained HuBERT teacher model and a HuBERT student model trained to predict the quantized contextualized representations generated by the teacher” make training order clear.

• "Agglomeration" vs "SylBoost": This paper appears to use these two terms interchangeably. Agglomerative clustering is apparently also used (line 183). This makes it difficult for the reader to tell when "agglomeration" is mentioned, whether the authors intend to refer to SylBoost or just the clustering.

You are correct that “Agglomeration” and “SylBoost” are used interchangeably, and we will replace these instances of “Agglomeration” with “SylBoost.” We will maintain that we always refer to clustering for discrete tokenization with K-Means as the full phrase “K-Means and Agglomerative Clustering”

Answering "Is LossPred really necessary?"

• While Table 4 shows that LossPred is a better initialization strategy for SylBoost, would a cheaper initialization strategy (like feature similarity or even random) produce equally good segmentation given more iterations?

In rows 2 and 3 of Table 4, we demonstrate that the second iteration of SylBoost using feature similarity results in worse performance than the first iteration so we do believe that LossPred is necessary.

• Suppose the answer to the previous question is yes, would it make sense to run SylBoost but using the final activation of HuBERT instead of an intermediate layer (line 154)? The final activation may be better semantic features.

Although the answer to the first question is no, Pasad et al [43] shows that the semantic features from mean-pooling across ground truth syllable boundaries perform best at Layer 9 and not the final layer activations.

If you agree that these minor edits address your major concerns about the paper’s clarity, given that you state that our method and results in this paper would make a good paper for NeurIPS, we kindly ask that you consider raising your score.

We sincerely thank the reviewer for their feedback. We hope to clarify any additional concerns they had below:

Weaknesses

1. As pointed out by the authors, the proposed LossPred and SylBoost methods seem to be restricted to speech representation learning. It might be difficult to apply these methods to music, singing voice, speech with noisy backgrounds.

We entirely agree and hope that future work can tackle these challenges.

2. LossPred is slow in evaluating the loss prediction matrix. Each sentence requires about 200 Transformer network evaluations.

This is entirely true. Fortunately, the forward passes can be batched and the first iteration of SylBoost works well with only 10% of LibriSpeech labeled by LossPred.

3. LossPred is highly heuristic. There seems to be no theoretical guarantee that the HuBERT model combined with LossPred reveals syllabic boundaries instead of revealing only phoneme or word boundaries.

We agree that there is no guarantee of the types of units that LossPred predicts. We think that an especially interesting case is how the two words “at a” in Figure 1 get mapped to a single cluster by SylBoost. We are hopeful that future work can inspect these phenomena with respect to the statistics of spoken language.

Questions

1. The equation above line 127 is rather confusing. The description of the procedure for computing the loss prediction matrix is confusing in general.

We sincerely apologize that there is a typo in the referenced equation, and the letter $t$ should be replaced with $c$ in said equation and its description starting on line 128. We acknowledge that LossPred is a detailed algorithm, and fully understand any misunderstanding that could have resulted from this typo.

In LossPred, $k$, the number of segments, is chosen to be proportional to the sequence length. What happens when a speaker is speaking very fast or very slow? It seems that $k$ should be determined by the number of syllables in the utterance.

We would be interested in a future approach that could attempt to count the number of distinct regions generated by $C$ in LossPred however found that using an empirical $k$ performs well and is compatible with prior cut algorithms such as that used by Peng et al.

We thank the reviewer again for their feedback.

We thank the reviewer for their valuable feedback and for stating the proposed approach shows clear benefits in terms of performance and efficiency. We respond to each of the reviewer’s concerns below.

Demonstrating efficiency: As efficiency is also one selling point of the paper, it would be great if the authors can demonstrate the training efficiency and low-bitrate benefits in a more comprehensive way, like visualizing the GPU training time vs Performance, and also bitrate vs unit quality for certain tasks.

We demonstrate the GPU training time vs Performance on the semantic tasks of sWUGGY, sBLIMP, and tStoryCloze in Table 3. We demonstrate bitrate vs unit quality for WER and CER measures of quality in Table 2 and further ablate controling for the number of units. We also demonstrate bitrate vs unit quality on sWUGGY and sBLIMP in Table 6, controlling for changes in the number of tokens and for changes in sample rate. To better visualize the role of bitrate in Table 6, we will add a column containing bitrate, however we note that bitrate is directly calculable from the provided unit frequency and number of unit clusters. If you are asking in particular for visual diagrams, such as a GPU-Hour vs sWUGGY accuracy graph, please let us know and we would be more than happy to adapt the results from these tables into a graph in our appendix.

Limited use cases: The proposed approach focuses on learning semantic units for speech applications. It’s unclear if the proposed methods can be applied to other important non-speech use cases like understanding acoustic environment, and understanding speaker’s identity and emotion.

We fully agree with this limitation and acknowledge it our work. The speech domain is both difficult and important, and so we believe that expanding this paper to evaluate other domains may cause it to lose focus. We believe that focusing in-depth on speech applications throughout our paper provides a strong starting ground for future work to adapt LossPred and SylBoost to other domains, which we eagerly await.

Understanding Unit Quality: To demonstrate the unit quality for synthesizing the audio and for generation, should the author also compare with other related works (like 1 and 2) in terms of reconstructing the original signal? Like in 1 (see Table 1), the authors compare the different approaches in terms of reconstruction performance using a couple of metrics like MEL, STFT and ViSQOL score, and also semantic task performance.

We believe that evaluating the acoustic quality of our unit reconstruction is not a task best suited for this paper. Like Hassid et al, we evaluate our units solely on semantic tasks instead of metrics like MEL, STFT, and ViSQOL. Codecs geared toward natural sounding speech generation, such as 1 and 2, are orthogonal to our method and operate at a significantly higher bitrate. Our paper seeks to improve the textual understanding of models, and units like those of 1 and 2 are compatible with downstream vocoding in cascaded networks such as our Interleaved-Vocoder-LM or the fine acoustic modeling stage of Boros et al.

We also point out that the WER of the concurrent work (1) referenced is 19.6% at 360bps compared to our 7.0% WER at 81bps (albeit on different datasets), which we hope additionally clarifies how our units cover different problem niches.

If the reviewer believes that the provided response adequately addresses their concerts, we respectfully request that they consider raising their score to reflect that.

Thank you for the review. Below, we will respond to several questions you raised about our work.

line 189 -> Superior compared to what?

This is answered in lines 189-190: “Superior performance in high difficulty settings compared to other Neural Codec Lanaugae Models like VALL-E [54]”

Table 1 -> What happens if you apply SylBoost to Feat-Sim?

These results are the focus of Table 4, where we evaluate SylBoost from both FeatSim and LossPred and find that initializing with FeatSim boundaries results in SylBoost converging to lower-quality boundaries.

Questions

How does the performance on the syllable boundary detection task relate to the unit re-synthesis and language modeling performance? In other words, is having well-defined boundaries necessary for the speech language modeling task?

Higher quality boundaries are essential for low error rates, a comparison which we draw from the WER of SD-HuBERT and SylBoost+HuBERT units in Table 2. Having exact syllable boundaries is not necessarily essential for the language modeling task after word error rates are low, however, as seen through evaluating on a wide range of unit rates (5.0-8.33 Hz) in Tables 2, 6, and 7.

It is difficult to disentangle the effect of the number of units and temporal resolution on the overall performance in Tables 2 and 3. Despite having lower resolutions, the results from the proposed methods are not much better than baseline methods (e.g., AudioLM and TWIST). Can the authors elaborate on this?

We respectfully disagree with the framing of this question. First, we heavily outperform TWIST-CI-90M and TWIST-300M with our proposed models, where we can afford to match training compute. We also beat AudioLM in sWUGGY with less than 10% of the pretraining compute, and are the first work to come within 1% of AudioLM on sBLIMP. SyllableLM only starts losing to TWIST models when they reach 7B parameters, 3x the data, and 20x the pretraining GPU compute hours.

Second, we disagree with “Despite having lower resolutions.” Lower resolutions provide a significant speedup as generation and training run on up to 5x fewer tokens so even equivalent performance would be significant. Using fewer tokens is a challenge, and our work is the first to reach below 19.5Hz.

What happens if we directly use LossPred units in the LM framework? In other words, how important is the boundary refinement stage for the language modeling task?

LossPred only provides boundaries and not units, as it is calculated only using a model loss instead of a cut-algorithm on model features. Even if we used the LossPred boundaries for HuBERT pooling, LossPred is too expensive to extract across an entire dataset, which is one reason we created SylBoost.

Keeping everything fixed, how does the number of units impact the re-synthesis and language modeling results?

Resynthesis results, keeping everything else fixed, are in the “+Increase #Units” row of Table 2. The language modeling results are in “Table 6: Holding number of units and unit rate constant.”

How does the masking probability and span used when training the HuBERT model impact the quality of the discovered boundaries, and what impact does it have on your approach?

We do not have the compute to pretrain a HuBERT model with the requested modifications and we are not aware of any public checkpoints that would facilitate this, so unfortunately we cannot answer this question.