REBORN: Reinforcement-Learned Boundary Segmentation with Iterative Training for Unsupervised ASR

We sincerely thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.

The authors propose a very complex methodology. These are not a weakness per se, but point towards the overall complex picture of the proposed method.

To ensure the reproducibility of our work, we will release the codes, training and evaluation scripts, and model weights.

Why the authors try to sell this as an iterative method, instead of follow-up stages to (significantly) improve on wav2vec-u.

Why do you state in 4.3 to only use 1 to 2 iterations, while figure 2 shows results with improvements until 5 iterations?

While the performance gain is significant in the early iterations, Figure 2 shows that further improvements can still be achieved with more iterations. In Table 1, we use only two iterations to emphasize the effective integration of REBORN and self-training, as well as to demonstrate the efficiency of REBORN, i.e., achieving improved performance only requires few iterations.

Minor comments on writing:

We thank the reviewer for the detailed suggestions. We will revise the paper to fix them.

Why do you think that your model performs better on 5 languages but worse on Portuguese?

We hypothesize that our method performs slightly worse in Portuguese because the cross-lingual feature extractor (XLSR-53) uses the least amount of Portuguese data compared to other languages during self-supervised pre-training.

Why did you decide to mean-pool the segment before predicting the phoneme?

Our phoneme prediction model is initialized from wav2vec-U, which uses mean-pooled features. To ensure consistency with wav2vec-U, we also use mean-pooling in REBORN. Exploring other pooling methods is interesting and potentially beneficial, which is left for future work.

The ultimate goal of UASR is to enable speech technology for digitally rare languages without good labeled data. Although this work studies UASR on 6 languages, they are all from rich European countries.

We thank the reviewer for pointing this out. Indeed, we have not yet tested REBORN on a digitally rare language. Referencing prior works [1-3], we tested REBORN on the most commonly used datasets in UASR. We will revise the paper to include this limitation in Appendix A.

References

[1] Chen et al. “Completely Unsupervised Phoneme Recognition by a Generative Adversarial Network Harmonized with Iteratively Refined Hidden Markov Models.” Interspeech, 2019.

[2] Baevski et al. "Unsupervised speech recognition." NeurIPS, 2021.

[3] Gao et al. "Euro: Espnet unsupervised asr open-source toolkit." ICASSP, 2023.

We sincerely thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.

The author’s (mis)use of the word “phoneme” is problematic. What the authors may have had in mind is phonetic segmentation instead.

We thank the reviewer for pointing this out. We will thoroughly revise the paper to ensure the correct use of these terms.

The authors claim that “the effect of feature segmentation on phoneme prediction is non-differentiable”, however, it can be.

We would like to clarify that in our paper, we meant that the process of predicting phonemes based on predicted segments is naturally non-differentiable. As the reviewer pointed out, some techniques could allow for approximating the backpropagation gradients. We will revise our claim in the paper to avoid confusion and discuss the work [Bhati et al., 2021] referred by the reviewer.

Could REBORN be leveraged in a semi-supervised setup?

We thank the reviewer for this interesting idea. We believe that REBORN could benefit especially when the labels of the limited supervised data are noisy. In this way, the phoneme predictor might be non-ideal and could possibly get improved through REBORN’s iterative paradigm. We would love to investigate this idea in the future thoroughly.

It should be presented as a “refinement” stage for existsing models, but the title and the abstract don’t mention that.

We will rethink the paper's title and abstract to clarify this. We appreciate the reviewer’s suggestion.

We sincerely thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.

How do you get from phonemes to words? ... it needs a lexicon. If lexicon is used, is it fair to call this unsupervised ASR? I think this should be made much more clear.

We thank the reviewer for raising this concern. Our work does utilize an additional phonemizer (L668-L670) to obtain lexicon, following the standard setup in UASR [1-5]. We acknowledge that using a lexicon is a limitation of recent UASR works, including ours, which is discussed in Appendix A (L540). As suggested by the reviewer, we will revise the paper to clearly discuss this in Section 2 and Section 4.3.

The definition of the phoneme predictor is unclear. Is it autoregressive or non-autoregressive?

It is non-autoregressive. Precisely, it is a one-layer CNN following wav2vec-U. We will revise the paper to include this information in Section 4.3 to make it clear.

How do you sample from the phoneme predictor for policy gradient? Does this involve beam search?

When training the segmentation model using policy gradient, we obtain the phoneme predictor by taking the argmax prediction from the phoneme predictor. Since our phoneme prediction model is non-autoregressive, we do not perform a beam search. We will revise the paper to include this detail in Section 3.

Needs wav2vec-U for initialization

We acknowledge that relying on a reasonably effective phoneme prediction model for initialization may present a limitation, as noted in L539. However, we would like to clarify that our method is not limited to the wav2vec-U for initialization; instead, we have experimented with using different feature extractors, such as HuBERT [6] and WavLM [7] following EURO [4], and starting from different phoneme predictors on LibriSpeech (see Appendix C.5). The results show that REBORN achieves comparable performance improvements even with a completely different backbone model and a less effective phoneme predictor.

References

[1] Liu et al. “Completely Unsupervised Phoneme Recognition by Adversarially Learning Mapping Relationships from Audio Embeddings.” Interspeech, 2018.

[2] Yeh et al. "Unsupervised speech recognition via segmental empirical output distribution matching.” ICLR, 2019.

[3] Baevski et al. "Unsupervised speech recognition." NeurIPS, 2021.

[4] Gao et al. "EURO: Espnet unsupervised asr open-source toolkit." ICASSP, 2023.

[5] Liu et al. “Towards end-to-end unsupervised speech recognition.” SLT, 2022.

[6] Hsu et al. "Hubert: Self-supervised speech representation learning by masked prediction of hidden units." TASLP, 2021.

[7] Chen et al. "Wavlm: Large-scale self-supervised pre-training for full stack speech processing." JSTSP, 2022.

We sincerely thank the reviewer for the thorough and constructive comments. Please find the response to your questions below.

ASR don't need accurate segmentation, e.g. CTC model is peaky -- the peak of a phoneme can appear at any frame for this phoneme.

While recent supervised ASR does not need explicit segmentation, under the unsupervised scenario, learning the mapping from very long speech features to short text transcriptions has been found to be very challenging. To mitigate this difficulty, segmentation is critical to ensure good unsupervised ASR performance and stability, which is evident from early works [1-3] to recent studies [4-5].

To investigate the importance of segmentation in UASR, we compare the training stability of UASR algorithms: wav2vec-U, wav2vec-U 2.0, and REBORN. wav2vec-U 2.0 is the only UASR algorithm that does not segment the speech features. wav2vec-U uses k-means segmentation (see Appendix D), while REBORN uses a segmentation model learned using RL. Following the EURO setup described in Section 4.1, we use each algorithm to train UASR models on LibriSpeech. Since each algorithm has its own configuration and search space, including the random seeds, we iterate through these configurations, with each resulting in one model. For each algorithm, we report the "percentage of models yielding PER < 40%" among all training configurations and random seeds. The results are shown in Table 1 below.

Table 1: The importance of segmentation in current UASR. Results are obtained from LibriSpeech following the EURO setup. As the table indicates, wav2vec-U and REBORN are fully converged in PER with proper segmentation.

From Table 1, we observe that wav2vec-U 2.0, which does not have segmentation, is highly unstable. Conversely, REBORN and wav2vec-U (with k-means segmentation) always result in a UASR model with PER lower than 40%. Additionally, removing the segmentation from wav2vec-U makes the algorithm unstable and never yields a model with PER lower than 40%.

For this reason, it is not so clear to me why improving segmentation/phoneme granularity can help improve UASR so much.

The results in Table 1 have justified the importance of segmentation in UASR. Moreover, we found that the quality of the segmentation strongly affects the performance of UASR (Table 4 in the paper): using hand-crafted rules or separately learned segmentation boundaries yields suboptimal UASR performance while using the oracle phoneme boundary greatly improves the UASR performance. Motivated by the above observation, we thus propose to use RL to learn segmentation that is tailored for the phoneme prediction model. The segmentation boundary in REBORN is learned with feedback from the phoneme prediction model, which ensures the boundary is useful for improving the UASR performance and attains state-of-the-art results on many widely used UASR datasets.

The predicted segments are not more accurate than other approaches and probably are not good to be used to obtain accurate timestamps.

As suggested by the reviewer, we use TIMIT’s human-annotated phone-level timestamps for evaluation and report the results in Table 2 below.

Table 2: Boundary evaluation results on TIMIT. REBORN with boundary merging (Figure 1-(c)) is close to the existing SoTA method.

The results show that the initial boundaries learned by REBORN (before boundary merging in Figure 1-(c)), which are specifically optimized for the phoneme prediction model, already achieve a high recall. With boundary merging, i.e., consecutive segments with the same phoneme prediction are merged as illustrated in Figure 1-(c), REBORN achieves better alignment, bringing our results close to existing SoTA unsupervised segmentation methods.

It would be better to add a table for TIMIT which comes with word/phone-level timestamps.

We thank the reviewer for suggesting this experiment, which strengthens the contribution of our method. We will revise the paper to include this result.

if we use sub-phoneme structures in wav2vec-U 2.0, it may be as performant as the proposed approach

Since wav2vec-U 2.0 uses a fixed stride 1D-CNN to downsample the speech features and does not segment the speech features, we are unsure how to incorporate sub-phoneme structures in wav2vec-U 2.0. We would appreciate it if the reviewer could provide more details. We are willing to implement, analyze, and discuss it if time allows.

Figure 1 presentation suggestions

We thank the reviewer for the suggestion. We will revise Figure 1 based on the suggestion.

References

[1] Liu et al. “Completely Unsupervised Phoneme Recognition by Adversarially Learning Mapping Relationships from Audio Embeddings.” Interspeech, 2018.

[2] Yeh et al. "Unsupervised speech recognition via segmental empirical output distribution matching.” ICLR, 2019.

[3] Chen et al. “Completely Unsupervised Phoneme Recognition by a Generative Adversarial Network Harmonized with Iteratively Refined Hidden Markov Models.” Interspeech, 2019.

[4] Baevski et al. "Unsupervised speech recognition." NeurIPS, 2021.

[5] Gao et al. "Euro: Espnet unsupervised asr open-source toolkit." ICASSP, 2023.

[6] Strgar et al. “Phoneme segmentation using self-supervised speech models.” SLT, 2022.

It's a little surprising to see so few segmentation results except Table 4 (also raised by reviewer TC8W), given how much the emphasis is on segmentation.

Our work mainly aims to improve the performance of unsupervised ASR. Hence, the evaluations center around UASR performance. To investigate the intermediate results, i.e., segmentations, Table 4 analyzes the role and performance of segmentation. Specifically, Table 4 reveals some of the interesting findings of the REBORN boundary, such as more frequent segmentation, as discussed in L320-L329. Considering that the readers might also be interested in the boundary evaluation score on the human-annotated datasets, we will revise the paper to include the rebuttal’s Table 2 in response to Reviewer TC8W.

The results in Table 4 are weird. 1) Worse segmentation results actually lead to better PERs, contradicting to the claim in the paper that better segmentation leads to better PERs.

Although our method is motivated by the idea that “a better segmentation can lead to a better UASR performance,” we consider that there are two factors that affect the “goodness” of segmentation in UASR:

• The quality of the segmentation, which can be evaluated by boundary metrics.
• The suitability of the segmentation for the unsupervised phoneme predictor.

While the oracle boundary in Table 4 has better boundary F1 than the REBORN boundary, it is not tailored for the unsupervised phoneme prediction model. Note that the phoneme prediction model used in Table 4 is obtained from wav2vec-U, which uses the k-means boundary when training the phoneme prediction model. As a result, naively replacing the k-means boundary with the oracle boundary cannot yield the optimal PER. Contrarily, the segmental boundary learned by REBORN is better suited for the phoneme prediction model, yielding a better PER.

2) The proposed approach is over-segmenting a lot and does not seem to perform well compared to something as simple as k-means.

The lower F1 score for REBORN compared with k-means is because the segmentation is tighter. However, over-segmentation is not an issue since most segments result in consecutive phoneme predictions, which do not affect PER as they are de-duplicated, as noted in the paper (L132-L133). Phoneme deduplication is a common practice in UASR for generating more reasonable results ([1-3]), and is also involved in the training process of REBORN.

3) The paper deliberately emphasizes that the result is without boundary merging. Why not report both with and without?

Adopting boundary merging can lead to higher boundary F1, but in Table 4, we focus on the original RL-learned boundary to emphasize the importance of tailoring segmentation for the unsupervised phoneme predictor. Merging boundaries directly changes the segmentation, making it less suitable for the current phoneme predictor. This technique is introduced to stabilize stage 2 training rather than boost boundary F1 performance (see REBORN stage 2 in Table 7, where PER std is larger without boundary merging). Nevertheless, to provide a comprehensive boundary evaluation, we will include Table 2 from the rebuttal in our next revision.

4) How come using the oracle segmentation leads to a worse PER?

As introduced earlier, given that the phoneme predictor is non-ideal, directly using the oracle segmentation, although it still greatly improves the PER, may not be the optimal solution. REBORN segmentation, which is tailored for the non-ideal phoneme predictor, results in better improvements. However, referring to the first rows of Table 1 and Table 2, if we train the phoneme predictor directly on the oracle segmentation, the GAN-trained predictor is more suitable with the oracle boundary, thus surpassing the performance of REBORN.

References

[1] Baevski et al. "Unsupervised speech recognition." NeurIPS, 2021.

[2] Gao et al. "Euro: Espnet unsupervised asr open-source toolkit." ICASSP, 2023.

[3] Liu et al. “Towards end-to-end unsupervised speech recognition.” SLT, 2022.

We sincerely thank the AC for the thorough comments. Please find the response to your questions below.

It's unclear how the approach is different from Cuervo et al. (2022) (cited as [15] in the submission). Given that the authors are aware of this paper, the paper clearly fails to address the novelty (also raised by reviewer AM1A). Even the reward functions are very similar to the ones in Cuervo et al. (2022).

Our work significantly differs from Cuervo et al. (2022) [15] in the following aspects:

• Task: The method presented in [15] is designed for acoustic unit discovery, whose goal is to learning better-segmented representations. On the other hand, our work aims to address unsupervised ASR, which aims to directly solve the distribution matching problem between speech segments and phoneme sequences. The differences between these two tasks are specifically discussed in Section F of the Appendix.
• RL reward: The reward proposed in [15] uses contrastive loss (Equation 2), $\mathcal{L}\_H = - \sum\_{k} \sum\_{m=1}\^{M} \frac{\exp(p\_m\^T u\_{k+m}\^{q})}{\sum\_{i \in \set{k+m-1, k+m+1}} \exp(p_m^T u_i^q)}$, which drastically differs from our reward design. Specifically, we design the perplexity difference reward calculated by a phoneme language model, along with the edit distance reward and the length difference reward as regularization rewards: $R = c_{\text{ppl}} \cdot R_{\text{ppl}} + c_{\text{edit}} \cdot R_{\text{edit}} + c_{\text{len}} \cdot R_{\text{len}}$, where R_{\text{ppl}} = \text{PPL}\_{\theta-1} - \text{PPL}\_{\text{\theta}}, $R_{\text{edit}} = -\frac{d\_\text{Lev}(Y'\_{\theta-1}, Y'\_{\theta})}{|Y'\_{\theta-1}|}$, and $R_{\text{len}} = 1 - \frac{\left| |Y'\_{\theta}| - |Y'\_{\theta-1}| \right|}{|Y'\_{\theta-1}|}$.

Our work and [15] only share a similar high-level idea, which is using RL for boundary prediction. The main contribution of our work compared to [15] is that we find that in UASR, the segmentations that are tailored for the phoneme prediction model are crucial to the PER, and we use RL to allow the segmentation model to tailor the segmental boundary for the phoneme prediction model. Designing rewards for UASR to learn a segmentation model is non-trivial, and the reward design is completely different from [15] and not seen in any prior works in acoustic unit discovery.

The positive reviews are mostly due to the strong results, while being sparse on the soundness of the paper and where the improvement is coming from.

Reviewer TC8W is not convinced that the improvement is coming from segmentation, while reviewer q9jx and AM1A think the improvement is from the iterative training.

We would like to point out that the numeric scores of soundness in the review form show that all four reviewers consider this paper to be sound, with soundness scores 3, 3, 3, and 4. Moreover, in the latest response to the author’s rebuttal about segmentation, Reviewer TC8W agrees with us on the criticality of segmentation in UASR.

We believe that REBORN improves PER, compared to wav2vec-U, mainly from the segmentation model. Specifically, REBORN trains the segmentation model to tailor the segmental boundary. Under an unsupervised ASR setting, learning the segmental boundary for the phoneme prediction model without ground truth transcription is highly non-trivial. To overcome this difficulty, we construct carefully designed rewards to guide the segmentation model. We say the segmental boundary is tailored for the phoneme prediction model because the segmentation model is trained by the guidance of the phoneme prediction model (and with the external LM). With extensive experimental results presented in Table 4 and the rebuttal’s Table 1, we have verified that proper segmentation tailored to the phoneme prediction model is critical to the UASR performance, which is acknowledged by Reviewer TC8W. Also, Reviewer AM1A considers our reward design to be interesting and effective, which is listed as a strength of this paper.

Last, we would like to respectfully point out that the reviews from Reviewers AM1A and q9jx do not attribute the improvement of REBORN to iterative training. Reviewer AM1A did not mention the word “iterative”. Reviewer q9jx considers the improvement of REBORN in the early stage to be very significant and questions if it is necessary to frame the method as an iterative training method; this is contrary to attributing the improvement to iterative training.