Unsupervised ASR via Cross-Lingual Pseudo-Labeling

We would like to thank Reviewer GYQT for their time. Please find below our comments:

It would be more valuable to demonstrate that a single source AM can be effectively applied to multiple target languages (e.g., en -> {sw, ha, X, Y, Z}) rather than showcasing the application of multiple source AMs to the same target language (e.g., {en, es, ge, fr} → sw):

Thanks for the interesting suggestion! We will investigate this in the future.

It appears that this method is effective primarily for the language pairs { *-> sw} and {be -> cs}, which raises concerns about its generalization to other low-resource languages.

We wish we could have included more low-resource languages settings. However, the main obstacle is the availability of the unlabeled audio data and unpaired text for the same language. Apart Common Voice, we are not aware of many freely available datasets with 30h or more unlabeled audio per language, and large available text corpora.

In contrast to wav2vec-u, the concept of "similarity" might restrict the applicability of this method, potentially preventing it from identifying an appropriate source language for a specific target language.

We agree. wav2vec-U 2.0 and our approach have different tradeoffs. We trade (phoneme lexicon, GAN training, and large amount of unlabeled target audio) for (simple pseudo-labeling, limited amount of labeled source audio). In practice, as labeled source audio is always available, our approach is most likely applicable. In contrast, wav2vec-U 2.0 requires a phoneme lexicon and large amount of target audio, which may not always be available. As for which source language to choose, zero-shot performance on target language seems to be a good indicator of the cross-lingual transfer difficulty (see Figure 3).

In the paper, it is claimed that "for African languages, we use Kinyarwanda only as a source language (as text data are not available in the Common Crawl dataset)." How is the Kinyarwanda AM trained without text data?

Kinyarwanda has labeled audio data (with transcriptions) in Common Voice, which is used to train the source acoustic model for Kinyarwanda. However, no available unpaired text data to train a language model is available. If we had used the audio transcriptions for that matter, then audio and language model would have the same training text data, in which case the problem could be viewed as a simple alignment problem rather than unsupervised ASR.

Table 3 indicates that the performance of the de degrades when language model decoding is applied. Is there any analysis provided in the paper to explain this

Thanks for the question! As German uses compound words, the language model dictionary has limited word coverage. This shows at beam-search decoding time, when unknown words (not in the dictionary) are then decoded into multiple words allowed by the dictionary. We found that if we allow the beam-search decoding to produce unknown words, then it mitigates the issue, at the expense of an additional hyper-parameter. For simplicity, and consistency with other languages which did not require it, we sticked to beam-search decoding restricted to the original dictionary.

We would like to thank Reviewer mAuZ for their time. Please find below our comments:

It would be good to compare performance on more languages, prior work has investigated FLEURS. Comparing performance on this data set would enable clearer comparisons.

As prior work on unsupervised ASR did not report numbers on FLEURS, we did not consider it, as we would not be able to compare with them. Also, as far as we know FLEURS is used to compare multilingual systems trained on data of internet-scale, which is not the type of system trained in this paper.

There is a reliance on the unidecode to “romanize” the character sets of various languages to Latin script. The recognition performance in a language should be in its own script, not a romanized version. It does not seem as though unidecode inverted prior to CER calculation. (Though I suppose this could also be a “question” rather than a “weakness”

We used unidecode only to “romanize” characters which are not in the native character set of each of the language considered. In practice, we found this occurs rarely (< 0.01%). Note that Common Voice datasets come unnormalized (for example, they contain punctuation), and using unidecode is a way to normalize the Common Voice datasets in a generalizable manner (across different languages), with little effect on the WER (given those events occur rarely). We will release the normalization procedure for reproducibility.

Is the example in Figure 1 a real example or made up to demonstrate idealized behavior? This wasn’t clear from the context.

It is a real example from the es → en model. We will clarify it.

In Section 2.0 it is claimed that ‘unsupervised ASR is viable, as long as source and target language share enough ”similarities”’. Which similarities are critical for performance here? Acoustic, lexical, other?

Based on our empirical results in Figure 3, languages which are known to have acoustic similarity transfer better. In addition, if the target language has simple pronunciation rules (like Spanish), transfer works better in practice. In contrast, French (known for its highly irregular orthography (Adda-Decker et al., 2005)) is a difficult target language, as inferring those rules is hard.

Zero-shot performance on target language seems to be a good indicator of the cross-lingual transfer difficulty (see Figure 3).

Just a note for Section 5.6 – the title describes transfer across “Alphabets”, however, not all writing systems are alphabets. Would this approach extend to abugida, abjad or logographic writing systems?

Using an adequate transliteration system, one could convert logographic writing systems to other alphabets, and still use our approach. For example, converting Chinese hanzi → pinyin (e.g. ”你好“ → ”ni hao“) or Japanese kanji → romaji (e.g. ”日本“ → ”Nihon“). As the point of the paper is to show how and when end-to-end unsupervised ASR via cross-lingual PL can work, we did not investigate cases where complex transliteration systems are required though. We will change ”across alphabets“ in the section title to ”across writing systems“.

Note: it would be nice if Figure 4 used the same Y axis in both tables. (It’s more understandable that the X axis varies)

Thanks for the suggestion! We will fix it.

We would like to thank Reviewer 26fF for their time. Please find below our comments:

Although the topic is very important, the technique itself does not have sufficient novelty as an ML conference paper. The topic is specific to ASR, and the technique is based on one particular ASR method (i.e., CTC). The connection to general ML problems is not clear. for example, some experimental results (e.g., the use of n-gram LMs, etc.) are specific to CTC, and it does not seem to be generalized to the other architectures.

Unsupervised end-to-end ASR that practically works is a significant novelty brought by this paper. End-to-end ASR alleviates the need for any phoneme lexicon required in other successful unsupervised ASR approaches (like wav2vec-U 2.0). Another notable difference with wav2vec-U 2.0 is that we do not rely on adversarial training at all. We will highlight better these two points in the paper.

We truly think that speech-specific papers have their place at ICLR. Prominent work in ASR such as wav2vec 2.0 or wav2vec-U were published at NeurIPS and Whisper was published at ICML.

Our method is generalizable to any acoustic or language models which can output posterior probabilities (see Eq. 1). There is nothing specific in our approach which limits ourselves to CTC (or Transformers acoustic models).

Most experimental findings are expected, and there are not so many new findings (e.g., it's a bit trivial that large data help the performance, etc.).

As stated above, unsupervised end-to-end ASR that practically works is not trivial, and has not been fully addressed in previous literature.

The methodology is not very new. Although there are several differences, some prior studies try to transcribe unseen languages with seen language ASR systems (e.g., Hasegawa-Johnson, Mark A., et al. "ASR for under-resourced languages from probabilistic transcription." IEEE/ACM Transactions on Audio, Speech, and Language Processing 25.1 (2016): 50-63.).

Thanks for the reference. As for most previous work in unsupervised ASR, pointed paper uses ASR with pronunciation dictionaries. In contrast, as stated above, we introduce a practical approach for training end-to-end unsupervised ASR models. In addition, pointed paper i) uses now outdated GMM ASR ii) states that “self-training (ST) [...] costs very little, and benefits little”, while self-training (a.k.a. pseudo-labeling) is a core component of our approach (critical for low WER performance). We will add the reference.

Did you use SSLs? Since SSLs are obtained by unsupervised training, combining this method and SSL would be very powerful.

Indeed, it has been shown that SSL and pseudo-labeling are complimentary for monolingual semi-supervised ASR settings, see Xu, Q., et.al. Self-training and pre-training are complementary for speech recognition. ICASSP 2021 and Zhang, Y., et.al Bigssl: Exploring the frontier of large-scale semi-supervised learning for automatic speech recognition. Journal of Selected Topics in Signal Processing 2022. We plan to study this in a follow up paper.

Can you explain why this method uses CTC? Is there a particular reason, or is this method applicable to the other methods (e.g., HMM, attention-based encoder-decoder, RNN-T)? I think that this part makes the paper's scope narrow.

We picked CTC for simplicity. As mentioned above, our method is generalizable to any acoustic (seq2seq, RNN-T...) or language models which can output posterior probabilities (see Eq. 1).

We would like to thank Reviewer UAMi for their time. Please find below our comments:

The contribution of this work is limited in cross-lingual scenarios.

Baseline system setting is relatively limited. In the core validation experiments (section 5.2), the baseline is only zero-shot evaluation w/ and w/o target LM plus the fully supervised training. It would be more informative if other unsupervised training methods can be compared side-by-side.

One of the main contribution of our paper is to show that end-to-end unsupervised ASR is practical (in the sense that it may almost match fully supervised ASR). In contrast, successful previous unsupervised ASR approaches were phoneme-based. We compare with the best available end-to-end unsupervised ASR system, which is wav2vec-U 2.0 on LJSpeech (see Sec. 5.7).

The paper does not demonstrate sufficient novelty. The author sets an assumption: the target language has no labelled speech accessible but has a fair amount of text data to train an LM. The paper reports how an existing IPL method works under this assumption.

Unsupervised end-to-end ASR that practically works is a significant novelty brought by this paper. End-to-end ASR alleviates the need for any phoneme lexicon required in other successful unsupervised ASR approaches (like wav2vec-U 2.0). Another notable difference with wav2vec-U 2.0 is that we do not rely on adversarial training at all. We will highlight better these two points in the paper.

Note also that wav2vec-U 2.0 concludes that “Future work includes simplifying the self-training pipeline and removing the need for a phonemizer.” We demonstrate in this paper a way to do this.

The LMs seem to be trained and fixed for experiments. Have you tested the correlation between the LM's PPL VS. final WERs?

Correct. We did observe a correlation between LM PPL and WER, in the context of n-gram-based word LMs. We also observed that increasing the dictionary LM size (number of words) improved WER. We will report these results.

Please check and fix typos (e.g. section 5.2: "any Indo-European language, any Indo-European language")

Please give pointers to the tables/figures in line. E.g. in section 5.5 "the results below" doesn't correctly point to the corresponding table.

Thanks for spotting this, we will fix it!

Dear Reviewers,

The authors have submitted the response to all reviews. Would you please read the authors' responses and acknowledge it, respond to them early on in the discussion, and discuss points of disagreement? Thank you!

AC