Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models

We sincerely appreciate Reviewer 8Y8C for the valuable and constructive comments. Please find detailed responses below:

• Weakness 1 & 2

  Thanks for your feedback. We clarify that the true-label models use all labeled data from the training set of each domain. Therefore, how good this upper-bound is also depends on the size of their training set, with more training details provided in Appendix F.

  Furthermore, fitting a 1B model to a few hours of labeled data can lead to overfitting issues. Without corresponding measures to mitigate this, the model's performance in other domains would significantly degrade. However, the training labels of STAR are self-generated, and reweighting is merely an attempt to selectively update without forcing the model to overfit to a new data distribution. This is also why STAR avoids catastrophic forgetting.

• Question 1 & 2: Which layer or head of the attention weights are used for the indicator A.

  We appreciate your suggestion. In our experiment, the self-attention matrix is drawn from the last transformer layer of the foundation model, averaging on all heads. We also found that the performance is not sensitive to the selection of layers or heads. The last two layers, with a single head, can also achieve comparable performance.

  For the confusion matrix, we confirm that the attentive score from the last five layers is very similar to Figure 2. The earlier layers can also show similar patterns but are somewhat less indicative of the pseudo-label quality, as the high layers learned more linguistic semantics.

We appreciate Reviewer LswZ for valuable and constructive comments, and we believe our detailed responses below can solve your concerns on missing details. Please let us know if you have further questions or recommendations.

• Q1: Several missing details: Are the attention weights from cross-attention or self-attention used? How is the attention matrix calculated given multiple layers and heads?

  Thanks for your question. We use self-attention weight as we introduced in the paper (line 65 and line 180).

  In our experiments, the attention matrix is obtained by averaging all heads in the last transformer layer, as the high layers learned more linguistic semantics. We also found that the performance is not sensitive to the selection of layers or heads. The last two layers, with a single head, can also achieve comparable performance. More details can be found in our submitted code.

• Q2: Do you need to tune the threshold for each corpus, or simply use a fixed threshold for all datasets? What is the specific value for the threshold lambda?

  Thanks for your question. We apply a fixed threshold of lambda$=2$ for all corpus and it works well generally, as mentioned in line 264.

• Q3: The definition of conflict is $A_l^2/C_l$, which seems intuitive. Is there any particular reason for using this formulation?

  Thanks for your question. The design of $A_l^2/C_l$ can be decoupled into two terms, $A_l/C_l$ and $A_l$, which means to match the “conflict” criterion, we not only require $A_l$ to be much larger than $C_l$, but the absolute value of $A_l$ should also be large. This is to avoid special cases with two small scores where one is many times the other (e.g., $A_l = 0.1, 0.01$). We have mentioned this particular design in line 212 to 214 and footnote 2.

  The specific equation definition of “conflict” is not unique, if only the above condition is met roughly.

• Q4: Many studies have attempted to use entropy from the output layer instead of probability for confidence estimation in end-to-end ASR systems.

  Thanks for your question. Entropy is indeed an alternative choice for confidence estimation. However, in our case, since the probability $P$ cannot reliably indicate pseudo-label quality (Figure 2, Figure 5, and Figure 6), the minus entropy $P*logP$ that is positively correlated to $P$ cannot serve as a reliable indicator either. Our preliminary results have confirmed this, but considering $P$ is a more typical and intuitive choice, we only select $P$ as the confidence score baseline.

• Q5: I am interested in whether the informed fine-tuning shown in Equation (4) is proposed in this paper. The confidence score is used to filter utterances and weigh the importance of each word. Do you have any ablation studies demonstrating the effectiveness of this approach?

  Thanks for your question. The informed finetuning with confidence score in Eq. (4) is followed from prior works as discussed in the Introduction (line 59 to 62). However, we observe that the confidence score is unreliable in assessing pseudo labels so we propose a reliable attentive score based on the self-attention matrix. Finally, we combine conventional confidence and proposed attentive score as our final STAR indicator, which guides the token-level reweighting. On the other hand, utterance-level filtering is implemented by Monte-Carlo Sampling introduced in Section 3.3.

  For experiments, Table 1 presents the ablation study of utterance-level filtering, and token-level reweighting (confidence score, attentive score, STAR score).

• Q6: It would be better to combine Equations (6) and (7) into an equation array to clearly state the formulation.

  Thanks for your suggestion and careful check, we will revise it in the next version.

• Q7: In Equation (5), the attention score is computed using the attention weight for the current word to the previous words and from future words to the current word. Is there any particular reason for this choice? Is it acceptable for the A_l to be larger than 1?

  Thanks for asking this question. For the choice of history and future tokens in calculating the attentive score (Eq. (5)), we use both of them to capture the comprehensive global context to better assess the role of the current token, in terms of its semantic significance in the entire sentence. Table 7 presents an ablation study on this choice, indicating that both history and future tokens are helpful. The $A_l$ will be normalized using sequence length after calculation so that the absolute value of $A_l$ does not matter, we will add these details in the next version.

Dear Reviewer LswZ,

Thank you for your kind efforts in providing initial reviews for our work! We have taken them into careful consideration and provided the responses accordingly.

Since the deadline of author-reviewer discussion period is approaching, could you please take some time to confirm whether our responses have satisfactorily addressed your concerns? If you believe so, may you please consider adjusting your initial score accordingly? Please do not hesitate to reach us if you have any further questions or comments.

Best,

Paper 10759 Authors

We sincerely appreciate Reviewer zXq9 for your valuable and constructive comments. Please find detailed responses below:

• Q1: References to previous work

  We sincerely appreciate your feedback. While this is not a completely new topic, our proposed method only requires a downloadable foundation model and one hour of unlabeled speech to enhance the model in this domain without catastrophic forgetting. With the introduction of more speech foundation models, we believe this is a meaningful exploration.

• Q2: What would have happened if we had adapted a purely LibriSpeech-trained model to the domains?

  Thanks for your question. Allow us to point out that the robustness of a model trained on LibriSpeech is insufficiently robust to serve as a pseudo-labeler. For instance, using the ATIS test set as an example, the LibriSpeech ASR model cannot recognise many city names. In comparison, Whisper, which is a widely used foundation model trained on much larger and more diverse datasets, is a lot more robust and can achieve one-to-many adaptation to accommodate different speech domains. Related discussions are included in Appendix A (Q1 & Q2).

• Q3: What happens to STAR without Utterance-level Filtering?

  Thanks for your question, the ablation study without Utterance-level Filtering is in Table 8, and discussed in Appendix D. Since Table 1 can no longer accommodate additional columns, and Utterance-level Filtering is merely a process of removing particularly difficult-to-recognize samples. The effect is similar across different methods, so we have placed it in the appendix.

• Q4: Why was the LM-rescoring baseline not used?

  Thanks for your suggestion. We need to point out that in the STAR experiments, each domain used only about one hour of unlabeled speech, which is equal to a few hundred utterances (Figure 3). This amount of text is insufficient to directly train or even adapt a useful LM for LM rescoring. Furthermore, all of our experiments were based on a single end-to-end model, while the additional LM-rescoring stage will increase the complexity making it no longer a purely end-to-end system.

We sincerely appreciate Reviewer vnSx for valuable and constructive comments. Your suggestion is also instructive for further analysis and our future work.

Please find detailed responses below:

• Q1: Appling STAR for CTC or RNN-T-based ASR models.

  Thanks for your feedback, As the illustration in section 3, applying STAR to CTC or RNN-T, auto-regressive Whisper is still employed as a pseudo-labeler to provide pseudo transcription, as well as token-level or utterance-level indicators. Therefore, the RNN-T model can be adapted based on the provided information. We acknowledge that the process requires compositional efforts, but this connection to popular ASR variants also has high potential application value.

• Q2: Discussion of the differences and similarities between STAR and noisy student-based training [1][2].

  Thanks for your feedback. We will add a paragraph of related discussion to the paper. Although both STAR and NST contain a pseudo labeling and filtering process, they have several distinctions listed as follows:

  • Motivation: STAR focuses on domain adaptation, and NST focuses on leveraging abundant unlabeled data to improve general ASR accuracy, where the unlabelled data are from similar domains (LibriSpeech and LibriLight datasets).
  • Data: STAR only requires 1 hour of unlabeled data from any target domain, NST requires both labeled data (960 hours) and abundant unlabeled data (60k hours).
  • Method: STAR focuses on the token-level indicators for informed finetuning (which could be attributed to the strong linguistic ability of the Whisper decoder), and NST focuses on utterance-level data filtering.

• Q3: Missing a normalization term in Equation 5.

  Thanks for your valuable question. When we obtain the attention matrix, it is normalized according to the sequence length. We will clarify this point with a note.

• Q4: Why not use the cross-attention matrix?

  Thank you for this discussion. As we described in the paper, self-attention in the decoder better reflects the linguistic acceptability of the predicted transcription. As for cross-attention, it is more directly linked to the acoustic representation–vulnerable to variable speech domain. In contrast, self-attention is text-based and Whisper's multi-task pre-training also includes text-only learning objectives. Therefore, self-attention potentially gives more linguistic knowledge. Our preliminary results also confirmed that the cross-attention matrix could not provide a reliable indicator.

• Q5: Difference between Table 1 and Table 2.

  Thanks for your question. Table 1 shows the main results where Whisper is both finetuned and tested on each domain individually. Table 2 aims to explore the forgetting issue of STAR, where Whisper is finetuned on the CHiME-4 dataset and then tested on other datasets. Therefore, Table 2 presents worse results than Table 1. We will clarify this in the captions of the tables.

• Q6: How many beams are used in the beam search? Are the beam search-based pseudo-labels also used for self-training baselines?

  Thanks for your question. For beam search, we set beam$=5$. Our preliminary results show that beam search-based pseudo labels show almost the same self-training performance as the greedy search ones. Therefore, for higher efficiency, we use greedy search only.

• Q7: ​​Can Whisper-Large generate training resources for Whisper-Tiny?

  Thanks for the question. Yes, we confirm that STAR can distill knowledge from large models to smaller models (but not vice versa). We will include this advantage in the next version.