Advancing Test-Time Adaptation for Acoustic Foundation Models in Open-World Shifts

Thanks for the careful review and constructive suggestions. Below are our detailed responses to your questions:

Q1: "writing quality could be very improved"

A1: We appreciate your feedback regarding the writing quality. We have carefully revised the citation formats and typos in the new version of our paper.

Q2: "The related speech processing references are missing", "A well-known recent insight involves latent space alignment via Wasserstein measurements, yet the related discussion is absent here"

A2: Thanks for highlighting these references. We have incorporated a discussion of them within the related work section in the revised version.

Q3: "how's the proposed methods different with reprogramming and jointly reprogramming plus adapter in the prior works?"

A3: In summary, the key difference is that our proposed method performs unsupervised adaptation-based entropy minimization while reprogramming and jointly reprogramming plus adapter perform supervised optimization with access to annotated target data pairs. Furthermore, while reprogramming-based methods focus on training additional small parameter sets for data input parts or adapters within the model, our method focuses on the feature modulation on norm layers without additional parameter introduction.

Q4: "...how different pre-trained speech and acoustic models...", "provide studies different pre-trained acoustic models to have a deeper look", "...only Wav2Vec 2.0 Large has been evaluated. The term "foundation model" is often used for models with over 1 billion or even 100 billion parameters to showcase emergent abilities, which is not the case here"

A4: Thanks for the good suggestion. We choose the Wav2vec2 models due to their popularity in the community and we acknowledge the importance of evaluating diverse acoustic models. Hence, we conduct additional experiments using, Hubert-Base, Hubert-Large, WavLM-Base, and WavLM-Large to compare the adaptation performance under different model sizes, and training data sources. The experimental results are discussed in Appendix A.5 in the new version.

Furthermore, we believe that there hasn’t been a clear definition for the "acoustic foundation models" in terms of model sizes. This perspective -- "The term "foundation model" is often used for models with over 1 billion or even 100 billion parameters" -- is often discussed in the context of large language models. We refer to "acoustic foundation models" as those large acoustic models that serve fundamental roles and can be extended to various downstream speech tasks.

Q5: "Similar parameter-efficient baselines work on frozen pre-trained adaptation diagrams, making it difficult to justify the novelty on applying trainable layer norms"

A5: We hope to kindly clarify that parameter efficiency and applying trainable layer norms are not our main contributions even though our method is indeed parameter-efficient. Rather, we investigate TTA within the audio modality and discover that non-silent frames with high entropy contain vital semantic content that cannot be dropped. This motivates us to develop a learning-based adaptation approach to address it.

Besides, there is a key difference in the experimental setting. That is, the mentioned parameter-efficient baselines are trained using supervised signals, while our method performs unsupervised optimization.

Q6: "what is the trainable parameters in both settings?"

A6: In the CEA, trainable parameters include affine parameters associated with layer normalization and the feature extractor. In the STCR, trainable parameters are affine parameters associated with layer normalization in the model, as illustrated in Figure 2.

Q7: "in terms of acoustic modeling, any acoustic classification or speaker-level open set has been studied in the proposed method?"

A7: Thanks for this good suggestion. We mainly evaluate ASR in our work since it is one of the most typical problems in speech processing. We definitely agree that acoustic classification and speaker-related tasks are also interesting and could be extendable tasks in future work.

Q8: "I am curious how the entropy based additional losses training gonna impact the latent space distance measured from w2v2 pre-trained to target open set"

A8: We believe such latent space distance might increase as entropy-based additional losses decrease. Basically, it might not be meaningful for a direct comparison of such distances as we cannot correlate larger distances to better adaptation performance. Instead, it might make sense for the distance comparison among different adaptation methods. This might be an interesting future work.

We hope these responses comprehensively address your queries, and we are open to any further discussion or feedback.

Thanks for authors for the responses.

• I am happy to see the paper draft format (citations latex issues and model size issues) and quality have both improved largely.

Besides, there are issues I confirm the authors would have replied with imprecise information. I provide below for further references / help for improvements.

1. Does layer norm / feature modulation introduce new additional associated with trainable parameters?

• Actually, Layer Normalization (Layer Norm) does have trainable parameters. Specifically, it has two parameters: the scale (denoted as $\gamma$) and the shift (denoted as $\beta$). As the cases in this work, applying an additional layer(s) of Layer Normalization to a frozen model can be considered as introducing new parameters for model adaptation.

This extra norms provides a way to adapt the model to new (open domain) data or tasks without altering the original, pre-trained parameters of the model, which is a strategy often used in transfer learning and fine-tuning scenarios.

As a frozen model adaptation study, the additional loss based training upon the additional trainable parameter is standard for having adapters baseline as the reviewer’s initial review suggestion.

Also, the authors’ response of no extra parameter introduction is imprecise (e.g., difference between acoustic model reprogramming and input prompting).

Please consider to correct it since it will potentially mislead further readers.

2. The wav prompt reference

• The wav prompt reference is not a rigorous format of in-context learning, where ICL is required sequence stacking in previous rounds forward propagations. Referring as prompting learning is much formal.

3. Language usage of foundation models

• I disagree with the authors’ viewpoint on non large scale (1B+) statement. General purpose pre-trained model and foundation model in terms of scaling effects benefited representation learning are different, since many frozen model adaptation characteristics are highly related to scales of model and data. This point does not affect my final score.

In short, this is borderline paper for me. The authors have made largely improvements from a clear reject initial draft. In the latest draft, there is still a gap in terms of deeper understanding of parameter efficient adaptation that missed other multi-loss based parameter efficient adaptation analysis.

I am ok if the paper got accepted but I would keep my recommendation as borderline reject.

The authors could consider to put trainable color blocks into red and frozen colors blocks into blue / gray for better visualization toward ICLR audiences.

Thanks for the authors clarified the layer norm has been parts of the frozen pre-trained model instead of adding new layer norms. My original interpretation is indeed with this confusion.

Meanwhile, the partially released setting of layer norm / feature modulation is related to bias-only tuning and P-tuning in parameter efficient adaptation.

In recent understandings, parameter efficient learning (e.g., like this partially frozen adaptation work) performing better in few-shot and low-resource adaptation is due to the generalization loss over universal approximation without modifying the decision boundaries. The data efficiency is not the main focus on the performance-driven parameter efficient learning. I would like to point out that the connection between existing frozen representation learning is important here.

P-tuning in Q, V would often take longer time to process experiments. If the authors would add some preliminary results on their additional losses training works for bias-only tuning (or commit to add it in their final version), I would consider the disconnection of equivalent techniques have been satisfactorily resolved to increase my score to six.

The aforementioned Chen et al. [5] and A. Pasad et al. ASRU 21 have discussed this relationship. The author may consider to add some related large pre-trained AM theoretical findings discussion to have broader impacts in their final version.

Again, thanks the author for the replies. I would avoid to use foundation model for rigor definition and encourage the authors spend some time reading the original foundation model paper.

Thank you for your meticulous review and constructive feedback. Below are our responses to your queries:

Q1: "One of the paper's goals is heuristic-free test-time adaptation. As for me, the authors mainly move existing heuristics into loss functions, which makes it a learnable approach and gets rid of hyperparameters that need to be manually set (at least when ignoring optimization hyperparameters such as learning rate, etc.). But the heuristics remain, don't they?"

A1: Thanks for this good question. We acknowledge that the heuristics still remain. Our "heuristic-free" refers to removing the filter-based method for unreliable data with high entropy. Our original intention is to make this heuristic-based method into a learning-based method. We will clarify our description and revise it in the new version.

Q2: "... However, the only twist that the authors add is an auxiliary loss based on some heuristic for speech signals' short-term consistency ...", "Why do we need the auxiliary loss in Eq.(5) to improve the consistency of speech signals?"

A2: The rationale behind the short-term consistency regularization is grounded in the inherent characteristics of speech signals. Typically, neighboring speech frames often correspond to the same phoneme or character. As such, this consistency regularization aids in fostering reliable predictions and maintaining coherence in speech frames.

Q3: "...Is this all we can/need to do for sequential data?", "Could we get rid of this heuristic by using a proper sequence-level loss (e.g., sequence-level instead of frame-level entropy in Eq.(2))?"

A3: Thanks for the great suggestion. We agree that adopting the sequence-level loss, such as sequence discriminative training loss might further improve the TTA performance. We consider exploring this as a future work.

Q4: "a few issues with the equations"

A4: We are sorry for the confusion. Eq.(3) and Eq.(4) should be frame-level quantities. We have revised Equations (3) and (4), as well as the cardinalities in Equation (5) in the updated version of our paper. Thanks for pointing out these issues.

Q5: "For Figure 3, is the classification done once on the initial, non-adapted model or repeatedly on the respective adapted model?"

A5: The classification is done repeatedly on the respective adapted model.

Q6: "How are the hyper-parameters (e.g., loss coefficients) selected?"

A6: Hyper-parameters are selected using standard search techniques. The implementation details of hyper-parameters can be found in Appendix A.2.

Q7: "Could you please clarify what LS-C stands for: LibriSpeech Noise Corrupted (Figure 3) vs synthetic dataset (Sec. 5.1)?"

A7: We are sorry for the confusion. LS-C stands for LibriSpeech test-other set Corrupted by Gaussian noises, as initially mentioned in Sec. 5.1. We have made appropriate revisions in Sec. 5.1 for greater clarity.

Q8: "Figure 3: Is percentage in [0,1] or [0,100]?"

A8: The percentage value is in the range [0,1].

Q9: "Something is off with the inline citations." "Typo: "model training phrase""

A9: Thanks for pointing out these issues. We have revised them in the new version.

Q10: "...This makes me wonder what the lowest achievable WER (e.g., human WER) is, in particular on the heavily corrupted data sets? This information might be useful to understand the empirical results."

A10: This is a good question. We agree that the human WER bound can indeed assist in understanding the experimental results. However, it is difficult for us to obtain such numbers as it requires conducting human subjective studies. Besides, heavily corrupted speech data may not be common cases in the real world. Our primary aim in this experimental setting is to evaluate the trend of model robustness under varying noise levels, and our results demonstrate that our proposed method outperforms all baselines in this regard.

We hope these responses adequately address your concerns and we are open to any further feedback or discussions.

We appreciate your thoughtful review and valuable suggestions. Our responses to your questions are as follows:

Q1: "The contributions seems minimal, as the core idea of TTA are already available in the literature"

A1: We recognize the existence of TTA-related works in the literature. However, our work focuses on acoustic modeling, compared to the prevalent focus on vision tasks in previous studies. Specifically, our key contributions include:

• Identifying that noisy speech frames with high entropy often contain non-silent segments crucial for semantic understanding.
• Introducing a learning-based adaptation approach, complemented by short-term consistency regularization, to augment the adaptation performance.

Q2: "In comparison to different vocabulary sizes, such as employing a BPE model with a larger vocabulary, how does this work perform? Would the entropy for non-silent frames significantly increase under such circumstances?"

A2: Thanks for the question. Our proposed method is generalizable to models with large vocabulary sizes. Theoretically, the maximum entropy for non-silent frames is expected to increase due to the larger number of classes. Practically, this might also depend on the test input and models. We conduct an additional experiment using the Conformer-CTC model with BPE tokenization, observing an increase in entropy for non-silent frames from 59.4% to 70.0%, as illustrated in Table 2.

Table 2: Entropy Distribution at Step 0 for models with different vocabulary sizes.

Q3: "Could this approach be adapted to function with a model architecture similar to RNN-Transducer? What specific modifications would be necessary to enable its compatibility with such an architecture"

A3: We think that our proposed approach is adaptable to architectures like RNN-Transducer. The primary modification would involve basing the prediction probability and pseudo labels on the decoder outputs.

Q4: "... Would it be a viable alternative to the proposed TTA approach to fine-tune the wav2vec2 encoder using test utterance audio as a baseline? What advantages or disadvantages might this alternative approach present"

A4: Thanks for the question. Self-supervised fine-tuning wav2vec2 encoder could serve as a baseline. However, in our experimental setting, we consider a more practical setting where we only have access to a single utterance for adaptation before test-time inference. In this setting, self-supervised fine-tuning wav2vec2 using such a single utterance poses two major disadvantages:

• The insufficiency of a single test utterance for effective batch construction, leading to data imbalance issues.
• Potential distortion of the models' distribution upon fine-tuning the entire wav2vec2 encoder.

Q5: "Could you provide further clarification on the decoding strategies outlined in Section 5.4 of your paper? Specifically, post-adaptation, how do different inference strategies, such as beam search or greedy decoding, impact the inference process, and what are the key differences in their outcomes?"

A5: The performance differences between the two decoding strategies are detailed in Table 4, Sec.5.3.2. Beam search incorporates an additional language model in the decoding process, while greedy search does not. This language model, with its inherent text distribution prior, can help eliminate impossible text combinations during decoding. Moreover, the inclusion of a language model provides a mechanism to address text-domain shifts.

We hope these responses comprehensively address your concerns and we remain open to further discussion and feedback.

We are grateful for your insightful feedback and offer our responses as follows:

Q1: "...The importance of the two components could be different in the noise shift than the accent shift. what do you think?"

A1: Thanks for this question. We agree that the importance of two components may indeed be different in the noise shift and accent shift because these shifts come from distinct sources. Therefore, we conduct additional ablation studies for them respectively. Specifically, we conduct experiments using the dataset with level-3 Gaussian noises in LS-C for noise shift and speech of speaker BWC with Mandarin as L1 in L2-Arctic for accent shift. All experiments are carried out using the Wav2vec2 Base model. The results are detailed in Table 1:

Table 1: Ablation study of core components in the noise shift and accent shift

These experimental findings indicate a pronounced efficacy of our method in mitigating noise shifts as opposed to accent shifts. We conjecture the reason could be that the shift caused by Gaussian noises for each frame is consistent while other shifts such as accent shift could be different within frames.

Q2: "Do you consider language as a shift? could a foundation model be adapted to a new language?"

A2: Thanks for the interesting question. In our current experimental setting, we mainly address acoustic shifts including Gaussian and environmental noises, accents, and sung speech. We consider the text-domain shifts within the same language, reflecting variations in the linguistic content or context of the speech data. Adapting the model to a new language is an important research problem in multilingual acoustic models. We consider it as future work.

Q3: "what is a practical way to get alpha in equation 5?"

A3: Alpha represents the weight for short-term consistency regularization and is a hyperparameter in our experiments. We employ standard grid search techniques to search the value ranging from 0.1 to 1 with 0.1 an interval. The details can be found in Appendix A.2.

We hope that these responses satisfactorily address your questions and look forward to any further feedback or suggestions you may have.