OWLS: Scaling Laws for Multilingual Speech Recognition and Translation Models

Thank your for your insights and comments.

I'm not quite convinced with the claim related to ICL ability. Whisper-style models like OWLS are trained primarily for ASR and translation, not instruction-following...

While Whisper-style models are not trained for instruction-following, instruction-tuning is not a requirement for models to have the capability to perform ICL, although it will certainly improve their ICL capabilities by minimizing the distribution shift between prompts and pre-training data [1]. Early LLMs, such as GPT2 and GPT3 [2,3], were not instruction-tuned and were only trained on pure language modeling. ICL can be described as a byproduct of auto-regressive modeling, which is how ASR/ST are formulated in auto-regressive encoder-decoder frameworks such as Whisper. ICL is thus a reasonable expectation for the OWLS models because they may be able to use in-context examples (speech-text pairs) as a prior for mapping sounds to words. Examples we provide in our "Strengths And Weaknesses" response to Reviewer ogkW illustrate this capability. [4] also shows preliminary results for this, although we believe that their weaker results are due to the much smaller model size of the tested models.

One limitation is that the method lacks detailed justification for scaling the model specifically up to 18B parameters...

We chose the upper of 18B primarily due to budget constraints. At 18B, we are still able to perform training without sharding the model parameters across GPUs, which is why we are able to maintain such high training efficiency (training time of 17 days). If we performed model sharding, the expected training time would have been ~2 months, which was unfeasible. Additionally, we were still able to maintain the same mini batch size at 18B without performing gradient accumulation at our compute budget, as gradient accumulation would have also slowed down training significantly. We want to maintain identical batch sizes to minimize experimental variance from the hyper-parameters.

In terms of the model architecture, we equally allocate parameters between the encoder and decoder for two main reasons:

• It is the approach adopted by current SOTA models such as Whisper
• It was found to be the optimal setting in encoder-decoder scaling for NLP [5,6]

Finally, we chose the Transformer architecture due to its ease of scaling. We experimented with other architectures, such as Conformer, but found it was extremely difficult to tune the hyper-parameters and get the model to converge. This is a similar finding to that of Zhang et al., when they trained Google USM [7]. Other works [8] has shown that the advantages of Conformer diminish as the model size grows. Overall, we believe that our design choices are indeed optimal when considering both effectiveness and efficiency. We will add all of this information in the final draft.

it is pretty obvious to discover the scaling laws with language-specific model size and scaling data with diversity

However, a clear weakness is the minimal methodological novelty... Moreover, most findings (i.e., better performance as the model, data, or compute scales) are obvious...

We respectfully argue that many insights of the paper are not predictable. While the notion that a larger model/data/compute leads to better performance is indeed obvious, our results showing that the exact performance improvements w.r.t downstream metrics like WER can be modeled as a scaling law is a novel insight. Many previous scaling law papers model only show scaling laws w.r.t test cross-entropy loss, which is a more predictable but less actionable result for practitioners and researchers. We also want to emphasize that the degree to which performance improves at scale is also non-obvious. For example, En to X ST BLEU increases by a very large factor of 21.8% on average when scaling from 2B (already larger than most SOTA speech models like Whisper Large (1.5B) and Seamless (1.2B)) to 18B parameters, showing that there is much to gain at scale.

Every figures contain excessively small font sizes, making them difficult to read..

We apologize for the inconvenience. We will make sure to rectify this in future editions.

[1] An Explanation of In-context Learning as Implicit Bayesian Inference (Tie et al., ICLR 2022)

[2] Language Models are Unsupervised Multitask Learners (Radford et al., 2019)

[3] Language Models are Few-Shot Learners (Brown et al., 2020)

[4] Can Whisper perform speech-based in-context learning? (Wang et al., ICASSP 2024)

[5] Scaling Laws for Neural Machine Translation (Ghorbani et al., ICLR 2022)

[6] Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Raffel et al., 2019)

[7] Hearing the AGI: from GMM-HMM to GPT-4o. (Yu Zhang, 2024). https://youtu.be/pRUrO0x637A.

[8] Revisiting Convolution-free Transformer for Speech Recognition (Hou et al., Interspeech 2024)

Thank your for your insights and comments. We organize our response by section:

Supplementary Material

We believe that we have all the possible information for the embedding extraction process in the appendix, although we acknowledge that the description may be unclear and hard to follow through text alone. For further clarity, we will add the following figures to the manuscript to better visualize the embedding extraction method: https://anonymous.4open.science/r/owls-samples-13A3/icl_figure.md . The code to perform ICL will also be released for reproducibility.

Claims And Evidence:

1. We understand the reviewer’s concern. While our definition was indeed focusing on the fact that scaling leads to a better model for many languages, we do acknowledge that this definition may be too narrow. We will instead rephrase the fairness claim to “Scaling to larger models can improve ASR performance across the board, in both low and high resource languages, with significant improvements in the former category.”

2. We perform inference with the model after 15K, 30K, 60K, 120K, 240K, 480K training steps, and use these datapoints to fit a power-law function. We then extrapolate the power law to 675K steps, which is where the final checkpoint is taken from, and compare it to the true WER of the final checkpoint. In Fig 9, each datapoint corresponds to each checkpoint that we evaluated (note that some early checkpoints are not visible since we cut off the chart at 100 WER). The training FLOPS of each checkpoint is then calculated by profiling each model size for a few steps and scaling the result to 15K-675K steps. We will adjust the manuscript to make these details more clear.

3. We also arrived at the conclusion that the clusters are largely distinguished by character set. However, we believe that linguistic similarity likely also plays a role. Russian, despite using Cyrillic letters, is more linguistically similar to English and the bottom cluster languages (which are all European), than the top cluster languages. We believe there may be some miscommunication for Arabic due to our presentation format. Arabic is the red line in the middle of the top cluster and certainly suffers from higher CERs. We will update the chart with different shapes for each language to make it less confusing.

Other Strengths And Weaknesses

Fig. 4, was there an experimental issue in the AMI (or Tedlium, cannot quite tell from the color) ASR runs?

We addressed below in our response to Question 3.

In-context learning is happening to some extent but the given example may not be sufficient to claim this ability.

We have included a few samples on example ICL generations https://anonymous.4open.science/r/owls-samples-C53F/README.md , which should better illustrate the improvements in ASR output as the number of in-context examples increase. The most obvious change is from pure zero-shot (k=0) to on-shot ICL (k=1), where the model changes from outputting a completely wrong language / character set to text that looks like Quechua. With more examples, the models also start to learn more complex phone-to-orthographic mapping ("ai" -> "ay") and white space placement ("ruraikunatukunapa -> ruraikuna tukunapa").

Other Comments Or Suggestions

1. Yes for both. Apologies for the mistake. We will fix this typo.
2. Yes, the caption should also read PN-CER. We will fix this typo.

Questions For Authors

1. We believe that the large increases in WER/CER are due to two main factors: 1) domain mismatches between the additional YouTube data and FLEURS evaluation data and 2) interference from similar languages/dialects (ie. Chinese and Cantonese).

2. Addressed above in "Claims and Evidence" point 1.

3. The fluctuation is in AMI, where the data is largely spontaneous and conversational, as the audio is sourced from business meetings. Since our training data is largely read speech and multilingual, it can be expected that there will be some small fluctuations on individual test sets despite average performance improving with scale. We will make the figure easier to read by using different shapes for the points of each dataset.

4. Addressed above in "Claims and Evidence" point 2.

5. Addressed above in "Claims and Evidence" point 3.

Thank you for the review and insights.

some details about data preprocessing are missing

Reviewer uTgG raised similar concerns. Please refer to our response to their question 3.

code-switching improvements appear inconsistent across languages.

Since we are performing multilingual multi-domain training and evaluation, some per-language fluctuation is expected. However, the average code-switching CER clearly decreases with increases in model size. We will better illustrate this by including a line for the average performance, along with a table with raw scores in the appendix.

the role of domain mismatch in dataset expansion should be further explored

While we agree that the domain mismatch should be further explored (and plan to do so in future work), we believe that further experiments are outside this paper's scope, which is focused on multilingual neural scaling rather than studying the role of data domains. The domain shift finding is a consequence of doing such large scaling analyses, rather than a hypothesis we explicitly designed an experiment to test.

The analysis of emergent abilities would be stronger with qualitative examples.

Table 9 includes example mondegreen generations. We have uploaded the accompanying audio for reference in https://anonymous.4open.science/w/owls-samples-88B2/ . We also added sample generations for ICL (Table 5): https://anonymous.4open.science/r/owls-samples-C53F/README.md . We will include these in the final manuscript.

More details on hyperparameter selection and convergence criteria

We use the same hyper-parameters as OWSM v3.1, which generalized to all model sizes. We did not have explicit convergence criteria, but instead cut off training after 675K steps for fairness and budgeting. All models appear to have converged by this point in both validation and training loss. We will include convergence plots in the final draft.

does not discuss some recent multilingual speech models (SeamlessM4T and MMS)

We reference Seamless in 2.3 and discuss how they differ from Whisper-style models. We also reference MMS at a high-level in 2.3, but will update the draft to make more direct comparisons. WER comparisons with Seamless are shown in the able below.

performance appears relatively uncompetitive compared to Audio-LLMs, such as Qwen2Audio...does not provide a direct comparison with other efficiency-focused models, such as SenseVoice

Below are comparisons with these models for 3 languages using FLEURS and non-FLEURS test sets. Chinese/Japanese use CER, English uses WER. We use the official reported scores when possible, designated by a *, otherwise we run inference ourselves. Since SenseVoice-L is not publicly available, we only include their official results.

OWLS models are competitive with Qwen2Audio: OWLS 18B v2 outperforms QwenAudio2 on 4/6 test sets. We note that we were also unable to reproduce the results in the Qwen2Audio paper, since they did not release their prompts (we obtained a CER of 14.2 on FLEURS zh with Qwen2Audio, while official is 7.5 CER), further emphasizing the importance of the open nature of the OWLS suite. We found that Qwen2Audio was more susceptible to hallucinations compared to the other models, which is in line with previous studies on LLMs for ASR [1].

Questions

1. OWLS is similar to Seamless and SenseVoice-L in terms of algorithmic efficiency, since they're all encoder-decoders.

2. All benchmarks we use are in the manuscript. Code-switching and Quechua are out-of-domain.

3. While it is difficult to compare the training cost to all SOTA models due to the proprietary nature of training details, the below table shows a comparison for the models where the information is available:

Note that training hours are not fully comparable due to differences in GPU and cluster environment. Canary also uses a pre-trained speech encoder to decrease the training time, which is not considered in the final budget. Nevertheless, we believe the OWLS training is well-optimized, as we had a limited compute budget. All of these optimizations will be open-sourced.

[1] Performance evaluation of SLAM-ASR: The Good, the Bad, the Ugly, and the Way Forward. (Kumar et al., ICASSP 2025 SALMA Workshop).

Thank you for your valuable insights.

most of the conclusions are predictable, limiting the overall insight gained from the study

We respectfully clarify that many of our insights are not predictable. While the notion that a larger model leads to better performance is indeed obvious, showing that exact performance improvements w.r.t downstream metrics like WER can be modeled as a scaling law is a novel insight. Many previous scaling law papers model only show scaling laws w.r.t test cross-entropy loss, which is a more predictable but less actionable result for researchers.

concerns about the analysis, as the transcriptions are sourced from YouTube, and their accuracy remains uncertain... Scaling trends observed here might not hold when using large-scale, high-quality transcribed audio data.

All of our evaluation results are performed on standard benchmarks with high-quality transcriptions, such as FLEURS or CoVoST, commonly used in evaluating many large-scale models such as [1,2]. We only use YouTube transcriptions for helping train two models in this paper (OWSL 1B and 18B 360K), which means scaling curves in figures 1-4,6,7, and 9 are not affected.

Questions

1. Most of our training data is high quality [4], as it comes from common academic benchmarks. We only use noisier transcriptions for OWLS 1B 360K and OWLS 18B 360K. The purpose of Figure 2 is to show scaling patterns across all languages, and that it is unfeasible to fit a language-agnostic universal scaling law. This is a reasonable expectation, since some languages are naturally more difficult to model for ASR than others (i.e Spanish vs Chinese) [5], so they will require more training data. We acknowledge that this point was not clearly made in the paper, and will better word it in the future drafts.

2. While we believe that the statement is indeed true for traditional ASR/ST models, we respectfully argue that such intuitions may not hold true for large-scale models like OWLS 18B. For example, [6] found that LLMs only require a small amount of leaked bilingual data to learn text translation. It is thus not completely out of the question that a sufficiently large ASR/ST model may be able to overcome traditional data limitations. Due to the potential openness of this question, we believe that this result is valuable evidence and proof to improve the scientific community’s understanding of large-scale models.

3. A breakdown of all datasets is found in Appendix A/Table 6. The OWSM datasets are high quality [4], as they come from common academic benchmarks, such as LibriSpeech, VoxPopuli, MuST-C, and CoVoST. For YODAS, we obtain a cleaner subset from the dataset authors. They use text language identification on the transcripts (to check that the text/audio language is aligned) and CTC segmentation on the audio with a pre-trained OWSM-CTC [7] model. The worst 10% of the utterances by CTC segmentation score are then discarded. We will make this information more apparent in the main paper body in future drafts.

4. We apologize for the oversight. Canary is 1B and Whisper is 1.5B. We are using Whisper Large V3, and will make it more clear in future drafts.

5. We ran each model in Table 2 on LibriSpeech test-other with beam size=1, and compare the results with that of Table 2 below, showing that WER is indeed sensitive to beam size. We will include this analysis in future drafts.

6. We can assert that the observed trends are consistent across multiple languages, as we perform multilingual evaluations in almost every figure/table.The only exception is beam search (Table 2), which we will rectify by including the below results on FLEURS for Spanish and Turkish. Larger models with smaller beam sizes have better WER than smaller models with larger beam sizes. The only exception is 4B and 9B, where 4B already outperforms 9B with beam size=1.

[1] Robust Speech Recognition via Large-Scale Weak Supervision (Radford et al., ICML 2023).

[2] Gemini: A Family of Highly Capable Multimodal Models (Gemini Team., 2024).

[4] On the effects of heterogeneous data sources on speech-to-text foundation models (Tian et al., Interspeech 2024).

[5] Language Complexity and Speech Recognition Accuracy: Orthographic Complexity Hurts, Phonological Complexity Doesn't. (Taguchi et al., 2024).

[6] Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM’s Translation Capability (Briakou et al., ACL 2023).

[7] OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification (Peng et al., ACL 2024).

Thank your for your insights and comments. We organize our response by section:

Claims And Evidence:

2. We understand the reviewer’s concern. While our definition was indeed focusing on the fact that scaling leads to a better model for many languages, we do acknowledge that this definition may be too narrow. We will instead rephrase the fairness claim to “Scaling to larger models can improve ASR performance across the board, in both low and high resource languages, with significant improvements in the former category.”

3. We have included the accompanying audio for the mondegreen samples in Appendix Table 9 here: https://anonymous.4open.science/w/owls-samples-88B2/ , which we hope better illustrate the phenomena. If there any particular items that can be further clarified, we will be happy to make those changes as well.

4. We have included a few samples on example ICL generations https://anonymous.4open.science/r/owls-samples-C53F/README.md , which should better illustrate the improvements in ASR output as the number of in-context examples increase. While there is not a clear monotonic trend, we note that there are 3 clear clusters in ICL CER that are aligned with model size. 0.25B and 0.5B both have ~33.8 CER, 1B to 4B all have ~31.5 CER, and 9B to 18B have ~27.7 CER.

Experimental Designs Or Analyses:

For the Mondegreen section, I am not particularly sure how PPL/MOS relates to ASR performance, or in other words, how they are useful. Does this phenomenon actually help reduce WER/CER, or is it simply an artifact of large-scale training? If not, why is this ability important enough to be discussed in this paper?

PPL/MOS do not have a relationship with ASR performance, since the only goal of these metrics is to measure the English coherence of the mondegreen generation. We note that this phenomenon likely does not have any impact on overall WER/CER, and is probably indeed an artifact of scaling. However, we believe that it is important to discuss and present this evaluation, because it allows researchers to better understand the properties of neural networks that emerge at scale, and how it relates to the human perceptions of spoken language. By including this evaluation, we can show that there is much we do not understand about ASR models and how they scale, opening up potential new areas of research that were previously not possible due to lack of access to such models.