Scaling Analysis of Interleaved Speech-Text Language Models

We would like to thank the reviewer for reviewing our manuscript and providing meaningful feedback. We are happy the reviewer finds our scaling analysis and findings useful to the community.. We are also pleased the reviewer finds our paper to presents comprehensive experiments to justify the analysis setting and conduct the scaling analysis.

Regarding (1) placing the figure in the introduction, and (2) optimistic trends on the sub-figures: (1) We wanted to give a quick overview to those familiar with the domain but will reconsider this per your suggestion. (2) We should clarify - results for interleaved SLM are better than speech-only consistently per compute-budget. Furthermore, for sSC the slope of the trend is larger thus “more optimistic trend”. For tSC the results are near the saturation limit (which is probably less than the theoretical 100, and humans and textLMs don’t get perfect scores), thus making the slope harder to address explicitly. We will clarify in the final paper.

Regarding the usage of synthetic data, sTineStories: We can see why the phrasing could be confusing, we use a diverse set of real datasets (as specified in Appendix A.1) and the synthetic dataset (See Table 1). We wanted to highlight the use of synthetic data as in the context of scaling and data requirements - synthetic data is often easier to collect than real data. We used specifically this synthetic dataset for consistency with prior work [3]. We will clarify for the camera ready.

Regarding cross-model evaluation: This follows the existing evaluation protocol of [1, 2, 4]. Essentially, this is much like the uni-modal story cloze where we wish to see whether the “real” continuation is assigned a higher likelihood by the SLM, compared to a distractor. However, for the cross-modal setting the prompt is a specific modality (for instance text), and the continuation is a different modality (speech). In other words, given a specific text prompt the SLM needs to select the more likely speech continuation. We will clarify for the final manuscript.

Regarding L174, real and synthetic data in SLM training: Yes, your correction is in place. We mean that it reduces the real data needed which is often harder to obtain (and standardise, and clean), compared with synthetic data which is becoming progressively easier to create at scale with high quality and decent controllability.

Regarding Figure3: The trend the interleaved SLMs outperform Speech-only SLMs, for good model families, was prevalent in many preliminary studies as well Figure 3. In Figure 5, we see that for a weaker base model - OPT125, the speech-only model starts with better performance but the interleaved model closes the gap and even overtakes it on some of the metrics. This again shows the interleaved models improve more with larger compute-budget. Figure 5, also shows another example where the TWIST initialisation starts with a larger gap for small steps, but this gap closes with more steps.

Regarding what makes SIMS better than other joint speech-text SLMs: Great question! It is somewhat hard to pin-point specific reasons, as many aspects change between models (the approach for joining speech and text, the training data, base model, hyper-parameters). We suspect that many aspects of the training recipe itself - base model, data mixture, optimisation details give some benefit, see Figure 2 and also [5]. However, it is important to clarify that we do not claim that SIMS is better than these models, but rather comparable to them. Furthermore, we would like to highlight that we do not focus on creating a state-of-the-art model, but rather to conduct scaling analysis of interleaved SLMs. The fact that most models are a similar size, but notably more tokens, could also suggest that the impact of more tokens decays faster as we hypothesised.

---

[1] Nguyen, Tu Anh, et al. "Spirit-lm: Interleaved spoken and written language model." Transactions of the Association for Computational Linguistics 13 (2025): 30-52.

[2] Zeng, Aohan, et al. "Scaling speech-text pre-training with synthetic interleaved data." arXiv preprint arXiv:2411.17607 (2024).

[3] Cuervo, Santiago, and Ricard Marxer. "Scaling properties of speech language models." arXiv preprint arXiv:2404.00685 (2024).

[4] Cuervo, Santiago, et al. "Text-Speech Language Models with Improved Cross-Modal Transfer by Aligning Abstraction Levels." arXiv preprint arXiv:2503.06211 (2025).

[5] Maimon, Gallil, Avishai Elmakies, and Yossi Adi. "Slamming: Training a speech language model on one gpu in a day." arXiv preprint arXiv:2502.15814 (2025).

We would like to thank the reviewer for reviewing our manuscript and providing valuable feedback. We are happy the reviewer finds our paper thoroughly examines the landscapes of Interleaved SpeechLMs and provides useful observations for building a scalable model. We are also pleased the reviewer finds the empirical results provide several key insights for building a scalable SpeechLLM efficiently.

Regarding small subcommunity: Getting SpeechLMs to listen and speak is gaining popularity across both academic research and industry [9]. Indeed, there are many studies on the best ways to integrate speech and text LMs [1, 5, 9], with many specifically focusing on the interleaving approach [1, 2, 9, 10]. Understanding the scaling dynamics of such approaches helps understand the real world applicability compared to other methods.

Regarding mathematical formulations of scaling laws for text initialized SLMs: Indeed conducting scaling analysis when limited to existing pre-trained text model sizes poses a challenge, as we mention in Section 1 (line 47). Most models do not come in very diverse (small) sizes, especially when considering only single, high quality model families. The best, but most unfeasible, way to do this is to train from scratch the base textLMs across diverse model sizes, but this is very costly and out of scope. Another option would be to try to consider different model families which are somewhat similar, for instance use Qwen3 as well as Qwen2.5 which could have slightly different sizes. Our analysis shows a large impact to model family making this possibly impractical. We also note that the mathematical formula not fitting perfectly, is not necessarily a lacking data issue but perhaps a real formulation change. As this scaling analysis is more like adding a new language (and tokens), the scaling properties could be notably different! For instance, the impact of tokens might not decay exponentially because of the impact of the initialisation (further encouraging less steps to prohibit catastrophic forgetting).

Regarding the ratio of mixing speech and text: The ideal ratios between speech and text in training are indeed an interesting question. We follow [1, 2] in using equal ratios of speech, text, and interleaving tokens. We focus on the total number of tokens out of a given compute-budget. While interesting, running a proper evaluation of ratios jointly with scaling analysis requires a large compute budget, and we leave that for future work.

---

[1] Nguyen, Tu Anh, et al. "Spirit-lm: Interleaved spoken and written language model." Transactions of the Association for Computational Linguistics 13 (2025): 30-52.

[2] Zeng, Aohan, et al. "Scaling speech-text pre-training with synthetic interleaved data." arXiv preprint arXiv:2411.17607 (2024).

[3] Lin, Guan-Ting, et al. "Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback." arXiv preprint arXiv:2411.01834 (2024).

[4] Cuervo, Santiago, and Ricard Marxer. "Scaling properties of speech language models." arXiv preprint arXiv:2404.00685 (2024).

[5] Défossez, Alexandre, et al. "Moshi: a speech-text foundation model for real-time dialogue." arXiv preprint arXiv:2410.00037 (2024).

[6] Guo, Yiwei, et al. "Recent Advances in Discrete Speech Tokens: A Review." arXiv preprint arXiv:2502.06490 (2025).

[7] Hassid, Michael, et al. "Textually pretrained speech language models." Advances in Neural Information Processing Systems 36 (2023): 63483-63501.

[8] Lakhotia, Kushal, et al. "On generative spoken language modeling from raw audio." Transactions of the Association for Computational Linguistics 9 (2021): 1336-1354.

[9] Arora, Siddhant, et al. "On the landscape of spoken language models: A comprehensive survey." arXiv preprint arXiv:2504.08528 (2025).

[10] Cuervo, Santiago, et al. "Text-Speech Language Models with Improved Cross-Modal Transfer by Aligning Abstraction Levels." arXiv preprint arXiv:2503.06211 (2025).

We would like to thank the reviewer for taking the time to review our paper. We are happy the reviewer found our paper well-written and easy to understand, technically sound, and with useful advice on scaling LMs.

Regarding the impact of choice of acoustic units: This can definitely impact the performance, but most scaling laws papers do in fact on a single tokeniser [4] as the experiments are very costly and there are so many different tokenisers used [7]. We focus on one of the most commonly used tokenizer [1, 3, 8, 9] and more importantly the same one as used for existing SpeechLM scaling laws [4] allowing a direct comparison.

Regarding using a single speaker TTS for synthetic data: We should clarify - indeed using diverse TTS could improve performance or robustness even further. We follow existing works in using a single speaker TTS [4] for consistency purposes. The main point was that much of the data can be synthetic leading to less harsh requirements on real data. The exact best characteristics of this synthetic data (e.g diversity) is left for future work.

Regarding 4 speakers may not be sufficient to test speaker independence: The four speakers include British Male, British Female, American Male and American Female thus fairly diverse across gender and accent. While this of course does not capture the full diversity of human speech, it does go beyond most common approaches which use a single speaker [1, 2, 3, 4, 8]. Per your suggestion we report here the specific results per-speaker and show that while there are gaps between speakers, the performance does not change drastically. This is likely because HuBERT also removes much of the speaker's information. Additionally, per-your suggestion we randomly sample 4 more speakers and report the results here (left 4 are the original, right 4 are new voices).

Regarding conclusion from Line 234: The benefit to performance of increasing $D$ - the number of training tokens (and subsequent number of steps) decays faster when there is TWIST init vs without. This can be seen by the gap closing between models with it and without, as training tokens increase. This was consistent across many preliminary studies and results in Figure 3, and can also make intuitive sense as when D->\inf it is likely that the importance of the initialisation decays. If indeed the impact of D decays faster, so for a given compute budget the relative compute for parameters should be larger. We will clarify this in the final version. This is also evident from Figure 4, that training models with more parameters and less tokens leads to better performance, compared to the speech-only SLM scaling laws.

Regarding additional evaluations: We follow the standard evaluation methods used in speech-text interleaving models [1, 2, 3], which include both perplexity based, modelling metrics, and generative metrics. These are also in line with existing SpeechLM scaling laws which we believe is important for comparability of results [4]. We would like to highlight that the SpeechLMs evaluated and analysed in this paper are base models (trained for speech/text continuation and speech-text interleaving), and not Speech-Aware LMs [5] or Instruction tuned SpeechLMs [6]. Hence these models are less suitable for instruct-based evaluation tasks (ASR, TTS, SpokenQA, etc.), as they were not trained for such tasks.

We would like to thank the reviewer for taking the time to review our paper. We are happy the reviewer found our paper investigating an important in the field, contianes well-supported empirical evidence, and of high-quality presentation.

Regarding additional evaluations: We follow the standard evaluation methods used in speech-text interleaving models [1, 2, 3], which include both perplexity based, modelling metrics, and generative metrics. These are also in line with existing SpeechLM scaling laws which we believe is important for comparability of results [4]. We would like to highlight that the SpeechLMs evaluated and analysed in this paper are base models (trained for speech/text continuation and speech-text interleaving), and not Speech-Aware LMs [5] or Instruction tuned SpeechLMs [6]. Hence these models are less suitable for instruct-based evaluation tasks (ASR, TTS, SpokenQA, etc.), as they were not trained for such tasks.

---

[1] Nguyen, Tu Anh, et al. "Spirit-lm: Interleaved spoken and written language model." Transactions of the Association for Computational Linguistics 13 (2025): 30-52.

[2] Zeng, Aohan, et al. "Scaling speech-text pre-training with synthetic interleaved data." arXiv preprint arXiv:2411.17607 (2024).

[3] Lin, Guan-Ting, et al. "Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback." arXiv preprint arXiv:2411.01834 (2024).

[4] Cuervo, Santiago, and Ricard Marxer. "Scaling properties of speech language models." arXiv preprint arXiv:2404.00685 (2024).

[5] Tang, Changli, et al. "Salmonn: Towards generic hearing abilities for large language models." arXiv preprint arXiv:2310.13289 (2023).

[6] Défossez, Alexandre, et al. "Moshi: a speech-text foundation model for real-time dialogue." arXiv preprint arXiv:2410.00037 (2024).