Scaling Speech-Text Pre-training with Synthetic Interleaved Data

Thank you for your review. We are pleased to hear that you find this work solid and well-motivated. Please let us know if your concerns and questions have been addressed—we would be happy to engage in further discussions!

There is a lack of performance evaluation for the text-to-tokens model. For example, after converting a piece of text into tokens and then decoding it into speech using a vocoder, what is the ASR-WER of the resulting speech? This result is necessary to demonstrate the semantic representation capability of the tokens generated by the text-to-tokens model.

We have added more details about the evaluation of text-to-token model in Section 2.2. Table 2 shows the ASR-WER of the results speech. For the standard VCTK dataset, we observed a lower WER of 3.20. However, the WER for speech spans generated from the text pre-training data was higher. We attribute this discrepancy primarily to the format and content of the text pre-training data, which can include text that is challenging to pronounce accurately.

Based on my experience, tokens generated by text-to-models lacks diversity, and the speech instruction dataset SpeechDialog-90K in the SFT stage is also synthesized by TTS, so I am concerned about whether the model can understand real speech input. I checked the evaluation datasets in this work, all of which were synthesized through TTS, lacking evaluation on real speech input (such as AIRBench [2]).

Thanks for point out this! We explain this in two aspects:

• First, we have conducted additional evaluations on ASR benchmarks, specifically LibriSpeech for English and AISHELL-1 for Chinese, using our base model. The results demonstrate competitive performance, indicating that our architecture is fully capable of understanding real speech input.

  Method	LibriSpeech (test-clean)	LibriSpeech (test-other)	AISHELL-1 (test)
  whipser-large-v3	2.50	4.53	9.31
  9B-base (ours)	2.82	7.66	2.46

• Second, we were unable to find open-source speech dialogue data with real and diverse speech input. Therefore, to demonstrate the model's potential as an end-to-end spoken chatbot, we synthesized speech dialogue data using TTS model. Regarding benchmarks, while AIRBench focuses more on audio understanding tasks (e.g., speaker gender recognition, music classification), we believe it may not be suitable for evaluating a spoken chatbot system. Since the primary focus of this work is on pre-training, we synthesized speech through TTS for evaluation purposes. However, we are committed to exploring new datasets and benchmarks in the future to further evaluate and enhance our model.

The quality of the output speech is not satisfactory, as evidenced by the poor ASR-WER in Table 4. In comparison to llama-omini, which was only trained on 100 hours of speech data, this model was trained on a much larger scale of 700k hours of speech data. The author needs to provide a reasonable explanation for why the ASR-WER is so poor.

We apologies for this. The poor ASR-WER was caused by an implementation mistake in the speech reconstruction process. This issue resulted in some speech tokens being randomly omitted, leading to mispronounced words. It is important to note that this mistake only affected the spoken chatbot results. We have corrected the error and updated Table 4 with the accurate evaluation results. The updated results show that our model achieves a lower WER of 7.83 compared to LLaMA-Omni's 7.95. We believe the WER could be further reduced with the use of a more advanced TTS model for speech dialogue data construction.

Thank the author for the detailed reply.

I have seen the evaluation of the text-to-tokens model, the evaluation on the ASR task, and the corrected results in Table 4 in the new version. I'm glad to see that the final end-to-end model can achieve a relatively low WER on the ASR task. However, it can also be seen that the text-to-tokens model has a relatively poor semantic modeling ability on real text corpora.

Nevertheless, I'm still quite curious about why the author designed the speech tokenizer to be casual. During inference, speech is segmented into chunks of 2 seconds before being fed into the LLM, rather than being in a completely streaming manner.

Despite some limitations of this article, I'm still inclined to accept it.

Thank you for highlighting the weaknesses in our work and providing valuable suggestions to improve our writing and presentation. Based on your feedback, we have updated the paper and made several improvements, including adding more details about our method to enhance clarity and understanding. Specifically, we have added more details about the interleaved data construction process (Cf. Section 2.2) and better established this concept in the abstract and introduction. Additionally, we have included detailed training data statistics (Cf. Appendix A) and provided comprehensive training details for our model (Cf. Appendix B). We hope these changes better clarify our method and address your concerns. If they do, we would greatly appreciate it if you would consider raising your score. Thank you once again for your valuable feedback!

While there is a good ablation analysis, there aren't any explanations for why the architectural / training parameters where chosen as they where. I suggest to add a dedicated subsection or add this into the methodology section where the parameters are introducted.

We have add explantations into method section, including sampling rates (Cf. Section 2.1), span corruption ratio (Cf. Section 2.2) and the amount of synthetic interleaved data tokens (Cf. Section 2.2).

The training datasets are lacking in clarity: For table 1, is it unclear on which dataset it was evaluated and why MOSNet scores are so low. It is unclear what Supervised speech-text data are used to train the model. It is unclear what datasets areused to train the text to speech tokens model. It is unclear what datasets are used to fine-tune the tokenization encoder and decoder. These should be added in the section specifying the training pipeline or in a dedicated table / figure.

We haved add more details about the training dataset in Appendix A. We list the source and during of supervised speech-text data (including ASR and TTS data) in Appendix A.2. We list the dataset use to train text-to-token LM in Appendix A.3. We also add a section about the the training details of the speech tokenizer, speech decoder and speech language model in Appendix B.

The experimental results are lacking in clarity: the origin of the baseline numbers in all tables is lacking, are these from other papers or from independently evaluating? I would suggest adding these directly to the table or in the caption.

We have added the source of baseline numbers to the caption of Table 1, Table 3 and Table 4. Thanks for your suggestion!

In table 3, speechGPT and Spectron are speech to speech methods, while the results are stated in speech to text. in Table 1 MOSNet was used while in table 4 UTMOS is used. This reason for this should be explained in the paper or have uniformity between them.

The reason we choose different metrics is because we follow the settings of previous work for speech reconstruction and spoken chatbot. For the speech reconstruction, we adopt the approach from Moshi [1], utilizing the MOSNet metric. For the spoken chatbot evaluation, we follow the settings from Llama-Omni [2], using UTMOS as the evaluation metric. We haved add explantion to the following section.

[1] Défossez, Alexandre, et al. "Moshi: a speech-text foundation model for real-time dialogue." arXiv preprint arXiv:2410.00037 (2024).

[2] Fang, Qingkai, et al. "Llama-omni: Seamless speech interaction with large language models." arXiv preprint arXiv:2409.06666 (2024).

Human evaluations of speech quality, such as MOS or MUSHRA evaluations where humans will rate the speech quality of the proposed method compared to the baseline.

Thank you for your suggestion. As the focus of this work is on pre-training, we fine-tuned the pre-trained model only on synthesized speech dialogue data, with the output speech generated using an open-source TTS system (MeloTTS in our experiments). This was intended to demonstrate the model's potential as an end-to-end spoken chatbot. Therefore, we did not conduct human evaluations of speech quality. We will consider adding human evaluations in future revisions.

Thank you for your detailed review and for providing many practical suggestions on improving the paper's presentation! We have made several changes to the paper, and we hope these updates better clarify our method and address your concerns. We would also be happy to engage in further discussions!

Currently there's no sample page (unless I missed something). Consider creating a sample page with samples on speech continuation (audio prefix, audio GT continuation, and the model's audio continuation). Also consider adding examples of spoken question answering (audio question, audio GT answer, the model's prediction). Examples from the spoken chatbot evaluation would also be great. Moreover, you can visualize how the interleaved samples sound like (paragraph with audio tokens that were decoded in it).

• In Section 2.1 and Table 1, we provide additional details about the evaluation process, including the dataset name. Specifically, "LS" refers to LibriSpeech. To evaluate semantic retention, we measure the Word Error Rate (WER) using the ASR model provided in Expresso [1]. The speech decoder is trained on a multi-speaker TTS dataset (see Appendix A.2). Since the tokenizer is trained in a supervised manner, it effectively retains semantic content but does not preserve speaker identity.
• In Section 2.2, we reorganized the section and included more details to describe the interleaved data construction process.
• In Section 2.3.2, we added more details about the filtering process, including training hyperparameters.
• In Section 3.1, we expanded on the spoken chatbot evaluation, specifying that we use whisper-large-v3 to transcribe generated speech into text for assessment.
• We have included sample pages for spoken question answering, the spoken chatbot, and speech-text interleaved data. See Appendix E for more details.

[1] Nguyen, Tu Anh, et al. "Expresso: A benchmark and analysis of discrete expressive speech resynthesis." arXiv preprint arXiv:2308.05725 (2023).

what’s the difference between \emptyset and “-“?

We use $\emptyset$ to indicate tasks and modalities not supported by the model, and - to indicate scores that are not publicly available. We have add this explantion to the caption of Table 5.

Regarding audio tokenization - consider reducing the dimension (e.g. from 1024 to 8/16) before quantization to prevent codebook collapse.

We indeed observed codebook collapse in our early experiments, we solve this issue by applying the random restart trick (Cf. Sec 2.1). We will try your solution in our following experiments.

Moreover, I suggest applying text-tokenization algorithms (BPE?) on the speech units, to produce variable-length representations with a more balanced distribution, and further shorten the audio sequence.

We tried applying BPE on the speech units in our early experiments, but the compression ratio was not ideal. We learned a BPE tokenizer with 16k vocabulary on the 25Hz speech tokenizer with 4k vocabulary. The compression ratio (sequence length before BPE vs after BPE) is only 1.23, much smaller than BPE tokenizers for texts. Therefore we choose to reduce the sampling rate of speech tokenizer from 25Hz to 12.5Hz to reduce the sequence length.

Thank you for your review and the positive assessment of our paper. We are pleased that you recognize our contributions to speech language models. If you have any further questions or suggestions, please don't hesitate to let us know!

While the whisper ASR model has achieved excellent performance on a range of tasks, it does have its limitations, especially with regards to unseen or low-resource languages. That is not an issue for this paper which seems to focus on English (although there was quite a bit of Chinese data used as well). Have the authors given any thought as to how to extend this work to cover more languages?

Adding a new language requires specific data for that language. ASR data is needed to train the speech tokenizer, and unsupervised speech data is used for the speech decoder. TTS data is required to train the text-to-token model, while text pre-training data is used to synthesize interleaved speech-text data. Finally, dialogue data is necessary to build a spoken chatbot in the target language.

Are there any plans to open source the tokenizer, text-to-token model, or the speech LM itself?

We plan to release the tokenizer and the speech LM in the camera-ready revision if accpected.

Also, it would be nice if the authors could describe the amount of computation required to pretrain the speech LM.

We pretrain our 9B LLM on a total of 1 trillion tokens, which corresponds to approximately 5.4e22 FLOPs.

Thank you for highlighting the weaknesses in our work and providing valuable suggestions to improve our writing and presentation. Based on your feedback, we have made several improvements and update our paper. Specifically, we have added more details about the interleaved data construction process (Cf. Section 2.2) and better established this concept in the abstract and introduction. Additionally, we have included detailed training data statistics (Cf. Appendix A) and provided comprehensive training details for our model (Cf. Appendix B). We hope these changes address your concerns, and if so, we would be grateful if you would consider raising your score. Thank you again for your valuable feedback!

The paper mentions "we are first to use supervised semantic tokens for SpeechLMs". However, one of the baselines, Mini-Omni also uses a whisper-based speech tokenizer.

Mini-Omni [1] employs different speech representations for input and output. Specifically, it utilizes the encoder from Whisper-small to generate continuous speech embeddings for input, while adopting reconstruction-based discrete tokens from SNAC [2] for output. This inconsistency in speech representation prevents Mini-Omni from performing speech pretraining in large unsupervised speech corpus. In contrast, our approach uses a unified supervised discrete speech token for both input and output, which is better suited for speech pretraining.

[1] Xie, Zhifei, and Changqiao Wu. "Mini-omni: Language models can hear, talk while thinking in streaming." arXiv preprint arXiv:2408.16725 (2024).

[2] Siuzdak, Hubert, Florian Grötschla, and Luca A. Lanzendörfer. "SNAC: Multi-Scale Neural Audio Codec." arXiv preprint arXiv:2410.14411 (2024).

The details on how the speech and text modalities are interleaved are missing. As an important part of the process, the details of the text-to-token model are missing—for example, model architectures, training schemes, etc.

We tokenize both modalities into discrete tokens and speech and text are interleaved at word level. We additionally include samples for interleaved data on Appendix D.3 for better understanding. More details have been provided in Section 2.2 for constructing interleaved data. We also have add the training data statistics (Cf. Appendix A.3) and training details of text-to-token model (Cf. Appendix B.3). Besides, we also include a evaluation of the text-to-token model (Cf. Section 2.2).

The large amounts of speech tokens generated by the text-to-token model are still from existing datasets and speech-synthesized audio from text. How is this process different from generating speech tokens from synthesized speech audio using large amounts of text? For example, llama-omni also uses cosy-voice to synthesize speech audio to augment training data. What's the innovation here between text-to-speech-to-token and text-to-token?

The innovation can be summarize as follows:

• Efficiency: Llama-Omni uses CosyVoice to synthesize speech audio for instruction data, generating only 200K samples. However, synthesizing hundreds of billions of tokens using a text-to-speech-to-token pipeline is highly inefficient. In contrast, we train a text-to-token LLM and leverage the SGLang framework for efficient and scalable synthesis. To demonstrate the efficiency, we conducted an experiment comparing the generation speed of the two methods on an H800 GPU. The results show that our approach is 70x faster.

  Method	Speed (tokens/s/GPU)
  CosyVoice (text-to-speech-to-token)	360
  Ours (text-to-token)	25000

• Simplicity: CosyVoice uses a text-to-token language model to generate speech tokens from text and a speech decoder to reconstruct speech from these tokens. By directly using a text-to-token LLM to generate speech tokens, we simplify the pipeline, improve efficiency, and avoid accumulated errors introduced compared to the text-to-speech-to-token pipeline.

We greatly appreciate the time and effort you have dedicated to reviewing our work. We hope that all the points raised in your feedback have been adequately addressed. If there are any remaining concerns or areas requiring further clarification, please do not hesitate to let us know—we would be happy to provide additional details or explanations.