UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation

The claim their work is "the first unified pre-training framework for speech representation learning and generation" is arguable. For example, the paper "Speechgptgen: Scaling chain-of-information speech generation." introduced an approach to train a discrete representation that works for both understanding and generation. The paper "Moshi: a speech-text foundation model for real-time dialogue" further improved the approach.

We thank the reviewer for raising the question on our claim. We would like to clarify that both SpeechGPTGEN and Moshi are multi-modal (speech-text) pre-trained model, whereas UniWav focused on single-modal pre-training. Multi-modal models like SpeechGPTGEN and Moshi rely on single-modal pre-trained backbones (e.g., SpeechTokenizer in SpeechGPTGEN and Mimi codec in Moshi both relied on speech self-supervised model), and our work can be viewed as the upstream pre-training to serve for multi-modal models.

---

the DinoSR part in Figure 1 is confusing since the softmax layer is not clearly shown since the encoder, according to the text description, does not include that layer. In contrast, the figure in the original DinoSR paper is very clear.

Figure 1 did not include details like the softmax layer for simplicity, we are sorry if it is causing confusion. The prediction head and softmax are listed explicitly in line 126, which also appears in the overall objective of encoder (Eq.4).

The majority of Section 2.1 is describing DinoSR. However, notation is not clearly explained sometimes. for example, what is "k" in s^k_v right below eq. 3? it's unclear why Euclidean distance (instead of COS distance) is used given that Cos distance is usually more preferred in higher-dim spaces.

The “k” was a legacy typo from earlier version of the draft, we have removed it. We thank the reviewer for pointing it out. For measuring distance in high-dim space, we believe Euclidean distance is equally common for clustering (e.g., kmeans), assuming there is no special normalization before quantization.

the footnote under Table 1 is not clear. alignment is only used for generation tasks?

We thank the reviewer for raising the question, we have add more words to clarify it.

eq 11 is wrong? argmin returns an index instead of a representation.

Although argmin in python returns index, in math notation Argmin is formally defined as the function that returns the element of a set which provides the minimum value of some criterion (https://en.wikipedia.org/wiki/Arg_max). In short, argmin in our case returns the centroid, which is a vector, that is closest to the input embedding.

---

When researchers train a representation model they usually keep the encoder fixed when using them in downstream tasks such as ASR. however, in this work the encoder is finetuned. This causes the claims weaker. Similarlly if flow matching is used in the generation part the quality of the generated speech will of course become better. However, this gain comes from the flow matching model instead of the way the representation is learned. Some clarification here is needed.

We thank the reviewer for the helpful comments, we have added experiments where speech representation is kept frozen (i.e., the encoder is not finetuned) using the existing speech representation benchmark to Section A1.

---

In section 3.1, "We extract the prequantized latent feature from a 16kHz EnCodec (D´efossez et al., 2022) encoder, pre-trained on the same dataset, to serve as the input for our model." This is contradictory to the claim in early sections that the input to your model is mel spectrum? In addition, this makes the comparison with other approaches unfair since you used another model as the pre-processing module (i.e., total model size is actually increased significantly).

In line 108, we use mel spectrogram as an example for surface feature. In practice, our method works with either Encodec latent or mel spectrogram and encodec latent. As shown in Appedix A.1, the choice of input feature (spectrogram or Encodec feature) have little to no impact on the behavior of the model on ASR/TTS. We are sorry for the confusion, and have edited line 108 accordingly.

For the discriminative task of speech recognition, the authors fine-tune the encoder on 100-hour and 960-hour splits, showing competitive results compared to other methods. However, they do not provide results for low-resource scenarios (e.g., 10 hours or 1 hour of data) or results using a frozen encoder (SUPERB benchmark). These two setups are essential to support their claim that the learned encoder representations are effective for speech recognition tasks. Because with large finetuning data, effect of pre-training is less.

We thank the reviewer for the helpful comments, we have added experiment using SUPERB to Section A1. Due to time constraint, we selected phone recognition and speaker identification for evaluating speech representation. In short, results are consistent with our findings in speech recognition and layer-wise analysis.

---

Regarding speech tokenization, it’s unclear if the comparison is entirely fair. For speech tokenization, the input to the SpeechTokenizer is raw audio, while for UniWav, the input is Encodec encoder features, which are further transformed by UniWav’s 10-layer encoder. It is as if UniWav is heavily overparameterized and most of the parameters are used for generation task.

We thank the reviewer for raising the concern and would like to clarify the use of parameters. While UniWav builds on top of a pre-trained Encodec model, SpeechTokenizer also distills from a pre-trained WavLM. So from a high-level perspective, both models required a CNN-based encoder plus a deep transformer to effectively learn discrete tokens.

It is also worth mentioning that results in Appedix A.1 have suggested that the choice of input feature (spectrogram or Encodec feature) have little to no impact on the behavior of the model on ASR/TTS. Based on the results, the # of parameter used in total can likely be further reduced by switching to spectrogram.

---

On line 108, the authors claim that the model uses mel spectrograms as input, but on lines 239–240, they mention using Encodec features instead.

As shown in Appedix A.1, the choice of input feature (spectrogram or Encodec feature) have little to no impact on the behavior of the model on ASR/TTS. We are sorry for the confusion, and have edited line 108 accordingly.

---

We hope the responses and updates on the paper have addressed the concerns raised by the reviewer.

domain

This works only evaluates on Librispeech, i.e., the audio book domain, which has very homogenetic speaker conditions. The observation that ASR performance closely matches HuBERT, WavLM, etc, does not take into account the robustness these models have to other domains. The authors could discuss this limitation or add experiments with the SUPERB benchmark.

We thank the reviewer for the helpful comments, we have added more discussion on the limitation in Section 5. We have also added experiment using SUPERB to Section A1. Due to time constraints, we selected phone recognition and speaker identification for evaluating speech representation. In short, the results are consistent with our findings in speech recognition and layer-wise analysis.

---

modelling

As the proposed method is an extension of DinoSR, it would be nice to see UniWav w/o decoder, as in Table 3 for 200k steps, in Table 1 with 600k steps of pre-training. Moreover, Speechflow also uses an encoder-decoder model. This encoder could in theory be fine-tuned for ASR. The original work did not do this. Can the authors comment on expected results when the SpeechFlow encoder would be fine-tuned for ASR, and how this would compare to UniWav? Could this be discussed in a future work section?

For UniWav without decoder, we did not train it for full 600k updates, but instead trained more different combinations of encoder/decoder sizes to accommodate our compute budget, we hope the review can understand our rationale. We would also like to note that UniWav without decoder is still (slightly) different from DinoSR; the former is trained on Encodec feature and the latter relies on deep convolution that is trained from scratch jointly with the representation model.

For fine-tuning SpeechFlow as ASR, we would first like to note that fine-tuning SpeechFlow as ASR can be less intuitive, for example: the input of the model, noise-infused speech and the time embedding specifically, is conditioned on the flow matching time step t. For discriminative tasks, it is unclear what is the meaning of sampling t.

Nevertheless, we tried fine-tuning as the reviewer suggested. This is done by fixing t at zero. Using 100hr to fine-tune the model (as in Table 1), we observed that the word error rate failed to reach below 40%. Although there might be a better way (e.g., fix to a different t), our preliminary result suggested that generative-only pre-training is not useful for speech

---

run superb speaker recognition

(done, see first section of response)

---

Questions

Based on Figure 2, have the authors considered only conditioning the decoder on the first, e.g., 12 layers of the encoder?

We haven’t tried it, but this idea is interesting and worth exploring in the future.

Can the authors clarify how speaker similarity is computed? Is it the average cosine similarity between the WavLM speaker embedding of pairs of ground-truth and synthesized utterances? If so, I think it would make sense to share the standard deviation as well.

Speaker similarity is computed through the cosine distance between the WavLM speaker embedding of ground truth and synthesized utterances. The variance of this score should be stable across different methods in general, we, therefore, omit it to save space in the table following the convention [A,B,C]. For your reference, the std of the score is 0.119/0.118 for SpeechFlow/Uniwav respectively.

[A] https://arxiv.org/abs/2306.15687

[B] https://arxiv.org/abs/2301.02111

[C] https://arxiv.org/abs/2310.16338

Can the authors comment on their batch size and how they are sampled (e.g., like wav2vec 2.0)? This would help with reproducibility, and figuring out how much data is seen throughout pre-training, and where UniWav lies in Figure 2 of the DinoSR paper (trade-off between performance and data efficiency)

We are sorry for missing the detail, the batch size is 312.5 seconds (5M frames at 16kHz) per GPU, where audio files are randomly sampled and randomly cropped into chunks no longer than 20 seconds. We have updated Section 3.1 to include the detail.

Line 233, "sinusoidal pos. enc. is appended to the input." Is this appended to the feature dimension or the time dimension? Isn't it normally the case that positional embeddings are summed with the input features? Can the authors comments on this design decision?

Sinusoidal embedding is used to encode the time in ODE function (i.e., t in Eq.5), different from the common positional embedding used in transformers. It is treated as a single vector that is prepended to the input sequence of decoder, following the design of Voicebox and SpeechFlow. We are sorry for the confusion, we have make this more clear in Section 3.1.

---

We would like to express our appreciation for the careful review and helpful comments, we have edited the submission to address issues and concerns raised in the review accordingly.

Lastly, the metrics used for evaluating downstream tasks are insufficient. For example, in the speech generation task, subjective evaluations such as Mean Opinion Score (MOS) should be included, as this is a critical metric. For speech tokenization, additional metrics related to speech reconstruction, such as Mel/STFT distance, could be incorporated.

We thank the review for these helpful suggestion, we have updated the paper accordingly:

• For TTS, we have added subjective evaluation results to Section A.1 and Table 6. Result confirmed that our model is indeed matching the performance of SpeechFlow, which is a generative-only pre-training method.
• For reconstruction, we have added ViSQOL to Table 2. ViSQOL is an objective and intrusive metric that measures audio reconstruction quality, similar to Mel/STFT distance but more widely adopted by codec models (including Encodec and DAC, which were added to Table2 as suggested by the reviewer).

---

We hope the responses and updates on the paper have addressed the concerns raised by the reviewer.

Reference

[A] https://arxiv.org/abs/2211.09117

Firstly, I have some doubts regarding the motivation and novelty validation in this paper. ...

As clearly stated in the limitations, we acknowledge the fact that performance drop in recognition tasks compared to existing self-supervised learning models at similar scales is not ideal. Nevertheless, we believe this work is still valuable for different reasons:

• Unified pre-training has shown great value in other application fields but has not yet been explored in speech. For example:

  • In NLP, next-token prediction has become the predominant pre-training method. Back then there were different pre-training approaches for different purposes (such as contrastive learning for retrieval, masked language modeling for representation learning, etc.), much like the current research landscape of speech. The unified pre-training method has significantly boosted the progress of NLP in both understanding and generation tasks.
  • In image processing, joint representation learning and generation have also been studied more recently [A]. Similar to our work, [A] found a trade-off between image classification accuracy and generation quality. However, their effort is still recognized and many later works followed the spirit to push the boundary of unified pre-training.

• Unified pre-training also has its own advantage

  • This is related to the question raised by the reviewer:

  If these advantages cannot be clearly demonstrated, then why combine the two tasks?

  In certain tasks, having both good representation and generation are equally important. Our experiment on speech tokenization and resynthesis is an example. We show a clear gap between our method and other methods, where speech representation and generation are optimized separately.

To summarize, we believe these are strong evidence to support the importance of the research direction this work is pursuing. Once again, we acknowledge the fact that this model is not state-of-the-art when competing against specialized models. Instead of chasing state-of-the-art through pre-training for either discriminative or generative tasks, we contribute to the field by developing a different way of pre-training that is more general. Therefore, we argued that our work, as the first step towards unifying pre-training for speech representation learning and generation, is well-motivated and novel.

---

Secondly, the selection of baselines in the paper is quite limited, especially for the speech generation and tokenization tasks. Speech generation could be compared with Natural Speech 3 [1]

We thank the reviewer for suggesting more baseline models to compare against.

For speech generation, we would like to point out that all state-of-the-art models are not open-sourced to the best of our knowledge, and we therefore needed to reproduce one baseline ourselves. We choose to reproduce SpeechFlow (instead of Natural Speech 3) for several reasons:

• it also falls in the pretrain-and-finetune paradigm, using the same labeled and unlabeled datasets (that are both publicly available) as our work; Natural Speech 3 requires 60k hours of labeled data where the annotations are not publicly available.
• Architecture-wise it is more comparable to the proposed work, whereas Natural Speech 3 has a 4-stage diffusion design with disentangled input feature
• Following SpeechFlow’s evaluation protocol (continuing zero-shot TTS) allows us to compare directly with VALL-E and Voicebox without the need for model weights.

the speech tokenization task could benefit from comparisons with the latest models like Funcodec [2], Encodec [3], and DAC [4].

For speech tokenization, we targeted SpeechTokenizer as it was the most recent work in this direction at the time. We have added the earlier works suggested by the reviewer to Table 2, including Encodec and DAC. We would like to note that, unlike the original models studied in Table 2, the models suggested by the reviewer are NOT developed for speech tokenization but for general audio compressing. Therefore they inevitably performed poorly on low-bit-rate speech-only tokenization, especially on encoding spoken content (see WER in Table 2).

Furthermore, the paper should include comparisons with the experimental results of DinoSR for both the speech recognition and tokenization tasks, as the encoder component of the paper is primarily based on the DinoSR model.

For DinoSR, the original work focused on a 12-layer model only and did not provide a recipe for training a 24-layer model. We also note that we failed to train a 24-layer model DinoSR with the official code, bottlenecked by training stability. (we did not extensively sweep the hyper-parameters due to resource restriction) Nevertheless, a comparison between DinoSR and UniWav can be found in Table 3 by comparing the fine-tuned WER with and without decoder during pre-training. Note that DinoSR is modified here to taking Encodec feature as input.

---

(To be continued)

The results of the method in this paper do not show an advantage over existing methods in speech recognition and speech generation.

As clearly stated in the limitations, we acknowledge the fact that performance drop in recognition tasks compared to existing self-supervised learning models at similar scales is not ideal. Nevertheless, we believe this work is still valuable for different reasons:

• Unified pre-training has shown great value in other application fields but have not yet been explored in speech. For example:
  • In NLP, next-token prediction has become the predominant pre-training method. Back then there were different pre-training approaches for different purposes (such as contrastive learning for retrieval, masked language modeling for representation learning, etc.), much like the current research landscape of speech. The unified pre-training method has significantly boosted the progress of NLP in both understanding and generation tasks.
  • In image processing, joint representation learning and generation have also been studied more recently [A]. Similar to our work, [A] found a trade-off between image classification accuracy and generation quality. However, their effort is still recognized and many later works followed the spirit to push the boundary of unified pre-training.
• Unified pre-training also has its own advantage.
  • Having both good representation and generation are equally important in certain tasks.
  • Our experiment on speech tokenization and resynthesis is an example. We show a clear gap between our method and other methods, where speech representation and generation are optimized separately.

To summarize, we believe these are strong evidence to support the importance of the research direction this work is pursuing. Once again, we acknowledge the fact that this model is not state-of-the-art when competing against specialized models. Instead of chasing state-of-the-art through pre-training for either discriminative or generative tasks, we contribute to the field by developing a different way of pre-training that is more general. Therefore, we argued that our work, as the first step towards unifying pre-training for speech representation learning and generation, is well-motivated and novel.

---

In the speech tokenization section, there is a lack of experiments related to the modeling performance of the speech language model (LLM-TTS) as mentioned in SpeechTokenizer[1]. Such experiments could effectively evaluate the potential of the tokenizer proposed in this paper when applied to autoregressive speech synthesis. [1]

We thank the reviewer for the helpful comment, and we will add this to our future work. We also note that it is hard to provide an apple-to-apple comparison between different codec models in training token-based auto-regressive TTS, since there are different in many aspects:

• Different bitrate results in different TTS configuration. E.g., [1] required modeling 400 tokens per second, whereas our method required 50 or 100 tokens per second.
• For RVQ-based codec TTS such as [1,B,C], training 2 models (typically an auto-regressive model for semantic codes and a non-auto-regressive model for remaining codes) for text-to-speech is often necessary to deal with multiple codebook. In this case these TTS system are not fully auto-regressive.

---

Can the method proposed in this paper be applied to general audio such as sound and music?

We believe the generation part of UniWav can be easily applied to sound and music, as flow matching have been explored in the [D]. The representation learning part should also be applicable to sound and music. However, the input feature choice and the masking strategy might need to be revisited to adapt to those different domains [E]. We thank the reviewer for pointing out an important future direction, we have updated Section 5 to include the idea.

---

Reference

[A] https://arxiv.org/abs/2211.09117

[B] https://arxiv.org/abs/2410.00037

[C] https://arxiv.org/abs/2301.02111

[D] https://arxiv.org/abs/2312.15821

[E] https://arxiv.org/abs/2110.09784

I believe that the joint pretraining of audio understanding and generation is a promising and exciting research direction. I appreciate the authors' attempts in this area. However, my major concern is that the pretraining framework proposed by the authors does not show significant advancements over previous methods (like TTS, ASR, and tokenizers). Therefore, I have doubts about whether the framework is an effective representation learning method.

• ASR: For the ASR task, the proposed method does not show advantages over Data2Vec 2 and even HuBERT.

• TTS: Regarding the TTS task itself, the current state-of-the-art methods are NaturalSpeech2/MASKGCT and CosyVoice/FishSpeech (a non-autoregressive and an autoregressive model). The paper does not include these baselines for comparison.

• TTS Tokenizer: For the tokenizer, the current state-of-the-art open-source tokenizers for TTS are CosyVoice and DAC. The authors also do not include these as baselines. Moreover, for the TTS tokenizer, the frame rate (how many tokens per second) is more important than the bitrate for LLM-based CodeTTS models, which could be included in Table 2 for comparison. Additionally, recent low-bitrate codec models such as WavTokenizer and Mini are also not included in the paper for comparison.

Indeed, I want to emphasize that I do not believe a unified model must achieve optimal results across all aspects, especially when the scale is not significantly increased. It would be more beneficial to demonstrate some potential synergies. For instance, combined training for understanding and generation could significantly assist each other, or the trained tokenizer might possess properties that other training methods in semantic and acoustic processing do not have. However, based on the scores presented in the paper, the performance across various metrics appears to be somewhat unsatisfactory.