RepCodec: A Speech Representation Codec for Speech Tokenization

Thanks for the valuable suggestions from the reviewer. We respond to your questions as follows.

• “Meaning of semantic quality”

  We believe we do not directly mention "semantic quality". Instead, we mention the “quality of semantic tokens”, which refers to suitability of the semantic tokens for downstream tasks like ASR or TTS. When integrating LLMs with semantic tokens, we need to discretize speech into tokens to perform different downstream tasks. Some discrete tokens may not be suitable for downstream tasks and leads to inferior downstream task performance.

• “Speaker information of RepCodec”

  We agree with the reviewer that the semantic units should carry as much semantic information as possible and as less speaker information as possible. Otherwise they may affect the speaker effect brought by other components, e.g. speaker embedding, acoustic tokens. We verify the speaker information carried by our units and the baseline k-means units through the voice conversion task:

  Methods	Speaker Similarity ↑	MOS↑
  k-means(baseline)	0.771	2.82±0.36
  RepCodec	0.764	3.48±0.74

  In our voice conversion experiment, the speaker characteristics of the generated speech is controlled by the speaker embedding. However, the speaker information carried by the semantic units (if any) may also affect the result. In the above table, we measure the similarity of the voice converted speech and the original speech from the same speaker. A higher similarity score implies less speaker information carried by the units. It can be seen that the similarity scores are very close but RepCodec units have a much higher MOS score due to the improved semantic part. This means that RepCodec does not carry extra speaker information which is harmful to the task.

• “Limitation of k-means”

  The limitation of k-means is reflected in the degraded performance of the ASR modeling. Even if we train the seq-to-seq model with large amounts of data, using k-means tokens as input always results in much worse WER. The explanation to this empirical observation is that k-means discards some information of speech and makes the speech recognition of some similar words impossible so that the WER is inferior to using raw speech as input.

• “Referred number of 2.5%”

  We refer to Table II of the AudioLM paper, where WER of the original speech is 2.5%. However, the reconstructed speech using k-means has a much worse WER of 6.0%.

• Equation (5)”

  Thanks for your advice. We will add explanations to symbols $\tilde{n}_k$ and $e_k$ in our revision. They represent the moving average of EMA. $\tilde{n}_k$ represents the moving average of the number of clusters $k$ and $e_k$ represents the moving average of the code book of cluster $k$.

• “What kind of representations are suitable for clustering?”

  From our experiments, we believe that encoders trained with SSL objective function may be more suitable for clustering. Whisper is trained by end-to-end encoder-decoder architecture so that its representation from the encoder may require complicated transformation to be decoded as text. Therefore, its representation may be less suitable for clustering.

• “Examples of speech resynthesis”

  We have uploaded some resynthesized speech to supplementary material.

Thanks for the feedback from the reviewer. We have the following response to your comments.

• “Novelty of the paper”

  We believe our paper proposes completely new methods to generate semantic units and provides new understanding of the quality of the semantic units. Before RepCodec, semantic units can only be produced by traditional k-means methods following HuBERT [1], which was proposed about two years ago. Our paper, despite technical similarity to SoundStream, provides a novel method that largely outperforms long existing k-means clustering.

  In addition, we also provide a new understanding about the semantic units. In previous papers such as HuBERT [1] and AudioLM [2], similarity to phonemes is used to measure the quality of the semantic units. Our paper, however, shows that PNMI is not the only metric for evaluating the semantic tokens. Instead, the reconstruction loss provides more meaningful indication for the performance of downstream tasks. It brings new insight for future study of semantic tokens.

• “Ablation Study of $(\lambda_q, \lambda_r)$”

  As shown in the following Table, we change the weights of reconstruction loss $\lambda_r$ to train the encoder of RepCodec and report its WER on the downstream task of ASR modeling with representation from the 18th layer of data2vec large model. The results show that RepCodec is robust to the changes of the weights of $(\lambda_q, \lambda_r)$.

  $\lambda_q$	$\lambda_r$	PNMI	Reconstruction Loss	WER of ASR Modeling
  1.0	30.0	0.3689	0.2091	2.91
  1.0	45.0	0.3714	0.2100	2.87
  1.0	60.0	0.3697	0.2090	2.83

• “Confusing details of the paper: Figure 1”

  Our method is compatible with different architectures of the encoder and decoder in Figure 1. To allow variants of the network architectures, we show in Figure 1 that dimension reduction can be applied. We will add more detailed illustrations in Figure 1.

• “Confusing details of the paper: Equation (5)”

  Thanks for your advice. We will add explanations to symbols $\tilde{n}_k$ and $e_k$ in our revision. They represent the moving average of EMA. $\tilde{n}_k$ represents the moving average of the number of clusters $k$ and $e_k$ represents the moving average of the code book of cluster $k$.

• “Confusing details of the paper: Equation (7)”

  Thanks for pointing out our typo. We will fix it in our revision.

• “Using WER to evaluate speech resynthesis”

  We follow previous papers and use WER to evaluate the quality of semantic units such as mHubert [3] and AudioLM [2]. As the role of semantic units is to provide the semantic information of the speech, the best way to evaluate the semantic information is to measure the content, i.e. ASR-WER, of the output speech.

[1] Hsu, Wei-Ning, et al. "Hubert: Self-supervised speech representation learning by masked prediction of hidden units." IEEE/ACM Transactions on Audio, Speech, and Language Processing 29 (2021): 3451-3460.

[2] Borsos, Zalán, et al. "Audiolm: a language modeling approach to audio generation." IEEE/ACM Transactions on Audio, Speech, and Language Processing (2023).

[3] Lee, Ann, et al. "Textless Speech-to-Speech Translation on Real Data." In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2022).

We sincerely appreciate your positive review and the valuable feedback you have provided. We hope our response completely addresses any concerns you may have.

• “Zero-Shot TTS such as VALL-E”

  We agree with the reviewer that applying semantic units of RepCodec to zero-shot TTS is an interesting experiment. However, VALL-E does not use semantic units as the inputs for semantic information. Instead, they use phoneme sequence to control the semantic content of the output speech. Our experiments include both speech resynthesis and voice conversion tasks, which demonstrate superior performance to the traditional k-means clustering. We believe current experiments are enough to demonstrate the superiority of RepCodec, and we leave the experiments of zero-shot TTS to further work.

• "Comparison to SpeechTokenzier"

  Thanks for your advice. We will add a citation of SpeechTokenizer in our paper. However, we would like to point out that SpeechTokenizer and RepCodec focus on different categories of the discrete units. RepCodec aims to improve the quality of semantic units, trying to preserve more information for downstream tasks. On the other hand, SpeechTokenzier targets at improving the quality of acoustic units. Therefore, we do not think directly comparing numbers of these two methods is meaningful.

Thanks for the constructive comments from the reviewer. We address your concern as follows.

• “Insufficient evaluation metrics.”

  First, our evaluation metrics follow previous papers on evaluating the quality of semantic units such as mHubert [1] and AudioLM [2]. They all use word error rate (WER) to measure the quality of semantic units. As the role of semantic units is to provide the semantic information of the speech, the best way to evaluate the semantic information is to measure the content, i.e. ASR-WER, of the output speech. Other metrics such as speaker similarity or mean-opinion score (MOS) measure the acoustic quality of the speech, so they do not reflect the quality of the semantic units.

  While we do not think speaker similarity and MOS can measure the quality of RepCodec units, we still perform the experiments to calculate the speaker similarity, F0 error, and mean-opinion score on speech resynthesis task using LJSpeech. As shown in the following Table, RepCodec largely outperforms k-means on all these metrics, proving the superiority of the acoustic quality of RepCodec units in the speech generation tasks.

  Methods	F0 VDE↓	F0 FFE ↓	Speaker Similarity ↑	MOS↑
  k-means	0.199	0.248	0.771	2.82±0.36
  RepCodec	0.174	0.185	0.764	3.48±0.74
  Original Speech	-	-	-	4.32±0.92

• “Insufficient downstream tasks.”

  First, we would like to point out that our experiments do focus on practical downstream tasks. In addition to ASR and speech resynthesis, we also perform voice conversion in Table 3(b). ASR is a critical task for speech understanding, and RepCodec can be corporated into frameworks like PolyVoice for speech-speech translation. Voice conversion is also a practical task that has wide applications.

  Completely performing the experiments like language modeling of audio, text-to-speech, speech-to-speech translation needs a lot of computation resources and a large amount of time to train the model, which makes it infeasible to complete the experiments in such a short time of the rebuttal. However, our experiments have verified both the input stage and output stage in the framework of PolyVoice for speech-to-speech translation, we believe it is sufficient to show that RepCodec is superior to k-means in these downstream tasks.

[1] Lee, Ann, et al. "Textless Speech-to-Speech Translation on Real Data." In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2022).

[2] Borsos, Zalán, et al. "Audiolm: a language modeling approach to audio generation." IEEE/ACM Transactions on Audio, Speech, and Language Processing (2023).

I appreciate the authors for addressing my comments. However, since the focus of this study is not on more powerful self-supervised representation learning, but rather on producing effective discrete units from representations of self-supervised pretrained models, I believe it should have further validated its effectiveness in language modeling on speech data. Considering both the strengths and limitations of this research, I will maintain my current rating.