DC-Spin: A Speaker-invariant Speech Tokenizer For Spoken Language Models

We sincerely appreciate the reviewer's detailed feedback and the opportunity to address the concerns raised.

Response to Weakness 1
In Table 2, the performance gains of our method are notable relative to prior work. The speech tokenizers proposed in NAST and Gat et al. are more complex, often requiring extensive hyperparameter tuning, while we found Spin and DC-Spin to be more stable, with minimal tuning required.

In Table 3, our approach demonstrates comparable or better results than high-resource speech LMs, even with significantly less data and compute, suggesting that our tokenizer effectively lowers the training requirements.

Response to Weakness 2
We acknowledge the limitation of not including SLM-based TTS evaluations. While time constraints prevent us from conducting this experiment, we reference another paper on self-supervised speech encoder-based tokenizers [1], where consistent results between speech resynthesis and TTS suggest that our speech tokenizer would likely show similar performance in TTS.

Response to Weakness 3
While we understand UTMOS is only a proxy for MOS, policies at some authors' institutions make human MOS evaluations difficult within the rebuttal period.

Response to Weakness 4
We conducted some initial scaling experiments, shown in the table below, which indicate slight improvements with larger models. (tokenizer: HuBERT Base + DC-Spin$_{200,4096}$)

However, larger models tend to overfit, particularly given our smaller dataset compared to prior works, which typically use 10X more data. We would like to conduct further scaling experiments, though these may not be feasible within the rebuttal period.

Response to Weakness 5
Our DC-Spin training objective aligns with Spin, whose speaker invariance has been demonstrated in [2,3]. We rely on this property in our approach to achieving a speaker-invariant tokenizer.

Response to Question 1
Following Expresso [4], we use an ASR based on wav2vec 2.0 Large pre-trained on Libri-Light 60k hours dataset and fine-tuned with LibriSpeech 960 hours corpus (Section 4.3). The high ASR-WER is due to out-of-domain speaking styles in the Expresso dataset, such as laughter and whispering.

Response to Question 2
The proxy tasks in Section 4.7 (bitrate, n-gram predictability, ABX, PNMI, and CNMI) do not need additional SLM training, mitigating the need for 12 hours of training on 8 GPUs. These proxies can be computed within an hour, allowing for faster development and evaluation.

Response to Question 3
As explained in Section 3.3 and Table 1, HuBERT offers a relatively better initialization for DC-Spin and has stable training compared with EMA-based data2vec. We suspect that the HuBERT training objective is more similar to Spin and DC-Spin, contributing to the success of these fine-tuning methods, i.e., predicting discrete tokens. The only difference between the Spin objective and HuBERT is that Spin's prediction targets are learned during training, while HuBERT has fixed targets.

Response to Question 4
In [3], the authors have shown that fine-tuning more layers leads to model collapse. We also observed a similar pattern in our experiments. This phenomenon is caused by a trivial solution to the Spin objective that can be easily achieved. The trivial solution is to ignore all input speech information and possibly encode only the relative position of each frame. Hence, freezing some layers of a pre-trained model enforces the encoder to preserve the original information of speech.

Response to Question 5
The nature of each downstream task has led us to choose the final DC-Spin codebook size carefully. Here are some properties in the tokenizer required for each task.

1. tSC and sBLIMP benefit from smaller codebooks for coarse-grained semantic and syntactic information.
2. sWUGGY and Resynthesis require finer phonetic and acoustic details, favoring a larger codebook.

According to Figure 1, we chose 500 to obtain the best overall performance across tasks. Note that this choice is based on the downstream tasks and the tokenizer's operating framerate (50Hz), a different setup requires a different codebook size.

---

[1] Shi, Jiatong, et al. "MMM: Multi-Layer Multi-Residual Multi-Stream Discrete Speech Representation from Self-supervised Learning Model." arXiv (2024).
[2] Chang, Heng-Jui, Alexander H. Liu, and James Glass. "Self-supervised fine-tuning for improved content representations by speaker-invariant clustering." arXiv (2023).
[3] Chang, Heng-Jui, and James Glass. "R-Spin: Efficient Speaker and Noise-invariant Representation Learning with Acoustic Pieces." arXiv (2023).
[4] Nguyen, Tu Anh, et al. "Expresso: A benchmark and analysis of discrete expressive speech resynthesis." arXiv (2023).

We sincerely appreciate the reviewer's detailed feedback and the opportunity to address the concerns raised. We believe, however, that certain assumptions may not fully capture the recent advances and diversity in spoken language model (SLM) research, as a range of approaches for integrating speech into LMs are actively being explored.

---

(I) Response to the Reviewer's Comments on Speech Tokenization
The reviewer suggests that connecting a speech encoder to an LM should involve continuous speech representation, not discrete tokens. While continuous representations are indeed common, recent work has shown that discrete token-based methods are both valuable and effective. For example:

1. Spirit LM [1]: Uses K-means clustered units in training a speech and text interleaved LLM.
2. Moshi [2]: Proposes Mimi, a neural audio codec that transforms speech into discrete tokens for their full-duplex spoken dialogue framework's input and output units.
3. hertz-dev [3]: Similar to Moshi, also uses a neural audio codec.

Moreover, a recent study comparing discrete and continuous representations for SLM-based ASR found both approaches to offer comparable performance [4]. Additionally, one of the reviewer's own references (https://arxiv.org/abs/2309.00169) supports the use of discrete tokens in ASR, reinforcing our approach. We hope this clarifies the value of discrete tokens in SLM.

---

(II) Response to the Reviewer's Comments on Speech Generation
The reviewer notes that modern TTS focuses on high naturalness and voice cloning, expressing concern that removing speaker information could reduce effectiveness. However, the referenced papers also use discrete tokens as TTS output units, which seems to contradict the concern raised. Furthermore, in Tables 4 and 19 of our paper, we show that our speech tokens outperform neural audio codecs under similar bitrates.

The ASR-WER in our work is higher due to two main factors:

1. We used expressive speech data, such as laughing and whispering, which is not standard in ASR training and naturally yields higher WER.
2. The vocoder we used was not carefully optimized, as our focus was on demonstrating the tokenizer's effectiveness. Future research could improve resynthesis quality using enhanced vocoder architectures (e.g., a diffusion transformer).

Additionally, regarding voice cloning, the goal of removing speaker information is to enhance content representation; speaker details can be reintroduced in the vocoder stage, as outlined in Section 4.1.

---

Response to Question 1

How do you handle speaking style, which is also a kind of speaker-variant?

Our approach removes speaker identity from speech tokens to encourage the tokenizer to encode better content information. We did not consider perturbing speaking styles like loudness and emotional speech. Hence, we leave this part for future studies because these kinds of perturbations require a more complex data augmentation / generation process. E.g., controllable voice conversion or TTS.

Response to Question 2

In the most right hand side of Figure 2, it is not clear to me how the codebook is formed and how the discrete speech tokens are generated? Is K-means clustering still be used?

As described in Section 3.2, we first explained that we directly use the tokens from a Spin codebook without K-means. Next, we described that DC-Spin has two Spin codebooks and share the same encoder. We can add more detailed illustration in the appendix to clarify how DC-Spin works.

---

We believe the weaknesses mentioned do not constitute grounds for rejecting the paper. While we respect the practical perspective of the reviewer, we encourage the reviewer and Area Chair to consider the academic contribution and future impact of our work.

---

[1] Nguyen, Tu Anh, et al. "Spirit-lm: Interleaved spoken and written language model." arXiv preprint arXiv:2402.05755 (2024).
[2] Défossez, Alexandre, et al. "Moshi: a speech-text foundation model for real-time dialogue." arXiv preprint arXiv:2410.00037 (2024).
[3] https://si.inc/hertz-dev/
[4] Xu, Yaoxun, et al. "Comparing Discrete and Continuous Space LLMs for Speech Recognition." arXiv preprint arXiv:2409.00800 (2024).
[5] Shi, Jiatong, et al. "MMM: Multi-Layer Multi-Residual Multi-Stream Discrete Speech Representation from Self-supervised Learning Model." arXiv preprint arXiv:2406.09869 (2024).

We sincerely appreciate the reviewer's thorough feedback and are grateful for the opportunity to clarify and address these points.

Response to Question 1
We observed the auxiliary codebook in DC-Spin behaves similarly to the single large codebook used in the original Spin model, with comparable quality in SLM and codebook metrics. However, the primary codebook's performance has improved in DC-Spin (Table 14). Additionally, we have conducted an ablation study with varying auxiliary codebook sizes, as detailed in the revised paper's Appendix B.1 (Figure 4). These results underscore the auxiliary codebook's effectiveness.

Response to Question 2
We follow the robustness evaluation pipeline in [1] (code at https://github.com/ShovalMessica/NAST), and we followed their official setup as described in Section 3 of [1]:

(i) time-stretch using a Phase Vocoder method to stretch or shrink the time domain signal in the range $[0.8, 1.2]$ without changing the pitch; (ii) pitch-shifting the speech signal by four semitones using the resampling method over the time-stretched signal; (iii) reverberation following similar setting as simulated via the pyroomacoustics audio room simulations package; and (iv) noise injection using a randomly sampled Signal-to-Noise Ratio (SNR) in the range of $[5, 15]$. Background noises are sampled from the Deep Noise Suppression (DNS) challenge which includes a diverse set of noise types from AudioSet, Freesound, and Demand.

Response to Question 3 and the Weakness
To explore DC-Spin's potential at higher bitrates, we experimented with residual vector quantization (RVQ) to expand bandwidth and incorporated Mel spectrogram reconstruction training (Appendix B.3). We applied the Spin loss to the primary codebook to enhance speaker-invariant content, while additional codebooks were designated for capturing fine-grained acoustic details. Despite these modifications, zero-shot SLM performance did not improve significantly, though we believe further studies could yield higher resynthesis quality. A possible direction for future work is outlined in [2].

---

[1] Messica, Shoval, and Yossi Adi. "NAST: Noise Aware Speech Tokenization for Speech Language Models." arXiv preprint arXiv:2406.11037 (2024).
[2] Shi, Jiatong, et al. "MMM: Multi-Layer Multi-Residual Multi-Stream Discrete Speech Representation from Self-supervised Learning Model." arXiv preprint arXiv:2406.09869 (2024).

We sincerely appreciate the reviewer's detailed feedback and valuable insights, which have helped us clarify key aspects of our work.

Response to Weakness 1
In Table 2, the improvements of DC-Spin over Spin become more pronounced with smaller primary codebook sizes (e.g., units = 50 or 100). While the improvements are less evident at 500 units, they are still present. Additionally, the chunk-wise streaming approach illustrates the ease with which our tokenizer adapts to streaming applications. For this reason, we provided the detailed explanations in the appendix. We will revise the contribution section in the introduction to emphasize this more clearly.

Response to Weakness 2
Although the baseline methods we used, EnCodec and SpeechTokenizer, are relatively recent (2022 and 2023) and remain widely used, we acknowledge that there are newer alternatives. We can possibly try to experiment with DAC, but due to the rebuttal time frame, we might not be able to report the results. Additionally, WavTokenizer is considered concurrent work since it was first posted to ArXiv in late August this year (https://iclr.cc/Conferences/2025/FAQ).

Response to Weakness 3
As noted in Section 3.2, larger Spin codebooks have been shown by Chang et al. (2023) to better capture phonetic representations. Since both the primary and auxiliary codebooks share the same encoder, we expect the auxiliary codebook to indirectly assist the primary codebook in encoding high-quality phonetic units. We have clarified this mechanism in the paper. For example: "Moreover, Chang et al. (2023) found larger Spin codebooks capture better phonetic representations in the encoder." Although we found that DC-Spin helps capturing phonetic units, there is no evidence about ignoring acoustic details. Besides, in Tables 4 and 19, we show that DC-Spin consistently outperform K-means in speech resynthesis.

Response to Question 1
The auxiliary codebook is used only during training and is not involved in token extraction. When tokenizing speech with DC-Spin, we rely solely on the primary codebook, resulting in a bitrate comparable to that of K-means$_{500}$.