dMel: Speech Tokenization Made Simple

We sincerely appreciate your insightful review and for recognizing this as pioneering work that may open a potentially new track of TTS research. Your understanding of our core motivation - addressing the challenge of capturing both semantic and acoustic information without architectural complexity - perfectly aligns with our research goals.

We appreciate the reviewer bringing MELL-E (https://arxiv.org/abs/2407.08551) to our attention. However, we would like to clarify two important points:

• MELL-E was released on July 11, 2024. According to ICLR 2025's policy, this qualifies as concurrent work, as it was published during our paper's preparation period. However, the concurrent emergence of similar ideas from different teams often suggests the research direction is promising
• While both works address speech representation, MELL-E employs a more complex architectural approach and operates on continuous mel features. Our method operates on discrete features (opposite direction to continuous features) and prioritizes simplicity and efficiency while achieving competitive performance. Also, our work demonstrates broader applicability across both ASR and TTS tasks

We believe these concurrent developments validate the importance of exploring mel-based representations for language model-based speech processing.

We sincerely appreciate the reviewer noting our work's soundness and contribution, particularly in recognizing the novelty of our mel-spectrum approach. We've carefully considered your feedback and would like to address each point comprehensively:

1. Baseline Comparisons

While we understand the interest in Whisper/VALL-E comparisons, there are fundamental methodological reasons these were not included:

• Neither is open-source, preventing reproducible research. Whisper uses proprietary, undisclosed training data.
• Such comparisons wouldn't validate our core scientific contribution
• Our focus is on advancing open, reproducible speech research

2. On Architecture Experiments:

We apologize if Table 10's extensive architecture experiments weren't sufficiently highlighted, but we reported results across multiple standard architectures in Table 10:

• Transformer Decoder (most popular architecture)
• CTC-based models
• Sequence-to-sequence (encoder-decoder) models (whisper falls into this type of models, but was trained on ~400x more data, so we actually compare with the family of models whisper is based on)

Our experiments demonstrate dMel's effectiveness across multiple standard architectures in a reproducible setting with open data.

3. Response to questions

For RichTTS and RichASR, what about the implementation details like architecture/training data (compare to speechgpt?)

The architecture is introduced in Table12. The training data is LibriSpeech for most of the Tables except Table 5 and 6, as we introduced in Section 3.1. Table 5 and 6 are using different datasets to compare the results with different models.

Is there any open-source plan to support the community?

We will definitely open-source our full codebase and we are undergoing necessary steps to release full code.

Thank you once again for your thoughtful review and feedback! As we approach the end of the discussion period, we want to ensure that our previous responses have fully addressed all your concerns. If you have any additional questions or unresolved issues that we can clarify, please don’t hesitate to let us know. We’re more than happy to assist further!

Thank you for your detailed review. We address your concerns and questions below:

1. Regarding Bit Rate and Compression

We cannot agree on different bit-rate leading to unfair comparison as bit-rates don't necessarily correlate with better downstream task performance. Several work have observed this:

• Moshi [1] observed: "Across our experiments, we make the somehow counter-intuitive observation that this gain gets more significant as we lower the bitrate."
• DASB [2] similarly reported: "Interestingly, higher bitrates, when available (e.g., for EnCodec and DAC), tend to degrade performance."
• Another evidence is in VoxtLM [3] paper, they found using k=200 centroids achieved 3.5 CER for TTS but using k=1000 centroids resulted in a higher 6.1 CER for TTS. This also indicates higher bit-rate doesn’t mean better results.

2. It's important to note that traditional bit-rate comparisons may not be directly applicable to dMel due to its distinct architectural features:

   • Encoder-free Design: Unlike conventional approaches, dMel operates without an audio compression model, making traditional compression rate metrics less relevant and inaccurate to measure dMel.
   • Parallel Processing: While dMel utilizes 80 channels per frame, these channels are processed in parallel during both encoding and decoding. Therefore:
     • Computational complexity is primarily determined by frame rate rather than bit-rate
     • Traditional bit-rate calculations do not accurately reflect the model's efficiency

We respectfully suggest that dMel's performance should be evaluated within the context of its novel modeling approach rather than compression method, where frame rate serves as a more meaningful metric than bit-rate. We would like to add discussion about the bit-rate and frame-rate in our paper too.

3. Choice of Centroids for HuBERT K-means

Regarding the number of centroids used in our HuBERT k-means implementation, our choice of 200 centroids follows the methodology established in VoxTLM[3], which demonstrated superior performance with this configuration. Specifically, VoxTLM's ablation studies (Table 6) show that:

• k=200 centroids achieved 3.5 CER for TTS
• k=1000 centroids resulted in a higher 6.1 CER for TTS

This empirical evidence supports our choice of centroid count.

2. Regarding Baseline Comparisons

We appreciate the reviewer's comments about baselines, but several points need clarification:

• Regarding baseline selection:
  • SpeechTokenizer is a very recent work (ICLR 2024)
  • Even Moshi [1], which you cited, builds upon SpeechTokenizer's methodology, acknowledging "inspiration from previous work on SpeechTokenizer"
• Regarding Spoken Language Modeling (SLM): The assertion that SLM is a "standard evaluation task" for speech tokenization papers is not accurate. Many significant works do not include SLM evaluation yet:
  • EnCodec
  • SpeechTokenizer
  • Recent works mentioned by reviewer 6UMS (SPECTRAL CODECS, SemantiCodec, APCodec, Single-Codec)
• Regarding cited papers:
  • [4] (AudioLM) doesn't propose new tokenization methods but focuses on modeling existing tokens
  • [2] focuses on representation robustness for SLM rather than tokenization
  • [1] (Moshi) was released just one week before submission deadline and it is a 67 pages paper works on both speech tokenization and SLM.

3. Regarding Writing

Thanks for spotting places with less formal style in our paper. We will correct accordingly the style, so please let us know if you found any other places. At the same time, we are concerned about feedback on usage LLMs for writing as this could violate the privacy.

[1] Moshi: a speech-text foundation model for real-time dialogue
[2] DASB - Discrete Audio and Speech Benchmark
[3] Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation tasks.
[4] Borsos, Zalán, et al. Audiolm: a language modeling approach to audio generation.

Response to the Questions:

What are the hyperparameters for extracting log Mel spectrograms? Window size? Stride?

The stride size is 1000/frame_rate. For our 40HZ feature, the stride size is 25ms, the window size is 50ms. We have introduced these in the caption of Table 10.

Why are the model names "RichASR" and "RichTTS?" Any specific reasons?

We named our model informally because of internal reasons, not because of any scientific reason - sorry if that caused confusion.

What is the codebook utilization rate or distribution of dMel? The proposed quantization approach divides the intensity into equally-spaced bins. However, a potentially better way is to assign bin sizes according to the data distribution for a uniform codebook utilization.

We ablated dMel uniform binning with a percentile-based discretization method, which computes bin boundaries based on channel-specific statistics from the LibriSpeech training data. The latter showed competitive but slightly inferior performance compared to our proposed method and we have left this for future exploration.

Does pre-training the LM with speech-only data help downstream performance? In spoken LM applications, it is common to pre-train the LM on speech tokens with large unlabeled data.

While speech-only pre-training could be valuable future work, our current focus is specifically on:

• Demonstrating effective mel feature discretization
• Establishing a decoder-only architecture for conditional sequence generation, where speech is generated from text input (TTS) or text is generated from speech input (ASR)

Our strong ASR/TTS results demonstrate these core contributions are effective without additional pre-training. Future work could explore both pre-training for multi-task scenarios and unconditioned generation tasks.

Are there any decoding techniques involved in RichTTS and RichASR? E.g., beam search.

Thanks for pointing out. We use top-p (p=0.95) sampling instead of beam search for RichTTS while no beam-search is used for RichASR and simple greedy decoding is done. We will add this important implementation details in our manuscript.

We sincerely appreciate your thorough review and recognition of our novel contributions. Below we address your key concerns:

1. Regarding Bit Rate and Compression

You raise an important point about bit rate discussion. While dMel operates at approximately 12.8 kbps (40fps × 80 mel filters × 4 bits/filter), we'd like to highlight several important considerations:

1. Recent research has demonstrated that higher bit rates don't necessarily correlate with better downstream task performance. For instance:

   • Moshi [1] observed: "Across our experiments, we make the somehow counter-intuitive observation that this gain gets more significant as we lower the bitrate."
   • DASB [2] similarly reported: "Interestingly, higher bitrates, when available (e.g., for EnCodec and DAC), tend to degrade performance."

2. It's important to note that traditional bit-rate comparisons may not be directly applicable to dMel due to its distinct architectural features:

   • Encoder-free Design: Unlike conventional approaches, dMel operates without an audio compression model, making traditional compression rate metrics less relevant and inaccurate to measure dMel, since the encoders participate in the compression scheme of the other models.
   • Parallel Processing: While dMel utilizes 80 channels per frame, these channels are processed in parallel during both encoding and decoding. Therefore:
     1. Computational complexity is primarily determined by frame rate rather than bit-rate
     2. Traditional bit-rate calculations do not accurately reflect the model's efficiency

We respectfully suggest that dMel's performance should be evaluated within the context of its novel modeling approach rather than compression method, where frame rate serves as a more meaningful metric than bit-rate. We would like to add discussion about the bit-rate and frame-rate in our paper too.

2. Regarding Content and Contributions

We appreciate your feedback about the paper's focus on ASR & TTS systems. We would like to emphasize three key aspects that demonstrate the broader impact of our work:

• Novel Architecture: While mel features have been extensively studied, our work is the first to investigate modeling them with a decoder-only architecture. The dMel + decoder combination represents a fundamental architectural innovation in the field.
• Superior TTS Performance: Contrary to the expectation that our approach might underperform traditional mel spectrograms, Table 5 demonstrates that RichTTS (dMel + TransformerDecoder) achieves lower WER compared to popular open-source mel-based TTS models including VITS, FastSpeech2, and Tacotron2.
• Innovative Applications: By aligning TTS model design with Language Model architectures, dMel enables the application of numerous LM techniques to speech synthesis, including:
  • Speculative decoding
  • KV-caching
  • Multi-turn capabilities
  • And potentially many more

We will revise the manuscript to better emphasize these broader implications and their potential impact on the field.

[1] Moshi: a speech-text foundation model for real-time dialogue https://arxiv.org/abs/2410.00037

[2] DASB - Discrete Audio and Speech Benchmark https://arxiv.org/abs/2406.14294

3. Questions

Is there a fundamental difference between finite scalar quantization (FSQ; [1]) and dMel's quantization? If not, I think the FSQ paper should be acknowledged.

We recognize that our quantization method shares some similarities with FSQ, but also differs in key aspects:

1. FSQ employs scalar quantization in a learned latent code space, primarily focusing on image data modeling. In contrast, our work pioneers the successful application of scalar quantization technique to the original raw mel-frequency band (mel-fb) space, serving as a training-free speech tokenization approach.
2. FSQ necessitates an additional bound operation to limit the range of the latent codes. Moreover, the dimensionality of the code in FSQ is considerably smaller (≤6) compared to ours (80). This higher dimensionality is crucial for maintaining speech quality across a broader frequency range.

We will make sure to cite the FSQ paper appropriately in our revised manuscript to acknowledge this prior work. Our primary contribution is not the quantization method itself, but rather:

• Successfully applying it to mel-frequency features for speech discretization
• Developing an integrated architecture that enables decoder-only speech modeling
• Demonstrating strong empirical results across multiple downstream tasks

We thank reviewer for the comments and recognition of our approach. We have thoroughly examined each point and would like to provide detailed responses:

1. Evaluation with Recent Benchmarks

Evaluate with recent Benchmark such as Codec-Superb and DASB. Compare with more baselines include: SPECTRAL CODECS and SemantiCodec.

We highly appreciate the reviewer for pointing out these relevant work and benchmarks, and we definitely would like to discuss these work in our next revision. However, we believe that these works should be considered as concurrent work, and according to ICLR's concurrent policy [https://iclr.cc/Conferences/2025/FAQ], that paper published within the last 4 months are considered as contemporaneous. Also, paper not published in peer-reviewer proceedings or journals are not required to compare. Therefore, we believe this should not be considered as a main weakness.

2. Evaluation on Other Audio Domains

Evaluation on other domain audio data includes music and general audio

As indicated in our title and our limitation section, the current scope of this work is primarily on speech. We will consider extending dMel to music and general audio as our future work.