ALMTokenizer: A Low-bitrate and Semantic-rich Audio Codec Tokenizer for Audio Language Modeling

We thank the reviewer for recognizing our contributions.

Q1: latent space optimized for AR modeling...it seems that not using the LM loss is beneficial for several metrics.

A: We appreciate this comment. We acknowledge that introducing the autoregressive (AR) loss may slightly impact reconstruction metrics. As discussed in the Limitations section, we emphasize this trade-off to encourage future research on achieving a better balance between reconstruction quality and modeling efficiency.

Q2: Methods And Evaluation Criteria: However, there are many training/evaluation datasets utilized for different tasks ... mention which evaluation set it used ...

A: We appreciate this comment. We will update our paper to explicitly specify the evaluation set used for all tables and figures in the final version. Furthermore, both Table 1 and Table 6 report results on the VCTK dataset. In Appendix Table 12, we present a reconstruction performance comparison between our proposed method and previous works on the LibriTTS test set. Additionally, we have reported results with 95% confidence intervals for subjective evaluations. For objective reconstruction metrics, since they are deterministic, confidence intervals were not previously included. However, for generation experiments, we will incorporate confidence intervals by sampling multiple times in the final version.

Q3: I have one concern, both Encodec and DAC are used at 1.5kbps...potential as well.

A: We appreciate this comment. In line with MimiCodec, we will include results for Encodec and DAC in two configurations: (1) using 3 RVQ layers, as reported in our paper, and (2) using 8 RVQ layers to demonstrate their full potential.

Q4: The training strategy seems to be too wasteful... initialize the depth transformer...

A: We appreciate this constructive comment, which provides valuable insights for improving our work. In particular, leveraging the AR decoder to initialize the deep transformer in language modeling is an interesting idea. We find this direction highly promising and plan to explore it in future work. Additionally, we will incorporate this discussion into the final version of our paper to inspire further research in this area.

Q5: The variable bitrate strategy is useful for the model as a pure codec...

A: We appreciate and agree with the reviewer that the applicability of the variable bitrate strategy for downstream language modeling tasks remains unexplored. Our proposed method primarily facilitates the selection of codec models with different frame rates, providing greater flexibility in bitrate allocation. We will add this discussion into the Limitation part.

Q6: Table 7, codebook size and FPS columns seem to be interchanged.

A: Thank you for your help to find this mistake. We will update it in the final version.

Q7: Line 216: "we found ...of this statement?

A: In our experiments, we observed that a high masking rate negatively impacts reconstruction performance. We evaluated three masking rate ranges: 10–20%, 20–30%, and 30–40%. As shown following, higher masking rates (30–40%) improve semantic representation but degrade reconstruction quality. Based on these findings, we adopt an intermediate masking range of 20–30% to balance semantic preservation and reconstruction fidelity.

Q8: I think the subjective evaluation ... it could be one automatic and the MUSHRA score.

A: We appreciate and agree with the reviewer that subjective evaluation performance should be presented as the primary result in the paper. Accordingly, we will update our manuscript to include the MUSHRA score results in the main text.

Q9: It is better to include an explanation of the wastefulness...

A: As we discussed in Q4, we will incorporate this constructive discussion into our final version, stating the existing wastefulness and listing the potential solutions.

Q10: The paper relies a lot on the Appendix...several details might be missed.

A: We appreciate this comment. Since the final version of ICML allows an additional page in the main text, we will move more experiments from the Appendix into the main text, such as (1) Table 10 (the subjective evaluation results) and (2) Table 11, the LM-based sound and music understanding and generation results.

We thank the reviewer for recognizing our contributions. We do appreciate the constructive comments the reviewer provided to us to further improve our paper. We are delighted to have the following discussion with the reviewer.

Q1: One of the claims is that the "semantic priors" avoids distillation as done by previous work. The semantic prior involves training a k-means on self-supervised embeddings and using the centroids as the fixed codebook of the first VQ. This is a form of distillation, yet less costly as during training one does not need to pass audio through a teacher embedding...

A: We appreciate and agree with the reviewer that semantic priors can be regarded as a form of distillation. Their advantages include: (1) reduced training cost, as they do not require audio to be passed through a teacher embedding during training; and (2) the flexibility to integrate multiple teacher models, such as Wav2Vec2 for speech semantic priors and BEATs for general sound semantic priors. In contrast, previous methods like SpeechTokenizer and MimiCodec rely on a single teacher model and primarily focus on speech semantic priors, although they can be extended to both speech and general sound.

We will revise our previous claim that 'semantic priors avoid distillation' to clarify that semantic priors are indeed a form of distillation. Additionally, we will highlight their advantages and application scenarios is different with previous methods.

Q2: Presentation of results: Human evaluations of audio quality are put in Appendix...

A: We appreciate this comment and agree that human evaluation performance should be presented as a primary result in the paper. Accordingly, we will update our manuscript to include the MUSHRA score results in the main text.

Q3: L230: what does "continuous" mean in that setting? that it's continuously trained? This requires more details in particular I guess a stop gradient is applied to avoid degenerate solutions?

A: We appreciate this comment. The term 'continuous autoregressive (AR) transformer' is used to distinguish our approach from traditional discrete AR models, which operate on discrete token sequences and are optimized using cross-entropy loss. In our study, to facilitate gradient backpropagation, we apply the AR transformer directly to continuous features (i.e., the quantized features) and optimize using mean squared error (MSE) loss. We will put these details in our final version.

Q4: Also, does "predicting the third VQ from the first and second" mean that this small model is only autoregressive along the VQ axis or is it also autoregressive along time?.

A: Yes, the AR model is only autoregressive along the VQ axis.

Q5: L357: 12.5kHz -> 12.5Hz.

A: Thank you for your help to find this mistake. We will update it in the final version.

Q6: Also, Mimi uses Transformers in the encoder and the decoder, is this included in this ablation? The baseline described in "The Effectiveness of Query-based Audio Compression" seems to be purely convolutional. Making sure those transformers are included would demonstrate the usefulness of the query-based proposal wrt a simple transformer.

A: We appreciate this comment. In our ablation study, The Effectiveness of Query-based Audio Compression, we compare our approach against a reproduced version of MimiCodec, which incorporates convolutional and transformer layers. The details of our reproduced MimiCodec implementation are provided in Appendix B.4. Table 7 presents the performance of reproduced MimiCodec at three different frame rates: 50 Hz, 25 Hz, and 12.5 Hz.

We greatly appreciate the reviewer's time and patience with our paper. We are delighted to solve the concerns of reviewer one by one.

Q1: Several essential ... query token is not clear to me.

A: We appreciate your feedback regarding the clarity of the "query token" concept. Below, we provide a detailed explanation.

we first revisit the workflow of previous tokenization methods, such as Encodec and SoundStream. As shown in the left part of Figure 2, the input audio data is first processed by the encoder module, transforming it into a series of audio frames, denoted as $\boldsymbol{e} \in \mathcal{R}^{T \times d}$, where $T$ represents the number of frames. The value of $T$ is determined by the down-sampling strides in the encoder module. Notably, previous works treat all audio frames equally. However, in such a setting, reducing a low frame rate (e.g., 12.5 Hz) requires increasing the down-sampling stride to 1920, which significantly degrades reconstruction due to: (1) ignoring the fact that different audio frames encode varying levels of information; and (2) failing to leverage contextual dependencies across frames.

Thus, we propose a query-based compression strategy. Instead of employing large down-sampling strides, we first use a Patchify module to segment the audio into frames. Our approach then introduces learnable query tokens, which dynamically extract information across multiple frames. Notably, query tokens are learnable embedding vectors that are updated throughout the training process. As described in Section 3.2, these learnable query tokens are combined with the audio frames and processed by a transformer encoder, where they adaptively aggregate important information. After that the original audio frames will be ignored, and these query tokens will be used for RVQ and decoder.

In summary, query tokens serve as learnable embedding vectors designed to capture holistic contextual information from the audio frames. Since the number of query tokens is smaller than the number of audio frames, which effectively reduces the frame rate while preserving essential information. The concept of query tokens is analogous to BERT [1], where a [CLS] token is placed at the beginning of a sentence to capture its semantic representation (In contrast, in our stdudy, we introduce multiple query tokens by Query token interleaving). To enhance clarity for readers, we denote this query token as [CLS] in our paper (line 205).

Q2: It seems the bit-rate is...

A: As discussed in Q1, the low bitrate is not achieved through hyperparameter tuning. Instead, we introduce query tokens that summarize contextual information across frames, where the number of query tokens directly determines the bitrate.

Q3: Second, the ... same is not clear to me.

A: We appreciate this comment. As discussed in Q1, we employ query tokens to capture holistic audio context information from audio frames. Similar to BERT, semantic information is effectively aggregated into the query tokens with the aid of a transformer encoder.

Q4: The use of transformers, MAE, .. seems novel ... audio conferences and journals.

A: We thank the reviewer for recognizing our contributions. Although our approach involves audio processing techniques, its core contribution lies in machine learning methodologies—specifically, transformer-based compression and representation learning for audio modeling. Our method is not limited to audio; the query-based compression strategy can be extended to other sequential modalities. Given the increasing importance of efficient tokenization strategies in large-scale multimodal models, we believe our work is highly relevant to the ICML community.

Q4: Essential References Not Discussed...

A: We appreciate and agree with the reviewer that tokenization can also be applied to audio retrieval, further highlighting the potential of our research for multimodal tasks. We are pleased to discuss the application of tokenization methods (such as DUSTED and wav2tok) in audio retrieval in our revised version.

Q5: Contributions: I am not convinced about contributions to ML.

A: Reviewer can refer to Q4.

Q6: What are query tokens? ...for speech synthesis?

A: The reviewer may refer to Q1 for further details. For the text-to-speech task, the corresponding query tokens can be predicted based on textual conditions. Because we use RVQ to quantize the query tokens into discrete IDs, enabling them to be modeled by an LM-based audio generation framework.

Q7: The paper does an impressive number of experiments...

A: We thank the reviewer for recognizing our esperiments and contributions. The theoretical contributions to ML can refer to Q4.

[1] Devlin J, et al. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL. 2019.

We thank the reviewer for recognizing our contributions. We do appreciate the constructive comments the reviewer provided to us to further improve our paper. We are delighted to have the following discussion with the reviewer.

Q1 : Although the paper’s ablation study suggests that each proposed technique contributes, the simultaneous use of multiple methods (semantic prior VQ, MAE, LM loss, and two-stage training) complicates fair comparisons.

A : We thank the reviewer for acknowledging the contributions of our proposed techniques. To systematically validate each component's effectiveness, we have conducted extensive ablation studies (Table 6 of manuscripts), including:

(1) Query-Based Framework : We constructed a MimiCodec-style baseline [1] with identical convolutional patchify/unpatchify modules and transformer-based encoder/decoder. The key distinction lies in our proposed query-based framework, which demonstrably improves audio compression and semantic performance (Table 6, Row 3).

(2) MAE and LM (AR) Losses : Ablation experiments confirm that both losses enhance semantic performance (Table 6, Rows 4) and Figure 3.

(3) Two-Stage Training : Compared to one-stage training, our two-stage strategy consistently improves reconstruction and semantic metrics (Table 6, Row 6).

These experiments clearly clarify the impacts of our techniques.

Q2: For example, while semantic VQ priors provide only marginal improvements in semantic tasks, they slightly degrade audio reconstruction quality.

A: We agree with the reviewer’s observation regarding the trade-off between semantic performance and reconstruction quality. One of the potential reasons is that we fix the VQ codebooks during training, which is different from the traditional VQ training (as noted in Section 3.2). This aligns with our discussion on limitations part: while semantic VQ priors improve semantic performance, optimizing the semantic information and minimize reconstruction loss remains an open challenge. We highlight this trade-off to encourage future work on balanced solutions.

Q3: Additionally, MAE, LM loss, and two-stage training could likely be integrated independently into other neural audio codecs.

A: We appreciate and agree with the reviewer that our proposed technique contributes, such as MAE loss, LM loss, two-stage training strategy can be integrated into other neural audio codec models, such as Encodec and MimiCodec. As we discussed in Q1, we have conducted ablation studies to validate the effectiveness of each part. We hope these contributions will inspire broader adoption in the audio codec community.

Q4: The paper does not sufficiently ..., analyzing these variations would help determine whether the approach is robust or ... under different configurations.

A: We appreciate this suggestion and provide additional analyses:

(1) Mask Rate in MAE Loss. Inspired by MAE [2], we tested three group of mask rates ranges: (10–20%), (20–30%), and (30–40%). The experiments as following Table shows. Results indicate that higher rates (30–40%) benefit semantics but harm reconstruction, leading us to adopt an intermediate range (20–30%).

(2) The hyperparameters of MAE loss and AR loss weighting. To better understanding the influence of hyperparameters of MAE loss and AR loss ($\lambda_1$ and $\lambda_2$), we design 3 group settings (1, 1), (0.5, 0.5), (0.5, 0.1). The experimental results as the following Table shows. We can see that (0.5, 0.1) obtains better performance, thus we empirically choose $\lambda_1=0.5$ and $\lambda_2=0.1$ as our default setting.

[1] Défossez A, et al. Moshi: a speech-text foundation model for real-time dialogue[J]. 2024. \

[2] He K, et al. Masked autoencoders are scalable vision learners CVPR 2022.