Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding

We sincerely appreciate your time and the effort you’ve put into helping us improve our presentation.

Novelty (It appears that there are some misapprehension): Our quantizer fundamentally differs from plain RVQ by employing a multi-band split directly at the latent level, guided by psychoacoustic features such as content, formant articulation, and speaker characteristics. Unlike other multi-band codecs (e.g., Spectral Codec, which splits at the input level, or HALL-E and PyramidCodec, which use multi-scale down-sampling), our approach as accurately pointed by other reviewers that it performs a spectral split in the latent space. This design allows the codec to construct psychoacoustical disentangled features, with each codebook optimized specifically for different frequency bands (supported by theory and empirical observations), clearly distinguishing from previous work. As demonstrated in Theorem 3.1, our method not only enhances the elegance and robustness of the codec but also pushes the boundaries of performance. We believe this novel perspective is an important contribution to the community and bridge the gap of traditional work of MP3.

Fairness (It appears that there are some confusion but we are happy to clarify): Baseline Alignment-We intentionally align our work to HiFi-Codec (baseline) by matching codebook used of 4. HiFi-Codec represents SOTA speech performance, making it a rigorous and relevant benchmark. Additionally, we retrained the baseline on the same dataset to ensure robust performance and to control for any potential data bias, thereby ensuring a fair comparison. Benchmark other work: For our comparisons with other codecs (e.g., DAC and EnCodec), we adopt the “early codebooks” approach, consistent with recent studies such as WavTokenizer, SemantiCodec, and Speech Tokenizer. MUFFIN performs well even when evaluated across other papers of same dataset, showing consistency in the literature.

Computationally- RVQ (benchmarked model) mathematically encourages each codebook to be as self-sufficient as possible. Each stage minimizes its own L1/L2 reconstruction error without “looking ahead” to future stages. This independent optimization ensures that the early codebooks in any model, whether a model is trained with 4 or 32 codebooks, are directly comparable, since each stage is forced to capture as much residual information as possible.

Lastly, MiMi is compared with the officially reported count of 8 codebooks to further validate our results. Overall, our comparative tables are constructed on a scientifically fair basis, allowing for meaningful insights rather than merely demonstrating superior performance or any attempt to over-claim our work.

Streaming: Our streaming capability is built on a fully CNN-based model that follows exactly like Encodec’s design so it will be inaccurate to say that ours is non-causal. Specifically, our system processes audio in small window frames (3.5 seconds) that are non-causal within each frame, i.e., using such global window context, but causal over past windows for all streaming applications. Since CNNs are fundamentally local feature extractors, they do not inherently capture global context as transformers do. This locality is advantageous in streaming applications, as it allows for more stable and consistent performance when operating under strict causal constraints. By contrast, self-attention models (wavtokenizer), although trained in a non-causal manner, must be adapted to causal computation during streaming, which can lead to instability. Our approach leverages the stability of CNNs in local processing, ensuring robust streaming even when constrained to causal operation.

Following LLASA: We fully agree with the reviewer’s concern regarding the importance of these metrics. Due to space constraints, we did include the detailed table in the original submission Appendix F, where we demonstrate their impact comprehensively with latency metric of MACs and model size as well. However, we will update the references raised and the typo in line 034.

Ablation of Snake Activation: While modified snake activation is not our core contribution in this work, we agree that showing the ablation performance helps to improve the quality of the presentation. We will include the results in appendix C while showing the performance as below

LibriTTS (test-clean)

We appreciate your dedication to carefully scrutinize our work and it means a lot to us.

Disabling MBS: We agree that conducting an ablation study by disabling “MBS” is important to better demonstrate its contribution to the reconstruction performance. Part of this analysis has already presented in Table 5, where we compare MUFFIN with plain RVQ to evaluate WER, STOI, and the behavior of each individual codebook. To further strengthen our empirical evidence, we will include detailed reconstruction performance results in a new appendix, as shown below focusing on speech reconstruction.

LibriTTS (test-clean)

LibriTTS (test-other)

IEMOCAP

From the above tables, there is consistent improvement over using MBS demonstrating the effectiveness of MUFFIN (supported by theorem).

---

Using SECS for evaluation: We acknowledge that our current evaluation metrics for the codec do not include human evaluations, and we agree that SECS may be a viable addition. The metrics we adopted follow previous work (e.g., Codec-SUPERB), which argued that these objective measures provide sufficient coverage.

Nevertheless, we appreciate your suggestion and have attempted it. However, given the STOI scores of the reconstruction results and the nature of our task (i.e., not generating entirely new speech but reconstructing existing ones), it can be challenging for human evaluators to reliably distinguish subtle quality differences, as shown in the demos (especially without cherry-picking samples). Similar issue has been discussed in [1].

Therefore, we find it difficult to implement and believe that relying on objective metrics is more appropriate for evaluating the codec’s performance with more precise distance metrics. However, we are also careful with our evaluations and have indeed used human evaluation for our TTS outputs, as shown in Table 6, where naturalness (MOS) and speaker similarity can be meaningfully assessed. We trust that this is a common valid concern and will discuss in a new appendix section to dismiss some misunderstanding/issue with evaluating codec performance with human while evaluating/citing SECS.

[1] Varadhan, Praveen Srinivasa, et al. "Rethinking MUSHRA: Addressing Modern Challenges in Text-to-Speech Evaluation." arXiv preprint arXiv:2411.12719 (2024).

---

Annotation of off-the-shelf models in tables and snake activation references: It makes perfect sense and we will update the manuscript with the provided references to enhance the credibility of our results. We deeply appreciate your effort in pointing out our weaknesses.

---

We thank you for your constructive comments and thoughtful concerns, which help to improve the impact of this work and spark further discussion.

Using larger audio set: We agree with the valid concern regarding the size of the environmental audio dataset, and we welcome further discussion on this topic. Our findings indicate that integrating both speech and music data help to overcome low-resource while achieving SOTA environmental audio performance, even when using a smaller set, comparing to various off-the-shelf models (DAC, EnCodec) trained on much larger collections (see Table 4). This can be attributed to our training that follows existing work, which uses short 1s segments that capture brief vocal or instrumental passages. These segments tend to be somehow similar in terms of their audio characteristics to some environmental audio, thereby reducing the reliance on distinctive larger audio datasets. Moreover, our interest is on the vocal, where psychoacoustic features, such as vocal timbre and articulation, are the highlight of the neural psychoacoustic codec. Thus, we will not consider using larger audio dataset in this work as compared to speech and music (covering singing vocal).

Human evaluation and latency: We agree that the absence of human evaluations is a common concern. To address this, we will add a dedicated subsection in Appendix F to explain how the objective metrics, widely adopted in the literature, correlate with human perceptual quality, thereby demonstrating the self-sufficiency of our report without human for codec. Further detailed clarifications will also be provided in response to Reviewer grYU (using SECS for evaluation). Similarly, our latency benchmarks, presented in Appendix F, are based on MACs and model parameters, which provide a good objective measure of inference time while normalizing for factors such as GPU specifications.

Transient audio: We appreciate the reviewer’s thoughtful observation. While psychoacoustic studies have primarily focused on speech, we agree that applying similar analysis to non-stationary or transient sounds, such as those in music, is both important and intriguing. To explore this, we extended our decomposition approach to a variety of musical genres, including singing, classical, jazz, and symphonic music. Consistent with the psychoacoustic framework used in speech analysis, we observed that:

• Codebook 1: primarily captures vocal content and coarse rhythmic beats.

• Codebook 2: emphasizes vocal clarity and mid-frequency information.

• Codebook 3: encodes pitch details reflective of the singer’s unique characteristics.

We have updated our demos to include samples that support these observations. Interestingly, instrumental content does not clearly separate across Codebooks 2 and 3, suggesting that our psychoacoustic-guided representation is particularly effective in disentangling vocal attributes (speech and singing), but less so for purely instrumental channels. This finding reinforces the theoretical value of psychoacoustic principles for modeling vocal properties, an area that remains underexplored in neural codecs. While applying this framework to instrumental music remains challenging, we believe this opens new research directions. Further investigations, beyond the scope of the current study, will be discussed in our future work section to encourage continued exploration of this promising line of research in appendix.

Practical limitation of 12.5 Hz: Achieving high compression rates in audio codecs often challenges the preservation of the full spectrum of human hearing, potentially leading to muffled sounds or perceptible artifacts (especially so for streaming with reference to MiMi's performance). To address this, integrating psychoacoustic models can enhance reconstruction quality by optimizing compression across multiple frequency bands, focusing on perceptually significant components. However, implementing such models typically necessitates more complex and deeper neural networks to effectively quantize and encode the nuanced psychoacoustic information without significant loss. This increased complexity can lead to larger model sizes, which may offset the benefits of efficient compression by demanding more computational resources and storage capacity. Therefore, a careful balance must be struck between leveraging psychoacoustic properties for improved audio quality and managing the trade-offs related to model complexity and compression efficiency. This could spark more research work to investigate in this area.