Towards Codec-LM Co-design for Neural Codec Language Models

From the motivational perspective:

1. the theoretical foundation underlying the independence of the features in the encoder is notably lacking. The authors must provide a rigorous explanation for this, as emphasized in the limitations section. This deficiency also raises two potential issues: first, there is a concern that the model may lose its ability to capture contextual information, leading to semantic omissions in the codec; second, the complete independence of the receptive fields may introduce boundary issues when reconstructing relevant textual information.
2. the rationale for using "longer frame durations" and the choice of a codebook size(32768) of warrant clarification. If the codebook is excessively large, it may result in decreased utilization and, consequently, adversely affect TTS performance.
3. The LM codebook level dropout appears more as an engineering experiment with limited applicability, particularly since architectures like VALL-E and SoundStorm utilize random selections of quantizers during training.
4. The innovations presented in these three tricks are not particularly groundbreaking within the codec domain. From the CodecLLM perspective, attention might be better directed toward bridging the gap between reconstruction paradigms and LLM generation paradigms, as well as investigating the impact of integrating acoustic tokens on the degradation of text-based LLM performance. These three tricks in this paper appear to be unrelated to the broader objectives(and considering the substantial existing work on codec compressibility).

From the experimental perspective:

1. the rationale for employing internal datasets among numerous open-source options raises questions about the potential biases introduced by dataset selection.
2. Additionally, both codec and TTS evaluations lack a number of relevant metrics; for instance, the codec assessment should include PESQ, VISQAL, UTMOS, and STOI results, while TTS evaluations should incorporate MOS metrics.
3. Based on Table 1, the effectiveness of the tricks employed in this paper appears to be modest, with no significant differences observed in WER, NISQA, and Spk metrics for TTS tasks. The codec's reconstruction quality also does not demonstrate substantial improvements.
4. Figure 3 indicates that CL drop does not enhance performance and may be influenced by training biases.
5. There is the need for an analysis of codebook utilization similar to that found in the WavTokenizer paper in Table 2, particularly concerning frame duration and codebook space design.
6. As plug-and-play tricks, it is essential to validate its effectiveness across various codec models or assess its adaptability within different codec TTS architectures (e.g., diffusion models, VALL-E). Alternatively, the introduction of additional baselines could be beneficial.

In conclusion, I eagerly anticipate witnessing innovative paradigm designs in the acoustic codec domain (e.g., Encodec, SpeechTokenizer) or models that deliver remarkable results (e.g., DAC). However, this current work still exhibits certain gaps that need to be addressed.

This is purely an investigation paper without theoretical or architectural advancements. I also think that the observation from point (3) is within expectation and does not bring much impact to the community. While point (1) can be regarded as an architectural improvement, more exploration would be required to be fully convincing.

• The framewise encoder fundamentally limits the codec quality and sets an upper bound on the quality achievable by the codec-based TTS model. Then, could it be possible that better training of codec-LM (better hyperparameters, better model architecture, or just more training compute) could reverse the observation that "framewise encoder is better"? If so, could you add further discussion on this concern?
• Please include WER, NISQA, and Spk Sim for codec reconstruction in Tables 1 and 2.
• Please add standard deviations for all rows in Tables 1 and 2. It is odd to include them only occasionally.
• For completeness, I’d suggest adding the setting where the encoder is not framewise, and the decoder is framewise for Table 1. Without this, the statement in Section 6, “The all-frame setting, … suggesting that framewise receptive fields (see Sec. 4.1) should not be applied to the codec decoder” is not fully supported.

Minor comment:

• It is not appropriate to report speaker similarity in percentages because it is not a notion ranging from 0 to 100. You can simply report the number as is (similar to Mel-L1).

1. The techniques primarily focus on the multi-codebook tokenizer structure in RVQ. However, their effectiveness, particularly when employing the framewise codec encoder and longer frame duration approaches, requires further validation when applied to single-codebook speech tokenizers in combination with LMs.
2. The use of LM codebook level dropout (CL drop) and longer frame duration are common techniques in both speech LMs [1,2] and codec models. Presenting them as key contributions may lack sufficient innovation.
3. The paper's Table 3 presents only four permutations of the three techniques. The remaining two combinations should also be explored and reported to provide a comprehensive understanding of the effects.
4. The experiments are predominantly focused on TTS applications, limiting insight into how the techniques perform in other audio generation tasks. Additionally, these experiments are mostly conducted on a single codec architecture variant, potentially restricting the generality of the findings.

[1]:Wang C, Chen S, Wu Y, et al. Neural codec language models are zero-shot text to speech synthesizers[J]. arXiv preprint arXiv:2301.02111, 2023.

[2]:Borsos Z, Sharifi M, Vincent D, et al. Soundstorm: Efficient parallel audio generation[J]. arXiv preprint arXiv:2305.09636, 2023.

1. Current Codec training also uses layer drop, similar to the CL Drop proposed by the authors. Perhaps the use of layer drop during Codec training will also have an impact on the effectiveness of Codec LM. The author could explore this further.
2. The trick of increased codec frame duration is not novel, and it is just an engineering trade-off. In my opinion, the efficiency comparison in the paper should not consider different frameshift, because this is a definite efficiency improvement.
3. The entry point and approach make sense, however, my main concern is that there is no apple-to-apple comparison. The paper uses stacked Mamba2 and Transformer blocks, instead of existing TTS architecture such as VALL-E or AudioLM. In addition, the evaluation of audio quality did not report the results of PESQ or VISQOL. I hope to see the efficiency and effectiveness compared with existing works under the same model architecture with these tricks.