Audio Large Language Models Can Be Descriptive Speech Quality Evaluators

Thank you for the time and effort you've put into reviewing. Based on your valuable feedback, we have summarized the following issues and responded to each one to address your concerns.

Q1: The question about formulate 1 and formulate 2.
Thanks for your question. We would clarify that formulate 2 is also token-level. The probabilities of entire sequences $\pi (y|x)$ are products of token-level probabilities. When we apply the logarithm, the product of probabilities becomes a sum of log probabilities for each token. Expanding formulate 2 using this role, the KL divergence is a token-level constraint applied to each token's probability difference between the model and the reference, accumulated over the sequence.

Q2: Why does the proposed ALLD give improved results over Full-FT?
The benefits brought by ALLD stem from two main reasons:
(1) Token-level distillation. We observed that after SFT, the diversity in response language from the audio LLM was not sufficient, particularly in A/B testing tasks. This was mainly due to insufficient data and the comparatively weaker language description ability of audio LLMs compared to standard LLMs. Therefore, token-level distillation uses an external LLM to further enhance its descriptive capabilities, improving the quality of the generated descriptions.
(2) The exposure bias problem. Due to the teacher-forcing algorithm, the model predicts the current token based on ground-truth sequence during training, while having to depend on its own predicted sequence during inference due to the unavailability of ground truth. This mismatch results in exposure bias, and the RL-based ALLD approach contains a sampling process that can alleviate this issue, and is widely used in sequence generation tasks [1-3].
Additionally, the motivation of this work is that we found mainstream audio LLMs can not perceive all details of the audio and perform speech quality evaluation, due to the lack of suitable training corpus. Based on this, we introduce this corpus based on the existing human rating dataset, which is suitable for audio LLM learning.

We would be delighted to receive any additional suggestions or comments during discussion phase.

Reference

[1] Bahdanau D, Brakel P, Xu K, et al. An actor-critic algorithm for sequence prediction[J]. arXiv preprint arXiv:1607.07086, 2016.
[2] Rennie S J, Marcheret E, Mroueh Y, et al. Self-critical sequence training for image captioning[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 7008-7024.
[3] Prabhavalkar R, Sainath T N, Wu Y, et al. Minimum word error rate training for attention-based sequence-to-sequence models[C]//2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018: 4839-4843.

Thank you for the time and effort you've put into reviewing. Based on your valuable feedback, we have summarized the following issues and responded to each one to address your concerns.

Q1: The decision criteria for generating preference judgments.
To faithfully clarify, the proposed our method does NOT require preference judgment. In ALLD, the description generated by the LLM based on meta information is always considered a positive example, as the meta information is derived from the ground truth of NISQA, and the LLMs’ descriptive ability is accurate and strong. Conversely, the description produced by the audio LLM from raw audio is treated as a negative example, as it must extract information from the raw audio, making it impossible to be more precise than the ground truth-based description. Thus, although our method is inspired by DPO, it can skip the preference judgment step. A similar setting can be found in SpeechAlign [1]. In addition, We would like to clarify that ALLD is not a form of "preference optimization." Instead, we found that the audio LLM exhibits some shortcomings (e.g., lack of descriptive diversity) when learning quality evaluation, particularly in A/B test tasks. Therefore, our goal with ALLD is to align the output of the audio LLM with that of the LLM, rather than selecting a better one between them. By leveraging token-level distillation, ALLD were able to achieve this alignment effectively.

Q2: A potential bias in the interpretation of the BLEU scores.
Thank you for your insightful question. The model we used for the generation was Llama 3.1-70B, as we found it produced the highest quality among models of 70B size. For distillation, we used Qwen2-7B—primarily to save memory during training, and also because the tokenizers of Qwen2 and Qwen-Audio2 are inherently aligned, which facilitates token-level distillation. Additionally, we also experimented with using GPT-4 to generate the descriptions and found that the generated descriptions were similar to those of Llama 3.1, likely because we included the scoring criteria and in-context learning examples in the input.

Q3: Empirical support for "Human listeners cannot distinguish between original and edited speech produced by VoiceCraft."
Thank you for your question. We refer to the results in paper [2]. In Figure 1, human listeners preferred VOICECRAFT-edited speech over the original real recording 48% of the time in side-by-side naturalness comparisons. Specifically, in the 1-2 word results shown in Figure 3, VOICECRAFT accounted for 39.68%, raw speech for 40.95%, and Neutral for 19.37%. Since most of the edits in our work only contain a single word, this result provides meaningful evidence for this view.

We would be delighted to receive any additional suggestions or comments during discussion phase.

Reference
[1] Zhang D, Li Z, Li S, et al. SpeechAlign: Aligning Speech Generation to Human Preferences[J]. arXiv preprint arXiv:2404.05600, 2024.
[2] Peng, Puyuan, et al. "Voicecraft: Zero-shot speech editing and text-to-speech in the wild." arXiv preprint arXiv:2403.16973 (2024).

Thank you for your detailed and thoughtful responses to my queries. I have updated my score based on these responses and commend the authors for their work. I look forward to seeing these additional details incorporated into the paper for further clarity and transparency.

Thank you for the time and effort you've put into reviewing. Based on your valuable feedback, we have summarized the following issues and responded to each one to address your concerns.

Q1: Explain the reason for lower LCC and SRCC for LIVE and P501 datasets.
Thank you for your question. The lower LCC and SRCC are compared with their in-domain (test-sim) performance, primarily because the speech from these two subsets was not included in the training set. Since their speech domains are different, a slight drop in performance is inevitable.

Q2: How to generate descriptive text for audio data maintaining uniformity.
Thank you for raising this important topic. We would like to share some observations and insights to address your concern:
(1) The demonstrations for in-context learning are useful for uniformity. As mentioned on line 205, we provide several golden examples (e.g., example in line 072) to guide the LLM in understanding the expected format of the generated responses. Additionally, the text-based scoring criteria are provided as input to the LLM for reference. As a result, the language style generated by the same LLM tends to be consistent.
(2) We have also considered the issue of subjectivity in language descriptions. Initially, our approach was to use different LLMs for the descriptions to simulate linguistic diversity. However, we found that even though this task may seem straightforward, only Llama 3.1 (among models of around 70B parameters) performed well enough. GPT-4 also produced high-quality text, but due to cost considerations, we ultimately opted for an open-source model.
(3) In our subsequent research, we design a mechanism to increase sample encourage diversity in the generated responses by adjusting the top-p and temperature factor during LLM inference. We generated a total of five variations to simulate five different annotators. The results of MSE, LCC and SRCC metrics remained largely consistent with previous results (0.17, 0.93, 0.93), while BLEU scores showed a slight improvement (average 26.4 on 5 test sets).

We would be delighted to receive any additional suggestions or comments during discussion phase.

Thank you for the time and effort you've put into reviewing. Based on your valuable feedback, we have summarized the following issues and responded to each one to address your concerns.

Q1: the MOS should be an integer between 1 and 5.
Thank you for your question. In the MOS prediction task (dataset), the ground truth MOS is the average score given by multiple listeners. This setting stems from the subjective variation of human perception, the MOS scores for the same sample could vary, thus averaging these scores provides a more objective reflection of speech quality. Moreover, almost all existing MOS prediction works [1] focus on predicting the decimal MOS score. Therefore, aligning with these works provides quantitative references and reproducible comparison.

Q2: Lack of Novelty: Prior works, such as SALMONN, have already covered a wide range of audio understanding tasks.
As we highlighted in the Abstract (line 14), speech quality evaluation has been ignored in these audio LLMs, which would be due to the lack of suitable datasets. Additionally, in the main text (line 225) and Appendix C, we demonstrated that even with in-context learning, a series of audio LLMs are not able to evaluate speech quality, including SALMONN. Based on this, we introduce a dataset suitable for audio LLM learning and also propose new methods to improve the learning effectiveness. Therefore, both the motivation and contributions of our work are distinct from those of works like SALMONN, and we believe it fits well with ICLR, a conference that encourages innovation.

Q3: ALLD contains STF and RL.
From a rigorous perspective, ALLD cannot be defined as a typical RLHF method—we defined it as a “distillation method” to align the behavior of the audio LLM with that of an LLM (line235). We also explain its differences from traditional RLHF (starting from line 243). Apart from the reference model mentioned, the positive samples used in the optimization process are ground-truth rather than sampled by the model itself. In other words, ALLD is essentially an enhanced version of SFT, motivated by the goal of improving the quality of the responses generated by the audio LLM rather than to optimize its generation preferences.
Additionally, even in other RLHF methods, RL is not the only needed in RLHF. The importance of using the SFT as an auxiliary module has been explored in [2], and the open-source implementation of DPO includes a hyperparameter “rpo_alpha ”for SFT loss (as seen here: https://huggingface.co/docs/trl/dpo_trainer). Thanks for your question, and we will clarify further in the next version.

Q4: Unclear about "Full-ft" and "SFT".
We would like to clarify that “full-FT” and “SFT” are not the same concept. “Full-FT” refers to fully fine-tuning all the parameters of the model. It highlights “how many parameters are trainable”, as listed alongside LoRA tuning and encoder-only tuning in Table 1. On the other hand, “SFT” refers to supervised fine-tuning with labelled data, which is a primary method for enabling LLMs to follow instructions. We will add further clarification to avoid confusion.

We hope the above discussion has clarified the perspectives the reviewer was interested in exploring further. We would be delighted to receive any additional suggestions or comments.

Reference
[1] Lo C C, Fu S W, Huang W C, et al. Mosnet: Deep learning based objective assessment for voice conversion[J]. arXiv preprint arXiv:1904.08352, 2019.
[2] Pang R Y, Yuan W, Cho K, et al. Iterative reasoning preference optimization[J]. arXiv preprint arXiv:2404.19733, 2024.

Thank you for addressing some of my concerns. I understand that previous audio LLM studies did not include audio quality evaluation tasks. However, the methodology for this task seems similar to other tasks, offering limited new ideas and insights. With that being said, I still appreciate the effort of this work. I would like to improve the score to 5, leaving room for improvement to meet my higher expectations for novelty in this work.