Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedback

Q6: Different proxy for rewards to optimize the WER and SIM.
Thank you for your question. Yes, recognizing the importance of positive-negative selection, we have experimented with various selection rules, including WER and SIM. The results indicate that intelligibility and quality are highly positively correlated in VOICECRAFT's synthesized speech. Specifically, cases with MOS below 3 tend to have high WER, while samples with high MOS typically exhibit low WER. However, in practice, using WER+MOS or SIM+MOS as selection criteria did not result in significant performance improvements. That said, once quality is ensured, the proxy can be customized based on specific objectives. For example, in our results on emotional TTS, we employed valence or arousal metrics as proxies to optimize the emotional expressiveness of the synthesized speech.

Q7: Generalization to in-the-wild data.
Thanks for your question. The last 8 samples shown on our demo page (https://uno-tts.github.io/listening-examples/) are the emotional TTS experiments generated from totally unseen speakers with unseen emotions (from ESD[1]). UNO can improve the Valence+MOS and Arousal+MOS.

Reference
[1]Zhou K, Sisman B, Liu R, et al. Seen and unseen emotional style transfer for voice conversion with a new emotional speech dataset[C]//ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021: 920-924.

Thank you for the time and effort you've put into reviewing. Based on your valuable feedback, we have summarized the following issues and responded to each one to address your concerns.

Q1: Proxy model vs. reward model.
Thank you for your question. Let us first clarify the role of the simulation proxy model: its primary function is to provide a binary classification of "desirable" or "undesirable." Ideally, this task would be carried out by human annotators, with uncertainty represented by the variance among their judgments. However, due to the substantial labor demands, we used either EDL or I-CNF as a proxy. As noted by the reviewer, current MOS prediction models often struggle with generalization when applied to newly developed TTS systems, as the publicly available MOS prediction datasets are typically based on older, more traditional TTS models. Consequently, we did not use these models as reward models for generating precise numerical values. Instead, we applied a threshold to convert their outputs into binary classifications. In this capacity, they are reliable; for instance, synthesized speech with a MOS score below 3.0 is very likely a poor case, while scores above 4.5 generally indicate near-perfect quality.

Q2: Generalization of the proxy model.
As we explained in Q1, the proxy model is not accurate but suitable to be a binary classifier. We also have human listeners to determine "desirable" or "undesirable" in Table 2, which shows the efficacy of the proxy model.

Q3: Whether the improvements come from using the LibriSpeech dataset or RLHF?
Thanks for your question. We believe the improvement comes from RLHF post-training. The reason is that we did not use any “labels” (which are ground-truth speech audios here) from the LibriSpeech dataset. Actually, RLHF finetuning learns from human feedback instead of labels, which is its key difference from SFT.
Furthermore, the 830M model can be viewed as the STF baseline, because it adds part of the LibriTTS model in its training set (https://huggingface.co/pyp1/VoiceCraft_830M_TTSEnhanced), which archives 4.30 zero-shot performance. However, UNO and UNO-RI improved it to 4.41 and 4.55, respectively.

Q4: Novelty of UNO on TTS task.
Thank you for your suggestion. We agree with your perspective that UNO can be viewed as a TTS-specific adaptation RLHF method. In particular, we have carefully designed the methodology to enhance sampling efficiency by exposing the model to a greater diversity of speakers, which significantly improves the robustness and performance of the SFT TTS model. We believe our contributions are specifically tailored to the TTS domain, providing sufficient novelty in this context.

Q5: Powerful TTS models exhibit strong diversity in synthesized speech.
We appreciate the reviewer’s insightful question and would like to clarify two distinct meanings of the term “diversity” in this context. State-of-the-art zero-shot TTS models can generate natural and diverse speech by mimicking a speaker's voice characteristics and speaking style based solely on the input speech prompt (typically the first three seconds of speech). The diversity here in the generated speech arises from the evidence provided in the speech prompt, which can be expressive enough to convey emotional speech. However, this type of diversity is different from the diversity involved in sampling pairwise positive-negative examples based on the same speech prompt.
In practice, we observed that for a given input speech prompt, the generated samples tend to be either all positive or all negative, with only minor prosody variations. This is because, given the prosody in the initial three seconds and the text to be spoken, the pronunciation and prosody of the subsequent speech are generally predictable. Consequently, it is rare to find training samples that share the same speech prompt and text inputs but exhibit significantly different pronunciations or prosodies.

Thank you for the time and effort you've put into reviewing. Based on your valuable feedback, we have summarized the following issues and responded to each one to address your concerns.

Q1: The diversity of the zero-shot TTS system.
We appreciate the reviewer’s insightful question and would like to clarify two distinct meanings of the term “diversity” in this context. State-of-the-art zero-shot TTS models can generate natural and diverse speech by mimicking a speaker's voice characteristics and speaking style based solely on the input speech prompt (typically the first three seconds of speech). The diversity here in the generated speech arises from the evidence provided in the speech prompt, which can be expressive enough to convey emotional speech. However, this type of diversity is different from the diversity involved in sampling pairwise positive-negative examples based on the same speech prompt. In practice, we observed that for a given input speech prompt, the generated samples tend to be either all positive or all negative, with only minor prosody variations. This is because, given the prosody in the initial three seconds and the text to be spoken, the pronunciation and prosody of the subsequent speech are generally predictable. Consequently, it is rare to find training samples that share the same speech prompt and text inputs but exhibit significantly different pronunciations or prosodies.

Q2: Q2: WER of VALL-E-UNO.
We apologize for any misunderstanding caused. Here is a typo and the real result should be 15.8 rather than 157.8, and we will correct it in the next version.

Q3: Could you provide experimental results to support UNO’s application to other diffusion-based TTS models?
We thank the reviewer for the suggestion. As stated in the limitation section, we believe UNO may be feasible for TTS models with diffusion. However, current zero-shot TTS models have mostly used the autoregressive approach, which is often not yet widely combined with diffusion. Moreover, UNO, as a reinforcement learning approach, benefits from alleviating the exposure bias issue caused by error propagation in autoregressive decoding. Current TTS methods with diffusion are mostly non-autoregressive, which may not suffer as many instability issues as the autoregressive ones.

Q4: Provide comparative statistics on these failure rates?
Thanks for your question. As we reported in Section 6: If we define WER>15% as “failure rates”, then the 330M drops from 18.5% to 4.9%, and the 830M mode drops from 4.6% to 1.9%. With UNO-RI, this figuare reduce to less than 1%. A similar reduction is also observed if we use the MOS value as the threshold. However, due to the lower unreliability of MOS estimation, we select WER as this threshold.

We would be delighted to receive any additional suggestions or comments.

Q4: Why does real human feedback lead to poorer performance?
This phenomenon occurs because, unlike the fixed threshold used for the EDL model, we could not establish a consistent threshold for human raters. Instead, listeners were instructed to select two desirable and two undesirable samples from a batch of four, which resulted in less effective selection. Furthermore, since each audio sample was evaluated by only three listeners, the uncertainty calculation was relatively imprecise, as explained in Appendix E.
Additionally, we found that sometimes the model itself is better suited as a judge of synthesized speech than human listeners. As we did in Section 6.4, when people were unable to determine the quality difference between two synthesized speech samples, using the TTS model itself for reverse inference often led to selecting better positive samples.

Q5: Clarity on the accuracy of EDL and I-CNF.
For mean MOS prediction, we have highlighted that no model currently provides accurate MOS values due to the rapid progress made in developing TTS systems. Therefore, we use these models as binary classifiers to filter representative examples. Regarding uncertainty, we visualized a comparison of I-CNF's uncertainty predictions with other methods in Appendix G. Combined with the metrics presented in the main text, this evidence demonstrates that I-CNF is the state-of-the-art method for estimating MOS uncertainty.

Q6: A TTS model optimized for a high naturalness score might still struggle with intelligibility, leading to possible issues in accurately mapping the generated speech to the text content.
We apologize for any misunderstanding caused. In VoiceCraft's synthesized speech, intelligibility and quality are highly positively correlated. Samples with a MOS below 3 typically exhibit high WER, while those with high MOS generally have low WER. The reviewer can refer to our listening demo page, where samples with improved quality do not show a decline in intelligibility. Furthermore, during our initial explorations, both WER and SIM scores were considered for sample selection. However, SIM was found to be unreliable, and combining WER with MOS as a standard did not result in significant performance improvements.

Q7: Highlighting any key differences or improvements between UNO and KTO, and the contributions of incorporating uncertainty.
As stated in line 390, our approach is inspired by KTO—a method that does not rely on pairwise comparisons when designing the reference point. Using a reference point has been a common approach in reinforcement learning optimization since its introduction in the 1990s [1]. Even in recent years, for optimizing sequence generation, subtracting a reference point from the reward to stabilize training remains a standard practice [2]. Additionally, the differences between our method and KTO include:

• KTO primarily focuses on addressing the bias inherent in human feedback, particularly the well-documented tendency for humans to be loss-averse. To account for this, they incorporate a loss aversion coefficient to model this aspect of prospect theory. However, this approach was not effective in our experiments, leading us to omit this setting.
• UNO set beta is related to the uncertainty of each data point, and the importance of dynamic beta is also explored in paper [3]. The first advantage of a dynamic beta is filtering to safeguard against the influence of outliers. As the EDL and I-CNF models provide a rough judgment of whether a sample is desirable or undesirable, if an ambiguous sample (typically characterized by high uncertainty) appears, its high uncertainty prevents the model from over-adjusting towards this outlier during the learning process.
• The second advantage is shown in the comparison with UNO-null. Though UNO-null works well with the 330M model, however, it fails to improve the performance of the 830M model (Figure 4). This is mainly due to the 830M model's ability to produce relatively stable sampling results. (Typo here) Both EDL and I-CNF could provide sufficient differentiation in these cases. However, a dynamic uncertainty metric allows for further quality improvement.

We hope that the above discussion can clarify the reviewer's misunderstandings and address proposed concerns. We would be delighted to receive any additional suggestions or comments.

Reference
[1] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. In Machine Learning, pages 229–256, 1992.
[2] Rennie S J, Marcheret E, Mroueh Y, et al. Self-critical sequence training for image captioning[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 7008-7024.
[3] Wu J, Xie Y, Yang Z, et al. $\beta$-DPO: Direct Preference Optimization with Dynamic $\beta$ [J]. arXiv preprint arXiv:2407.08639, 2024.

Thank you for the time and effort you've put into reviewing. Based on your valuable feedback, we have summarized the following issues and responded to each one to address your concerns.

Q1: Question about EDL and I-CNF, and the selection of positive and negative examples.
In short, EDL and I-CNF predict a MOS distribution (mean and variance). The SOMOS dataset includes the individual MOS scores from each listener, which supports these models in predicting the distribution. Regarding the positive and negative selection, we provide the following observations and discussions to address your concern:
(1) Although the predicted means of EDL and I-CNF may not serve as accurate MOS values due to the mismatch between their training data, produced by older TTS systems, and the outputs of recent TTS systems, we found that they are well-suited to function as binary classifiers for identifying positive or negative samples. For instance, when the predicted MOS is below 3.0, the synthesized speech usually has various issues, whereas samples with a score above 4.5 generally sound nearly perfect. Therefore, we use them to automatically identify negative samples.
(2) In the experiments of the 330M model, we set thresholds of 3.5 and 4.0 to determine whether a sample was “desirable” or “undesirable”. For samples whose MOS scores are from 3.5 to 4.0, EDL and I-CNF may not be able to provide meaningful differentiation. Fortunately, the optimization algorithm is not sensitive to these intermediate samples—they are usually observed relateive high uncertainty to limit the model update. Therefore, whether they are included in the positive or negative sample pool has little impact on the final outcome.
(3) In the 830M-model experiment, we found that the synthesized speech quality was generally very high. Therefore, we raised the two thresholds to 3.9 and 4.4. Under these conditions, we observed that uncertainty could provide additional differentiation. Specifically, we aim for the model to optimize towards high MOS scores while maintaining low uncertainty. Therefore, UNO surpasses the UNO-null due to the contribution of uncertainty.
(4) In Section 6.4, we further explored a method for refining the sample selection when a large number of samples are available—namely, reverse inference. Using this approach, we could further identify the truly "desirable" examples, ultimately resulting in improved outcomes of 830M model. In general, the selection criterion of "desirable" and “undesirable” is flexible: For the 330M model, it’s important to filter out those bad cases to improve the robustness. For the 830M model, more strict rules are needed for selecting real "desirable" samples.

Q2: The question about emotional TTS and listening examples.
The model we used for the prediction of valence and arousal is in the footnote of page 19: (https://github.com/audeering/w2v2-how-to). As we mentioned, when the 830M model can synthesize speech with high MOS, the emotional TTS actually adds more rules for sample selection—we want the model to be optimized toward the direction with high MOS and high valence (or arousal). The motivation of experiments is to show the flexibility of our optimization strategy. When the quality issue is solved, UNO can also be used for preference optimization. Please find the listening examples on our demo page in the footnote of page 8 (https://uno-tts.github.io/listening-examples/).

Q3: Evaluation discrepancies for Voicecraft.
We would like to clarify the confusion here. The 4.17 MOS reported in VoiceCraft is evaluated with their 830M model. Our main experiments were conducted using the 330M model, and we also have the experiments on the 830M model, the MOS score of zero-shot synthesis is 4.30, even better than the number reported in the original paper. Additionally, the 830M model actually is enhanced by a part of LibriTTS, while the 330M model we used is only trained by Gigaspeech. Therefore, the proposed method handles the slight domain mismatch and unseen speakers, achieving similar performance with the 830M model without any ground-truth speech on LibriTTS.

Thank you for your response. However, it does not provide the accuracy of EDL and I-CNF. I understand that there may be some mismatch between their training data and your synthesized data, but this is the potential issue of whether EDL and I-CNF can correctly predict human-like annotations and human’s associated uncertainty of your synthesized data. Furthermore, as you noted, the EDL model relies on a fixed threshold, whereas human listeners do not. We don’t know the accuracy of EDL and I-CNF on predicting human feedback and your UNO-Human experiment demonstrates that the EDL model behaves differently from human listeners. it is questionable whether UNO truly enhances zero-shot TTS with human feedback as you claimed.

Additionally, I still have questions regarding the contribution of your work. As you noted, your method is inspired by KTO and is quite similar to it. One key difference is the inclusion of uncertainty, achieved by setting a beta parameter related to the uncertainty of each data point. However, your experiments do not clearly demonstrate whether the inclusion of uncertainty significantly enhances performance compared to a method without uncertainty:

1. UNO-null can improve the 330M model just like UNO-ICNF
2. For the 830M model, neither UNO-null nor UNO-ICNF results in a significant performance improvement (as shown in Figure4)
3. You state that neither EDL nor I-CNF provides sufficient differentiation in these cases.

Given these observations, how can you confidently claim that the inclusion of uncertainty significantly enhances performance? What is the contribution of your method?

Thank you for your response. We would clarify that The EDL and I-CNF serve as Proxy model to reduce dependence on human resources. In this case, the criterion for determining whether they are qualified lies in whether they can improve the optimization performane, and the comparative exprement with real annotator also validates their effctiveness. The threshold they use is because as neural networks, they have to use scores to show the preferences, unlike annotator can directly indicate which one is better. But there is essentially no difference between these two judgment methods. For example, if we employ a language model as a judgment model to perform online perference optimization. Then, if this LM can achieve similar (even better) results as real annotator, we believe it is a qualified proxy, and it is NOT impotant whether LM use a socre or directly indicate a candidate.

We apologize for the misunderstandings caused by "neither". What we intended to declare was that they could provide sufficient differentiation (sorry for the typo in the last round response). For 830M model, since the sampling results are good enough and the uncertainty can provide further differentiation for these samples. We also highlight that though 4.30 to 4.41 is not a significant MOS improvement, the bad case ratio (WER>15%) drops from 4.6% to 1.9%, further improving the robustness.

Ours main contirbution is to propose a TTS-specific adaptation RLHF method. In particular, we designed the strategy to enhance sampling efficiency by exposing the model to a greater diversity of speakers. As KTO's data format support the setting of this sampling strategy, we remove some ineffective KTO components and introduece the TTS-specific uncertanty, to make the new method effective on different models.

Thank you again for joining the discussion. We would be delighted to receive any additional suggestions or comments.

Thank you once again for your valuable feedback. We believe that our previous responses have addressed your concerns and clarified some misunderstandings. As the discussion phase is coming to a close, we kindly request if you would consider adjusting your rating. At the same time, we welcome any further comments or suggestions you may have, and we will do our best to address them before the conclusion of the discussion phase.

Thank you for the time and effort you've put into reviewing. Based on your valuable feedback, we have summarized the following issues and responded to each one to address your concerns.

Q1: The performance of VALL-E-UNO is unreasonably bad.
Thank you for pointing out this typo. The correct result should be 15.8, not 157.8, we will correct it in the next version.

Q2: the definitions of EDL and I-CNF.
Thanks for your advice, we are willing to include more details to improve its readability.