SpeechAlign: Aligning Speech Generation to Human Preferences

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. Your insights and comments have been invaluable to refining our research. Our responses are as follows:

Q1: How does the distribution gap relate to SpeechAlign?

There is a distribution gap between golden AR tokens and synthetic AR tokens, which adversely impacts the performance of the TTS model. SpeechAlign calibrates the output of codec language models to the golden codec distribution and bridges the distribution gap by preference optimization, which brings performance improvements.

Q2: Why Best-of-N Sampling (BoN) is at the same level of PO methods?

Here, our BoN selects the sampled result with the highest reward model score as the output. The aim is to utilize a reward model trained on preference data to align large language models with human preferences, thereby enhancing output quality. Essentially, it is a form of decoding-time alignment, which is why we have placed it in this section.

Q3: How are the training parameters (lr, bs) chosen?

For the design of the learning rate, we set approximate magnitudes of the learning rate for different models based on other concurrent RLHF works and our own training experience. Subsequently, we conducted experiments with different learning rates to ensure that the results provided in this paper are optimal. For selecting the batch size, it should be as large as possible to accelerate training, under the condition of ensuring that the GPU memory is not fully occupied.

Q4: The vocoder that generates speech from the acoustic tokens.

Sorry to confuse. The vocoder we use is the pre-trained decoder of SpeechTokenizer[1] model. We'll add that in our paper.

Q5: Will the PO datasets be shared ?

Yes, we will open source all these datasets for further research.

[1]Zhang, Xin, et al. "Speechtokenizer: Unified speech tokenizer for speech large language models." arXiv preprint arXiv:2308.16692 (2023).

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. Your insights and comments have been invaluable to refining our research. Our responses are as follows:

Q1:The description of the preference optimization algorithms.

Thank you for your valuable feedback. We apologize for any confusion caused by the descriptions of the preference optimization algorithms. Due to space constraints in the paper, we did not provide a detailed explanation of these methods. We will offer a detailed explanation in the subsequent versions of the paper. Brief introductions of each algorithms are as follows:

Chain-of-Hindsight (CoH): CoH improves the quality of responses by constructing a "chain of hindsight" that enables the model to learn from past experiences, especially user feedback. The CoH technique involves converting all types of responses into a sequence of sentences and fine-tuning the model by leveraging the language model's reading comprehension abilities. For instance, in the paper lines 159-162, we design different prompts for positive and negative samples during training, and during inference, we use the prompt for positive samples to guide the model in generating correct responses.

RLHF-PPO: RLHF-PPO is a reinforcement learning method based on human feedback. It involves training a reward model on preference data to score responses based on quality and then training a policy model (the model requiring RLHF) to generate responses that maximize rewards. This process is complex and often unstable.

Direct Preference Optimization (DPO): DPO is a new method for optimizing language models based on human preferences. Unlike RLHF, DPO does not rely on explicit reward modeling or reinforcement learning. DPO works by increasing the log probability of preferred samples and decreasing the log probability of non-preferred responses. In contrast to traditional methods that use a preference model to train a reward model and then train a policy based on that reward model, DPO directly defines preference loss based on the policy.

Q2:The variance of evaluation results.

Thanks for the reminder. We provide the variance here and we will add the variance score of each result to our final paper.

Q3:The meaning of reference model.

The reference model is initialized with a model trained through SFT, and typically, it is the same model as the policy model. The reference model remains frozen and does not participate in training. Its role is to ensure that during the reinforcement learning process, the distribution disparity between the policy model and the reference model should not be too large.

Thank you for your kind response. Regarding Q2, I realize I may have misunderstood the point. Nonetheless, I appreciate the experiments you conducted to increase the data. Additionally, thank you for the comparison with current high-performing models in response to Q3. I will raise the score to 5 points.

I have an additional question: Is your methodology applicable to non-autoregressive state-of-the-art models (e.g., NaturalSpeech, Voicebox, SoundStorm, HierSpeech++, etc.)? If so, have you attempted to apply it to any of these models?

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. Your insights and comments have been invaluable to refining our research. Our responses are as follows:

Q1: Which attributes cause this distribution difference between golden AR tokens and synthetic AR tokens?

The distribution gap between golden AR tokens and synthetic AR tokens arises from several factors during the training of the AR model. Insufficient training, limited data availability, or constraints imposed by cross-entropy can prevent the model from accurately learning the distribution of golden AR tokens. Consequently, the model lacks the capability to generate AR tokens that fit within the golden distribution. It is important to note that this issue is unrelated to our use of SpeechTokenizer tokens.

Q2: Scalability and Dataset Size

Our baseline model, SpeechAlign-sft, was trained on the large-scale dataset Multilingual LibriSpeech, which encompasses 40,000 hours of speech data. SpeechAlign primarily enhances speech language model performance through preference optimization, which typically requires significantly less data than what is used for pretraining. To examine scalability, we conducted experiments on the GigaSpeech dataset using SpeechAlign-DPO. The results, as shown below, indicate that increasing the dataset size can lead to further performance improvements:

Q3: Baselines

SpeechAlign is not orthogonal to other zero-shot TTS approaches; rather, it can be applied to enhance any zero-shot TTS method and brings iterative self-improvement to them. Thus, the performance of SpeechAlign is closely linked to the base zero-shot TTS approach. In our follow-up work[1], SpeechAlign was applied to Voicecraft, a state-of-the-art personalized speech synthesis method, showing promising improvements.

The comparative results with some current state-of-the-art Zero-shot TTS systems are as follows:

[1] Chen C, Hu Y, Wu W, et al. Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedback. arXiv preprint arXiv:2406.00654, 2024.

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. Your insights and comments have been invaluable to refining our research. Our responses are as follows:

Q1: Performance Gap: The performance of the proposed method still falls short of the ground truth.

We acknowledge that there is still an inevitable distribution gap between the synthetic AR tokens generated by our model and the ground truth AR tokens, as illustrated in Figure 1(b). Despite the optimization by SpeechAlign, which brings the distribution of synthetic AR tokens closer to that of the ground truth AR tokens, they do not completely overlap. This remaining distribution gap contributes to the performance gap between the proposed method and the ground truth. Although we conclude in Section 5.1 that Iterative Self-Improvement has an upper bound, we believe the reason is that as the number of iterations increases, the negative samples are of too high quality and too similar to the positive samples, which increases the difficulty of reward modeling and thus degrades the performance of the reward model. However, we believe that by increasing the number of parameters and the amount of data for the reward model during the iterative PPO process, the reward model can maintain strong capabilities. Consequently, the model's performance can continuously improve and approach the ground truth following preference optimization.

Q2: Limited Scope: The method is specifically tailored to the VALL-E based TTS model.

The core of SpeechAlign lies in utilizing ground truth samples and synthetic samples to construct a preference dataset, which, combined with preference optimization, allows the model to continuously self-improve. This approach is not limited to the VALL-E based TTS model, which comprises AR and NAR models. In [1], SpeechAlign was applied to an AR-only TTS model under the Voicecraft framework, resulting in promising performance improvements. This demonstrates that SpeechAlign is not exclusively tailored to the VALL-E based TTS model.

Q3: Hyperparameter Tuning:

For all the baseline systems, different hyperparameters were tuned to ensure the results presented in the paper were optimal.We tuned the hyperparameters for all baseline systems to ensure that the results reported in the paper represent their optimal performance.

Q4: Reward Model Accuracy

We selected 200 speech samples from VCTK test set to measure the reward model's accuracy, which resulted in an accuracy score of 0.87.

Q5: Iterative Training: Why does only DPO have iterative training?

PPO training exhibits instability and requires considerable time and computational resources to succeed. Therefore, we only conducted iterative optimization on DPO. We plan to include iterative PPO experiments in the camera-ready version if our paper is accepted.

Q6: Generalizability: Can the method generalize to other single-stage codec-based TTS systems like Voicecraft?

In our follow-up work [1], SpeechAlign was applied to Voicecraft, demonstrating improvements in zero-shot TTS. The results are as follows:

Q7: The method is more like a specific method to alleviate the domain mismatch or error propagation of the pipeline codec-based TTS model.

The answer is the same as that of Q2.

Q8: overlooking other high-quality token sequences that can also serve as the chosen samples

SpeechAlign focuses on exploring methods to make speech language models self-improve iteratively without external data, so we don't use other high-quality token sequences in our paper. But the framework is compatible with other high-quality token sequences.

[1] Chen C, Hu Y, Wu W, et al. Enhancing Zero-shot Text-to-Speech Synthesis with Human Feedback[J]. arXiv preprint arXiv:2406.00654, 2024.

Hello, Reviewer. The author has submitted a response to your comments. Whether or not it addresses your concerns, it would be greatly appreciated if you could acknowledge that you have reviewed the reply.