Do Not Mimic My Voice: Teacher-Guided Unlearning for Zero-Shot Text-to-Speech

3. The authors acknowledge that model performance degrades significantly as the number of “forgotten” speakers increases, which raises concerns about the practicality of this approach. The models do not work well when forgot speakers number increase.

We understand the concerns on the practicality. The model performance does change depending on the number of “forget” speakers, as described in L517~519 , often causing differences in required training time for desired results.

The experiment was conducted to forget 10 speaker identities (appx. 15 min per speaker) out of train dataset of 6,736 speakers (appx. 50,000 hours in total). An unlearning approach to remove specific identity in image generation [3] forgets 1 identity image at each experiment on a model pre-trained on 70,000 images of human identities. Another research [1] removes 1 specific concept or art style using 1000 images on Stable Diffusion model pre-trained on 5.85 billion image-text pairs for each experiment. Similar to these settings, in practice, the subjects requiring unlearning will likely be few.

The relationship between model performance and the size of forget dataset is a well-observed phenomenon across all domains of machine unlearning. A recent study [2] investigates this as a challenge in machine unlearning methods in Fig.2a and Table A6. Figure 8 of another study [1] also illustrates that increased size of forget dataset comes with significant limitations. This trend parallels the general observations in AI, where models tend to exhibit diminished performances when trained on smaller datasets.

We fully concur with points raised in your assessment, which had served as the motivation for introducing TGU as compared to SGU. While our work initially implemented a more naiive approach SGU, a method which already had obtained a higher unlearning performance than baselines NG and KL, we were inspired to derive main method TGU. TGU focuses on enhancing training efficiency while ensuring improved evaluation under same conditions. Notably, TGU was able to achieve the highest performance in fixed settings faster while others struggled at maintaining model’s retain scores, underscoring our contribution to this domain.

We agree on your concerns and believe this challenge of forget set size and unlearning performance should be addressed in all unlearning domains. In future work, we aim to prioritize achieving faster training times while accommodating larger forget sets.

[1] Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kaufman, and David Bau. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2426–2436, 2023. URL https://arxiv.org/pdf/2303.07345.

[2] Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, and Sijia Liu. Salun: Em- powering machine unlearning via gradient-based weight saliency in both image classification and generation. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. URL https://arxiv.org/abs/2310.12508.

[3] Juwon Seo, Sung-Hoon Lee, Tae-Young Lee, Seungjun Moon, and Gyeong-Moon Park. Generative unlearning for any identity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.9151–9161, 2024.

We thank the reviewer for taking the time to review our paper and provide valuable feedback. We hope to clarify some questions and are excited to improve our paper with your valuable feedback.

1. The authors appear to misunderstand the concept of zero-shot TTS. In Figure 1, it seems that the model functions when the speaker is included in the training set.

Figure 1 seems to contain some ambiguity in its depiction. The figure was included to emphasize the difference of how machine unlearning becomes a more complex task in zero-shot generative tasks.

Machine unlearning addresses the challenge of ensuring a specific knowledge, concept, or dataset can be effectively “forgotten” by a trained model. In non-zero-shot tasks, where the model is explicitly trained on a specific dataset or domain, unlearning a particular data point typically suffices to prevent the generation of that data point during inference. Hence, the ultimate aim is often to suggest an approximate unlearning method as a pathway to efficiently achieve parameters equivalent to exact unlearning (with the retrained model serving as a benchmark).

However in zero-shot tasks, the situation is much more complex. Zero-shot models possess a broad general knowledge base that enable them to perform tasks without explicit exposure to the corresponding training dataset. This means that simply unlearning the data point is insufficient to prevent a model from generating unwanted content as the model’s inherent zero-shot capabilities allow it to reconstruct similar information. This creates a fundamental limitation in achieving “truly unlearned” model parameters, when the goal is to prevent unwanted data. Thus, an exactly unlearned model is nor a benchmark or a golden standard as can be observed in Table 1.

Our figure focuses on this complexity : Current research in machine unlearning often treats retrained models of exact unlearning as the golden standard. However, we address the challenges posed by zero-shot capabilities in large models in this paper.

The VoiceBox model, which we utilized as a base model, is a Zero-Shot TTS model capable of effectively reconstructing voice styles of speakers not seen in training. Based on thorough understanding of ZS-TTS, we would like to refer to statements in L348~350 that states LibriHeavy was used for training, while LibriSpeech-eval (never seen during training) was employed as the Remain Set for evaluation. These two datasets originate from entirely different corpora and do not share overlapping speakers. Thus, the evaluation includes speakers who were not included in the training set.

2. The only model they used are not open-sourced. Have the authors try their approach on open sourced models?

To address your question, we chose to experiment on a SOTA closed model, than a low-performance open model. There exists models such as YourTTS [1] which adopts a Speaker Encoder-Generation architecture has made its model and training code publicly available as open-source. Unfortunately, due to ethical concerns surrounding potential misuse of the released model (serving as the motivation for our paper), official codes and models for most high-performing ZS-TTS methodologies have not been made publicly accessible.

Thus, significant effort was dedicated to reconstructing a closed source SOTA model to ensure the validity of our experiments and arguments. Our method emphasizes implementation of randomness to remove forget dataset from model parameters and therefore neutralize model’s ability to clone voices. We believe this is an applicable method to be integrated in all ZS-TTS models. We agree it is an essential work to expand our implementation and hope to address this in future work.

[1] Edresson Casanova, Julian Weber, Christopher Shulby, Arnaldo Candido Junior, Eren Gölge, and Moacir Antonelli Ponti. 2022. YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone. In International Conference on Machine Learning, pp. 2709-2720. PMLR, 2022. URL https://arxiv.org/abs/2112.02418.

Thanks to the authors for their response.

Regarding Point 1, while the authors acknowledge the issue with Figure 1, they have made no effort to clarify it. Instead, they provide an extended discussion that is unrelated to my original question and concern.

For Point 2, they state:

"we chose to experiment on a SOTA closed model, than a low-performance open model" "There exists models such as YourTTS [1] which adopts a Speaker Encoder-Generation architecture has made its model and training code publicly available as open-source. " "Unfortunately, due to ethical concerns surrounding potential misuse of the released model (serving as the motivation for our paper), official codes and models for most high-performing ZS-TTS methodologies have not been made publicly accessible."

This argument is simply inaccurate. Numerous high-quality open-source TTS models are available. However, the authors only refer to a low-performing model from 2021, which is now three years old—and soon to be four.

Overall, the authors' response fails to address my concerns in any meaningful way and instead seems to reconfirm them. As such, I intend to maintain my scores.

W1. Evaluation Metrics: The introduction of spk-ZRF is a valuable contribution, as it provides a quantitative measure of the unlearning effectiveness. However, the paper could benefit from a more detailed explanation of how this metric compares to existing metrics in the literature.

We sincerely appreciate your observation. The motivation for proposing spk-ZRF is spread out across the paper, and we will ensure that this aspect is more clearly articulated in the revised version. To elaborate on L282-283, the primary motivation for introducing spk-ZRF stems from the absence of a sufficient existing metric.

Commonly used methods, such as Completeness, JS-Divergence, Activation Distance, and Layer-wise Distance, focus on calculating the distance between behaviors on the remain set and the forget set. These approaches, as exemplified in our Table 1, compare the model performance of the unlearned model with the original model. While low performance on the forget set and bigger gap between remain and forget set is often used as a measure of unlearning success, this alone does not necessarily confirm whether the model has truly removed the forget set's information. For instance, the scenario described in L 222–226 highlights an insufficient unlearning method where a forgotten speaker's voice can be easily reconstructed, yet the model may appear to be well-forgotten if evaluated solely on existing metrics.

Epistemic Uncertainty, another existing metric used in the unlearning domain [1], evaluates how little information about the forget dataset in the unlearned model. However, applying this metric on ZS-TTS models is not suitable as the representations in model layers contain deeply entangled information about speaker identity and the knowledge to generate audible speech from a given text transcript. Consequently, low epistemic uncertainty does not necessarily indicate that the model has successfully forgotten speaker-specific information.

Our focus was to design an evaluation metric specifically targeting the removal of speaker identity information while isolating this from the performance of audible speech generation. For these reasons, we developed spk-ZRF as suggested in this paper, intended to complement existing methods and provide a more targeted approach to evaluation. Thank you once again for your valuable feedback, which has helped us refine our presentation of this concept.

[1] Alexander Becker and Thomas Liebig. 2022. Evaluating Machine Unlearning via Epistemic Uncertainty. URL https://arxiv.org/abs/2208.10836.

We thank the reviewer for taking the time to review our paper and provide valuable feedback. We truly appreciate this opportunity to strengthen our paper with your insights.

Q1. Perform a computational analysis detailing computational costs, training time requirements, comparison of computational overhead with baseline approaches, and inference time and resource requirements.

Compuational Resources

We would like to provide additional details regarding the computational analysis of our study. VoiceBox has model parameter size of approximately 328M.

Training Duration and Performance Considerations

The total training time is shorter in NG and KL only due to the fact they were intentionally halted at 9,500 and 32,500 steps, respectively. This decision was based on our observations that further training led to diminishing returns in terms of model performance and effectiveness in unlearning, with both models struggling to maintain performance beyond these points.

Additionally, it is pertinent to mention that the KL and TGU methods incorporate a teacher model. This inherently extends the training time per step due to the additional computations required.

Inference

Regarding inference, we note that the unlearning methods implemented do not influence the size of the final unlearned models. Consequently, the inference time per sample remains consistent across all methods -0.71 RTF on A100(40GB).

We trust that these clarifications will assist in the thorough evaluation of our work. Thank you for your attention to these details, we have realize the necessity of this analysis in our revision of final paper.

W.3-4 Complexity of Implementation and Limited Scope of Forgetting

The focus on only preventing replication of specific voices may overlook broader implications, such as how unlearning affects overall model performance or its ability to generalize across different tasks. A more holistic approach could provide deeper insights into the trade-offs involved.

We agree that our experiments focused on replication of voice styles, despite ZS-TTS having more applications. To address potential trade-off in model's ability across different tasks and applications, we compare the unlearned model's performance on two tasks : (1) transient noise removal and (2) diverse speech sampling.

(1) Transient noise removal

In real world, ZS-TTS can be used to edit audio. For instance, if an actor were to misspeak a word or unintended noise was introduced during recording, ZS-TTS could be applied to generate clean audio. For this experiment, we purposely construct a speech with noise at a -10dB signal-to-noise ratio over half of each sample’s duration from LibriSpeech test-clean. Then, we prompted the model to generates clean speech for the corresponding noisy segment.

Based on WER and SIM, we can note that the TGU model shows decrease in model performance in reconstructing speech for noisy segments. This performance gap is quite similar to the gap observable in Table 1 of our paper. This is because the model is able to fully mask over the noisy segment and predict to infill with clean generated output. For these tasks, WER is an imperative metric as correct content should be generated to edit audios. However from both Table 1 and this experiment, WER seems to increase as randomness is implemented. While Exact Unlearning and Fine Tuning methods do not reach sufficient unlearning, they are able to maintain low WER scores whereas randomness increases WER with SGU and TGU.

(2) Diverse speech sampling

We also explore diverse speech sampling to measure whether the unlearned model maintains to synthesize different voice styles. Being able to generate diverse speech is also an important feature of ZS-TTS models as it ensures realistic and high-quality speech that resembles natural distributions. This is necessary in applications such as speech synthesis or generating training data for speech related tasks (e.g., Automatic Speech Recognition). The diversity of generated speech samples is measured with Fr´echet Speech Distance (FSD) as suggested in [1]. High FSD indicates lower quality and minimal diversity, while low FSD refers to high quality and more diversity. The experiment is conducted by giving only the text prompt, and no audio prompt of LibriSpeech test-other. Even with no audio prompt, the model should be able to generate diverse and natural speech with distribution similar to real samples. Further implementation details are in Appendix H.2 of revised paper.

In this experiment, TGU unlearned model shows worse quality and diversity distribution in generated speech than the original model. While TGU robustly handles noisier LibriSpeech test-other dataset with WER lower than Original model, the lacking diversity could affect future applications of ZS-TTS. Ideally, removal of specific knowledge should be performed with minimal changes in model's output distribution.

---

Implementing randomness for machine unlearning may result in a degradation of model performance in terms of both content generation accuracy and diversity. We attribute the observed lower performance in WER to the reliance on model-generated targets as guidance in SGU and TGU. If these targets are not generated accurately beforehand, the model is likely to learn erroneous patterns. Additionally, the reduction in diversity appears to be a consequence of removing the influence of the forget dataset, as this effectively reduces the model's overall knowledge. Future work could focus on preserving the diversity of speaker styles while effectively mitigating unwanted generation, potentially through disentangled representations. While we were able to maintain a comparable level of performance and diversity, the trade-offs are likely to intensify as the size of the forget dataset increases.

We thank the reviewer for strengthening our research and would like to note that we have added these experiments in the revised version. Please let us know if we have addressed your concerns, we look forward to your answer.

[1] Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, et al. Voicebox: Text-guided multilingual universal speech generation at scale. Advances in neural information processing systems, 36, 2024.

Thank you for your insightful comments and queries regarding the motivations and practical implications of our approach, particularly concerning the necessity to retrain models to remove certain voices. We appreciate your perspective and acknowledge the need to clarify these points more comprehensively.

1. What is the intended use case of this unlearning scheme?

Under what condition would the user (in this case, the model trainer or machine learning engineer) re-train the model with all the training data to remove some voices? What benefits does this method provide?

In response to your concerns about the conditions under which one would re-train a model to ensure voice privacy, we emphasize the current challenges within the ZS-TTS domain. The majority of ZS-TTS models are kept private, often lacking in publicly available code implementations or pre-trained models due to developers' legitimate concerns over potential malicious use [1]. This secrecy stems largely from privacy considerations, necessitating stringent safeguards before releasing models openly. We also introduce a real life case where a speech generating model led to a lawsuit due to speaker cloning issue of a certain celebrity [2]. We confidently state that methods to prevent model misusage contributed greatly to our motivation of this work. Our proposed method of unlearning certain voice data is actually a necessary step to enhance the practical utility and safety of ZS-TTS technologies.

Since this method requires the entire training data of the original model, 5 minutes of audio for the forget speaker, and more than a quarter of training iterations of the original model, the actual significance of this work is rather limited.

Step size

Our decision to use 145,000 steps is that this constitutes a sufficiently modest number of steps for fine-tuning a Flow-Matching Audio Generation model. In the context of related research, the AudioBox paper [3] employs approximately 200,000 steps to finetune a Flow-Matching SSL model for a Zero-Shot TTS task. SpeechFlow [4] uses 145,000 steps for finetuning a Flow-Matching SSL model. We decided to employ the smaller number of steps 145,000 with regards to the fact that it is considered a modest step size in domain of Zero-Shot TTS. The steps in NG and KL are relatively small only due to the fact they were intentionally halted at 9,500 and 32,500 steps, respectively. With further training, these methods worsened model performance and we plan to update the results for comparison in following days.

Training dataset

Regarding the utilization of the remain set for implementing unlearning in TGU, we regret any ambiguity caused by our initial presentation. To clarify, TGU does not utilize the full dataset originally used for training; they are trained on approximately 50% of the data. Additionally, as noted in the appendix D of our paper, our methods achieved convergence at around 60,000 steps, utilizing only 21% of the total dataset, contrary to the 145,000 steps mentioned. Our decision to use 145,000 steps was intended to constrain experimental settings across all methods.

Forget speaker audio duration

We thank the reviewer for the opportunity of extensive experiments that will strengthen our paper, and plan to update experimental results using a shorter audio duration of forget speakers during this period as well.

Significance of our work with domain-specific challenges

While it would be most ideal to achieve a level of unlearning in smaller iterations and dataset size, this study makes an important contribution by achieving unlearning in ZS-TTS, which is particularly intricate. From a given audio and transcribed text, ZS-TTS models learn not only to imitate that speaker identity, but also to convert text to spoken words at the same time. Thus a major challenge lies in selectively unlearning only the speaker identity while maintaining the capability to accurately vocalize text from the forget set as our objective in L209-219. A recent study [3] explores challenge of unlearning while the forget and remain set share indistinguishable representations (here, knowledge to vocalize from given text prompt). Shared knowledge in forget and remain set can easily cause unlearning in remain set as well - as depicted in baselines NG and KL. For example, if a forget speaker utters “I am happy”, we have a challenge to forget only the speaker identity, not the ability to correctly vocalize the text “I am happy”.

2. If a zero-shot TTS service provider wants to prevent the model from cloning certain voices, they can easily use a speaker verification model to check whether the provided prompt speaker collides with a database of speaker embeddings whose voices are not supposed to be used and stop providing the service if the provided voice is in the forbidden speaker database.

We appreciate the reviewer’s suggestion regarding the use of a speaker verification model as a mechanism to prevent the unauthorized use of certain voices in a zero-shot TTS service. While this approach provides a practical barrier to unauthorized usage, we argue that it does not fulfill the core principle of the “Right to be Forgotten.” Blocking the generation of the forbidden speaker’s voice does not necessarily imply that the model has truly “forgotten” the speaker.

Additionally, implementing such a system would require the service provider to retain a database of embeddings for the forbidden speakers, contradicting the notion of fully respecting the privacy of individuals who request their voice to be forgotten. Moreover, this approach introduces a significant security vulnerability; if the model was to be compromised, an attacker could exploit the system to replicate the voice of the forbidden speakers.

Our proposed unlearning framework aims to address these issues by ensuring that the model itself forgets the speakers at a fundamental level, rendering it incapable of generating that speaker’s voice, even under adversarial conditions. This not only respects user privacy but also mitigates potential risks associated with model exploitation.

3. Since this method requires the entire training data of the original model, 5 minutes of audio for the forget speaker, and more than a quarter of training iterations of the original model, the actual significance of this work is rather limited.

We express our gratitude to the reviewer for allowing us to explore practical applications with shorter audio durations.

In this figure, we illustrate the performance of unlearned model originally reported in the paper at two checkpoints : 145 K steps and 48 K steps. Additionally, we include an experiment where only an average of 1 minute of audio per speaker is used for the unlearning process. This experiment leverages the original ZS-TTS model to augment 1 minute of audio. Using this 1-minute audio as a prompt, we generate diverse speech samples during training to mitigate overfitting to specific transcriptions. With such a limited amount of audio,it is highly unlikely that the model will unlearn only specific speaker characteristics. The model is prone to overfitting on the unlearning dataset, which could result in forgetting only specific utterances spoken by the forget speaker. Consequently, the model may fail to generalize its forgetting to the forget speaker uttering a speech not seen in the unlearning train set. To address this, we implement data augmentation to enhance the effectiveness of unlearning.

Our results indicate that the unlearning is more successful when provided with longer duration of forget speaker’s audio. This is likely because using a limited set of audio prompt, even with augmentation, can only generate a constrained range of styles (rather than being able to portray forget speaker’s characteristics in different emotions, accents, rhythms). We agree that exploring an approach using less datasets per forget speaker would be very practical and thank the reviewer for providing an insight from views of industrial application.

4. How much data is needed to recover the unlearned speech in both the non-zero-shot setting (where the forget speaker is in the training set) and the zero-shot setting (where the forget speaker is not in the training set)?

Thank you for your suggestions of recover experiments. For recovering experiments, we halted at early steps because of significantly low model performances. Also, our proposed paper's experiment has not yet extended to unlearn speech on an out-of-domain forget set, so we present the following experiment.

Aligning with our motivation to make ZS-TTS models safe, we presume a scenario of a privacy attacker who attempts to retrieve the original model parameters. We train the unlearned checkpoints on all 10 of forget speaker’s dataset to recover the original model. We also presume a practical scenario as the reviewer has highlighted important, and attempt to recover the model performance using average of 1 minute for each speaker.

Given a long audio duration of 15 minutes for the forget speakers, we can note that the model has become overfitted to generate speech specifically mimicking the forget speaker’s voice. The recovered model fails to retrieve its original model’s zero-shot capabilities, and is more likely to generate wrong speech content with higher WER. This process resembles fine-tuning a zero-shot model for specific speakers rather than true recovery. Consequently, the original ZS-TTS model cannot be restored, and the attacker is essentially leveraging transfer learning to create a forget speaker-specific TTS model, provided sufficient training data for the forget speaker is available. However, with enough training data, the attacker could achieve similar results using any other non-zero-shot TTS model. Taking the reviewer's emphasis on practicality into account, we also consider a scenario where an attacker has access to 1 minute of the forget speaker’s voice sample. In this case, the model parameters also remain unrecoverable and the model fails to generate forget speaker’s voice. The model loses its zero-shot abilities hence the performance at early steps. Therefore, in practical scenarios where an attacker may attempt to train the model to clone an individual’s voice with short sample of speech (e.g., voice phishing), it would not be feasible to recover the model or successfully generate the forget speaker’s voice.

As previously discussed, current ZS-TTS models are often closed to prevent malicious usage. This issue is particularly critical given that these models have achieved notable performance, yet most remain inaccessible for open use. Therefore, prioritizing safety and privacy is imperative to enable widespread and responsible accessibility of these technologies. With the TGU unlearned model, it would not be possible for any user to recover the original ZS-TTS model unless the service provider explicitly grants open access to its training dataset. While this approach may not fully eliminate the risks associated with cloning individual voices in scenarios where extensive datasets are available, it represents a meaningful step toward enhancing the safety and ethical use of such advanced technologies.

We thank the reviewer for the time taken to thoroughly review our work and provide valuable feedback. We hope to take this opportunity to strengthen our work through your insights.

W3. There are inaccuracies in the bold results presented in Table 1; the WER-F and WER-R recorded for the TGU approach are higher than those of the Exact Unlearning and Fine Tuning methods.

We would like to address the concern labeled as 'Weakness 3' regarding the perceived inaccuracies in the bolded results in Table 1.

In our initial submission, we highlighted the Word Error Rates (WER-F and WER-R) for the TGU approach, despite these figures being higher than those observed for the Exact Unlearning and Fine Tuning methods.

We recognize that this decision might have conveyed an unintended message. Our rationale was focused on the comparative analysis within the subset of models that demonstrated a degree of unlearning (NG, KL, SGU, TGU). We apologize for any confusion this may have caused and are committed to correcting this in our revised manuscript to ensure clarity and accuracy in our comparative analysis.

W.2 The TGU mechanism resembles a distillation process, suggesting another potential baseline worth exploring.

We also thank the reviewer for suggesting another possible baseline for this research, and waiting for experimental results.

The approach could involve using the zero-shot TTS model to generate audio with the retained speaker style using text from the forgotten set, then employing the generated audio prompt to potentially address the issues mentioned in L234-236 about speaker confusion and privacy constraints.

As the reviewer suggested, we utilized retained speaker style to generate audio prompt as new targets for this experiment. For each forget data $(x^{spk=f}, y)$ used to unlearn, we generated a target by randomly selecting audio prompt from the forget set. The generated target should then be an audio file uttering same text $y$ as the forget prompt with frame-wise alignment; $(x^{spk\not=f},y)$. While the results of the newly suggested baseline outperforms the naiive SGU, it is noticeable that TGU proposed in our paper still demonstrates effective unlearning while maintaining zero-shot performances on remain identities.

We believe this is because RTU and SGU are more dependent on the remain set's speaker distribution to generate targets, while TGU uses randomly initialized noise. This strengthens our proposed TGU in addressing effective unlearning in ZS-TTS by implementation purely random behaviors - isolated from specific data distributions.

We hope these experiments and discussions address your concerns and look forward to your answer.