Do Not Mimic My Voice : Speaker Identity Unlearning for Zero-Shot Text-to-Speech

We sincerely thank the reviewer 87b9 for the thoughtful and constructive comments.

1. Human Subjective Evaluation

We deeply agree with the reviewer on that speaker similarity metric may not perfectly reflect human perception. Speech consists of pitch, prosody, and unique speaker characteristics. To better capture human perception, we employed CMOS (Comparative Mean Opinion Score) and SMOS (Similarity Mean Opinion Score) in Table 3.

CMOS was evaluated by asking participants to compare audio quality of outputs from the Original (Exactly Learned) model and outputs from SGU, TGU, and ground truth.

SMOS was evaluated by presenting participants with two speech samples, and asking whether the two voices sound similar. For ground truth cases, we used two samples from the same speaker. For other cases, we used one real sample and one synthesized sample generated with same speaker’s prompt.

Then, a well unlearned model should have very low SMOS-F and SIM-F, while keeping comparable scores to the Original model in SMOS-R and SIM-R.

We prepared 160 audio samples in total (10 randomly selected samples x 4 systems x 4 conditions). 25 participants with proficiency in English were recruited and compensated for the experiment. The detailed information on human evaluation are described in Appendix F.

2. Forget ID Detection

As pointed out, the challenge of detecting if the input prompt belongs to a forget speaker under limited data is a non-trivial. On top of that, our framework does not consist of an independent speaker classifier or encoder.

Instead of relying on explicit forget speaker classifier or detector, our model implicitly identify the forget speakers by by directly updating its parameters through our proposed training framework. Specifically, when given a forget speaker’s prompt, the model generates speech with a voice different from that of the forget speaker by treating it as noise and suppressing its identity. Thus, as the model generates random voices for forget speaker prompts, it becomes practically infeasible to reconstruct their original voices, ensuring stronger privacy protection.We verify this with multiple experiments. With proposed spk-ZRF, we can evaluate that TGU acts significantly randomly on forget speaker prompt.

To further evaluate the robustness of our method against potential false positives, we conduct experiments using remain speakers who have high speaker similarity with the forget speakers. The results confirm that our model reliably distinguishes remain speakers without mistakenly treating them as forget speakers. For detailed experimental results, please refer to Section 7. Similar Voice Handling in our response to reviewer (bkoi).

3. Computational Costs

Contrary to traditional machine unlearning, Exact Unlearning is not ideal nor practical in Zero-Shot TTS. Zero-Shot TTS models can replicate voices they have never seen during training; simply removing the forget speaker’s samples from training data does not suffice. Therefore, a comparable method to unlearn speaker identities does not exist.

Regarding the training cost, the Exact Unlearning and Original models were trained for 500K steps. When unlearning 1 speaker at a time, SGU and TGU for were trained for 10K steps (2% of pretraining steps) in Table 2. When unlearning 10 speakers at a time with LibriTTS dataset, TGU was trained for 10K steps, refer to Section 4. Generalizability in our response to Reviewer d6q4. Fine Tuning, SGU, and TGU for Table 1. were trained for 145K steps - with TGU showing convergence around 58K steps in Appendix D. Negative Gradient and Kullback-Libeler divergence were halted at early stages due to exploding gradients.

Nevertheless, our method is not confined to a specific fine-tuning method. Therefore, in practicality, computational cost can be minimized using diverse accelerating methods in the fine-tuning process.

4. SIM

SIM is calculated using the cosine similarity between speaker embeddings extracted from a speaker verification model. This evaluation is widely utilized in TTS research domain [1]-[6]. We use the same speaker verification model as [1]-[5] for evaluation. For SIM, the goal is to maintain a high performance in terms of ZS-TTS model evaluation.

For unlearning task, indeed SIM is not sufficient alone. To mitigate this limitation, we propose spk-ZRF as a new evaluation metric to measure model’s randomness in generation. Objective is for the model to “act as if they have no knowledge of forget speaker identity”. Having a high spk-ZRF indicates that the model is behaving randomly. Vice versa, a spk-ZRF lower than original model means the model has learned to act in a consistent way (e.g., synthesizing in a specific voice). Thus, we do not mark an arrow for spk-ZRF-R because maintaining original performance is ideal.

Once again, we greatly appreciate the reviewer’s insightful feedback, which helped us further clarify and strengthen our work.

We sincerely thank the reviewer d6q4 for the constructive feedback. Below we address each concern and clarify the raised points:

1. Similar Voice and Robustness

Please refer to Section 7 in our response to reviewer (bkoi)

2. Computational Resources

Please refer to Section 1 in our response to reviewer (bkoi)

3. Scalability and Length of Audio Samples:

We acknowledge concerns related to scalability, particularly regarding audio length requirements. From the perspective of service provider and user, we assume that the forget speakers voluntarily request their voices to be forgotten and are willing to provide audio samples specifically for the unlearning process.To alleviate the scalability concern under this assumption, we conducted experiments using data augmentation techniques leveraging the original model to generate additional training samples from as little as 1 minute of audio per forgotten speaker (as detailed in the "TGU w/ Augmentation" scenario). The results indicate substantial improvement in scalability, achieving a competitive SIM-R of 0.612 and SIM-F of 0.334 at only 48K training steps.

Although these results currently show room for further optimization, ongoing exploration into advanced data augmentation is expected to continue enhancing performance and practical viability, thus minimizing audio length constraints effectively.

4. Generalizability

Model Selection We selected VoiceBox due to its state-of-the-art (SoTA) performance at the time of conducting our research. We acknowledge the reviewer's concern regarding the generalizability and comprehensiveness of our evaluations. Given that MaskGCT [4] and F5TTS [3] were very recently introduced, we recognize their importance. F5TTS shares significant similarities with VoiceBox, being based on Flow-Matching with analogous model structures. We are currently preparing experiments using F5TTS to address this concern.

Reproducibility on Different Dataset

To address the reviewer's concern, we conducted additional experiments using another dataset; LibriTTS [10]. We trained the VoiceBox model on LibriTTS and 10 speakers were randomly selected as forget set, following experimental settings of Appendix A.

Due to time constraints, only TGU was evaluated at 10k steps (7% of pre-training steps). We plan to comprehensively evaluate additional baseline methods, including recent unlearning techniques, and add these results in the final camera-ready version of our manuscript.

5. Recoverability Analysis:

Thank you for finding our recoverability experiments valuable. We provide further analysis on recoverability experiments with additional metrics. Frechet Speech Distance (FSD) is used to measure Zero-Shot TTS’s ability to generate high quality speech with natural distributions. Low FSD indicates high quality and diversity in generated speech. Refer to Appendix H.2 for more information on FSD.

According to results, recovering the original model by re-training is not possible.

First, when recovering with audio duration of 1 min, the model fails to mimic any speaker. It even shows lower performance on the forget set in SIM-F.

Second, when recovering with audio duration of 15 min, the model overfits to the forget speaker. Instead of recovering to Original, the model is trained to be a speaker-specific TTS model.

6. Human Evaluation:

Please refer Appx F. and Section 1 in our response to reviewer (87b9).

7. Scalability - Forget Size

Please refer to our response to reviewer bkoi, specifically the section titled ‘5. Scalability ’

Again, we sincerely thank the reviewers for their insightful feedback, which will substantially strengthen the paper’s final version.

We thank the reviewer bkoi for the constructive feedback and valuable insights. We address the primary concerns as follows:

1. Practical Use Case

The 145K steps used during unlearning processes were chosen based on the common fine-tuning practices when adapting Flow-Matching-based pre-trained models to the Zero-Shot TTS domain [11]. In this context, such a step count is not considered excessive. Furthermore, as presented in Appendix D, we provide performance curves throughout the unlearning process. Notably, we observe that after approximately 58K steps (about 11% of the pre-training steps), the model already achieves sufficient unlearning performance with SIM-R and SIM-F scores reaching 0.588 and 0.174, respectively. Importantly, we also confirm that WER-R and WER-F remain stable throughout the entire unlearning process.

[11] Generative Pre-training for Speech with Flow Matching, ICLR, 2024

2. Adversarial Attacks

We agree that additional adversarial robustness evaluations would strengthen our method's validity. We plan to experiment this using adversarial attack methods in TTS to validate spk-ZRF’s resistance in the following week.

3. Latest Unlearning Methods

We provide results of latest machine unlearning methods with best scores of each methods in unlearning a single identity.

The poor results in [12] and [13] are expected as per the domain characteristics of ZSTTS. Recent architectures do not learn ‘speaker identity’ and ‘content generation’ separately. The features are deeply entangled within model parameters. Thus, removing the ability to mimic specific speaker identities without harming knowledge for content generation comes with an implicit but challenging task ; fine-grained feature disentanglement.

Current saliency-based unlearning methods perform poorly. In [4], even though remain and forget set were both used to create separate fisher maps - unlearning in forget set still impacted performance for remain set. In [5], random labeling with unaligned content as label is used alongside fisher map from forget set. In this case, the model was updated minimally.

We agree with your insights in adapting latest machine unlearning methods that can be adopted alongside Guided Unlearning. We plan to experiment saliency-based approach on SGU and TGU in the following week.

[12] Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement [13] SalUn:Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation

4. Speaker Verification Model

Please refer to Section 2 in our response to 4th Reviewer (87b9)

5. Scalability

Here, we provide an updated version of Table 2. K refers to the of forget speakers in the forget set.

To analyze the impact of the number of forget speakers on model performance, we conducted analysis by varying the forget speaker set size. As shown, our model maintains robust performance.

Importantly, using 10 forget speakers is not a small-scale scenario compared to prior generative unlearning studies [10,12-13], where the forget set sizes are generally 1, and at most, 3. We plan to report our active experiments on 50 forget speakers in the camera-ready version.

[12] Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models. arXiv, 2023.

[13] Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient, AAAI, 2025

6. Similar Voice and Robustness

In Appendix G, we provide a figure illustrating TGU's performance on remain speakers with varying similarity to forget speakers. Pearson analysis (F=0.1396, p>0.0003) validates that even when a remain speaker may have vocal features similar to a forget speaker, this does not affect model's performance for remain speakers. We also provide comparisons of model performance for robustness. We obtain SIM between samples of remain speakers and forget speakers. Then, we identified samples of remain speakers that show high similarity ( > 0.4 ) to forget speakers. We use these samples as input prompt for evaluation. We will update the project page and supplementary to contain samples of similar speakers, and the generated output of each method.

7. Generalizability

Please refer to Section 4 in our response to 3rd Reviewer (d6q4)z

8.spk-ZRF

Please refer to Section 4 in our response to 4th reviewer

We deeply appreciate the reviewer k36c's insightful comments and thorough review. Below, we address each comment clearly and concisely.

1. Clarification and Typos

As you correctly indicated, Exact Unlearning is an exact method rather than approximate. We will correct this with other typos and clearly state the distinction for revision. For supplementary, samples from Exact Unlearning and Fine Tuning methods will be included.

2. SIM-R and spk-ZRF-R

Please refer to Section 4. of our answer to 4th reviewer 87b9.

3. Statistical Significance

Here we provide statistical significance test results for Table 1 and Table 2. The result of ANOVA test on SIM and WER validates that the unlearning methods are effect SIM and WER. In Table 2, low significance in WER-R and WER-F validates that SGU and TGU both maintain comparable content accuracy. However, significance in SIM show that TGU is a more effective method.

4. Ground Truth for WER

Word Error Rate (WER) is widely used in speech synthesis domain to analyze accuracy in content generation [1]-[5]. The Ground Truth for WER in Table 1 is obtained by transcribing the target speech using the Automatic Speech Recognition model [6], then comparing the ASR result to the target speech transcription. The target speech is obtained from the test dataset LibriSpeech test-clean [7].

[1] Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale, NeurIPS, 2023 [2] Neural Codec Language Models are Zero-Shot Text to Speech Synthesizer, Arxiv, 2023 [3] F5-tts: A fairytaler that fakes fluent and faithful speech with flow matching, aAxiv, 2024 [4] Maskgct: Zero-shot text-to-speech with masked generative codec transformer, ICLR, 2025 [5] CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens, 2024 [6] HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units, 2021 [7] Librispeech: An ASR corpus based on public domain audio books, ICASSP, 2015

5. Missing Results in Table 3

Thank you for your feedback. We initially focused on methods with noticeable changes in SIM-F, but we acknowledge the value of assessing Exact Unlearning and Fine Tuning. Following Institutional Review Board (IRB) guidelines to ensure ethical evaluation takes time, and we will add these results in camera-ready version.

6. Discussion on Higher SIM-F

While SIM-R measures the zero-shot voice replicability on completely unseen speakers [9], SIM-F is evaluated with the test set from data [8] the model was pre-trained on. In the case of Original, SIM-F is inevitably higher due to similarity in data distribution with the train set. Exact Unlearning and Fine Tuning both fail to unlearn due to model's zero-shot capabilities, thus for similar reasons, SIM-F scores are higher.

[8] Libriheavy: a 50,000 hours asr corpus with punctuation casing and context, ICASSP, 2024

7.Unlearning Unseen Speakers

We experiment this Out-of-Domain Unlearning [9] scenario where the challenge precisely lies in the fact that ZS-TTS can replicate voices without explicit training data for speakers. We unlearn speaker identity never used in training and results demonstrate that our unlearning method is effective, even for unseen speakers.

[9] Generative Unlearning for Any Identity. CVPR, 2024. [10] LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech, Interppeech, 2019

8. Revisiting Machine Unlearning

We adopt the task of machine unlearning for Zero-Shot TTS, and introduce newly formulated problem as Speaker Identity Unlearning (SIU) in Section 3.2

SIU directly adjusts model parameters to remove the capability to reconstruct forget speaker’s voice. This is consistent with recent trends in generative unlearning. A recent study [9] demonstrates that unlearning can be applied to prevent identities from being visually generated. Similarly, our work prevents identities from being synthesized upon request. Also, the goal of machine unlearning is to safeproof model parameters from revealing information. Same goal is applied in SIU, with notable results in preventing traceability of forget set. Our methods are consistent with machine unlearning's problem formulation; prevent revealing unwanted identities, while retaining model performance on remain set.