We thank the reviewer for the insightful and valuable comments. We respond to each comment as follows and sincerely hope that our rebuttal could properly address your concerns.

Weaknesses

W1. The proposed method increases time cost due to the introduction of the Refinement stage.

We agree that the Refinement stage increases computational time compared to single-stage purification. However, voice cloning attacks are fundamentally offline processes. Attackers typically have hours/days to prepare critical audio samples (e.g., CEO's voice for fraud), making increased computation, which is seconds-level as shown in Table 8 still acceptable for these scenarios. For instance, an audio sample of 15 seconds only needs 13 seconds for our method to process.

W2. The terms "force alignment" and "force aligner" may require clarification.

• The Force Aligner mentioned in our paper refers to the Montreal Forced Aligner (https://github.com/MontrealCorpusTools/Montreal-Forced-Aligner), which is a tool used for aligning text with audio. Specifically, it works by first converting the text into a sequence of phonemes. Then, using a pre-trained speech recognition model, it aligns the start and end times of each phoneme to precise timestamps within the corresponding audio.

• Force alignment, on the other hand, refers to the process of performing this audio and text alignment using the Montreal Forced Aligner.

To prevent any confusion, we will provide explicit definitions for both "force aligner" and "force alignment" in Section 4.3 of the revised paper.

Questions (& Weakness 3)

Q1. Please analyze why DiffSpec and DualPure perform unexpectedly worse than DiffWave and WavePurifier in your task compared to other purification tasks.

The relatively poorer performance of DiffSpec [2] and DualPure [4] can be attributed to their mel-spectrogram processing parameters. These methods operate on spectrograms with 32 mel-bins, which were initially designed for classification tasks. While 32 mel bins are sufficient for classification (for the classification models are trained on inputs of the same dimensionality), reconstructing waveforms from such a low-resolution spectrogram can lead to significant distortion.

However, existing voice cloning models [7, 8] typically utilize mel-spectrograms with 80 mel bins or more as input, relying on high-fidelity spectral features. In contrast, WavePurifier [3] employs a higher resolution (256x256) spectral representation, which preserves the essential details for voice cloning, thereby performing significantly better than DiffSpec [2] and DualPure [4] in our experiments. DiffWave, operating directly in the waveform domain, avoids this type of distortion, similarly leading to better results.

We will include this analysis in Section 5.2 or the Appendix of the revised paper.

Q2. Omitted comparisons with the "Filter Power" and "LPC" methods.

Filter Power and LPC from [1] were omitted in our initial experiments due to:

• Filter Power functionality overlaps with BPF in [5] (both are band pass filter methods),
• LPC functionality overlaps with MP3 in [5] (both are compression methods).

We acknowledge the importance of direct comparison and conducted new experiments. As shown below, both methods underperform our proposed approach (results to be added to Table 5):

Q3. The "Parameter Value" for Downsampling (1600Hz) needs explaination.

The "1600Hz" entry in Table 4 is a typo. The correct downsampling rate is 8000Hz, aligning with common practices in prior voice cloning defenses and adversarial purification [2, 5, 6]. For completeness, we also evaluated [1]'s default parameter (6000Hz), achieving:

both of them underperform our proposed approach. We will correct this typo in Table 4.

Other Comments or Suggestions

Open-sourcing the code would greatly enhance the reproducibility of this work and facilitate further research in this area.

Yes, we will make our code publicly available once our paper is accepted.

[1] WaveGuard: Understanding and Mitigating Audio Adversarial Examples. USENIX21.

[2] Defending against adversarial audio via diffusion model. ICLR23.

[3] WavePurifier: purifying audio adversarial examples via hierarchical diffusion models. MobiCom24.

[4] DualPure: An Efficient Adversarial Purification Method for Speech Command Recognition. Interspeech24.

[5] Towards Understanding and Mitigating Audio Adversarial Examples for Speaker Recognition. TDSC22.

[6] AntiFake: Using adversarial audio to prevent unauthorized speech synthesis. CCS23.

[7] Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. ICML22.

[8] Better speech synthesis through scaling. arXiv preprint 23.

We thank the reviewer for the insightful and valuable comments. We respond to each comment as follows and sincerely hope that our rebuttal could properly address your concerns.

Weaknesses

W1. The process of purification, specifically unconditional diffusion, requires further explanation. For instance, Equations (3) and (4) should explicitly include $x_\text{adv}$.

In Equations (3) and (4), $x_t$ refers to audio sample at time $t$, $x_\text{adv}$ refers to adversarial audio sample, and $t$ ranges from $1$ to $T_\text{pur}$. The foward process starts with $x_{0}$ ($x_\text{adv}$) and generates noisy sample $x_{T_\text{pur}}$; the reverse process starts with $x_{T_\text{pur}}$ and generates $x_{0}$ ($x_{\text{pur}}$). We will clarify these terms in Section 4.2 as follows:

• (Revision) At each timestep $t$, the process is expressed as: $q(x_t | x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t} x_{t-1}, \beta_t \mathbf{I}), t = T_\text{pur}, T_\text{pur}-1, ..., 1$ where $\beta_t$ denotes the variance schedule, and $x_0 = x_\text{adv}$.
• The reverse process denoises the waveform $x_{T_\text{pur}}$ in the same $T_\text{pur}$ steps, and at each step, it is formulated as: $p_{\theta}(x_{t-1} | x_t) = \mathcal{N}(x_{t-1}; \mu_\theta(x_t, t), \sigma_t^2 \mathbf{I}),$ where $\mu_\theta(x_t, t)$ is the mean function parameterized by $\theta$, $\sigma_t^2$ is the time-dependent variance schedule, and $x_0 = x_\text{pur}$.

W2. The training details are insufficiently clear. For example, the statement "The Purification model is a pretrained unconditional DiffWave model" contrasts with the context, which states, "Our method trains two models separately." The authors should clarify the detailed settings.

Thank you for pointing out this ambiguity. We will clarify the description, avoiding the term "pretrained" to describe a model fine-tuned on a specific dataset as follows:

• (Revision) The Purification model is based on a pretrained unconditional DiffWave model which is then fine-tuned on the LibriSpeech dataset.

W3. In Figure 11, the authors note that "Existing purification methods tend to produce samples with similar patterns and blurred details," but the similar patterns are difficult to discern. This section also requires further clarification.

We apologize for not clearly explaining "the similar patterns." This was because the pattern is not evident from the small number of examples provided. We will address this by adding more examples in Figure 11 to clarify these patterns.

Other Comments or Suggestions

C1. The description of the summarized contributions could be refined, for example, by directly highlighting the specific risks identified.

We will refine the descriptions of our contributions in the introduction by directly highlighting the specific risks identified. Details are as follows:

• (Revision) We assess six VC methods and three protective techniques, revealing the risk that existing defenses potentially fail to prevent voice cloning attacks.

C2. Typos: In the caption of Figure 2, "(b-c), Purified" should be corrected to "(b-c), purified."

Thank you for pointing out this typo. We will correct it in our revised paper.

We thank the reviewer for the insightful and valuable comments. We respond to each comment as follows and sincerely hope that our rebuttal could properly address your concerns.

Claims And Evidence

Concern about subjective 'human' notation (H). Embedding proxy may not reflect human perception. Suggest adding author evaluation.

• We follow [1] using similar notation $H$. We will make this clearer in the paper: $H(\cdot)$ represents the perceived speaker identity evaluated by human listeners with given audio.
• To clarify, (clean, protected, purified) speech is the reference speech (VC model input). Synthesized speech is the VC model output, potentially used to deceive humans/SV models. We used the VC model's embedding as a proxy evaluation for 'purified speech' as it's the VC model's input; our goal is reduced distortion for the cloning model in 'purified speech' to facilitate successful voice cloning (deceiving SV models/humans). Humans don't evaluate 'purified speech'. However, synthesized speech (VC model output from source speech) may deceive humans.
• Therefore, we have a subjective test (Sec. 5.2) with 20 humans evaluating synthesized speech from (clean, protected, purified) inputs, aligning with our threat model's $H$. For the purified speech, we don't involve human evaluation in the threat model (Sec. 3), so we think it is enough to use VC model embedding as proxy evaluation.

Methods And Evaluation Criteria

Fig. 2 unclear on reduced inter-class separability. Show embeddings after step 1 and full 2-steps for comparison.

• We add some arrows pointing from clean to purified samples within Fig. 2(b-c) of anonymous link https://github.com/de-antifake/1/blob/main/2.png to better convey the reduced inter-class separability.
• Fig. 2(c) already shows the embedding results after applying only step 1 (our method uses DiffWave for Purification stage, which is consistent with AudioPure), while Fig. 2(d) shows the results of applying the complete 2-step method.

Experimental Designs Or Analyses

Discuss adaptive attack against purification like DiffAttack in the audio domain.

• We actively experimented with DiffAttack's core mechanisms (deviated-reconstruction loss and segment-wise forwarding-backwarding) following public code on an RTX A6000 (48GB VRAM). However, audio data's high resolution (e.g., 96k data points for 6s audio at 16kHz vs. 1k for DiffAttack's CIFAR-10) caused out-of-memory issues for DiffAttack on our task.
• Possible solutions include using a lower-resolution diffusion model as surrogate, or processing long audio in chunks to reduce memory overhead. Furthermore, using accelerated sampling methods as surrogate to reduce computational graph depth might also help. Overall, running something like DiffAttack in the audio domain is challenging for defenders due to the high cost of calculating diffusion model gradients. Due to time, we didn't do further experiments, but future work can explore these adaptive defense methods. We will add this discussion to Sec. 5.4 or App. D in the revised paper.

Weaknesses/Questions

W1. & W2. There is an assumption: diffusion-based removal leads to similar distributions for clean & protected audio in the paper, which allows training refinement on (clean, purified(clean)) and applying to purified(protected). This non-intuitive observation should be mentioned in the paper.

We show embedding distributions of Clean vs Protected, and Purified(Clean) vs Purified(Protected) Speech in anonymous link https://github.com/de-antifake/1/blob/main/3.png. We observed that diffusion-based perturbation removal brings clean and protected audio to similar distributions, which is a key assumption for our method's effectiveness. Thus, we don't need to generate adversarial data for training. We will mention this observation in the revised paper to clarify the Refinement stage's motivation.

Other Comments or Suggestions

Explain voice cloning model inference in the text.

We apologize for not providing a clear explanation of how VC models work. Our work considers two main types of voice cloning: Text-to-Speech (TTS) and voice conversion. Both leverage speaker embeddings extracted from the victim's audio to capture their voice characteristics.

• For TTS-based voice cloning, the attacker provides arbitrary text. The TTS acoustic model, conditioned on the victim's speaker embeddings, takes this text and generates corresponding acoustic features (e.g., mel-spectrograms). These acoustic features are then used by the vocoder to synthesize the cloned voice.
• For voice conversion, the model typically takes the acoustic features of an arbitrary speaker's utterance provided by attacker (representing the content) and transforms them using the victim's speaker embeddings to generate speech with the victim's voice.

We will provide a detailed explanation of these mechanisms in the revised paper.

[1] Antifake: Using adversarial audio to prevent unauthorized speech synthesis. CCS23.

We thank the reviewer for the insightful and valuable comments. We respond to each comment as follows and sincerely hope that our rebuttal could properly address your concerns.

Weaknesses

W1. The purification method employed is relatively outdated, specifically an unconditional DiffWave model. Could the proposed method be generalized to other audio diffusion models?

Recent audio diffusion models are often designed for specific tasks such as conditional generation or speech enhancement, and therefore, the majority of them are conditional models. We experimented with applying newer conditional models [1, 2] to our current task, and the table below shows the dSVA after applying only the Purification stage with different models:

We find that their performance was generally not as good as unconditional DiffWave. We infer that an unconditional model is more suitable as our Purification model, possibly because it lacks conditional constraints and might be more robust to unseen noise.

Despite this, we attempted to apply another audio diffusion model to our task, specifically the one from [2], to investigate the potential of our method to generalize to other diffusion models. The resulting dSVA values are presented in the following table. The results indicate that when using another audio diffusion model as the Purification model, our Refinement stage also improved performance. This suggests that our method has the potential to generalize to other audio diffusion models.

W2. It is suggested that the authors consider adopting some diffusion model acceleration sampling techniques or flow-matching models to further improve the speed of purification. Additionally, I am curious about the respective time costs of the purification stage and the refinement stage. Please list them.

• We list the processing time per second of audio for the Purification and Refinement stages separately in the table below. The results indicate that the Refinement stage takes up the majority of the processing time. | | Purification Stage | Refinement Stage (N=15) | Refinement Stage (N=30) | |----|----|----|----| | Time Cost (s) | 0.152 | 0.715 | 1.253 |
• Your suggestion regarding acceleration is very constructive. We agree that acceleration sampling or flow-matching models could be helpful for real-world efficiency. Due to time constraints, we will explore potential accelerated sampling techniques or more efficient models in our future work to enhance the practicality of our method for real-world applications and advance the field.

W3. In Section 4.2, the sampling process of the diffusion model is not clearly articulated. It is recommended that the authors replace Equation 5 with an iterative sampling formula (such as DDPM or DDIM).

We will clarify the sampling process in Section 4.2, including replacing Equation 5 with the following iterative sampling formula:

$x_{t-1} \sim p_{\theta}(x_{t-1} | x_t) = N(x_{t-1}; \mu_{\theta}(x_t, t), \sigma_t^2 {I}), t = T_\text{pur}, T_\text{pur}-1, ..., 1$

where $x_0 = x_\text{pur}$.

[1] Richter J, et al. Speech enhancement and dereverberation with diffusion-based generative models. IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP), 2023.

[2] Tian Y, Liu W, Lee T. Diffusion-Based Mel-Spectrogram Enhancement for Personalized Speech Synthesis with Found Data. IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2023.

We thank the reviewer for the insightful and valuable comments. We respond to each comment as follows and sincerely hope that our rebuttal could properly address your concerns.

Weaknesses

W1. More discussion about practical limitations.

Our implementation utilized an RTX A6000 GPU and Xeon Gold 6130 @2.1GHz CPU. The peak computational resource usage is detailed below:

These requirements indicate that executing our attack is not feasible on lightweight devices, and requires moderate computational resources as illustrated in our threat model (Sec. 3).

W2. (& Theoretical Claims) Consider developing theory for better understanding of proposed method's interpretability and limitations.

Thank you for your insightful suggestion. According to the proof in [1], as the forward diffusion time $t$ increases, the KL divergence between the clean distribution $p_t$ and the adversarial distribution $q_t$ is monotonically decreasing, meaning the clean and the adversarial distribution become closer. This provides theoretical support for our Purification stage to effectively mitigate adversarial noise. Building upon this, developing a systematic theory specific to our 2-stage method will lead to a better understanding of our method's limitations and interpretability. However, due to time/space limitations, we can only explore this issue in future work.

W3. (& Question 6) Do you think there is a fundamental limitation to protective perturbations?

Though our method shows limitations in existing VC defenses, we did not 100% success in bypassing the protections, indicating that current defenses still offer some level of protection. Furthermore, as is common in all areas of information security research, defenders will likely develop countermeasures against our purification. Therefore, we believe it is premature to assert a fundamental limitation of existing protections, and the arms race between attack and defense will likely continue.

W4. Define dSVA and xSVA with math formulation.

We will add definitions for SVA like follows: $dSVA = \frac{1}{N} \sum_{i=1}^{N} SV_d(x_{test}^i, x_{clean}^i)$.

Questions

Q1. Cross-lingual performance of the Refinement stage.

We show the results of a small-batch inference on Russian LibriSpeech (non-Latin script) below. Our Refinement stage remains effective outside the training data (English).

Q2. Future directions for robust VC defenses.

We think embedding protective perturbations in higher-level semantic features for robustness appears promising. Combining perturbations with other defenses like proactive watermarking for a multi-layered defense is another promising direction.

Q3. Optimizations for computational overhead.

Our paper focuses on revealing existing defense vulnerabilities, and we believe the current time cost (second-level) is acceptable for offline attacks. Your question is constructive, due to time, we will strive to optimize in future work to enhance real-world efficiency.

Q4. Effectiveness for few/many-shot VC attacks.

Although existing VC defenses [2-4] don't consider few/many-shot scenarios, we run a small-batch VC attack where attacker have 5 reference audio and find our method remains effective:

Q5. Reasons and potential solutions for gender disparity.

Figure in anonymous link https://github.com/de-antifake/1/blob/main/1.png show protective perturbations mainly affect higher frequencies where female voices have more energy. We infer the frequency overlap makes separating perturbations from female voice characteristics harder during purification, explaining the disparity. Potential solutions include hierarchical purification: use different purification steps for high/low frequencies could mitigate this.

Other Comments Or Suggestions

Explore trade-offs between attack success and audio quality.

We do not observe a clear trade-off between audio quality and attack success. We use PESQ to evaluate the audio quality of the purified audio using a small batch of data:

As purification steps increases, both PESQ and dSVA initially improve and then decline, reaching their optimal values at the same step. This suggests that the protective noises degrades both audio quality and attack success, and proper number of purification steps improves both.

[1] Diffusion Models for Adversarial Purification. ICML22.

[2] AntiFake: Using Adversarial Audio to Prevent Unauthorized Speech Synthesis. CCS23.

[3] Defending Your Voice: Adversarial Attack on Voice Conversion. SLT21.

[4] Voice Guard: Protecting Voice Privacy with Strong and Imperceptible Adversarial Perturbation in the Time Domain. IJCAI23.