I Can Hear You: Selective Robust Training for Deepfake Audio Detection

Q4 - Add a data sheet that outlines the exact sources and utterance lengths per source

Thanks for you suggestions. We have added a data sheet that outlines the exact sources and utterance lengths per source in revised version.

Q5 - Are the WaveFake test samples also part of the DeefFakeVox-HQ test set?

No, WaveFake utilizes six AI synthesis models: MelGAN, ParallelWaveGAN, Multi-band MelGAN, Full-band MelGAN, HiFi-GAN, and WaveGlow, none of which are covered in our test set.

Q6 - Clarify for WaveFake Dataset

We apologize for any confusion caused; our experiment utilized only the English portion of the WaveFake dataset. We have updated our paper accordingly, revising Table 1 to indicate that WaveFake includes both English and Japanese languages. Additionally, in Section 5.1, we have clearly specified that we used only the English component of the dataset in revised version.

Q7 - Audio Spectrogram Transformer performance on DeefFakeVox-HQ

Audio Spectrogram Transformer does not perform as well as RawNet3 on DeepFakeVox-HQ. We use the same training hyperparameters (learning rate schedule, optimizer, batch size, etc.) and same augmentation.

Q8 - Setting for training Wavefake in Table 2

Sorry for the confusion, the training settings for the WaveFake dataset in our paper differ from those in the original paper to maintain consistency across all datasets. We used all available sources and divided them into training, validation, and testing sets with a ratio of 7:1.5:1.5.

Q9 - Which software libraries have been used to implement this project?

For the detection model, we used standard libraries such as Torch, Torchaudio, Scikit-learn, and Librosa. Detailed information on each package and its version is included in the supplementary file env.txt, which lists all dependencies. The software libraries used by TTS (Text-to-Speech) and VC (Voice Conversion) models to generate deepfake audio, however, vary. If accepted, we will open-source our code and data and provide a comprehensive README.

We appreciate the reviewer's positive feedback on our paper. It is encouraging to see the recognition of our tackling a significant problem, the interest in our adversarial attack perspective, and our efforts to keep results current by integrating samples from commercial models. We discuss these contributions and address reviewer suggestions in the sections below.

W1 - MP3 compression removes high-frequency content, which may impact the effectiveness of F-SAT

Traditional compression algorithms like MP3 typically remove frequencies above 16 kHz. However, in our experiment, the audio is resampled to 16,000 Hz before being input to the model, capping the maximum representable frequency at 8,000 Hz (per the Nyquist theorem). Thus, MP3’s high-frequency removal has no impact on our data, as it already falls below this range.

W2 - Robustness to compression

Thank you for your suggestion. We have added results on compressed audio in the appendix. To evaluate the robustness of our detection model to compression, we tested two lossy formats: MP3 and AAC. The evaluation utilized RawNet3 combined with RandAug and F-SAT. As shown in the table, both MP3 and AAC compression had minimal impact on detection accuracy.

W3 - Dataset details like the length in hours or training hyperparameters like the learning rate are missing

Thank you for bringing this up. We have added dataset details and training hyperparameters in the revised version.

Training Hyperparameters for RawNet3 on DeepfakeVox

• Learning Rate (lr): 1e-5
• Epochs: 15
• Batch Size (bs): 16
• Optimizer: adam
• Augmentation Number (aug_num): 1 or 2
• Augmentation Probability (aug_prob): 0.9

LR Scheduler (Warmup Cosine)

• Warm-up Epochs: 1
• Warm-up LR: 1e-6
• Minimum LR: 1e-7

Attack Hyperparameters

• Attack Type: l_inf
• Epsilon: 0.005, Alpha: 0.002
• Gamma (control ratio of clean loss and robust loss): 0.1
• Attack Iterations: 2
• Restarts: 1

Mixup Hyperparameters

• Mixup Alpha: 0.5

Dataset Details

For the training set, we used six open-source models to generate data from four datasets: VCTK (12k samples), LibriSpeech-clean-100 (28k samples), AudioSet (narration) (12k samples), and In-The-Wild (real parts: 9k samples), resulting in a dataset nearly six times larger than the real samples and evenly distributed across these sources. Additionally, we used one commercial model, ElevenLabs, to generate 2,500 samples. The total duration is detailed below.

We also combined our generated data with existing public datasets to enhance its diversity. For the test set, we included 14 different fake sources, with approximately 200 samples each.

Q1 - Which model was used to create the plot on the left side in Figure 2

RawNet3 Model (without F-SAT)

Q2 - How many hours of speech does the dataset encompass exactly? How long are the samples?

The total duration of our dataset is approximately 2700 hours, comprising 1400 hours of real audio and 1300 hours of fake audio. The fake audio includes 300 hours from a previous dataset and 1000 hours we generated. Sample lengths vary across subdatasets, ranging from 4 to 15 seconds. Specifically, for VCTK and the corresponding fake audio, the average duration is about 4 seconds per sample. In contrast, for AudioSet (narration), the average duration extends to around 15 seconds per sample.

Q3 - Are the samples aligned? Do all models synthesize speech using the same input sentences?

For the high-quality deepfake samples we generated, alignment is ensured. All models synthesize speech from the same input sentences.

Dear authors,

My last remaining concern is with Q8 (Table 2). Is it possible to explicitly state that the results are not comparable to the original wavefake paper? Furthermore, Wavefake does not exclusively contain LJSpeech samples. It also has a JSUT part. Is it possible to fix the caption?

We appreciate the reviewer's insightful feedback on our work. Key strengths highlighted include our comprehensive dataset, advanced deepfake detection method, and robust defense against high-frequency adversarial attacks. These contributions collectively push forward the boundaries in deepfake detection research. Further details and additional insights are discussed in the subsequent sections.

W - contribution on our dataset

Thank you for your suggestions. We are glad that the reviewer recognize our dataset to be well organized and processed. As suggested, we will remove the claim of “largest” in the revised version. Our dataset comprehensively summarizes existing TTS and VC models, evaluating over 30 AI-synthesis models proposed within the last 2-3 years, including both open-source and commercial models. We utilized 14 of these models, which we believe are significant to the community and merit publication.

W - "Compared with other defense method

Thank you for the suggestion. We have included a discussion of the paper ‘High-frequency Adversarial Defense for Speech and Audio’ (2021) in the related work section and present comparative results in the experiments section. We appreciate Reviewer EGHT's recommendation for this comparison.

We compared our method against MAD smoothing[1], as suggested in the paper, and also included Gaussian smoothing[3], and adversarial training in the time domain[2], which are used as baselines in the reference paper. We employed the same parameters as those in [1]. The results of these comparisons are outlined below. As the table shows, our method outperforms all other methods.

We also found that robust deepfake voice detection lacks enough related work in recent years. However, with GenAI making the deepfake generation increasingly realistic and widespread, it is urgent and crucial to design tools to detect them in a robust way. As far as we know, Our work is the first to tackle deepfake audio detection in a real-world setup after the boom of recent GenAI. Our work will be important in publishing and mitigating the urgent risk from generated audio.

[1] Olivier, Raphael, Bhiksha Raj, and Muhammad Shah. "High-frequency adversarial defense for speech and audio." ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021

[2] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu, “Towards deep learning models resistant to adversarial attacks,” international conference on learning representations, 2018.

[3] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter, “Certified adversarial robustness via randomized smoothing,” in Proceed- ings of the 36th International Conference on Machine Learning, Kamalika Chaudhuri and Ruslan Salakhutdinov, Eds. 09– 15 Jun 2019, vol. 97 of Proceedings of Machine Learning Research, pp. 1310–1320, PMLR.

W - DeepFakeVox-HQ can not indicate out-of-distribution generalization

Apologies for any confusion. The DeepFakeVox-HQ test set featured in our paper comprises entirely out-of-distribution (OOD) samples. As noted in the caption for Table 2, our experiments include seven fake sources not utilized in the training set. This also applies to real samples from recent YouTube videos, which are OOD as well.

For future research, we provide additional test samples from seven fake sources that were included in the training set, although these were not used in the current study. We have clarified this at the beginning of the experiments section to prevent any further confusion.

Q1 - Whether attack is perceptible to human listener

Thank you for bringing this up. The attack we use to evaluate model robustness in our paper is imperceptible. We confirmed this through a human study where we presented 20 pairs of attacked and unattacked audio clips to 10 participants, who were asked to identify the attacked ones. The average prediction accuracy was 55% with a standard deviation of 7.58%, indicating that the attacked audio is nearly indistinguishable from the unattacked, resembling random guessing.

Additionally, following Reviewer EGHT's advice, we utilized Signal-to-Noise Ratio (SNR) to quantitatively assess the perceptibility of the attack. An SNR greater than 40 dB indicates that the attack is almost imperceptible.

Q2 - Consistency in adversarial training across baseline models.

Thank you for your question. Among all the baseline models we evaluated, RawNet3 emerged as the best. In Table 3, we've applied standard adversarial training (AT) to RawNet3 and compared it with F-SAT. F-SAT significantly outperformed standard AT on both unattacked and attacked data.

In the revised manuscript, we will include experiments applying adversarial training to all baseline models.

Q3 - How sensitive is F-SAT to the choice of hyperparameters, particularly the frequency ranges and perturbation magnitudes Thank you for highlighting this. We have included an ablation study of hyperparameters, such as frequency ranges and magnitudes, in Figure 9 (Figure 8 in revised version) of the main paper. The results demonstrate that our approach is not sensitive to these parameters.

We thank the reviewers for their insightful feedback. We are glad that the reviewers recognize the broad impact that releasing the DeepFakeVox-HQ dataset would have on the community. Additionally, our robust training method has been highlighted as a significant advancement in deepfake detection. We address further details and reviewer suggestions in the sections below.

W1 - Baseline models using contemporary adversarial methods would provide a fairer comparison

Thank you for your question. RawNet3 is the best baseline model among all the baseline we study. As reviewer suggested, we've applied standard adversarial training to RawNet3 and compared it with F-SAT in Table 3. F-SAT exceed standard Adversarial training method by 9% on orign data and by Average of 43% on attacked data.

We will include experiments that apply adversarial training to all baseline models in the revised manuscript.

W2 - The rationale behind the reliance on high frequencies The key rationale for our algorithm is that high frequency features help distinguish deepfake audio but are vulnerable to attacks, while low frequency features, though robust, are insufficient for training an effective detector. Our rationale is supported by three experiments:

1. We find the state-of-art Detection Model automatically rely on high frequencies for decision-making, as shown in Figure 2.
2. Figure 8(b) （Figure 7b in revised version） demonstrates that adversarial attacks on high frequencies reduce model performance more significantly, suggesting their vulnerability. Additionally, human ears are less sensitive to high frequencies, making those attacks even harder to notice, highlighting the need for secure high-frequency features.
3. Low frequencies feature alone cannot adequately distinguish deepfake audio. As indicated in the table, employing a Biquad-lowpass filter to remove high frequency features during training and testing will reduces accuracy on original data.

Therefore, to develop a powerful and robust detector, retaining high-frequency features is essential, but they must be secured. This is the rationale behind our F-SAT approach.

W3 - F-SAT's training efficiency

Thank you for your advice. We have included an evaluation of F-SAT's training efficiency in the appendix. F-SAT’s training efficiency is influenced by hyperparameters such as attack iterations and restart counts, which identify the worst-case perturbation. Drawing on insights from "Fast Is Better Than Free: Revisiting Adversarial Training," we optimized these parameters by setting restarts to one and attack iterations to one or two, while employing a larger attack magnitude to enhance robustness. Training time is shown in the table below.

Although F-SAT requires longer training times, it improves accuracy by an average of 9% on original data and 43% on attacked data compared to Standard Adversarial Training. We should not compromise accuracy merely to accelerate training.

Other Concerns — Open Source of Data and Augmentation Instructions

We have outlined our data augmentation setup in the file dataset_aug_all.py, included in the supplemental materials. The hyperparameters for augmentation and detailed instructions for running the commands are provided in the README.

Upon acceptance, we will open-source our code and data, and provide a comprehensive README to guide users.

W4 - Compared with other defense method

Thank you for directing our attention to the paper titled "High-frequency Adversarial Defense for Speech and Audio." [1] Following your suggestion, we compare ours methods with [1] and the baselines used in [1], including Gaussian Smoothing [3], and adversarial training [2] in the time domain—techniques. We employed the same parameters as those in [1]. The results of these comparisons are outlined below. As the table shows, our method outperforms all other methods.

[1] Olivier, Raphael, Bhiksha Raj, and Muhammad Shah. "High-frequency adversarial defense for speech and audio." ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021

[2] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu, “Towards deep learning models resistant to adversarial attacks,” international conference on learning representations, 2018.

[3] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter, “Certified adversarial robustness via randomized smoothing,” in Proceedings of the 36th International Conference on Machine Learning, Kamalika Chaudhuri and Ruslan Salakhutdinov, Eds. 09– 15 Jun 2019, vol. 97 of Proceedings of Machine Learning Research, pp. 1310–1320, PMLR.

W5 - SNR to evaluate how perceptible the adversarial attack is

Thank you for the suggestions. We have incorporated Signal-to-Noise Ratio (SNR) to assess the perceptibility of adversarial attacks in our main paper. The SNR for attacks in the time domain is approximately 58.4, while in the frequency domain it is 68.7. These values indicate the imperceptibility of the attacks.

W6 - Clarity issues

Thank you for the suggestions, we have revised the caption for Figure 9 (Figure 8 in revised version) and added results for the 0-8K range in Figure 8b (Figure 7b in revised version).

Q1- What frequency range is the spec-magnitude attack applied over in Figure 8a?

The spec-magnitude attack is applied across a frequency range of 0-8 kHz.

Q2 - Why Use F-SAT Instead of Adversarial Training in the Time Domain and Employ a Bandpass Filter for Post-Processing Perturbations?

Thank you for your question. Our F-SAT approach targets the magnitude component of audio signals, not affecting the phase. This focused strategy empirically enhances the effectiveness of our method.

Adversarial training in the time domain has been shown to be ineffective, as evidenced by our experiments (referenced in Table 3) and supported by reference paper, "High-frequency adversarial defense for speech and audio" (Table 1). The ineffectiveness stems from time-domain attacks impacting both the magnitude and phase of the audio. Even using a bandpass filter does not alleviate the phase change. Our studies show that phase-focused adversarial training in the frequency domain degrades the performance of the RawNet3 model on natural data.

Q3 - Why is the performance of the model trained on DeepFakeVox-HQ so low on the In-the-wild dataset

Training on all datasets except ‘In-the-wild’ and testing on ‘In-the-wild’ leads to poor performance. Conversely, training on ‘In-the-wild’ and testing on all other datasets results in even worse outcomes. This is likely due to significant distributional differences in the ‘In-the-wild’ dataset. Factors such as the synthesis models used, the quality of reference audio, and other methods of generating deepfake samples contribute to these disparities.

W1.4.5 - Quantitative measurement of synthetic speech quality using metrics like DNSMOS, or NORESQA.

Thanks for your advice, we have used DNSMOS [1] to quantitatively measure synthetic speech quality across these models, on a scale from 1 to 5, where higher values indicate better quality. We utilized sections of the VCTK and In-The-Wild datasets for this evaluation to assess model performance under both clean and noisy conditions.

Table: MOS Scores for VCTK Speaker_id: p244 Table for VCTK Speaker: p244

Table for In-the-Wild Speaker: Alan Watts

[1] Reddy, C. K., Gopal, V., & Cutler, R. (2021, June). DNSMOS: A non-intrusive perceptual objective speech quality metric to evaluate noise suppressors. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 6493-6497). IEEE.

W2 - The choice of accuracy as a metric Thank you for the suggestion. We will include both F1-score and EER in our revised version. The accompanying table presents comparisons with other defenses, detailing F1 and EER results. These trends align with our original performance metrics and relate to our response in reply-W4 to you.

W3 - previous work relevant to adversarial attacks in the frequency domain and reverting to the temporal domain

Thank you for bringing up the paper “Cross-representation Transferability of Adversarial Attacks.” While that study focuses on adversarial attacks, our work concentrates on defense. Additionally, our research conducts an in-depth study on the impact of adversarial training across various frequency domains, providing more insights and achieving state-of-the-art robustness results. We have cited and discussed this paper in the related work section of our revised version.

We thank the reviewer for their thoughtful feedback. We are glad that the reviewer finds our dataset to be significant and worthy publishing, as well as the effectiveness of our approach to improve deepfake dectection. We address the reviewer’s questions below:

W1.1 - The settings used for adversarial training during AT, F-SAT

Thank you for the suggestion. We have included the training hyperparameters for AT and F-SAT in our revised version, as shown in the table below.

W1.2 - Attack Settings for Evaluation and SNR-based Perceptibility Assessment

Thank you for the suggestion. We have detailed the attack settings for evaluation in the table below and included them in our revised paper.

W1.3 - Randaug Hyperparameter

Thank you for the advice. We have included the parameters of the augmentations used in RandAugment in the appendix. In our experiments, we configured aug_num=1 and set aug_prob=0.9.

W1.4.1 - Generate Deepfakes

Thank you for the advice. We have included the Dataset Details in appendix. We use the list of deepfake models (XTTS v2, StyleTTS v2, Metavoice, Whisperspeech, Vokan-TTS, and VoiceCraft and Elevenlabs) to generate deepfake voice for training set. to generate voices for the training set. Additionally, we employ Cosyvoice, PlayHT 2.0, Resemble, LOVO AI, and Lipsynthesis to create deepfake voices for the test set. we generate the noisy deepfakes with postprocessing augmentations.

W1.4.2 - The number of audios from each deepfake generation system

We utilize four real datasets—VCTK (12.0k), Librispeech-clean-100 (28.5k), Audioset (narration) (12.2k), and In-The-Wild (real parts: 9.3k)—to generate deepfake voices for the training set, with the number of samples from each source listed in the table below. For the test set, each source contributes 200 samples.

W1.4.3 - the demographic distribution of the real and fake speakers

We are sorry about that. Our dataset, drawn from multiple sources, makes it difficult to calculate the overall demographic distribution of speakers. However, for the deepfake audio we generated, the speaker distribution is a composite of VCTK, In-the-wilds, Audioset (narration), and Librispeech. This is because each source uses the same sentence inputs as the real audio.

W1.4.4 the number of utterances used from each of the datasets from prior works

The details of the reference datasets used are provided in the table below.

Thank you for providing detailed responses. I find that my concerns have been address and I have increased my score.

Please ensure that the tables below have been included and appropriately referenced in the manuscript, many of them have not yet been included. I have the following comments:

W1.4.3 -- For several of these datasets the speaker demographic data is available. I would advise that that data be compiled into a table and included in the paper. My main concern is that if the data is heavily biased towards a specific demographic, then the generalizability of the results may be questionable. For example, if most of the data is from male speakers, does that impact the ability to detect deepfakes with female speakers?

W4 -- please update results to use EER.

I would recommend that all plots and tables in the main text use EER.

Q2: Please include this explanation in the paper if it is not already there.