SpeechFake: A Large-Scale Multilingual Speech Deepfake Dataset Toward Cutting-Edge Speech Generation Methods

Q3: Performance on Multilingual Dataset

The low EERs in Table 5 result from smaller test sets and fewer generation methods, making the tests less challenging than the broader BD sets. AASIST, without pretraining, shows suboptimal performance on unseen languages (e.g., EER = 9.57% for Hindi, 4.90% for French), highlighting the dataset’s challenges. The lower EERs for W2V+AASIST reflect the advantage of Wav2Vec XLSR’s multilingual pretraining, demonstrating the dataset’s utility in evaluating models across language mismatches and unseen conditions.

However, it is inaccurate to conclude that the multilingual dataset has limited value. Without a multilingual pretrained model, the language gap presents significant challenges for detection. Even with such a pretrained model, testing on the complete dataset (including the hidden portion) for each language can result in reduced performance, as demonstrated in the table below.

Table R3: EER (%) results for each language across the entire dataset, using W2V+AASIST models trained on English and Chinese.

Q4: Confidence Intervals for Low EERs

We appreciate the reviewer’s suggestion to include confidence intervals for settings with very low EERs. In the updated Table 6, we have added confidence intervals to all EER values to provide a clearer understanding of the results. The updated table also shows that the conclusions remain consistent with the original analysis.

Table 6*: EER(%) results of cross-speaker testing trials.

Q5: About the Title

We chose the word “toward” to emphasize the dataset’s role in advancing speech deepfake detection by incorporating more advanced speech generation methods. While our dataset includes cutting-edge techniques, it also retains classical methods to ensure comprehensive coverage. Using “generated using” would inaccurately suggest the dataset exclusively features cutting-edge methods.

Q6: About the Contribution

We appreciate the reviewer’s observation regarding the third contribution. To address this, we have rephrased it to emphasize its distinct value: “SpeechFake provides a valuable resource for developing robust deepfake detection models, demonstrating superior performance on existing datasets. It also supports future research in improving model generalization and advancing detection strategies for emerging techniques.” The revised contribution now highlights SpeechFake’s immediate utility in demonstrating superior performance on existing datasets and its long-term support for advancing future research, such as improving model generalization and detection strategies. In contrast, the first contribution focuses on introducing the dataset’s scale, diversity, and features, while the second emphasizes the experiments and analyses conducted.

Q7: Symbol “-” in Table 1

The “-” in Table 1 indicates that the dataset does not specify the number of speakers or generators included. We have clarified this in the table caption for the revised version of the paper.

Q8 & 9: Description in related work

For the first part of the Related Work section, Speech Generation, the sentence “Speech generation, or speech synthesis, can be broadly divided into two main tasks: Text-to-Speech (TTS) and Voice Conversion (VC)” does not imply that these are the only speech generation tasks. Rather, it highlights TTS and VC as the primary tasks in this domain. Other tasks, such as speech content editing and style conversion, often build upon these foundational tasks and share similar methods with modifications tailored to their specific objectives. Regarding Neural Vocoder (NV), it is not a standalone task but a fundamental component of many speech generation methods. Since vocoded speech plays a significant role in detecting deepfakes, we included such data in our dataset to enhance its utility.

For the second part, Speech Deepfake Datasets, the differences between our dataset and existing ones are stated in the Introduction. To make this clearer, we have added an additional description in the revised version of the paper to explicitly highlight these differences within the Related Work section.

Q10: About “Speaker Information”

We have revised the description to “Providing identity labels for the real speaker or the generated voice.” for clearer expression.

Q11: Unclear statement for description of waveform / mel

Thank you for your feedback regarding the figures and descriptions in the appendix. Below, we briefly clarify the terms for the unclear parts.

For waveforms:

• Speed and Rhythm: Reflected in the density and spacing of peaks, with faster speech showing more densely packed peaks and slower speech wider spacing.
• Accent and Intonation: Variations in peak amplitude indicate stress or emphasis, characteristic of accents or intonation patterns.
• Background Noise: Irregular, low-amplitude fluctuations suggest noise, while smoother patterns indicate cleaner audio

For mel-spectrograms:

• Energy Distribution: Represented by brightness across frequency bands.
• Frequency Patterns: Harmonic structures show pitch (spacing between harmonics) and stability (consistency of patterns).
• Clarity: Defined by sharpness of features; blurriness indicates noise or distortion.

We have updated the descriptions in the revised paper for improved clarity.

Ethics Review: Privacy, security and safety

To clarify, the dataset does not contain deepfakes of identifiable real people. Regarding the voice, it may originate from training data (TTS), reference voice (VC), or original audio (NV).

• From training data: the TTS systems used are typically trained on publicly available datasets that do not include identifiable speaker information. In cases where systems are trained on private data, we can only select voices from the output without any connection to real individuals.
• From reference voice or original audio: The voices for VC and NV systems are similarly taken from publicly available speech datasets without any identifiable speaker information.

For the content, all text or speech samples are taken from publicly available speech datasets commonly used in speech generation research. These datasets do not contain harmful or sensitive content.

Besides, publicly released datasets in prior work follow a similar approach, using open-source tools and public speech datasets for generating deepfake data. These datasets have not been associated with privacy or security concerns.

The dataset will primarily be released under CC BY-NC 4.0, with certain portions licensed under GPL-3.0 to comply with the requirements of specific tools.

We sincerely thank the reviewer for their thorough evaluation of our paper and for providing insightful comments and constructive feedback. Below, we address each of the points raised.

W1: Empirical Comparisons with Other Datasets

In our initial experiment, we report baseline results for models trained on ASV19, as it is a widely used setup for this task, and many existing datasets are too small to serve as training sets and are primarily used for testing. Results for models trained on other datasets will be included in the next revision of the paper, as thorough training and evaluation require significant computational resources and time. Meanwhile, we have added results for models trained on ASV19 and SpeechFake and tested on additional test sets in Table 10 of the revised paper, with a partial table provided here. These evaluations span 10 test sets, covering both earlier and more recent datasets. For most test sets, models trained on SpeechFake consistently outperform those trained on ASV19, demonstrating its effectiveness in capturing diverse and challenging spoofing scenarios.

Table 10*: Performance evaluation (EER%) of model (W2V+AASIST) trained on ASVspoof2019 (ASV19) or Bilingual Dataset (BD) across multiple test sets.

W2: Narrow Contribution

We appreciate the reviewer’s observation and would like to clarify that our work extends beyond simply presenting a dataset and baseline results. In addition to establishing baselines, we conduct detailed analyses that explore critical factors influencing deepfake detection performance, such as generation methods, language diversity, and speaker variation. These analyses provide new insights into the challenges of generalization, a key issue in deepfake detection that transcends specific domains like speech.

W3: Dataset Construction Details

We have added more details about the generation process in Section 3.1 of the revised paper. Regarding the selection of multiple speaker IDs: For TTS systems, we use the voices provided by each method and balance the selection to include a mix of genders whenever possible. For VC systems, we sample reference voices from real datasets, ensuring diversity and representativeness. For NV systems, the generated speech retains the same voice characteristics as the original input audio from the real datasets.

Q1: Quality Filtering

The detection of low-quality sounds is performed through human verification. Before the batch generation, we generate over 100 diverse sample audio clips for each system. Systems producing artifacts, silences, or other issues were filtered out at this stage. Only after passing this filtering process did we proceed with batch generation to create data covering different languages and voices. After generation, we applied voice activity detection (VAD) to filter out speech segments with less than 0.5 seconds of active speech. Additionally, random samples from each system were manually reviewed throughout the process to ensure consistency and naturalness.

Q2: Different Detection Performance

The differences in detection performance across Tables 3, 4, 5, and 6 are due to variations in training and testing settings, as described in each experiment section:

• Tables 3 and 4 present results on the BD (Bilingual Dataset), using different subsets for training or testing to evaluate overall performance (Table 3) or cross-generator performance (Table 4).
• Table 5 focuses on the MD (Multilingual Dataset), which uses entirely separate train, dev, and test sets from MD instead of BD, with differences in languages and generation methods.
• Table 6 evaluates a smaller dataset selected from BD, using a single generation method but different generated voices to study cross-speaker performance.

The improved results in Tables 5 and 6 are primarily due to the smaller test sets and fewer generation methods covered, which make the evaluation less challenging compared to the broader and more diverse settings in Tables 3 and 4.

Q4 & Q5: Selection of Generation Systems and Human Verification

To select speech generation systems, we first identified open-source tools and APIs that claim advanced performance, prioritizing those supported by technical papers or reports. We implemented their released codebases or used their APIs to generate over 100 diverse sample audio clips for each system. These samples underwent human verification to ensure natural-sounding quality. Systems producing artifacts, silences, or other issues were filtered out at this stage. Only after passing this filtering process did we proceed with batch generation to create data covering different languages and voices. After generation, we applied voice activity detection (VAD) to filter out speech segments with less active speech. Additionally, random samples from each system were manually reviewed throughout the process to ensure consistency and naturalness.

We sincerely thank the reviewer for their thorough evaluation of our paper and for providing insightful comments and constructive feedback. Below, we address each of the points raised.

W1: Multilingual Focus

We acknowledge the reviewer’s point that English and Chinese are prioritized in our dataset. This is due to practical limitations, as most open-source generation tools primarily support English (and sometimes Chinese). To address this while enabling multilingual research, we structured our dataset into two parts: BD (Bilingual Dataset) for English and Chinese and MD (Multilingual Dataset), which covers 46 languages.

In our experiments, MD is used for cross-lingual evaluation to test whether deepfake detectors can generalize across languages while minimizing the influence of generator differences. The train and dev sets include only English and Chinese data, while the test set is divided into 10 subsets—9 for primary languages and one combined subset for the remaining 37 languages, covering all 46 languages.

In total, MD includes over 1 million samples across 46 languages, with approximately 20,000 samples per language. For our experiments, we selected 5,000 samples per language for testing, leaving the remaining samples as a hidden set. We plan to release this hidden set as well (for both BD and MD) to support further research.

W2: Coverage of Speech Generation Tools

Given the numerous existing speech generation methods, many of which overlap or combine variants, it is challenging to cover every method comprehensively. Older generation methods are already well-represented in existing datasets, making it redundant for us to include them again. Instead, we focused on advanced and underrepresented generation methods. We carefully selected better-performing, newer, or representative tools to maximize the diversity and relevance of the dataset.

As speech generation tools continue to evolve, new methods will undoubtedly emerge. While it’s impractical to update the dataset with each new method immediately, we plan to release updated versions periodically once a sufficient number of new methods have accumulated.

Q1: Generation Process Details

We have added more details about the generation process in Section 3.1 of the revised paper. The number of utterances generated for each tool is primarily determined by the selected text and corresponding speech from the real datasets, which form the foundation of the generated content. The basic number of utterances is guided by the languages supported by each method. For methods with additional features (e.g., OpenVoice, CosyVoice) or specific use cases (e.g., Tortoise for cross-speaker experiments), we generate supplementary data to explore these aspects. Finally, post-processing steps, such as quality filtering, may result in some utterances being excluded.

Q2 & Q3: Test on Other Datasets

We added results on 10 additional test sets in Table 10 for the revised paper, and provide a partial table here. An observation is that models trained on the full BD dataset do not always outperform those trained on its subsets. This may be due to the distinct generation methods in BD-EN and BD-CN, which align more closely with certain test sets. In contrast, the broader diversity of the full BD dataset can sometimes dilute this alignment.

Table 10*: Performance evaluation (EER%) of model (W2V+AASIST) trained on ASVspoof2019 (ASV19) or Bilingual Dataset (BD) across multiple test sets.

We sincerely thank the reviewer for their thorough evaluation of our paper and for providing insightful comments and constructive feedback. Below, we address each of the points raised.

W1: Dataset Contribution

We acknowledge that our work is, in some respects, an incremental addition. However, SpeechFake’s large scale, advanced generation techniques, and multilingual coverage address key gaps left by prior datasets and provide essential advancements for the anti-spoofing community. No previous dataset offers this combination of scale and diversity, which is vital for building detection systems that generalize across languages, generation methods, and real-world contexts.

W2 & Q1: Additional Analyses and Use Cases

We appreciate the suggestion to expand our analyses and use cases. For additional analysis, we prioritized establishing robust baselines and analyzing key factors—generation methods, language diversity, and speaker variation—to align with the dataset’s primary purpose of supporting generalizable and comprehensive evaluation. While further analysis could offer deeper insights, our focus was to provide foundational results that future studies can expand upon. For use cases, we have included several additional use cases in the appendix, such as applications in privacy-preserving deepfake detection, automatic speech recognition robustness, and adversarial attack resilience.

Q2: More data for TTS-Tortoise and VC-OpenVoiceTTS

These two systems contain more data because they support a wider variety of voices. For TTS-Tortoise, we generated additional voices within one architecture to study cross-speaker performance. For VC-OpenVoiceTTS, its multi-speaker and style transfer capabilities allowed for more varied outputs, enhancing analysis across voice and style variations in voice conversion.

Q3: About BD-UT

We separated BD-UT because it serves as an expansion of our test set, specifically designed as an unseen test set. BD-UT includes high-quality commercial data, and since we lack details about their implementation, it allows us to evaluate whether deepfake detectors can generalize to real-life generation methods effectively. Regarding version details for the APIs, many commercial services do not provide explicit versioning information for their tools. Instead, we include the voice names provided by the APIs in the dataset metadata.