SingNet: Towards a Large-Scale, Diverse, and In-the-Wild Singing Voice Dataset

The paper's primary limitations revolve around dataset accessibility and the comprehensiveness of the experimental work. While large-scale datasets are invaluable for research communities, the copyrighted nature of this dataset (88.7% from commercial sample packs, the rest from copyrighted music) prevents its release. This significantly diminishes the novelty claim of the dataset itself, as its inaccessibility hinders broader research impact.

The experiments lack sufficient breadth and depth in several areas.

• In transcription, the comparison between speech and singing voice pre-training on Wav2Vec2 shows only a 1% absolute difference after fine-tuning. Fine-tuning experiments on a more capable model, Whisper, are absent.
• In vocoding, the comparison methodology is inconsistent. While the authors fine-tune BigVGAN with the proposed dataset, they train NSF-HiFiGAN from scratch with both the OpenVPI and proposed datasets. A more balanced comparison would include fine-tuning experiments with the OpenVPI dataset as well.
• In voice conversion, the intriguing observation that the 35-hour subset outperforms the full 3500-hour set on some objective metrics warrants deeper investigation and discussion. No such is provided. This leads to doubts on the dataset quality distribution.

Given the dataset's closed nature and the questionable results from pre-trained models, the open-source data processing pipeline emerges as the paper's most potentially novel contribution. More detailed documentation on this is necessary, especially considering the potential high costs associated with reproducing the pipeline due to the use of multiple commercial plugins.

To improve the paper, the authors should provide a more detailed analysis of the data processing pipeline, including potential alternatives to commercial plugins for broader reproducibility. The experimental section should be expanded to include fine-tuning experiments on Whisper for transcription, a consistent comparison methodology for vocoding, and an in-depth analysis of the performance discrepancy between the subset and full dataset in voice conversion. Additionally, the authors should discuss potential ways to make the dataset or a subset of it accessible to the research community, considering copyright constraints.

A minor comment: please check spelling and uppercase/lowercase usage. Several examples include: Mert (Page 7) should be MERT, Wav2vec2 (throughout) should be Wav2Vec2, MERT is misspelled as MRTE on Page 8, Openvpi/OpenVPI is not used consistently throughout.

1. Lack of Innovation: The paper closely resembles another paper's process, Emilia, without clearly articulating its improvements over Emilia. Additionally, the paper primarily leverages existing work from others, with minimal novel technical contributions. The methods related to neural vocoding and data augmentation could benefit from more comprehensive referencing to acknowledge the contributions of earlier studies.

2. The data processing pipeline is crucial but is not described with sufficient clarity or detail, and no rationale is provided for the choice of tools, making it unconvincing. In-the-wild data, not recorded in controlled studio environments, may still contain noise, reverberation, and other audio quality issues after source separation and audio restoration, especially for more complex vocal segments. A more detailed explanation of the data processing methods is needed.

3. Although the paper compares the dataset with existing ones in terms of scale and diversity, it falls short in directly comparing its performance with other existing datasets. For instance, while the paper demonstrates SingNet's effectiveness in tasks like singing voice synthesis, it lacks direct comparisons with similar large-scale datasets on the same tasks, making it difficult for readers to ascertain SingNet's advantages in practical applications.

4. The reliability of the data processing results is questionable. The paper mentions multiple singing voice data processing pipelines but does not provide any intermediate audio demos, making it impossible to determine whether the pipeline truly yields high-quality audio. The assumption that certain modules or pre-processing techniques directly enhance performance is not thoroughly tested or validated. Furthermore, the remaining quantity of data after cleaning is not specified, making it unclear how much data is retained at each step.

5. The SingNet dataset contains a multitude of languages, It is unclear whether the language distinctions are entirely accurate, and what the error rates might be. This aspect is not detailed, yet for voice, especially singing voice, misjudging a song in one language as another can introduce bias in training.

6. The vocoder comparison for singing voices should ideally use singing data exclusively: The Large-Compilation data used in the vocoder comparison experiment primarily consists of speech data, which should be replaced with a compilation of previously published singing voice datasets for a more reasonable comparison.

1. The authors claim that the proposed dataset can boost the development of singing voice synthesis (SVS). However, the common definition of SVS is synthesizing singing voices from lyrics and melody annotations (such as MIDIs or F0s). There is no evidence or verification in this paper to support that the proposed dataset supports such SVS tasks.
2. A similar concern, in lines 33-35, the authors claim that the way ACEStudio creates singing voices is manpower-consuming and inconvenient for scaling up, because it requires tuning the in-detailed pitch, phoneme, and duration information. Even so, ACEStudio can generate singing voices from textual annotations. I can't see how the proposed dataset can be used to train a model that generates singing voices without such tedious tuning.
3. The comparison with current datasets is somehow unfair. Datasets, like NHSS, CSD, M4Singer, Opencpop, etc., consist of detailed annotations, such as words, phonemes, word/phoneme durations, MIDI sequences, F0 sequences, etc., which prevents the scaling up of the data. However, the authors did not propose a new automated method to accelerate this process or reduce its cost. Instead, they chose to bypass it altogether; that is, their preprocessing method does not include any text alignment processes like MFA, or MIDI extraction processes. This results in their dataset containing only audio, limiting their experiments to ASR (which further needs Whisper as the rater), vocoder, and SVC.
4. Even though the simplicity of the proposed dataset restricts the authors to conducting only semi-supervised or audio-only experiments, they are still not comprehensive enough. For example, for audio-only reconstruction or compression methods, the authors only investigate vocoders, such as BigVGAN and NSF-HiFiGAN. However, with the advancement of voice generation methods with language modeling techniques, discrete representations like audio codecs are also an important part of acoustic representation. This paper lacks evidence of the effectiveness of their data on codec tokenizers.
5. From the perspective of an audio-only dataset, this paper also lacks many statistical conclusions. For example, how many singers are in this dataset? What is the gender structure? What is the age structure? What is the voice part structure?

In conclusion, this paper lacks significant contributions or contains overstated contributions. From a methodological perspective, the data preprocessing approach is simple, being a direct application of various SOTA methods (not even applied in a new domain); from a dataset perspective, although this dataset is currently the largest, it lacks many of the basic annotations required for SVS and is missing crucial statistical information.

• The statistics of the dataset can be more fine-grained, such as the distribution of singer gender, the pitch distribution of each gender, and the sub-distributions of each language and style.
• Despite possible difficulties of doing this on large datasets, the lack of fine-grained MIDI and phone duration annotations makes it relatively challenging for applying this dataset to singing voice synthesis (SVS) tasks. Some MIDI notation and alignment models may help the annotation.