GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinement

Writing

• W1: Please use the citation format correctly. It should be \citep and \citet, depending on the usage. It seems like the current version is a direct transfer from NeurIPS, which uses \cite.
• W2: The organization of the paper is a little messy. There are too many bold texts with multiple hierarchies (for example Sec 3.1). At the same time, there are also numbered points in Sec 4. When finish reading the paper, I already get lost on what the contributions of the paper are. I suggest making the paper more organized. One simple improvement is to reference the section numbers in your contribution in the Introduction.
• W3: The discussion of related works is a little weak. You should mention other works that also automatically create audio datasets. For example, [1],[2], [3]. There should be at least one subsection in related works that discusses this topic.

Novelty

• W4: While I appreciate the release of a new dataset, I don't find the dataset curation approach much novel. In particular, everything besides the NST seems very trivial and purely engineering work. The proposed NST is similar to the approach taken by [3]. I would like the authors to provide a detailed explanation of how their approach differs from previous approaches.

Evaluation

• W5: $**This is the biggest issue: **$ How good exactly is the training dataset? There are only evaluation results of fine-tuned models on the test dataset, but to claim a usable dataset, you $**must**$ report the accuracy of the dataset (at least on a random subset). Currently, such disclosure on the training set is missing.
• W6: There have been concerns raised by previous works on training with synthetic datasets. For example, I would like to quote Whisper's paper [4]: "Recent research has shown that training on datasets of mixed human and machine-generated data can significantly impair the performance of translation systems (Ghorbani et al., 2021)." I would like the authors to provide a more detailed analysis of whether their NST approach would solve that problem. It will also be a significant finding if self-training will completely solve that problem, which means synthetical datasets could be used in the future.
• W7: If I understand correctly, Table 3 is comparing pre-trained models with fine-tuned GigaSpeech2 model. This seems a little unfair. If possible, I would like to see fine-tuned public models as well.

Reference

[1] Sun, Luoyi, et al. "Auto-ACD: A large-scale dataset for audio-language representation learning." ACM Multimedia 2024. 2024.

[2] Liu, Houjun, et al. "Automation of language sample analysis." Journal of Speech, Language, and Hearing Research 66.7 (2023): 2421-2433.

[3] Yeroyan, Ara, and Nikolay Karpov. "Enabling ASR for Low-Resource Languages: A Comprehensive Dataset Creation Approach." arXiv preprint arXiv:2406.01446 (2024).

[4] Radford, Alec, et al. "Robust speech recognition via large-scale weak supervision." International conference on machine learning. PMLR, 2023.

-this is overall a good recipe for collecting large and good quality speech resources, but it applies well-known techniques and eventually we get a resource with 3 languages covered only (extending language coverage is mentioned as future work)

-for readers not familar with NST method, section 3.2 is not easy to understand and would benefit from a bit more context on base NFT method before describing how it was modified by the authors

-the comparison to existing ASR systems is convincing for Thai but less convincing for Vietnamese and Indonesian (although it can be that commercial systems that outperform the Gigaspeech 2 based ASR system might have been trained on commonvoice and fleurs datasets)

-multidimensional filtering and its impact could have been more detailed (how much data is rejected for each specific filter, etc.)

Despite the usefulness of the dataset itself, the main weakness is that the general novelty is limited in this work. The pipeline itself does not have many components that are significantly different from the existing dataset collection procedures. For example, one of the main contribution authors claim is to apply Noisy Student Training iteratively to refine collected dataset, however, this is a quite standard approach and is already applied in one of the original Noisy student training paper (e.g. https://arxiv.org/pdf/1911.04252). It would be great that authors can provide more novelty how their refinement can be distinguished from the existing approach.

Another minor concern is this dataset only contains Thai, Indonesian, and Vietnamese, and their target audience might be too limited in the context of ICLR. It would be nice if authors can extend their approach to much more languages and demonstrate they are useful.

• Although GigaSpeech 2 spans multiple domains, the performance drops on out-of-domain test sets (Common Voice, FLEURS) in Indonesian and Vietnamese suggest limitations in cross-domain generalizability (Table 3).
• Even though the authors mention zero reliance on the labelled data, the approach is strictly dependent on having an initial good seed model which can be used for transcription and timestamp prediction; hence, this approach does not completely preclude the need for labelled data.
• The quality of the training data released as a part of this work depends on the robustness of the Whisper model

The paper describes construction of a language resource and empirical ASR results demonstrating essentially that training on more relevant data helps lower WERs. Evaluation was on 3 languages only. Most importantly, the paper does not present a scientific hypothesis or any advancement in methodologies; it is purely a resource and evaluation paper and as such would be better place in a speech conference or language resources conference.