Bridging the Data Provenance Gap Across Text, Speech, and Video

We are extremely encouraged by Reviewer BAwp’s positive feedback, and recognition that this work addresses central bottlenecks in sourcing quality data in the wild. We also appreciate that the reviewer finds our analysis rigorous and highly relevant to real world practices. The reviewer has suggested that we better justify the data collection procedures that underlie the analysis, and extend our analysis of synthetic data. We propose changes below to address these recommendations, which we will incorporate into the paper.

How do the data collection & scoping methodologies ensure the analysis is empirically sound?

This is an important question, and one we agree is at the heart of the analysis. We spent a lot of time discussing and testing strategies for data collection and scoping methodology before collection. We would like to emphasize that by nature, this problem is difficult to solve: under-scoping datasets to just those found in one journal/conference, or even those already in a set of journals, is easier to define but can miss out on massive and important datasets. Many popular datasets are posted on HuggingFace or GitHub, not even with an arxiv paper, far less published to journals/conferences. Accordingly, to capture the broader “ecosystem” (where more impactful research questions can be answered), we realized it is necessary to go beyond this, and that scoping and discovery would need to be extremely rigorous.

We addressed these concerns in the following way:

We structured the search with a dedicated team for each modality led by experts within that domain. Each team relied on (a) their own expertise within that domain, (b) a mix of search tags from HuggingFace Datasets, (c) survey papers, (d) survey repositories, and (e) workshop/conference proceedings. External experts were also reached out to, in each case, to look for gaps in the coverage. Note that where possible, we sought to leverage prior work. Survey papers and repositories were particularly useful, as many are community crowdsourced, and driven by usage more than peer review, like workshops.

Overall, it is virtually impossible to be absolutely certain that the dataset coverage is perfect, but we believe this methodology is the most rigorous attempt to capture the full ecosystem, and we are fairly confident all widely used datasets are captured for each modality, within their predefined scopes. Moreover, as Reviewer Kmty pointed out, this paper is one of the first in this direction and our tools and annotations serve a call to action for the community, future dataset creators and their users.

Further, our goal is to ensure the database can be easily reused or extended, to facilitate continued research, as new datasets are added or scopes expanded. We will revisit this discussion in the methodology to better justify the process and more thoroughly describe the details.

Have we analyzed the characteristics of the synthetic data?

Yes—we consolidated a few key points in our general response above, and will add them to the paper.

Thanks again for your positive feedback and comments! Please let us know if you have further clarifications or suggestions.

We would like to thank Reviewer AQ3r for their constructive feedback and their strong support of our submission. We are particularly encouraged that they recognized the novel insights, and larger implications of the findings to practitioners. Their feedback centers around further analysis on dataset task trends, synthetically-generated data, and correlations between observed features. Based on their questions, we have run some numbers on our dataset annotations, and are happy to add these deeper analyses into the paper and appendix.

Can the source of synthetic datasets (e.g. the models they were generated from) be determined?

Yes! We’ve consolidated this information in the general response. Based on your comment, we’re happy to clarify this distinction further in the appendix of the camera-ready version.

Why was the scope of the datasets limited to generative tasks?

This is a great question. We decided to ground the work in tasks with relevance to generation, to find a reasonable bound to the scope and concentrate on generative models rather than purely discriminative tasks. Note that relevance to generation includes many tasks that are also discriminative but have been repurposed—e.g. Text classification datasets are widely used for instruction finetuning, once reformatted. For video, classification tasks such as Action Classification are actually considered generation as the video-label pair can be used in a generator to actually train such a video generation model. We provide a detailed note of what types of video tasks datasets are considered in the appendix. . All speech datasets are primarily for speech recognition, which is not always modeled as a generative task. Many can also be used for other tasks, which can be generative or discriminative. We will add further details in each modality’s methodology subsection.

Was there a variation in the tasks represented by the data across the years? It would be interesting to observe emerging trends in AI research over the years (e.g. code generation and other domain-specific tasks).

Thanks for raising this point! We agree that task trends are very interesting. While it didn’t make it into the main paper due to page constraints, we actually did collect fine-grained task metadata for every dataset, and report the top-level category distributions for each modality in Figure 7 (Appendix: “Additional Results”).

Prompted by your question, we plotted the increase in tokens and hours for each modality. Some clear trends emerge. For text, we immediately noticed that Code Generation has grown significantly since 2020-2022 whereas many more traditional NLP tasks like Summarization, Multiple Choice Question, and Question Answering have relatively subsided. For Video we notice emerging trends in Video Captioning, Video Q&A and Video Summarization have grown immensely in the 2020s. Additionally, we also see that Video Classification as a task has subsided in popularity, compared to its rise in 2013-2016, as more complex tasks such as Captioning and summarization have seen more success. We’d be happy to add these plots and analysis into the paper and appendix.

Was the correlation between data sources and language/geographical diversity analyzed?

This is also a great question. We conducted a preliminary analysis, first mapping countries to continents and consolidating categories to reduce sparsity. We then performed $\chi^2$ tests of independence (with Bonferroni correction across modalities) and Cramer's V effect sizes. We observed statistically significant associations for both video ($\chi^2=70.67, p<.0001, V=.302$) and text ($\chi^2=8014.76, p<0.0001, V=0.258$) with moderate effect sizes, but not speech ($\chi^2=24.33, p=0.434, V=0.248$) due to the prevalence of human and video sourced data across continents.

However, these results should be interpreted with caution due to prevailing sparsity despite our category aggregation. We view this as an important preliminary finding that warrants further investigation, potentially with more coarse-grained analyses better suited to sparse data. A more thorough analysis may reveal ways in which creators from different regions systematically differ in their data sources, adding context to our understanding of how geographic impacts dataset creation.

Finally, we’d like to thank you again for your positive feedback and incisive comments! Please let us know if you have further clarifications or suggestions.

We would like to thank Reviewer Kmty for their constructive feedback and thoughtful critiques. We appreciate their recognition of the importance of the findings around geographical representation and multilingual representation. The reviewer asks that we provide clearer calls to action and justify the long term value of the contributions. Below, we address both by pointing out relevant but underemphasized parts of the current paper, and propose changes that (a) center concrete calls to action and (b) discuss the longer term foundation this work provides for future research.

Ensuring the analysis can keep pace with a rapidly growing field

We agree it is important to enable continued data measurement for these contributions to remain relevant. However, we also do believe that large-scale retrospective analyses are valuable to inform the community in their own right. There are multiple ways this work is designed to facilitate continuous ecosystem-wide “data measurement”:

• First, we are open sourcing not just the annotations, but several pieces that helped us scale up the collection and analysis: (a) all annotation procedures/instructions (including the legal analysis, designed by domain experts), (b) the code for aggregating metadata from GitHub, HuggingFace, Papers with Code, Semantic Scholar, and (c) our many contributed taxonomies and mappings for organization geographies, task hierarchies, or use restrictions for ~100+ license types. All of these are organized to enable faster iteration and expedite new analysis. We believe this to be an important contribution to the wider community.
• Second, we’ve made an effort to identify and retroactively annotate three decades of historical datasets (1990 - 2024). Future analyses can simply import our CSVs and compare their own dataset domains against these or contribute the incremental additions back to our collection.
• Third, we are continuing to facilitate new contributions, and are working on having it crowdsourced, with internal validation checks so this can scale through community effort going forward while maintaining annotation quality. Our hope and vision is that our work’s contribution is not a fixed snapshot in time, but also a continued research effort that rallies stakeholders from various sectors.

All this is to say, we agree with your point, and are working towards addressing it, as well as providing a solid foundation for new analyses to adopt and build on. There are many pieces to our audit that this part wasn’t as emphasized, but based on your feedback, we will include the points above.

We would like to thank Reviewer Yvyf for their thoughtful and constructive feedback. We are particularly encouraged that they appreciated the scale of our analysis and found our results surprising and informative to the community’s future directions. We believe there are several feasible changes that will address their core concerns, which we outline here:

Comparing synthetic vs non-synthetic data—are they distributionally different?

We agree that this is an important point and discussing it would add useful insights for practitioners. We have added some key findings, taken from our audit, into the general response above. We are happy to flesh out these observations in a subsection in the paper, if the reviewer agrees that it addresses their recommendations.

Why allocate resources to study data provenance rather than create a new dataset?

This is a key motivating question for our work and one we passionately believe in. While we see the value in creating new datasets, we believe studying data provenance is critically undervalued. Within our work, we hope to make a convincing case for this. We propose making two changes to make this case: (a) adding the text below early in the paper, and (b) stressing in the discussion section the benefits/future work this analysis provides.

“It is difficult to improve on what you cannot measure. Data is fundamental to machine learning systems, and understanding (measuring) their qualities in detail is critical to understanding many real concerns and risks: bias, representation, toxicity, legal risks, privacy, effects on data creators (copyright/licenses), contamination and of course, model capabilities. There is a long history of machine learning models being developed with such issues, which could have been diagnosed earlier (or prevented) had they leveraged training data audits [1, 2, 3, 4, 5]. These retrospective analyses have re-directed and guided subsequent training data curation practices in significant ways. Understanding data trends at the ecosystem level informs key gaps and limitations of the data that is out there, in ways that new dataset creation cannot. From this work, we empirically demonstrate key gaps in sourcing copyright/restriction-free data, and geographically representative data. Our released repository also provides practitioners with the tools to filter and search for the best available datasets that do fit these criteria—including those that are not as well known.”

We believe our new contributions and tooling for analyzing data sourcing, restrictions, and representation meaningfully justify these resources. Please let us know if this is convincing, and addresses your concerns!

Given that the study includes datasets from 67 countries, when studying data licenses or terms of use, are differences in legal regulations of copyrights between countries/regions taken into consideration?

This is an important point as copyright laws vary across countries—we thank the reviewer for raising this. We note that the Berne convention complicates this analysis because signatories (which include a majority of nations) will give foreign works the same protection as domestic works. As a result, what matters is not only the origin of the data but also where the usage is taking place. We will clarify this complexity and cite some resources for practitioners to understand the law in some major jurisdictions: [6, 7, 8, 9, 10].