DCA-Bench: A Benchmark for Dataset Curation Agents

1. Regarding Domain Scopes

Our DCA-Bench is an initial step toward this important topic and aims to provide some preliminary insights. A more detailed analysis of domain-specific data with unique properties would undoubtedly be valuable but might require deeper domain-specific knowledge and more refined designs. Modalities beyond text are likely to introduce additional challenges. However, we believe the core components and frameworks of the benchmark—such as issue description, issue context, and ground truth collected from user reports—can be adapted to other domains and modalities.

Incorporating domain-specific or multi-modal content would make issue detection even more challenging, emphasizing the need for carefully designed hints to enable more precise evaluation. Our LLM Evaluator could also be replaced with tools integrating multi-modal or domain-specific knowledge, thereby enhancing its ability to assess the performance of LLM agents.

In conclusion, our approach has the potential to generalize to other domains, such as image datasets. We aim to explore these possibilities further in future work.

2. Regarding Agent-Testing Framework

We would like to clarify that our baseline Curators are already implemented within an agent framework. The experiments for the Baseline Curators are based on the OpenAI Assistant API, which provides tools including a code interpreter, file processing, and storage. Therefore, the baseline Curators evaluated in DCA-Bench are LLM agents with certain tool usage, as opposed to pure LLMs.

We envision that considering more tools and introducing more agents for collaborative problem-solving (as mentioned on the left of Fig. 2) are meaningful and exciting next steps. The API design of DCA-Bench can accommodate the evaluation of more complicated Curator agents, as we have taken these into consideration.

3. Availability of Open Source Code

According to the ICLR 2025 Author Guide, we cannot provide any URL that might reveal our identity, so we uploaded all of our code as supplementary materials in our submission, where we also provided information about how to download our dataset. Please refer to the “Reproduction Guidelines” in the README file in the supplementary materials for more details.

4. Applicability to Datasets like ImageNet

Thank you for raising this point! Please see our response to point 1 for more details.

5. Plan to Publicize Our Benchmark

Yes, we will release our benchmark code and dataset once we get accepted.

6. Format Issue

We appreciate you bringing this to our attention! The formatting issue has been addressed in the updated draft.

1. Limited test set size

Many well-established LLM benchmarks have sample sizes at hundred levels: HumanEval [1] (164), MTBench [2] (80), and ArenaHard [3] (500). As an initial step toward the novel direction of automatic data curation, we believe the size of the current test set is sufficient for a robust evaluation. In addition, DCA-Bench features a 4-level hints design, resulting in actually 884 test cases in total. However, we recognize that increasing the dataset size could further enhance robustness and improve the balance of test cases across different categories, which is an important future work.

$1$ M. Chen et al., “Evaluating Large Language Models Trained on Code,” Jul. 14, 2021, arXiv: arXiv:2107.03374. doi: 10.48550/arXiv.2107.03374.

$2$ L. Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” NeurIPS 2023

$3$ T. Li et al., “From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline,” Jun. 17, 2024, arXiv: arXiv:2406.11939. doi: 10.48550/arXiv.2406.11939.

2. About Related Works

We appreciate your suggestions on the related works. These works are all innovative and meaningful; while they primarily focus on improving data management, processing, or problem-solving frameworks, our work is distinguished by its focus on identifying hidden issues in public datasets, such as factual or logical errors that require semantic understanding, and usability aspects like document quality and licensing—areas that are critical but often overlooked in related research. The detailed discussions of these relevant works are as follows:

• LLM-Enhanced Data Management [1] designed and implemented a data management system that applied LLMs for data management tasks, such as database diagnosis and data analytics, and also shows the potential of using such a data system to enhance the performance of LLMs. However, it is more relevant to database management where LLMs are asked to retrieve data for analytics and reasoning or reorganize data using a series of APIs and tools. This work didn't target detecting the issues within a single dataset, highlighting a different task scope between DCA-Bench and this work.
• Data Juicer [2] serves as a comprehensive data processing system for LLM training, which provides numerous operators that can be used to generate a high-quality dataset from a mixture of data sources. However, the data quality enhancers in Data Juicer are rule- or metrics-based filters or deduplicators, which cannot handle issues like factual errors or logical errors that require semantic understanding—what we refer to as "hidden corruptions" in DCA-Bench (please refer to Section A1.2 for more details). Besides, it also didn't focus on dataset-level issues, where usability—including the quality of the document and the presence of licenses—matters a lot. This highlights the novelty of the tasks in DCA-Bench.
• DS-Agent [3] proposes an agentic framework for solving data science problems, such as Kaggle competitions. While this work leverages LLMs for data science tasks, it primarily aims to develop a problem-solving agent framework. In contrast, DCA-Bench has different motivations and focuses on identifying and addressing dataset issues rather than developing agents for data science.

We have integrated these discussions into the revised draft, marked with blue under “Related Works”.

$1$ Zhou, Xuanhe, Xinyang Zhao, and Guoliang Li. "LLM-Enhanced Data Management." arXiv preprint arXiv:2402.02643 (2024).

$2$ Chen, Daoyuan, et al. "Data-juicer: A one-stop data processing system for large language models." Companion of the 2024 International Conference on Management of Data. 2024.

$3$ Guo, Siyuan, et al. "DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning." arXiv preprint arXiv:2402.17453 (2024).

3. About Writings

Thank you for your thorough review and feedback on writing! We have addressed all of them in our latest draft. For easier comparison, we have marked these changes in blue.

4. Applicability to different types of datasets:

Our DCA-Bench is an initial step toward this important topic, and we intend to provide initial insights. More detailed analyses of domain-specific data with special properties are meaningful but should need more domain-specific knowledge and more refined designs, which we leave for future works. Currently, we classify those issues according to the type of issues themselves rather than domains, where we provide clarifications and analysis regarding some challenging issue types, such as "cross-file discrepancy" where the agent needs to spot discrepancies of the same information presented in multiple files. Please see A3.3 for more details.

1. About Realistic Testing Environment and Coverage of Issue Cases

We agree with the reviewer that a more realistic testing environment can further help develop and test candidate models. At the current stage, we chose to simplify the setup to focus on common concepts. Regarding comprehensiveness, our DCA-Bench serves as an initial step in this direction. While it may not yet cover all real-world issue types, it already provides a very challenging and meaningful testbed, as demonstrated by our experiments. Moreover, its structured taxonomy of issue types and tags, detailed in Sections A.3.2 and A.3.3, provides further depth and utility.

2. Hints Design and the Performance of Baseline Agent

• We want to clarify that the design of multi-level hints is a feature of DCA-Bench; we didn’t mean to use it to improve the performance of the baseline agent. With credit to this design, we carried out a more detailed analysis of the overall performance with different hints in Section A.3.2.
• It is not uncommon that the baseline methods have relatively low performance in new LLM Benchmarks [1, 2] that target intrinsically difficult tasks. The dataset issue detection task is quite challenging for LLM agents, requiring them to reason under a very long context, as shown in Tab.1. We have also analyzed the challenges posed by the issues in DCA-Bench, categorizing them by issue tags as detailed in Section A.3.3.

[1]: Jimenez C E, Yang J, Wettig A, et al. Swe-bench: Can language models resolve real-world github issues? ICLR 2024

[2]: T. Li et al., “From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline,” Jun. 17, 2024, arXiv: arXiv:2406.11939. doi: 10.48550/arXiv.2406.11939.

3. Potential biases of LLM Evaluators

In Lines 379-381(Lines 390-393, in the revised version), we have considered two common biases among LLMs: self-preference bias [1, 2] and length bias [3,4]. In Section A. 2.4, we have carried out well-designed experiments for both of them. Our results showed that these biases are negligible in our LLM Evaluator.

• For the self-preference experiment, we used GPT-3.5-Turbo and GPT-4-0125-preview as the backend models for both the Curators and the Evaluators, leading to four combinations of Curator and Evaluator settings. For each Evaluator, we calculated the Average Odds Difference (AOD), a metric that measures fairness by assessing differences in true positive and false positive rates between two groups. A value of zero indicates perfect fairness, with equal model performance across both groups. Higher AOD values suggest increasing disparity, with the model performing unevenly between the groups. Derived from Tab.7 and Tab.8, we showed that our GPT-4-0125-preview based Evaluator has an acceptable AOD ($5.31(\pm2.76)\\% < 10\\%$), which indicates the bias is within an acceptable range [5]. Furthermore, we observed a great alignment between the GPT-4-0125-preview based Evaluator and human annotations, on both the GPT-3.5-Turbo based Curator and the GPT-4-0125-preview based Curator, as shown in Tab.9.
• For length bias, we used GPT-3.5-Turbo (0125) and GPT-4o-mini (0718) as the dataset curator agents. After the Curator generated the results, we fed its response to another LLM (gpt-4o-2024-08-06) to expand upon its generation. While the most common bias is the preference towards longer responses, we also tested the case where the response is more succinct. As shown in Tab.10 and Tab.11 of our draft, the changes in the success rate given by the LLM Evaluator are negligible in both cases, indicating that our Evaluator has no significant length bias.

$1$ A. Panickssery, S. R. Bowman, and S. Feng, “LLM Evaluators Recognize and Favor Their Own Generations,” Apr. 15, 2024, arXiv: arXiv:2404.13076. doi: 10.48550/arXiv.2404.13076.

$2$ L. Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” NeurIPS 2023

$3$ Y. Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto, “Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators,” COLM 2024

$4$ E. Durmus, F. Ladhak, and T. Hashimoto, “Spurious Correlations in Reference-Free Evaluation of Text Generation,” ACL 2024

$5$ S. Majumder, J. Chakraborty, G. R. Bai, K. T. Stolee, and T. Menzies, “Fair Enough: Searching for Sufficient Measures of Fairness,” Mar. 21, 2022, arXiv: arXiv:2110.13029. Accessed: Aug. 13, 2024. $Online$ . Available: http://arxiv.org/abs/2110.13029.

4. Scale to Multimodal Data

We appreciate the reviewer’s suggestion. While DCA-Bench is currently text-focused, we believe it can be scaled to multimodal data, such as image or audio datasets. However, this would require more refined designs to address the unique complexities of multimodal data quality issues. We consider this an important direction for future work.

We thank the reviewer for the detailed and thoughtful comments. We address your individual concerns below.

1. Regarding Evaluation Details of the LLM Evaluator

We want to clarify that the main point we want to deliver in L332 - L335 (L342 - L345 in the revised version) is that we didn’t optimize the prompt design of the LLM Evaluators on the test data used for Tab2. As we mentioned in Line 344 (L354 in the revised version), the test cases are randomly selected, thus we didn’t try to cherry-pick the test data to make the performance better. The reason that test data are randomly collected after the prompt design is precisely to avoid overfitting in the prompt design. We have also included the labeled test data under pipeline/label.json in our uploaded supplementary materials for your reference.

2. Taking High-quality Dataset into Consideration

We sincerely appreciate the reviewer’s observation and the opportunity to address this important concern. Our focus on datasets reported to have issues is based on two key considerations: annotation challenges and practical utility.

• Annotation Challenges: Labeling a dataset as “issue-free” is inherently challenging. Human experts tasked with identifying issues must thoroughly examine the dataset, yet subtle or latent issues may go undetected. Conversely, labeling a dataset as “containing an issue” is more straightforward, as a single verified anomaly suffices to confirm its status. This asymmetry in annotation reliability underpins our choice of evaluation datasets.
• Practical Utility: From a dataset curator’s perspective, optimizing for high recall (i.e., identifying as many potential issues as possible) is generally more beneficial than maximizing precision. False positives, while not ideal, are relatively easy for a human maintainer to validate or dismiss based on the auxiliary evidence provided. This prioritization aligns with the practical needs of curators, where the cost of missed issues often outweighs the effort of validating flagged data.