DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?

Data Source and Evaluation.

• Thank you for your comments. To clarify, the majority of tasks in ModelOff are finance-related and financial analysis constitutes a substantial portion of real-world data analysis tasks. Financial modeling tasks often serve as representative benchmarks due to their complexity, requiring sophisticated reasoning, integration of multimodal data (e.g., spreadsheets, textual descriptions), and precise quantitative analysis. These characteristics make financial analysis a particularly suitable domain for evaluating the capabilities of data science agents. Additionally, while ModelOff primarily focuses on finance, it also includes tasks from other areas, such as election statistics, adding some diversity to the benchmark. We will revise the manuscript to explicitly highlight this rationale.
• For data modeling, Kaggle was selected due to its status as the most popular platform for data science competitions, covering a broad range of domains such as healthcare, retail, and environmental science. While other platforms and data sources can indeed be utilized, Kaggle's extensive task diversity and established reputation make it a robust choice for constructing our benchmark.
• Regarding the use of industrial pipelines, such as those incorporated in Spider2-V, we agree that these represent realistic workflows. However, our benchmark deliberately avoids restricting the tools or systems that can be used. This design decision provides greater freedom for participants and encourages creative approaches to problem-solving. In contrast, benchmarks like Spider2-V emphasize LLM's tool-specific capabilities in controlled environments, which are excellent for evaluating system-specific proficiencies but differ from our goal of focusing on evaluating performance of different systems.
• We will clarify these points in the revised manuscript and appreciate your feedback in helping us improve the paper’s scope and explanation.

Better Presentation.

• Thank you for your valuable suggestions. Regarding Figure 4, the absence of reported accuracy for some models indeed indicates zero accuracy. The varying widths of the bars have no special significance, and I will add explanations to clarify this in the paper.
• As for Table 5, we agree that transforming it into a line chart with corresponding accuracy values would better illustrate the trend in accuracy over time. I will make this adjustment to enhance the clarity and visual presentation of the data.

Difference between tables and data files in Table 1.

• We apologize for any misunderstanding caused. In our context, "table" typically refers to data presented directly in the task instruction in textual form. These tables are generally simpler and contain smaller amounts of data. In contrast, "data files" refer to larger datasets provided in formats such as CSV or Excel. For evaluation, small tables in textual form can be directly concatenated into the prompt for processing. On the other hand, data files may either be concatenated into the prompt (if feasible) or require the agent to write code to interact with and process the file. We will clarify these points in our paper.

I also find the taxonomy somewhat difficult to understand. For example, "tables" and "excels" are categorized separately, and I hope the author can clarify the distinctions between these categories more clearly.

• Thank you for pointing this out. We understand that the distinction in our taxonomy may not be immediately clear and will revise the manuscript to provide better clarification. "Tables" refer to structured data presented directly within the task instruction in textual format. These are typically smaller datasets and can be processed directly within the prompt. "Excels" refer to larger, external data files provided in formats like xlsx file, which may require agents to write code for processing due to their size and complexity. We will update the manuscript to ensure this distinction is explicitly stated.

Thanks for the response:

1. Regarding the differences between tables and Excel, as well as clarifications on other visualizations, I believe the authors have addressed some of my concerns.

2. However, I still have reservations about the choice of source data and the evaluation metrics. I hope the authors can specifically address the issues related to Weakness 2.

Specifically: If you believe the overall score fully reflects what you mentioned about "extracting insights" and "proper data handling," could you provide relevant examples and analysis to support this? If not, I strongly recommend that the authors include a discussion of the limitations for current evaluation metrics. Overall, I have seen the authors provide a series of explanations, but they have not yet made adjustments in the paper or included necessary discussions and explanations about the aforementioned limitations.

I will still hold my original score as I think this paper is right on the edge of being acceptable, and I do not think I will raise my score further since it is high enough. But I really hope the authors could make the necessary revisions, as I believe these adjustments are not particularly difficult to implement.

Statistical Rigor in Dataset Scale

• Thank you for this insightful comment. We appreciate the suggestion to enhance the statistical rigor of our analysis. Regarding the sample size, 540 tasks was determined based on the availability of high-quality competitions from ModelOff and Kaggle and the evaluation cost of evaluated models, offering realistic and diverse data science challenges.
• To ensure the reliability of our current results, we specifically designed the inference phase to minimize randomness. For all experiments, we use greedy decoding with the temperature set to 0 and top_p set to 1, which reduces variability and ensures consistent results across tasks.
• We acknowledge that incorporating confidence intervals, bootstrap analysis, and power analysis could further enhance the robustness of our findings. In future work, we plan to conduct multiple experimental runs and report confidence intervals for performance metrics.

Evaluation Methodology

• Thank you for your insightful question. We agree that the human baseline performance could be further strengthened by incorporating more expert evaluators for each task. However, since expert annotation is both time-consuming and expensive, we sampled ten tasks for human evaluation, with each task being annotated by a single expert. The experts were selected based on their qualifications, specifically being either master's graduates in the data science track within computer science or current PhD students specializing in data science. In the future, we will recruit more volunteers to work on annotation.
• Regarding the evaluation process, we recognize the value of discussing potential biases in task selection and analyzing the distribution of task difficulty. We present the difficulty distribution in Sections 3.2.1 and 3.2.2. In the revised manuscript, we will add a more detailed discussion on these aspects, such the task sucess rate of the human for each task.

Suggest reversing the criteria for "Exec. Eval.", "Code only", and "Env. Fix." to negative statements.

• Thank you for your valuable comments. We will include this comment into our paper.

Concern of sustainability of the benchmark and more explanation for Table 5.

• Thank you for your comment. We agree that there is a chance that our benchmark could be incorporated into the training data for LLMs or VLMs, either intentionally or unintentionally. To address this, we plan to implement strict licensing (as shown in the following) prohibiting its use in training. Additionally, these data analysis tasks were released before the era of large-scale LLMs (models with over 7 billion parameters), so the low accuracy may indicate that current models do not yet contain these tasks in their training data. (Because if they had been trained on these data, their performance might be significantly higher.)
• DSBench Usage Terms The use of DSBench is strictly limited to research and evaluation purposes. Any use of the benchmark or its data for training machine learning models, including large language models (LLMs) and vision-language models (VLMs), is strictly prohibited. Redistribution for non-commercial research is permitted with proper attribution. Unauthorized use, including for model training, may result in legal action. For inquiries or permissions, please contact: xxxxx.

More explanation for Table 5.

• In Table 5, we observe that the difficulty of the challenges increases over the years. This trend can be attributed to the evolution of data technology and the escalating complexity of the questions. To verify this, we further show the human performance over different years, which demonstrates a similar trend with GPT-4o, i.e., the accuracy decreases as the years progress.
• We will add these explanations in our paper. If you still have confusion, please tell us.

Q1: In Table 3, what does "context length" refer to? Is it the number of English words, characters, or tokens?

• Thanks for your question. It's number of English words of the problem.

Q2: Is it possible for the LLMs to know the answers to the questions from pretraining? For example, the answers to the challenge may be discussed in a Reddit forum that has been scraped in pretraining.

• Thanks for your valuable comment. Because of the low performane of the existing models, we believe that these tasks are unlikely to be included in existing pre-training data. In addition, the data collection strategy is general and can be used to collect new data modeling tasks.

Q3:In Figure 2(b), what are A-F, A-I, and A-D?

• Thank you for your question regarding Figure 2(b). To clarify, in the figure, "A-F" indicates that the question contains five answer options, labeled as A, B, C, D, and F. Similarly, "A-I" represents questions with nine options, ranging from A to I, and "A-D" corresponds to questions with four options, labeled as A, B, C, and D. We will add this clarification to the manuscript.

Q4: L265: what is N in \hat{F} = \mathcal{G}(E,N,S,M)?

• Thank you for catching this typo. We confirm that in \hat{F} = \mathcal{G}(E,N,S,M), "N" should indeed be "A". We will correct this in the revised manuscript.

Q5: Figure 3: Can you define m0, m1, ..., m17, perhaps as a separate table in the Appendix?

• Thanks for your advice. Of course, we will add a Table in the Appedx for the defination of m0, m1, ..., m17 as follows.

Thanks for your further valuable feedback. We answer your question as follows.

Steps of how RPG is calculated

• Thank you for your comment. We compute RPG of human as follows, $1/N \sum_{i=1}^N \times max((h_i-b_i)/(g_i-b_i))$, where $N$ is the total number of competitions. $b_i$ is the performance of the baseline. $h_i$ is the human performance. $g_i$ is the highest performance value for the $i$-th competition. For example, $g_i$ is 1 when the metric of the competition is accuracy, $g_i$ is 0 when the metric of the competition is MSE (Mean Squared Error).

Possible data leakage.

• This is a good question since all existing benchmarks suffer the same quesiton. As noted in Table 5 of our paper, we observe a clear trend where tasks from older challenges yield different performance compared to newer ones, potentially supporting the hypothesis of varying exposure. We will explicitly discuss this in the revision to make the possible effects of leakage clearer.
• In addition, we are exploring the inclusion of entirely new data challenges in future iterations of DSBench to evaluate models without the risk of prior exposure. While we understand that a license alone may not completely prevent misuse, it sets a clear legal framework against incorporating DSBench into training datasets.

Thanks again for your good questions.

Novelty techniques or ideas.

• We sincerely thank you for your time, effort, and thoughtful comments in reviewing our work. We greatly appreciate the detailed feedback, which has helped us better articulate the contributions of our paper. While DSBench does build upon existing resources like ModelOff and Kaggle, we want to emphasize that it is more than a collection of existing tasks. Our work introduces a realistic and comprehensive benchmark framework, incorporating multimodal tasks, long-context reasoning, and end-to-end evaluations to better reflect real-world data science challenges. Additionally, we propose the Relative Performance Gap metric, a novel method for normalizing diverse evaluation metrics, enabling fair and consistent comparisons. We believe these elements represent meaningful innovations that go beyond engineering contributions and provide valuable insights for advancing data science agents. Thank you again for your thoughtful review and constructive feedback.

Different with the existing work.

• Thank you for your thoughtful feedback and for sharing your perspective on our work. We understand the difficulty in assessing overlap in a rapidly evolving field like automated data science. To clarify, DSBench is distinct from existing benchmarks due to its focus on realistic, multimodal, long-context, and end-to-end tasks that closely mirror real-world challenges, setting it apart from prior work that often emphasizes simplified tasks like code generation. Additionally, our introduction of the Relative Performance Gap metric addresses a critical gap in evaluating diverse tasks by enabling fair and consistent comparisons. By leveraging tasks from ModelOff and Kaggle, DSBench provides a practical and challenging benchmark that highlights the current limitations of state-of-the-art agents. We hope this clarification underscores DSBench’s significance and unique contributions to the field, and we greatly appreciate your time and consideration in reviewing our work.

The only relatively minor issue I'd like to point out concerns Section 3.3, which promises discussing the errors made by LLMs, but does not provide any details, really. I think this should be amended.

• Thank you for the suggestion. We show in-depth analysis with case study in Appendix H. In particular, we show 7 testing examples for in-depth analysis. For example, in Figure 9, we show the introduction of the challenge, the question, a screenshot of a portion of the data file, and the generation text from GPT-4o. Based on the generation from GPT-4o, we found that the model misunderstood the meaning of the introduction and misinterpreted the district code as the voter identity code, leading to the wrong answer. We conducted comprehensive error analysis, we will add a reference in Section 3.3 to the appendix.
• If you have other suggestions pls let us know. If it's the only concern, could you consider increasing the contributions/ratings.

Diversity of Kaggle Tasks.

• Thank you for raising this concern. While it is true that many Kaggle competitions focus on domains like retail and finance, the platform also hosts challenges from a wide range of other fields, as reflected in the dataset we collected. For instance: 1) Healthcare and Life Sciences: Competitions like Ventilator Pressure Prediction and COVID-19 Global Forecasting provide tasks directly related to healthcare and pandemic modeling. 2) Education and Natural Language Processing: Examples include CommonLit Readability Prize and Learning Agency Lab Automated Essay Scoring, which focus on NLP and educational challenges. 3) Environment and Urban Planning: Tasks like See Click Predict Fix address urban planning and environmental issues. 4) Gaming and Simulations: The Conway's Reverse Game of Life challenge explores complex systems and algorithms. 5) Miscellaneous Domains: Competitions such as TMDB Box Office Prediction and Spaceship Titanic cover unique topics, including entertainment and hypothetical scenarios. This indicates that our dataset include a diverse domains. The diverse kaggle competition metrics shown in Figure 3 of our paper also show the diversity of our tasks
• Besides, the user can further collect because our dataset can be continually updatable for the newly released competition and othter data competition platforms, which can further enhance the diversity of the Kaggle data.
• We will revise the manuscript to emphasize the breadth of domains represented in our dataset.

Explain why RPG can reflect actual performance across tasks.

• Thank you for your comment. The Relative Performance Gap (RPG) metric is designed to provide a normalized measure of model performance across various data types and task complexities, allowing for a consistent evaluation framework within DSBench. Unlike raw metrics in the kaggle tasks such as accuracy and RUC, RPG accounts for performance differences by comparing the model's results against both a baseline and the best-known human expert performance for each task. This approach enables RPG to highlight how closely a model approaches expert-level proficiency, regardless of the specific data domain or task difficulty.
• RPG is particularly valuable in DSBench's diverse setting, where tasks vary widely in complexity and metric requirements (e.g., ROC AUC for classification tasks versus Root Mean Squared Error for regression tasks). By normalizing performance relative to the best achievable score and adjusting for task-specific baselines, RPG captures nuanced differences in model efficacy. This makes RPG a robust metric for assessing agents’ ability to generalize across the wide range of challenges that data science tasks present, from data analysis to end-to-end modeling.

[1] MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation.