InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generation

7) Privacy, security and safety

• No real ServiceNow data was used, so there’s no risk of exposing personal information. The dataset was only based on the platform’s structure, so anonymization or IRB approval wasn’t needed.

8) It was encouraging to see that LLaMa-3-Eval produced consistent outcomes compared to G-Eval, which is known to be well aligned with human judgments. Was there any way to evaluate whether LLaMa-3-Eval output is also directly aligned with human judgments

• While LLaMA-3-Eval scores' consistency with G-Eval indicates that it also aligns with human judgement to at least some extent, we investigated example interpretations of LLaMA-3-Eval below for further evidence of human alignment (also added to the paper in Appendix C, Table 7):

  • Example 1 (for good generated insights):
    • Ground Truth (GT): There are significantly higher asset costs in the HR Department.
    • Generated: The average cost of assets in the HR department is \$4874.25, which is significantly higher compared to the average cost of \\$1967.26 in other departments.
    • LLaMA-3-Eval Score: 0.81
    • Justification: The generated text accurately conveys that the HR department has higher asset costs compared to other departments, with specific numerical values supporting the claim. The score is slightly less than 1 because the term "significant" is used in a general sense rather than in a statistical context. The generated text could be improved by providing additional context for the high asset costs.
  • Example 2 (for poorly generated insights):
    • Ground Truth (GT): There is a weak correlation between the number of users and the high cost of computer assets in the HR department.
    • Generated: The distribution of asset types within the HR department includes 19 computers, 5 web servers, and 4 servers, with computers being the most prevalent asset type.
    • LLaMA-3-Eval Score: 0.18
    • Justification: The generated text does not address the correlation between the number of users and the cost of computer assets, which is the key point of the ground truth. Instead, it discusses the distribution of asset types, which is irrelevant to the claim, resulting in a low score.

• These examples highlight how LLaMA-3-Eval provides detailed and interpretable feedback, with scores reflecting both the relevance and accuracy of the generated insights.

(3/3) End of Rebuttal

Thank you for recognizing this work as a key step toward robust enterprise-grade analytics agents, with a focus on real-world, multi-step reasoning and practical benchmarks using ServiceNow data. We also appreciate your acknowledgment of LLaMA-3-Eval’s transparency, AgentPoirot’s strong performance, and our rigorous quality assurance process.

Below we address your concerns.

1) Whether synthetic data represents the real-world context is questionable. The synthetic data generation part is less well-documented than other parts of the paper. The authors should explain better how their synthetic data generation process works and how they ensured the representativeness of the synthetic data.

• Our synthetic data was vetted by 12 ServiceNow experts that we partnered with that have deep knowledge of actual enterprise data to ensure it accurately represents real-world nuances. Additionally, we have conducted a human evaluation study (detailed in Appendix B.4) where enterprise data specialists rated our synthetic data highly for its quality, coverage, and relevance.

• We have found consistent relative performance results (w.r.t. L3-Eval metric) between AgentPoirot and PandasAgent on both our synthetic data and real ServiceNow datasets. The real-world data includes CSM, ITSM, and HR.

• We unfortunately can’t include real enterprise data due to the proprietary nature of the data.
• We have updated Section 2.1.3 and Appendix B.1 of the paper to include more details to clarify the synthetic data generation.

2) The authors mentioned the four types of analytics up front (descriptive, diagnostic, predictive, and prescriptive). However, this notion of four types of analytics was not as prevalent in the later part, where the results were presented.

• Thank you for pointing this out. Figure 4 now includes performance breakdowns by insight types. As shown in Figure 4c, agent performance decreases progressively from Descriptive insights to Diagnostic, Prescriptive, and Predictive insights, which reflects the increasing complexity of these analytics types.

3) Do you have any evidence that the distribution of your data (for example, insight types--incident, user, asset, finance, goal management) resembles real-world business scenarios?

• The list of possible values for various columns (such as insight types, ID types, and dataset categories) was carefully designed by analyzing statistical patterns observed in real-world enterprise data, particularly ServiceNow data. As a result, the distributions closely reflect real-world, realistic business scenarios.

4) Agent Poirot works best with a temperature of 0.2. This value looks interesting to me. Do you know why the agent works best at such a low temperature?

• We believe a temperature setting of 0.2 strikes an effective balance between accurate mathematical inference and controlled freeform insight extraction. Lower temperatures encourage more deterministic outputs, which enhances the precision needed for reliable quantitative analysis and minimizes unnecessary variability in responses. This helps Agent Poirot maintain a focus on accurate code generation while still allowing some flexibility for generating insightful observations.

(1/3) please see next comment for the rest of the rebuttal

We appreciate the reviewer’s recognition of the dataset's coverage across diverse business domains like finance and incident management, making it broadly useful even if not focused on a single domain. We also thank the reviewer for noting the expert-evaluated insights and two-way evaluation, which help build user confidence in the results.

Below we address each concern:

1) InsightBench datasets depend on synthetic data generation which can make the data quality lower as the data regresses from real world scenarios. Particularly, it is unclear how well the data represents the unhappy paths / scenarios often found in real world. One suggestion might be to further split up the datasets into Happy path and Challenge datasets , with challenge datasets using less synthetic data, and being more realistic of real world scenarios. How do you ensure that InsightBench datasets capture the complexity of real-world enterprise data?

• Our synthetic data was vetted by 12 ServiceNow experts that we partnered with that have deep knowledge of actual enterprise data to ensure it accurately represents real-world nuances. Additionally, we have conducted a human evaluation study (detailed in Appendix B.4) where enterprise data specialists rated our synthetic data highly for its quality, coverage, and relevance.

• The domain experts also helped ensure that the injected trends reflect the “unhappy”, “happy”, and “challenge” paths that occur in the real world. In that regard, our benchmark includes datasets at varying difficulty levels (easy, medium, hard) to represent a range of scenarios, including those with subtle or hard-to-detect trends. We conducted user studies with expert data scientists where they managed to identify more trends in "easy" datasets and fewer in harder ones, helping us assess the difficulty levels where current analytics succeed or fail.

• While privacy concerns prevent us from including actual real data, we are committed to collaborating with domain experts to ensure the datasets remain high-quality and realistic as the benchmark evolves.

2) InsightBench focuses on structured, tabular data, but increasingly other data formats like unstructured text, images, or complex graphs are becoming relevant for business analytics. Do you have plans to incorporate unstructured data (like text logs or emails) that is common in businesses?

• Unstructured data, like text, images, and graphs, is a very interesting area for business analytics, but it is beyond the scope of InsightBench and it is something we are working on as future versions of this work. We mention this as limitation in Appendix A.

• Our focus with InsightBench is on structured, tabular data because it is still a core part of many data analytics applications and decision-making processes that is far from being solved. This version of InsightBench is already steps ahead of existing benchmarks that focus on closed-form, templated answer evaluations.

3) Do you have a process in place for regularly updating the dataset, or have you considered partnering with industry experts to ensure the dataset remains relevant to current business practices?

• Our data generation pipeline is designed to be flexible and scalable, allowing us (with the help of domain experts) to easily generate new data for various business tasks by following the methodology outlined in the paper.

• We also plan to open-source the benchmark and actively collaborate with the broader community. This approach will help keep the benchmark up-to-date, incorporate new types of trends and insights, and maintain high-quality standards over time.

• We have partnered with leading enterprises like ServiceNow to ensure the quality and realism of the data, and we aim to expand our partnership to more enterprises in the future.

4) Could you elaborate on the methods used to simulate unpredictable or "noisy" data patterns?

• We use a systematic approach to simulate both “noisy” and “clean” trends, as detailed in Section 2.1.2 and Appendix B.1.
• We have included 15 datasets in our benchmark without dominant trends to evaluate whether an agent would produce false positives by generating insights where none exist. On these datasets, AgentPoirot scored an L3-Eval of 0.864, compared to PandasAgent’s 0.821, showing that AgentPoirot is better at avoiding false insights.
• Specifically, the agent generates insights like "There are no insights; nothing specific to analyze”, “There are visible trends in [Quantity XX]”, and “The dataset is missing [Quantity YY] required for analysis.” in those cases.
• We also included a set of 100 datasets for sensitivity analysis where we vary the trend intensities to evaluate what trend intensity is needed for the agent to identify it.

(1/2) please see next comment for the rest of the rebuttal

5) The LLaMA-3-Eval scoring method could benefit from additional explanation regarding its interpretability and stability, particularly in cases where scores may fluctuate with subtle prompt changes.

• LLaMA-3-Eval scores are highly interpretable as they are based on explicit criteria such as factuality, coherence and consistency. In terms of stability, our scores are based on 5 random runs where each leads to subtle prompt changes. The results showed very little variation, indicating consistent performance.

• Regarding interpretability, here are some examples with justifications generated by the LLaMA-3 (added to the paper in Appendix C, Table 7):

  • Example 1 (for good generated insights):
    • Ground Truth (GT): There are significantly higher asset costs in the HR Department.
    • Generated: The average cost of assets in the HR department is \$4874.25, which is significantly higher compared to the average cost of \\$1967.26 in other departments.
    • LLaMA-3-Eval Score: 0.81
    • Justification: The generated text accurately conveys that the HR department has higher asset costs compared to other departments, with specific numerical values supporting the claim. The score is slightly less than 1 because the term "significant" is used in a general sense rather than in a statistical context. The generated text could be improved by providing additional context for the high asset costs.
  • Example 2 (for poorly generated insights):
    • Ground Truth (GT): There is a weak correlation between the number of users and the high cost of computer assets in the HR department.
    • Generated: The distribution of asset types within the HR department includes 19 computers, 5 web servers, and 4 servers, with computers being the most prevalent asset type.
    • LLaMA-3-Eval Score: 0.18
    • Justification: The generated text does not address the correlation between the number of users and the cost of computer assets, which is the key point of the ground truth. Instead, it discusses the distribution of asset types, which is irrelevant to the claim, resulting in a low score.

• This example highlights how LLaMA-3-Eval provides detailed and interpretable feedback, with scores reflecting both the relevance and accuracy of the generated insights.

(2/2) End of Rebuttal

We appreciate the reviewer’s recognition of InsightBench as an innovative, clear, and impactful benchmark for multi-step analytics, promoting versatile and context-aware models while addressing key gaps in data analytics research.

Below we address each concern:

1) While InsightBench provides synthetic data and insights, the effectiveness of AgentPoirot and other evaluated agents in handling real-world, noisy datasets remains untested.

• We have evaluated on real ServiceNow datasets and we have found consistent relative performance results (w.r.t. L3-Eval metric) between AgentPoirot and PandasAgent on both our synthetic and real world data. The real-world data includes CSM, ITSM, and HR.

• We unfortunately can’t include real enterprise data due to the proprietary nature of the data.
• Please note that our synthetic data was vetted by 12 ServiceNow experts that we partnered with that have deep knowledge of actual enterprise data to ensure it accurately represents real-world nuances. Additionally, we have conducted a human evaluation study (detailed in Appendix B.4) where enterprise data specialists rated our synthetic data highly for its quality, coverage, and relevance.

2) How does InsightBench handle noisy or incomplete data?

• We have included 15 datasets in our benchmark without dominant trends to evaluate whether an agent would produce false positives by generating insights where none exist. On these datasets, AgentPoirot scored an L3-Eval of 0.864, compared to PandasAgent’s 0.821, showing that AgentPoirot is better at avoiding false insights.

• Specifically, the agent generates insights like "There are no insights; nothing specific to analyze”, “There are visible trends in [Quantity XX]”, and “The dataset is missing [Quantity YY] required for analysis.” in those cases.

• We also have a set of 100 datasets for sensitivity analysis where we vary the trend intensities to evaluate what trend intensity is needed for the agent to identify it (see Figure 8b).

3) What is the scalability of InsightBench for larger datasets? Since InsightBench datasets are capped at 500 entries, how would the benchmark scale to larger, more complex datasets? Would AgentPoirot's performance degrade with larger data volumes?

• We have evaluated AgentPoirot on real-world ServiceNow datasets with 8000+ rows and AgentPoirot maintained strong performance of 0.64 LLaMA-3-Eval compared to PandasAgent 0.58. This result demonstrates that the benchmark can scale to larger, more complex data volumes without degradation in the Agent’s performance.

4) Are there plans to incorporate real-world data into InsightBench?

• No, we do not plan to incorporate real-world data directly into InsightBench, as it may contain private or sensitive information. Instead, we prioritize the use of synthetic data to ensure privacy and security.
• However, to ensure the benchmark's relevance to real-life scenarios, we will continue to use feedback from human experts from enterprises to vet and validate the synthetic data, ensuring it meets the distribution and the high-quality standards of real world data.

(1/2) please see next comment for the rest of the rebuttal

We appreciate the reviewer’s recognition of InsightBench’s end-to-end design as a real-world analytics benchmark, the realistic and versatile datasets it has which includes actual enterprise trends and anomalies, and the cost-effective, reproducible evaluation provided by LLaMA-3-Eval. We also thank the reviewer for acknowledging AgentPoirot’s potential as a foundation for multi-step data analytics agents.

Below we address each of your concerns:

1) LLaMA-3-Eval offers advantages, but its consistency with GPT-4 evaluations could benefit from more rigorous comparison

• We have conducted that rigorous comparison in Table 5 in Appendix C, which displays score consistency between LLaMA-3-Eval and G-Eval across complex tasks that go beyond simple insight extraction such as detecting challenging trends and anomalies. Please let us know if there is any other specific comparison we could provide.

2) the benchmark may not fully capture the complexity of certain domains, potentially limiting generalizability to more complex or highly regulated fields like healthcare or law.

• Our benchmark is designed primarily for key business use cases, as stated in the title and abstract. However, our data generation pipeline and evaluation framework can be easily adapted for other complex domains like healthcare or law with the guidance of domain experts.

3) Synthetic data generated from ServiceNow templates may lack the nuances of actual enterprise data, which could affect the evaluation's applicability to real-world use cases.

• Our synthetic data was vetted by 12 ServiceNow experts that we partnered with that have deep knowledge of actual enterprise data to ensure it accurately represents real-world nuances. Additionally, we have conducted a human evaluation study (detailed in Appendix B.4) where enterprise data specialists rated our synthetic data highly for its quality, coverage, and relevance.

4) It would be valuable to test InsightBench on actual enterprise data and synthetic data to better understand its applicability to real-world tasks.

• We have found consistent relative performance results (w.r.t. L3-Eval metric) between AgentPoirot and PandasAgent on both our synthetic data and real ServiceNow datasets. The real-world data includes CSM, ITSM, and HR.

• We unfortunately can’t include real enterprise data due to the proprietary nature of the data.

5) Given the evolving nature of LLMs, a dynamic benchmark that updates or adjusts insight difficulty could be beneficial for testing agent adaptability over time.

• Our benchmark includes datasets at varying difficulty levels: easy, medium, and hard. The best performance on the most difficult datasets remains low (at around 50%), which suggests that there is room for improvement from more advanced data analytics agents in the future.

6) Further breakdown of evaluation scores (e.g., insight precision vs. general text similarity) could improve the interpretability and reliability of agent assessments.

• To assess insight precision, we have included 15 datasets in our benchmark without dominant trends to evaluate whether an agent would produce false positives by generating insights where none exist. On these datasets, AgentPoirot scored an L3-Eval of 0.864, compared to PandasAgent’s 0.821, showing that AgentPoirot is better at avoiding false insights.

• We also found that text similarity metrics (e.g., sentence-BERT embeddings with cosine distance) often yield high scores as they fail to measure factuality and accuracy of insights. Thus, we opted for the G-Eval metric, which has been shown to better align with human judgment for a more reliable evaluation [1].

  • [1] Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023, December). G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment. In Proceedings of EMNLP.