ChartBench: A Benchmark for Complex Visual Reasoning in Charts

Q5: Application on Real-world

It is suggested to discuss the usefulness of the ChartBench dataset in real-world scenarios, such as in time series prediction tasks.

This is an good suggestion! ChartBench contains many real-time series charts, such as stock K-line charts in the box type, which can support prediction tasks. However, existing models have low accuracy and reliability in numerical extraction, leading to low credibility in predictions based on these models. We will incorporate this suggestion as one of the evaluation tasks in ChartBench.

---

Q6: About the Distribution of ChartBench

By observing Figure 3, there is a significant difference in the distribution of charts, CSVs, and queries between ChartBench and ChartQA. Is this difference due to the fact that the ChartBench dataset is primarily simulation-based? Please explain this reason during the rebuttal.

Not exactly. Many of the datasets we compare against also contain a large number of simulation-based instances, such as PlotQA, ChartX, and UniChart. The primary reason for the differences in charts is the variety of chart types. We have expanded the types of charts as much as possible, and during clustering, we observe that charts of the same type tend to cluster together.

The main reason for the differences in CSVs is the different data sources. For example, ChartQA data comes from the web, whereas our data primarily comes from Kaggle and GPT. Additionally, our tables contain an average of 87.62 numbers, compared to an average of 22.11 numbers in ChartQA. The queries are similar because our main goal is to compare with ChartQA. By using similar questions to evaluate MLLMs on both datasets, we can assess whether MLLMs possess visual reasoning capabilities despite the differences in charts and CSVs.

---

We hope to have addressed your concerns!

Thank you for your review and detailed comments! Please let us know if you have any further concerns or suggestions.

---

Q1: About the Contribution of ChartBench

In addition to expanding chart types and providing a convenient ACC++ evaluation metric, ChartBench's more significant contribution lies in its focus on charts without data point annotations. In many real-world scenarios, charts provide rich data but often lack text annotations for all data points. This leads to MLLMs heavily relying on textual information within the charts rather than developing true visual understanding ability. The introduction of ChartBench aims to encourage the community to pay attention to the visual logical reasoning capabilities of MLLMs in handling chart data, ensuring that they can provide truly reliable numerical data for more rigorous chart analysis.

---

Q2: Why No Data Annotation?

However, I believe that data point annotations for chart-related tasks are important. ... Please explain the reason for excluding data point annotations.

Yes, data point annotations are crucial for the accurate understanding of charts. However, in real-world scenarios, many charts lack data point annotations and require MLLMs to reason using visual cues in the chart, such as stock K-line charts or information-dense line charts. In these cases, only a few positions have data point annotations, and for QAs about other positions, MLLMs always provide unreliable numerical values, leading to untrustworthy analyses based on it. This is the motivation behind ChartBench: to evaluate whether MLLMs can truly infer reliable numerical values using visual cues rather than relying solely on their OCR capabilities.

---

Q3: Comparison on ChartQA

Considering that the authors reported the results on ChartQA dataset, it is necessary to validate the effectiveness of the proposed ChartBench training set on the ChartQA dataset.

Thanks for your reasonable suggestion. In fact, we have provided the results for InternLM-XC-v2 and its version after SFT on the ChartBench training set. These results are currently in Appendix E.1, specifically in Table 16 (on MME, SEED, QBench, etc.) and Table 17 (ChartQA). Given the importance of these results for readers' understanding of ChartBench, we will move them to the main body of the paper.

---

Q4: More Experiment on Recent Models

Yes, we have added the results for recent top-tier VLMs (i.e., DeepSeek, InternLM-v2, Qwen2-VL) as shown in the table below:

We also provide the performance of these models on ChartQA to validate the claims of ChartBench:

We will include this content in the paper and continue to keep it updated. Thank you for your suggestions, which have made our paper more comprehensive.

Dear Reviewer uiYC,

With the discussion phase ending in 3 days, we would greatly appreciate any further guidance or confirmation from your side. We wanted to follow up on our previous response to your valuable suggestions. Your insights are very important to us, and we are eager to ensure that we have addressed all concerns appropriately.

Warm Regards, :-)

Thank you for your review and detailed comments! We are glad to see your recognition of our contribution. Please let us know if you have any further concerns or suggestions.

---

Q1: The Contribution of ChartBench

I'm not fully convinced that the designed question types and data collection methods are substantially different compared to existing works. It would be nice to see some arguments on why evaluating on this benchmark substantially address the limitations of existing datasets.

ChartBench's significant contribution lies in its focus on charts without data point annotations. In many real-world scenarios, charts provide rich data but often lack text annotations for all data points. Existing benchmarks often overlook the issue of data point annotations and may degrade chart understanding tasks to trivial OCR problems. This leads to MLLMs heavily relying on textual information within the charts rather than developing true visual understanding. ChartBench is the first benchmark to propose evaluating whether MLLMs can provide reliable numerical data and analysis using only visual cues in the absence of detailed data point annotations. The introduction of ChartBench aims to encourage the community to pay attention to the visual logical reasoning capabilities of MLLMs in handling chart data, ensuring that they can provide truly reliable numerical data for more rigorous chart analysis.

---

Q2: Difference with Other Datasets Examples

How evaluating on this benchmark directly address the limitations on existing datasets? For example, any side-by-side comparison examples?

Thank you for your suggestion! We provide a comparison of samples from ChartBench and ChartQA in Appendix A.1 and visualize the responses of different MLLMs on ChartBench in Appendix E.5. We will supplement the visualization of the same MLLM's performance on both ChartBench and ChartQA in subsequent updates. In fact, the results of ChartBench in Table 6, which show the performance with and without annotations, highlight the main claim of ChartBench: to evaluate whether MLLMs can provide correct answers using only visual logic without relying on annotations.

---

Q3: Diversity of Questions/Instructions

Is there any effort in ensuring that the collected questions/instructions is diverse?

To facilitate comparison with the model rankings on ChartQA, we strive to maintain consistency in the way questions are asked. Figure 2 shows the t-SNE distribution of charts, tables, and questions. As shown, our tables and charts differ significantly from those in ChartQA, but the questions are similar, meaning the instructions are consistent with those in ChartQA (especially on the QA task). This allows the differential performance of models on these two benchmarks to highlight that many models overly rely on OCR to solve chart problems rather than possessing true visual chart understanding capabilities.

---

Q4: Evaluation requirement

Since the benchmark is relatively large, any comments on the compute/time resources required to conduct the evaluation? Are there any potential subsets that could be representative in any way of the full dataset?

Thank you for your suggestion! We provide the inference speed and memory usage for each MLLM during inference in Table 12. For the entire benchmark, we have 2100 charts and 18,900 QAs (Table 10) in the test set, so some MLLMs indeed take a longer time to complete the evaluation. To address this, We have introduced a 20% toy QA set (2100 chart, 3780 QA) via random sampling to facilitate quick testing of MLLMs on ChartBench. We ensure that random sampling is performed evenly on different task types and chart types to construct a representative test subset.

Thank you for your review and detailed comments! Here is our response.

---

Q1: About Paper Novelty and Chart Generation Pipeline

How does the data generation pipeline differ from other chart-related datasets?

ChartBench's significant contribution lies in its focus on charts without data point annotations. In many real-world scenarios, charts provide rich data but often lack text annotations for all data points. This leads to MLLMs heavily relying on textual information within the charts rather than developing true visual understanding. The introduction of ChartBench aims to encourage the community to pay attention to the visual logical reasoning capabilities of MLLMs in handling chart data, ensuring that they can provide truly reliable numerical data for more rigorous chart analysis. Besides, we also expand chart types and provide a convenient ACC++ metric for chart evaluation.

---

Q2: About Data Generalization

Have you considered testing the models' generalization ability on charts from completely different domains or visual styles not present in the training data?

Yes, in fact, we have provided the general QA results for InternLM-XC-v2 and its version after supervised fine-tuning on the ChartBench training set. These results are currently in Appendix E.1, specifically in Table 16 (GeneralQA: MME, SEED, QBench, etc.) and Table 17 (Chart-related: ChartQA+ChartX). We performed supervised fine-tuning on InternLM-XC-v2 using the ChartBench training set and evaluated it on ChartQA and ChartX. The significant performance improvements clearly demonstrate the effectiveness of ChartBench (ChartQA +3.1%, ChartX +11.17%).

Given the importance of these results for readers' understanding of ChartBench, we will move them to the main body of the paper.

---

Q3: About additional Experiment Results

Yes, we have added the results for recent top-tier VLMs (i.e., DeepSeek, InternLM-v2, Qwen2-VL) as shown in the table below:

We also provide the performance of these models on ChartQA to validate the claims of ChartBench:

We will include this content in the paper and continue to keep it updated. Thank you for your suggestions, which have made our paper more comprehensive.

---

Reference:

[1] ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning, Arxiv 2024.

[2] MMC: Advancing Multimodal Chart Understanding with LLM Instruction Tuning, NAACL 2024

Dear Reviewer hEAv,

With the discussion phase ending in 3 days, we would greatly appreciate any further guidance or confirmation from your side. We wanted to follow up on our previous response to your valuable suggestions. Your insights are very important to us, and we are eager to ensure that we have addressed all concerns appropriately.

Warm Regards, :-)

Thanks for your detailed comments! We provide detailed responses to your questions and concerns as follows. We hope the following discussion can address your concerns!

---

Q1: About Technical Novelty

While the work expands the range of chart types and introduces the ACC++ evaluation metric, the technical innovation appears somewhat limited beyond these contributions. This raises questions about the proposed benchmark's overall novelty.

In addition to expanding chart types and providing a convenient ACC++ evaluation metric, ChartBench's more significant contribution lies in its focus on charts without data point annotations. In many real-world scenarios, charts provide rich data but usually lack text annotations for all data points. This leads to MLLMs heavily relying on textual information within the charts rather than developing true visual understanding. ChartBench aims to encourage the community to pay attention to the visual logical reasoning capabilities of MLLMs in handling chart data, ensuring that they can provide truly reliable numerical data for more rigorous chart analysis.

---

Q2: Concern between Synthetic and Real-world Chart

The substantial gap between synthetic and real data distribution poses a critical question: Can the robust generalization abilities of VLMs effectively bridge this gap? The heavy reliance on synthetic data potentially diminishes the benchmark's effectiveness in evaluating real-world performance.

It is a reasonable concern. Recent work has shown that using synthetic data to enhance model performance can be highly effective, as demonstrated by ChartVLM[1] and MMC[2]. When diversity and data quality are ensured, models can effectively improve on real evaluation data. To address your concerns further, we performed supervised finetuning on InternLM-XC-v2 using the ChartBench training set and evaluated it on ChartQA and ChartX. The significant performance improvements clearly demonstrate the effectiveness of ChartBench (ChartQA +3.1%, ChartX +11.17%).

We will include this content in the main body of the paper to make it more comprehensive.

---

Q3: More Experiment Result

The baseline models should be more systematically categorized. Specialized models optimized for OCR and document understanding (like mPLUG-Owl and DocOwl) warrant separate evaluation from general-purpose VLMs (such as Qwen-VL).

Thank you very much for your reminder. We will categorize the models being evaluated as follows:

1. OCR Optimized Models: mPLUG, DocOwl, CogAgent
2. Chart Optimized Models: ChartVLM, OneChart, ChartLlama
3. General Purpose Models: Internlm-XComposer, Internlm-XComposer-v2, Shikra, MiniGPT-v2, BLIP2, InstructBLIP, VisualGLM, SPHINX, LLaVA, CogVLM, Mini-Gemini, Qwen-VL

The experimental evaluation would be more comprehensive with the inclusion of recent top-tier VLMs like InternVL

Yes, we have added the results for recent top-tier VLMs (i.e., DeepSeek, InternLM-v2, Qwen2-VL) as shown in the table below:

We also provide the performance of these models on ChartQA to validate the claims of ChartBench:

We will include this content in the paper and continue to keep it updated. Thank you for your suggestions, which have made our paper more comprehensive.

---

Reference:

[1] ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning, Arxiv 2024.

[2] MMC: Advancing Multimodal Chart Understanding with LLM Instruction Tuning, NAACL 2024

Dear Reviewer QY2B,

With the discussion phase ending in 3 days, we would greatly appreciate any further guidance or confirmation from your side. We wanted to follow up on our previous response to your valuable suggestions. Your insights are very important to us, and we are eager to ensure that we have addressed all concerns appropriately.

Warm Regards, :-)

Thank you for your review and detailed comments! You can find our detailed response to your questions and concerns below. Please let us know if you have any further concerns or suggestions.

---

Q1: The Novelty of Value Extraction Metric Based on Acc++

Since the metadata has the full CSV annotation of the chart data, an effective method to evaluate the ability of MLLMs should be the comprehensive recognition of all elements in the chart. A binary classification for the value extraction task cannot fully demonstrate the visual understanding capability of MLLMs.

For charts with sufficient data point annotations (e.g. ChartQA), directly evaluating the prediction of all elements is feasible. However, the majority of charts in ChartBench lack data point annotations, meaning MLLMs cannot obtain all numbers through OCR. This results in extremely low accuracy when directly comparing predicted tables with CSV annotations, making model rankings meaningless due to the low accuracy.

StructChart[1] proposed SCRM (Structuring Chart-oriented Representation Metric), which evaluates table predictions using a method similar to IoU (i.e., correctly predicted elements / total elements). However, SCRM only requires a portion of the elements to be correctly predicted within a specified error margin. This means that even with high SCRM scores, the extracted tables are not reliable for numerically sensitive applications, such as financial report analysis.

Considering these factors, we propose using Acc++ to predict a single value from the chart. For charts without data point annotations, MLLMs must use visual logic to understand the chart and provide the final answer. This approach has three advantages:

1. Faithful Evaluation: Models with high VE (Value Extraction) scores on ChartBench always provide reliably accurate values.
2. Deployment Flexibility: Our evaluation does not require LLM as evaluators, making it cost-effective and flexible.
3. Avoiding Random Guesses: In cases where the model makes random guesses, traditional metrics like Acc tend to be around 50%, and Acc+ proposed by MME[2] tends to be around 25%, while our Acc++ tends to be around 0%.

---

Q2: The Diversity of Template-based Instruction

The template-based generation of instruction data somehow lacks diversity. As illustrated in Figure 3(c), the distribution difference between ChartBench and other benchmarks is less significant than the other two dimensions.

ChartBench differs from other datasets (e.g. ChartQA) primarily in whether the charts have detailed data point annotations. To ensure that the evaluation results on ChartBench are comparable to those on ChartQA, we mainly follow the instruction types used in ChartQA & ChartX and compare the performance of the same MLLMs on both benchmarks in Table 3. As shown, when models must infer results from charts lacking detailed data point annotations using visual clues, the model rankings change significantly. This indicates that some models rely on their OCR capabilities to generate candidate answers rather than possessing true visual chart understanding.

Hence, ChartBench differs from previous datasets in both the charts and meta tables but maintains consistency in instruction design. In fact, ChartQA itself includes questions related to chart type, value extraction, and global conception, but lacks clear categorization labels, making statistical analysis difficult. In contrast, ChartBench organizes instructions by task categories during the design phase.

---

Q3: About the Novelty of Pipeline

The generation pipeline of ChartBench lacks sufficient novelty compared with other benchmarks, e.g., ChartX.

Our pipeline incorporates previous excellent works, such as ChartX[3], ChartLLaMA[4], and MMC[5]. Here, we would like to emphasize the novelty of ChartBench, which is designed to evaluate whether MLLMs can handle charts lacking detailed data point annotations. It is a common scenario in fields like financial report analysis. This capability has been overlooked by the community, and we hope ChartBench will draw attention to the importance of understanding such charts. This reflects whether MLLMs truly possess visual logical reasoning capabilities, rather than relying solely on textual logic.

Q4: About ChartX in Table 1

In Table 1, ChartX also seems to be visually grounded. Could you further check it?

Thank you for your kind reminder. ChartX is an excellent work that provides high-quality charts, ample metadata, and diverse instruction data. However:

1. Not all the charts in ChartX are visually grounded, and the classification does not focus on specialized analysis of charts lacking detailed data point annotations.

2. The instruction design in ChartX emphasizes overall evaluations, such as converting charts to tables, summarizing, redrawing, and other Q&A tasks. As a result, there are fewer questions focused on numerical values, making it difficult to reflect the accuracy of MLLMs in extracting reliable numerical data.

Nonetheless, we will revise the description in Table 1 to highlight the features and contributions of ChartX.

---

Reference:

[1] StructChart: Perception, Structuring, Reasoning for Visual Chart Understanding, Arxiv 2023.

[2] MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models, Arxiv 2023.

[3] ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning, Arxiv 2024.

[4] ChartLlama: A Multimodal LLM for Chart Understanding and Generation, Arxiv 2023.

[5] MMC: Advancing Multimodal Chart Understanding with LLM Instruction Tuning, NAACL 2024

Dear Reviewer FpUV,

With the discussion phase ending in 3 days, we would greatly appreciate any further guidance or confirmation from your side. We wanted to follow up on our previous response to your valuable suggestions. Your insights are very important to us, and we are eager to ensure that we have addressed all concerns appropriately.

Warm Regards, :-)