Thank you for your constructive comments. We address your concerns below:

Concern 1: Evaluation on other Chart Tasks

Our primary focus is on the chart question answering task due to the availability of multiple benchmarks emphasizing mathematical and visual reasoning, such as ChartQA, CharXiv, FigureQA, DVQA, and PlotQA. These benchmarks align well with our aim of enhancing reasoning capabilities. On the other hand, existing chart summarization/captioning benchmarks (e.g., Chart-to-Text [1]) primarily measure descriptive abilities using BLEU scores which offer limited insights into reasoning performance. We acknowledge that chart fact-checking is another valuable task for evaluating reasoning. Hence, we conducted additional evaluations on the ChartCheck benchmark [2]. As shown in the table below, our models (BigCharts-R1-3B and BigCharts-R1-7B) achieve higher accuracies than all open-source models on this task. Compared to closed-source models, our models perform competitively: although slightly behind GPT-4o and Gemini Flash 2.0 on Test 1, BigCharts-R1-7B surpasses them on Test 2. This highlights the strong out-of-distribution generalization of our models on diverse reasoning-intensive chart tasks, even though our BigCharts dataset was not explicitly designed for fact-checking.

Concern 2: Risk of Model Bias, Inaccuracies of the Gemini Flash 2.0 model.

We acknowledge that relying on LLMs for synthetic data generation introduces certain limitations and potential inaccuracies. Nonetheless, synthetic generation remains essential for constructing large-scale datasets efficiently, given the high cost of human annotation.

Our novel conditional replotting approach helps to mitigate these risks compared to previous methods. By replotting charts and providing the vision-language model (Gemini Flash 2.0) with both accurate metadata and visually authentic re-rendered images, our method enhances the accuracy of the resulting question-answer pairs. In contrast, previous chart data generation methods (e.g., ChartGemma and TinyChart) generate Q/A pairs using only the chart image, which can lead to greater inaccuracies due to the LLM’s need to estimate data values directly from the image.

To empirically validate this advantage, we compared the performance of Qwen2.5-VL-3B trained with Q/A pairs generated directly from original charts against those generated using our replotting method. Both models are trained using the same hyperparameters reported in Section 5.1 to ensure a fair comparison. As shown below, our approach consistently achieves higher performance, largely due to the improved accuracy of the generated Q/A pairs.

We believe our conditional replotting method presents a promising solution not only for chart reasoning but also for generating more accurate and diverse datasets in other visual reasoning tasks, such as tables and geometric figure comprehension. We also used Gemini Flash 2.0 due to its affordability; teams with greater resources may wish to explore larger, more expensive, and potentially more accurate models such as Claude 3.5 Sonnet with our novel replotting approach.

Concern 3: Demand for Computational Resources

We acknowledge that our experiments required relatively large computational resources (e.g., 8×H100 GPUs), which is increasingly common given the size of modern vision-language models, even though we used the smallest Qwen2.5-VL models (3B and 7B). Researchers with limited resources can fine-tune smaller models on our dataset and leverage our reinforcement learning approach with more modest hardware.

Thank you for your thoughtful review. We hope our response has addressed your concerns and kindly ask you to consider raising your score to support the dissemination of our work within the ML community.

[1] Chart-to-Text: https://arxiv.org/abs/2203.06486

[2] ChartCheck: https://arxiv.org/abs/2311.07453

Thank you for your constructive comments. We address your concerns below:

Concern 1: Limited Technical Contribution

We would like to clarify that the main contribution of our paper is the BigCharts dataset, created through a novel data generation pipeline (conditional chart replotting) designed to address key limitations of existing methods, as illustrated in Figures 1 and 2 (see lines 75–81), such as visual homogeneity and estimation errors. While our training framework largely builds on established techniques, our focus is on dataset creation pipelines rather than new model architectures. Our experiments demonstrate that models fine-tuned on BigCharts consistently outperform those trained on prior datasets for chart question answering such as TinyChart and ChartGemma models. Additionally, to the best of our knowledge, we are the first to apply the GRPO reinforcement learning algorithm to the chart reasoning and question answering domain, and our results highlight its effectiveness in enhancing visual mathematical reasoning for charts.

Concern 2: More studies on BigCharts

Please refer to our responses to the questions below (Question 5)

Concern 3: More Clarification

Please refer to our responses to the questions below (Question 5)

Question 1: Mismatches in the Replotting Process

Our replotting approach is designed to approximate the visual style and content of the original chart, rather than achieve a perfect match. Replicating charts exactly is extremely challenging, even for advanced LLMs like Gemini, due to unavoidable data estimation errors. Our goal is to generate a similar chart that preserves diversity in both visuals and content; as a result, we use Gemini to produce code for a closely related chart with estimated data values, and subsequently discard the original image.

If current LLMs such as Gemini were capable of perfectly reproducing original charts and estimating all data values accurately, we could generate precise Q/A pairs directly from the chart images without the need for replotting. However, since perfect accuracy cannot be ensured, our replotting process allows us to generate similar chart images that maintain visual diversity while providing reliable, high-quality metadata (i.e., the underlying code).

To assess whether our conditional chart replotting process maintains visual and topical diversity, we analyzed the distributions of chart types and topics (as presented in Figure 3 and Section 3.2) by prompting Gemini Flash 2.0 to classify both the replotted and original charts. As shown in the table below, the distributions of chart types and topics are very similar. This demonstrates that our conditional replotting approach, despite its imperfections, effectively preserves the diversity of visual styles and content in the generated dataset, BigCharts.

Question 2: Filtering process for Common Crawl

Due to space constraints, we originally placed the details of the filtering process in Appendix A.2 (lines 611–627), where we list all datasets used for the first-stage classifier training—both positive (ChartQA, Chart-to-Text, AI2D, LRV-Chart, etc.) and negative examples (CC-12M, ImageNet). These datasets were chosen for their diversity and because they could be compiled without manual annotation. The first-stage classifier achieved 88% precision on a small validation set. To further improve precision, we manually annotated 5,000 challenging samples, primarily false positives, identified by the model and used them to finetune the classifier further. This raised the classifier’s precision to 97%. We opted not to use in-context learning with LLMs due to the high cost and latency associated with processing the massive scale of the Mint-1T dataset split (5.07 million samples). A ResNet-50 classifier provided a significantly more efficient and cost-effective solution for large-scale filtering. We will clarify these points in the main text in the revised manuscript.

Thank you for your constructive comments. We address your concerns below:

Concern 1: Justification of GRPO

We use GRPO for its computational and memory efficiency compared to PPO. As noted in lines 240-243, GRPO eliminates the need for a separate value model, typically as large as the policy model, by estimating the baseline directly from the scores of multiple sampled outputs [1]. Given the large size of the models used in our experiments (3B and 7B parameters), GRPO provides a practical and efficient solution under limited compute and memory resources.

Concern 2: Experimental Instability

We would like to clarify that we randomly shuffle the training dataset before fine-tuning to avoid any bias or inconsistency related to data order. We agree that the early peak and subsequent drop observed in the SFT curves in Figure 4 are primarily due to the small batch size (8) used in these experiments. We intentionally used this small batch size in SFT to match the batch size used in RL training for this particular ablation study in Section 5.3, allowing for a more direct and fair step-by-step comparison in Figure 4. Given the large size of our models (Qwen2.5-VL-3B and Qwen2.5-VL-7B), larger batch sizes were not feasible for RL due to hardware constraints. Nevertheless, RL training with the same small batch size produces stable learning curves and steadily improving validation accuracy, as shown in Figure 4, particularly on out-of-distribution benchmarks such as PlotQA and DVQA. This shows that our GRPO training is more stable than SFT when using small batch sizes.

Concern 3: Limited Analysis of Reward Function Design

In our initial experiments, we actually explored a variety of reward functions and combination strategies beyond those reported in the main paper. Specifically, we performed a set of RL experiments on Qwen2.5-VL-3B-Instruct using the ChartQA dataset for training and evaluation.

To assess the importance of the CERM reward, we conducted an ablation study where it was removed, leaving only the format reward. In this setting, model performance dropped to 0.48%, indicating that the model learned to follow the required response format but was not incentivized to produce correct answers.

Beyond the format and CERM rewards, we experimented with an Intermediate reward designed to encourage the model to extract the accurate underlying data from chart images during the model’s step-by-step reasoning. Specifically, for every generated response, we parsed all numeric values within the <thinking...thinking> tags. We then computed two sub-scores:

Completeness: The proportion of required data values the model was able to extract, calculated as 1 - |num_extracted - n| / n, where n is the number of expected values (set to the number of numerical values in the chart’s underlying data table).

Correctness: The percentage of these extracted values that exactly match entries in the ground-truth data table.

The Intermediate reward was defined as the product of completeness and correctness scores. The intuition was to reward the model for both extracting all necessary values from the chart image and ensuring their accuracy before proceeding to the final answer.

We incorporated this Intermediate reward in two ways: by simple addition with the other rewards (i.e., R_Format + R_CERM + R_Intermediate), and via custom weighting (e.g., 0.25 × R_Format + 0.5 × R_CERM + 0.25 × R_Intermediate). However, in practice, this led the model to focus too heavily on extracting correct data values, often at the expense of learning the mathematical reasoning skills required for complex questions (e.g., sum, average, difference) which are common in the benchmarks.

These results demonstrate that the simple summation of the CERM and format rewards provides a very effective RL approach for our models. This is also consistent with the findings of concurrent works in the visual instruction tuning domain [3]. We will clarify these additional analyses and findings in the revised manuscript.

Thank you for your constructive comments. We address your concerns below:

Concern 1: Effectiveness of Different Components

Synthetic (Replotted) vs. Real-World (Original) Charts

As discussed in the manuscript (lines 37-39 and 45-50), real-world charts offer superior visual and topical diversity compared to synthetic charts, which typically lack such variety. However, real-world charts usually lack underlying metadata necessary for accurate Q/A generation, which is readily available for synthetic charts. Our proposed BigCharts approach addresses this issue by generating synthetic charts grounded in real-world data, thus preserving visual diversity while providing essential metadata for accurate Q/A pair creation.

To validate our hypothesis, we conducted experiments comparing the performance of Qwen2.5-VL-3B trained using Q/A generated from original (real-world) charts directly versus our synthetic replotting approach that provide both the chart image and the underlying data. For consistency, Q/A pairs for the original charts were generated using Gemini Flash 2.0 and prompts similar to those used for the replotted charts (see Figure 6 in the Appendix). Both models were trained with identical hyperparameters as described in Section 5.1 to ensure a fair comparison. Results in the table below demonstrate that our replotted charts approach leads to improved performance, primarily due to more accurate Q/A generation enabled by metadata availability.

Effectiveness of the Rewards Function

Our reward functions, the Format Reward and Chart Error Rate Reward (CERM), provide complementary benefits. The format reward promotes structured, step-by-step reasoning, while the CERM reward encourages the model to produce accurate numerical predictions. To assess their individual contributions, we conducted an ablation study by training the Qwen2.5-VL-Instruct model with the GRPO algorithm on the ChartQA dataset using different reward combinations.

As shown in the table below, the CERM reward has the greatest impact on performance; removing it causes the model’s accuracy to collapse. The format reward mainly incentivizes the model to follow the required reasoning format, supporting step-by-step reasoning and better answer parsing. However, relying solely on the format reward is insufficient, as it does not incentivize correctness in the final answers, resulting in a significant drop in performance.

Concern 2: Fairer Comparison with existing works (ChartGemma, TinyChart)

We appreciate this suggestion and would like to clarify that our original results (Table 1) demonstrate the advantage of fine-tuning on our proposed dataset (BigCharts) compared to previous datasets (TinyChart and ChartGemma). However, we agree that a more rigorous and controlled comparison, eliminating the bias introduced by different backbone models (Qwen2.5-VL, PaliGemma, TinyLLaVa), provides a fairer evaluation.

To address this concern, we fine-tuned the same backbone model (Qwen2.5-VL-3B) on the publicly available ChartGemma and TinyChart datasets for one epoch and using the same hyperparameters reported in Section 5.1. For evaluation, we utilized the codebase and "program-of-thought" answer formats recommended by the authors of TinyChart and ChartGemma to ensure optimal and fair performance. The results in the table below confirm that fine-tuning with our BigCharts dataset consistently outperforms these datasets, particularly on challenging benchmarks such as ChartQA, PlotQA, and CharXiv.

Many thanks for the author's details response to my original questions. My concern is properly addressed and I will still maintain my original score to this paper which I think is a great contribtution to the community.