Distill Visual Chart Reasoning Ability from LLMs to MLLMs

We appreciate your valuable time, insights, and for highlighting our strengths (i.e., the CIT method and deconstruction of visual reasoning ability). In the following, we will carefully respond to your questions.

1. Question about the overlap between existing datasets.

   • Thank you for your valuable suggestion! We conduct a comprehensive analysis of the data overlap as follows:
     • Image similarity: Following ChartBench [1], we apply the SigLIP-400M encoder [2] to generate embeddings for images across datasets and visualize using t-SNE [3].
     • Query similarity: Similarly, we apply the NV-Embed-v2 model [4] to generate embeddings for the queries across datasets, also visualize using t-SNE.
     • We compare (1) ReachQA training set vs. other chart-related benchmarks, and (2) ReachQA training set vs. ReachQA test set. The visualization results have been updated to Appendix E (Page 16) of the revised paper.
   • Key findings:
     • Our dataset exhibits clear distinctions from existing chart-related benchmarks in both images and queries. While some degree of overlap is naturally expected due to the shared focus on chart-related tasks, these instances are limited and do not compromise the broader differences in distribution. Overall, we believe that it shows the validity of our main experiments.
     • Since our training and testing sets were synthesized using the same process—though independently conducted—some distribution overlap is expected. To address concerns regarding potential data leakage, we implemented a filtering and manual review process. Specifically, we identified the top 50 most similar image pairs (similarity > 0.9) and carefully examined these pairs along with their associated queries. Among these, only 2 examples exhibited notable similarities. We will exclude them from the test set in future versions and update the evaluation accordingly. For the remaining samples, our review confirmed clear differences in chart topics, data values, and query types, ensuring that no further data leakage or contamination is present.

2. Question about the additional benchmark evaluations.

   • In response to your suggestion, we have added additional benchmark evaluations following the setting of Section 5.3, with results provided as follows:

     Models	OCRBench	We-Math
     LLaVA-Next-Llama3-8B	54.5	19.37
     + ReachQA-20k	49.2	31.21
     + ReachQA-LLaVA-Mix-40k	56.4	34.08

   • Consistent with the trends observed in our paper, we found some performance decline on OCRBench after training on ReachQA. This is likely due to the significant difference in task objectives, as OCRBench focuses on direct recognition. However, performance quickly recovered and even surpassed the base model once fine-tuned with a mixed dataset incorporating general-purpose data. For reasoning-intensive tasks like We-Math, the fine-tuned model achieved better performance at first, and this improvement remained stable after incorporating general data.

Reference

[1] ChartBench: A Benchmark for Complex Visual Reasoning in Charts, arXiv:2312.15915

[2] Sigmoid Loss for Language Image Pre-Training, ICCV 2023

[3] Visualizing Data Using t-SNE, JMLR 2008

[4] NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models, arXiv:2405.17428

Thank you for your practical and professional suggestions, which helped us make our paper better and more thoroughly researched! We are also pleased that our previous clarifications were satisfactory. [Updated] In the following, we will carefully respond to your follow-up questions.

Question about detailed results on additional benchmark.

• In response to your suggestions, we conducted the corresponding experiments on OCRBench and We-Math based on the setting in Figure 3. The results are as follows:

  Note: During the OCRBench experiments, we identified and fixed a bug in VLMEvalKit, the evaluation toolkit used in our study. This bug occasionally caused our fine-tuned model to produce empty string outputs due to formatting. Results marked with (*) reflect the corrected scores, which do not alter the conclusions presented in our previous analysis.

  Data Size	Reas. vs. Reco.	OCRBench (Acc.)	We-Math (Acc.)	Insufficient Knowledge (↓)	Inadequate Generalization (↓)	Complete Mastery (↑)	Rote Memorization (↓)
  -	-	54.5	19.37	83.05% (436)	3.05% (16)	7.24% (38)	47.95% (35)
  20k	12k : 8k	52.5*	31.21	71.81% (377)	7.24% (38)	14.10% (74)	32.73% (36)
  40k	12k : 8k	56.9*	34.08	68.00% (357)	8.38% (44)	17.14% (90)	27.42% (34)
  8k	0k : 8k	54.3	33.74	69.90% (367)	7.24% (38)	17.14% (90)	25.00% (30)
  8k	2k : 6k	54.4	33.28	70.29% (369)	7.62% (40)	16.57% (87)	25.00% (29)
  8k	4k : 4k	53.9	31.72	72.76% (382)	6.86% (36)	15.05% (79)	26.17% (28)
  8k	6k : 2k	53.3	32.82	70.29% (369)	7.24% (38)	16.57% (87)	26.27% (31)
  8k	8k : 0k	52.4	33.28	69.33% (364)	8.00% (42)	17.33% (91)	23.53% (28)

• Key findings:

  • The OCRBench (Acc.) results are consistent with the observations in Figure 3. The model’s recognition performance strongly correlates with the proportion of recognition-oriented training data. Notably, excessive training or reasoning data ratio may lead to performance drops.
  • The We-Math (Acc.) results show greater variance but consistently outperform the vanilla model across all settings, demonstrating the generalizability of our method despite the domain differences. However, achieving optimal performance requires careful control of data volume and ratio.
  • On the additional evaluation metrics of We-Math (IK, IG, CM, RM), most results align with the observations of the original paper, such as the shifted challenges from IK to IG. Notably, we found that the model trained with 100% reasoning data demonstrated enhanced output reliability, with CM (↑) and RM (↓) reaching optimal values. This suggests improved robustness in reasoning processes (e.g. correct intermediate steps leads to correct final answers).

We will continue to refine this work to make it more worthy of acceptance. Thank you again for your consideration!

We appreciate your valuable time, insights, and for highlighting our strengths (i.e., the writing, the idea of CIT, and the effectiveness of our dataset). In the following, we will carefully respond to your questions.

1. Question about error analysis on models before/after ReachQA training.

   • Thank you very much for your suggestion! We have added an analysis of errors before and after training to provide a clearer understanding of the improvements. The results are as follows:

     Model	# Errors	# Reco. Error	Reco. Proportion	# Reas. Error	Reas. Proportion
     LLaVA-Next-Llama3-8B	1355	827	61.0%	493	36.4%
     LLaVA-Next-Llama3-8B + ReachQA	886 (-469)	534 (-293)	60.3% (↓)	334 (-159)	37.7% (↑)
     MiniCPM-V2.5-Llama3	827	513	62.0%	297	35.9%
     MiniCPM-V2.5-Llama3 + ReachQA	714 (-113)	414 (-99)	58.0% (↓)	278 (-19)	38.9% (↑)
     InternVL2-8B	655	335	51.1%	296	45.2%
     InternVL2-8B + ReachQA	439 (-216)	209 (-126)	47.6% (↓)	198 (-98)	45.1% (↓)

   • The results demonstrate that: (1) For different models, both types of errors show a noticeable reduction, indicating that the trained model improves in both recognition and reasoning abilities. This is consistent with the observations in Sections 4.2 and 5.2 of our paper. (2) Additionally, stronger base models (e.g. InternVL2-8B) exhibit a lower proportion of recognition errors, and after training with ReachQA, the reduction in recognition errors is more significant. This suggests that recognition, as a more fundamental capability, could benefit more directly from the training.

2. Question about dataset sufficiency for cross-modal training.

   • Thank you for pointing this out! We acknowledge that the ReachQA dataset volume is still limited, especially for training larger MLLMs. However, as our method focuses on automated data synthesis, it is feasible to scale this approach to larger datasets. The experiments in our paper can be seen as an initial attempt under resource constraints, and we will release our implementation to support future efforts to expand data volume.
   • Additionally, based on prior experience with MLLM training, we agree that data size is crucial for enhancing recognition abilities. That said, we also believe that a visually complex image can be worth many simpler ones. For instance, each chart in ReachQA is paired with six distinct questions on average, covering focuses from local to global, and information types from colors to numeric values and their combinations. These diverse questions require the model to recognize various aspects of the image, so that each image to be fully utilized during training.
   • Regarding the performance gained by training on ReachQA, as discussed in Section 5.2 and the supplementary error analysis, these two abilities—recognition and reasoning—likely interact and mutually reinforce one another. As the model’s recognition skills improve, more accurate identification can support more effective reasoning; likewise, improved reasoning abilities can help the model better identify the elements it needs to recognize in subsequent steps.

Thank you for your valuable feedback and your positive assessment of our first-round response! We are pleased that our previous clarifications were satisfactory. In the following, we will carefully respond to your follow-up questions.

1. Question about qualitative examples.

   • Thank you very much for your suggestion! To provide intuitive insights into the performance gains, we have conducted a case study from the perspective of attention. The study aims to visually demonstrate the mechanisms behind the improved performance of our fine-tuned model. The results have been added to Appendix F (Page 16) and Figure 6 (Page 18) of the revised paper for your review.

   • Specifically, we analyzed the model’s attention patterns to the image during next-token prediction [1,2], comparing the vanilla model with our fine-tuned version. Figure 6 presents a case where the vanilla model produces lengthy, redundant outputs with scattered attention and ultimately reaches incorrect conclusions. In contrast, the fine-tuned model effectively focuses its attention on key elements (e.g., labels, axes, values) at each step, leading to accurate reasoning.

   • This analysis indicates that the fine-tuned model not only mimics expert rationales but also develops robust attention patterns, enabling a synergistic interplay between recognition and reasoning. It identifies crucial elements during reasoning and utilizes them to guide subsequent steps.

2. Question about the cost and alternative models.

   • Thank you for raising this insightful point! Our approach is already cost-effective for reasoning-intensive multimodal data synthesis. Using GPT-4o, we were able to generate 3k images and 20k instructions for approximately $300—a fraction of the cost compared to manually collecting and annotating complex charts.
   • We have also experimented with several open-source models, such as Llama-3.1-70B-Instruct and Qwen-2.5-72B-Instruct. For intermediary code synthesis, these models initially achieved a pass rate of around 60%, which improved to over 90% through iterative self-repair. This demonstrates that some open-source models are already feasible for further reducing costs.
   • However, for instruction generation, these models performed significantly worse than GPT-4o or Claude 3.5 Sonnet. This limitation is likely due to their weaker understanding of complex code—particularly the evolved versions after our Evol-Instruct process—making it challenging for them to effectively “visualize” charts in their “mind”.

Reference

[1] EViT: Expediting Vision Transformers via Token Reorganizations, ICLR 2022

[2] ColPali: Efficient Document Retrieval with Vision Language Models, arXiv.2407.01449

We appreciate your valuable time, insights, and for highlighting our strengths (i.e., the novelty of our ideas, the clear steps behind our method, and the completeness of experiments). In the following, we will carefully respond to your questions.

1. Question about motivation and clarity in the problem statement.

   • Thank you for your insights and kind suggestions! We agree with your perspective; our paper indeed focuses primarily on the deficiencies and improvements of open-sourced MLLMs. As you pointed out, proprietary MLLMs like Claude and GPT-4o may be advanced in these ways. In the revised manuscript, we will address this by rephrasing the introduction to more explicitly highlight our focus.
   • At the same time, we also think that even these models also have room for improvement, particularly in reasoning. As shown in Table 4 of our paper, while some proprietary models may excel in recognition, they still fall short in reasoning-intensive tasks compared to humans (e.g., the performance gap on ReachQA-Reas. and CharXiv-Reas.). Additionally, IsoBench [1] also demonstrates that even models like GPT-4, Gemini Pro, and Claude 3 Opus achieve more than 15-point advantages with text input over image input, highlighting opportunities for improvement through cross-modal distillation.
   • In conclusion, although we could only validate our approach on open-sourced MLLMs, the consistent improvements across diverse base models (e.g., LLaVA -> MiniCPM -> InternVL2) suggest that our method and dataset have the potential to benefit a broader range of MLLMs, including proprietary ones. We hope the clarified narrative will address your concerns.

2. Question about limited novelty in the proposed method/framework, though the idea of CIT is appreciated.

   • We acknowledge that the CIT framework incorporates several existing techniques, as they are well-established and have been widely validated for their feasibility and effectiveness.
   • That said, we would like to gently shift the focus toward the proposed concept of “code-as-intermediary” rather than the specific steps of the framework. By translating images into textual representations, this approach facilitates the seamless integration of various text-based data synthesis methods—such as Tree-Instruct [2], OSS-Instruct [3], and potentially others—into our workflow. This characteristic enhances the adaptability and scalability of the framework.
   • Furthermore, we believe that with the advancement of controllable generation techniques, the “intermediary translation” concept could be extended to the synthesis of diverse multimodal data types across various domains.

3. Question about the analysis of question types.

   • Thank you for your suggestion! We intentionally avoid restricting question types during the generation process to enhance its diversity, though this may reduce transparency. In response, we have added question type statistics for 2k ReachQA test set questions, categorized by GPT-4o. The results are as follows:

     Question Type	# in Reco. split	Question Type	# in Reas. split
     Direct Lookup	177	Trend Inference	147
     Comparative Lookup	326	Causal Analysis	63
     Attribute Identification	123	Conditional Reasoning	55
     Pattern Recognition	61	Comparative Analysis	267
     Compositionality	313	Quantitative Evaluation	378
     Others	0	Others	90
     Total	1000	Total	1000

Reference

[1] IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations, COLM 2024

[2] A Preliminary Study of the Intrinsic Relationship between Complexity and Alignment, COLING 2024

[3] Magicoder: Empowering Code Generation with OSS-Instruct, ICML 2024

4. Question about the performance comparison with close models.

   • We acknowledge that there remains a performance gap compared to closed-source models, which we hypothesize could be due to several factors. One possible reason is the inherent limited capabilities of the base model we employed. Additionally, it is unclear whether the closed-source models have been fine-tuned on similar or even more extensive task-specific datasets, which might have provided them with a notable advantage in these benchmarks.
   • Another point worth emphasizing is that our approach prioritizes maintaining a balance between task-specific performance and broader generalization abilities. For example, while some chart-augmented models like ChartLlama [4] and ChartAssistant [5] achieve strong results on specific tasks (e.g. ChartQA) by leveraging significantly larger datasets (e.g., ChartAssistant uses 39M samples, approximately 2,000× our dataset), such specialization often comes at the cost of reduced versatility in other areas, as observed in tasks like MathVista.
   • Despite these constraints, our work demonstrates meaningful progress. As shown in Table 4 of our paper, the fine-tuned InternVL2-8B achieves an average performance level approaching GPT-4o Mini (48.35 v.s. 49.34), highlighting its potential to achieve competitive and balanced results even with a relatively small dataset and weak model.

5. Question about the simplicity of the self-repair prompt.

   • We opted for a simple self-repair prompt because around 80% of code generated by GPT-4o passed on the first execution. By appending error messages and self-repair prompting for correction, we raised this success rate to 95% (Line 274, 275). While more complex workflows [6,7] may further improve this result, they would also incur additional costs.

6. Question about error analysis in trained models.

   • Thank you very much for this suggestion! We have added an analysis of errors before and after training to provide a clearer understanding of the improvements. The results are as follows:

     Model	# Errors	# Reco. Error	Reco. Proportion	# Reas. Error	Reas. Proportion
     LLaVA-Next-Llama3-8B	1355	827	61.0%	493	36.4%
     LLaVA-Next-Llama3-8B + ReachQA	886 (-469)	534 (-293)	60.3% (↓)	334 (-159)	37.7% (↑)
     MiniCPM-V2.5-Llama3	827	513	62.0%	297	35.9%
     MiniCPM-V2.5-Llama3 + ReachQA	714 (-113)	414 (-99)	58.0% (↓)	278 (-19)	38.9% (↑)
     InternVL2-8B	655	335	51.1%	296	45.2%
     InternVL2-8B + ReachQA	439 (-216)	209 (-126)	47.6% (↓)	198 (-98)	45.1% (↓)

   • The results demonstrate that: (1) For different models, both types of errors show a noticeable reduction, indicating that the trained model improves in both recognition and reasoning abilities. This is consistent with the observations in Sections 4.2 and 5.2 of our paper. (2) Additionally, stronger base models (e.g. InternVL2-8B) exhibit a lower proportion of recognition errors, and after training with ReachQA, the reduction in recognition errors is more significant. This suggests that recognition, as a more fundamental capability, could benefit more directly from the training.

Reference

[4] ChartLlama: A Multimodal LLM for Chart Understanding and Generation, arXiv.2311.16483

[5] ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning, ACL 2024

[6] Self-Edit: Fault-Aware Code Editor for Code Generation, ACL 2023

[7] SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, NeurIPS 2024

Dear Reviewer cHNi,

We hope this message finds you well. If this email arrives during your vacation or outside your normal working hours, please accept our apologies for the interruption.

It has been 10 days since our last round of communication. We just want to kindly follow up to ensure that we have addressed your concerns or any remaining questions you might have. We are still here and welcome any further discussion or feedback, as your insights are incredibly valuable to us.

Thank you very much for your time and consideration.

Best regards,

Authors of Submission 5783

Dear Reviewer cHNi,

We hope this message finds you well. We truly appreciate the time and effort you have already dedicated to reviewing our submission.

As the Author-Reviewer Discussion phase is nearing its conclusion, we just want to kindly follow up one last time. If there are any additional points you’d like us to address or clarify, we are here and ready to respond promptly. Should the issues be resolved, we kindly ask for your consideration in adjusting your scores accordingly.

Thank you again for your time and consideration!

Best regards,

Authors of Submission 5783

Thank you for your thoughtful comments and insights into our work. We greatly appreciate your recognition of our efforts and the questions raised. However, we have a few follow-up questions for understanding:

1. We notice that in your review, the references marked as [1] and [2] were mentioned but not specifically listed. Could you clarify or provide the full citations for these references? This would help us better understand the context and address your query more accurately.
2. Regarding Weakness 2, we want to seek clarification to ensure we fully understand your concern. From our understanding, you may be suggesting that, as we aim to fine-tune general MLLM models, we should consider comparing chart data from a more general dataset, rather than a specific dataset discussed in Section 5.3. This might involve, for example, sampling an equivalent amount of chart-related data from a broader multimodal dataset like The Cauldron.

If our understanding is incorrect, we would appreciate any further details you could share, so we can thoroughly address your concern as soon as possible. Thank you again for your dedicated review and feedback.

1. Sorry for missing the references.

[1]. https://arxiv.org/pdf/2409.01704

[2]. WizardLM: Empowering Large Language Models to Follow Complex Instructions

2. Yes, or other datasets, such as Cambrian-10M.

We appreciate your valuable time, insights, and for highlighting our strengths (i.e., the sound approach and the effectiveness of ReachQA). In the following, we will carefully respond to your questions.

1. Question about the use of code for rendering charts and data augmentation.

   • Thank you for your thoughtful feedback and for highlighting these important references. We acknowledge that our approach shares similarities with GOT [1] in leveraging code for synthetic chart data generation. However, it is worth noting that using code as a rendering tool is a fundamental and widely adopted practice in chart generation. It is not unique to our work or GOT [1]; earlier works such as ChartLlama [2] and ChartAst [3] also used code to build datasets for chart-related tasks. The use of programming languages (e.g., Python, R) and libraries (e.g., Matplotlib, Seaborn) is a common choice in this field due to their flexibility and precision in chart visualizations.
   • The second paper you mentioned, WizardLM [4], is a pioneering work that introduced the concept of Evol-Instruct to enhance data complexity. While they focus entirely within the textual modality, our motivation diverges significantly. Given the current scarcity of visually complex and reasoning-intensive chart datasets, we adopt an innovative approach by introducing the concept of "code-as-intermediary"—translating images into textual representations and subsequently adapting text-based data synthesis techniques, including Evol-Instruct (see Section 3.1, Line 247). In summary, while we acknowledge the importance of [4] as an inspiring foundation, our contribution lies in adapting it to a novel use case, using code to bridge visual and textual modalities and tackle the unique challenges of chart-related tasks.

2. Question about fine-tuning with chart data from general datasets.

   • Thank you for your suggestion! To address your point, we conducted additional experiments using an equivalent amount of chart-related data from The Cauldron [5], a massive collection of 50 vision-language datasets designed for general MLLM fine-tuning. Specifically, The Cauldron's chart understanding subset is a mixture of seven distinct datasets, and we followed the setting in Section 5.3 of our paper to proportionally sample 20k data for training. The results are as follows:

     Models	Avg.	ReachQA	CharXiv	MathVista	Math-V
     Base Model	16.39	6.50	17.20	32.40	9.44
     + The Cauldron	18.61	10.10	19.10	35.60	9.64
     + ReachQA	20.74	11.10	22.50	38.10	11.25

   • The results show that training with this general mixed dataset achieves slightly better performance compared to using individual datasets (e.g., ChartBench, ChartAst). However, the benefits are limited due to the quality of the included data, which features relatively simple charts, straightforward questions, and inadequate rationale annotations. As a result, the performance still lags behind fine-tuning with high-quality synthetic datasets like ChartGemma and ReachQA.

3. Question about the ∞ chart type.

   • Sorry for this confusion, but there seems to be some misunderstandings. In Table 1 of our paper, we specify “# Chart Type: 10 / 32”, indicating that our method can synthesize 10 major chart types and 32 subtypes, all of which can be generated using Python’s standard or third-party libraries.
   • The use of “# Chart Topic: ∞” reflects that our method does not restrict chart topics. Through a Self-Instruct approach, the model can freely expand on themes based on its knowledge, enabling a wide variety of topics to be represented. If needed, we will update the description in the revised manuscript to ensure clarity.

4. Question about the success rate of code generation.

   • We appreciate your interest in this detail. As Section 3.2 (lines 274–275) mentions, the execution success rate of code generated by GPT-4o is approximately 80%, which improves to about 95% after applying a Self-Repair method.
   • We also experimented with other models, such as Llama-3.1-70B-Instruct, which initially achieved around a 60% pass rate. However, through iterative Self-Repair, this also improved to over 90%.

Reference

[1] General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Model, arXiv.2409.01704

[2] ChartLlama: A Multimodal LLM for Chart Understanding and Generation, arXiv.2311.16483

[3] ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuning, ACL 2024

[4] WizardLM: Empowering Large Language Models to Follow Complex Instructions, ICLR 2024

[5] https://huggingface.co/datasets/HuggingFaceM4/the_cauldron

Dear Reviewer U7uB,

We hope this message finds you well. If this email arrives during your vacation or outside your normal working hours, please accept our apologies for the interruption.

It has been 10 days since our last round of communication. We just want to kindly follow up to ensure that we have addressed your concerns or any remaining questions you might have. We are still here and welcome any further discussion or feedback, as your insights are incredibly valuable to us.

Thank you very much for your time and consideration.

Best regards,

Authors of Submission 5783

Dear Reviewer U7uB,

We hope this message finds you well. We truly appreciate the time and effort you have already dedicated to reviewing our submission.

As the Author-Reviewer Discussion phase is nearing its conclusion, we just want to kindly follow up one last time. If there are any additional points you’d like us to address or clarify, we are here and ready to respond promptly. Should the issues be resolved, we kindly ask for your consideration in adjusting your scores accordingly.

Thank you again for your time and consideration!

Best regards,

Authors of Submission 5783

Here are some additional actions we have undertaken during the discussion phase. We hope these updates further demonstrate the effectiveness of our approach and address the concerns raised. Thank you!

Supplementary Experiments

6. Adding analysis of qualitative examples. (Reviewer 4dEr, Round 2)
   • We conducted a case study visualizing the model's attention patterns to the image during inference, comparing the vanilla model with our fine-tuned version. As presented in Appendix F and Figure 6 of the revised paper, the vanilla model exhibits dispersed attention, leading to redundant outputs and incorrect conclusions, while the fine-tuned model focuses effectively on key elements, enabling accurate and efficient reasoning.
7. Adding detailed results on two new benchmarks. (Reviewer Yw4L, Round 2)
   • We conducted more evaluations on the OCRBench and We-Math benchmarks following the setting in Section 5.2 of our paper. Most of the results are consistent with the trends discussed in our paper and the original benchmark papers.