MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Engine

We sincerely appreciate your insightful review and recognition of our work. We have provided detailed responses to your comment and updated the relevant content in the revised manuscript, hoping to address your concerns.

Q1: As the data engine is a rule-based system, how to ensure the diversity of captions and questions in training data?

Thanks for pointing it out! we ensure the diversity of training data in two aspects:

1. GPT-generated Diverse Templates. While the construction of our captions and questions follows a template-based approach, we leverage GPT-4 to generate a broad range of diverse language templates. These templates are designed to include various linguistic expressions, syntactic structures, and phrasing styles to minimize repetitive patterns. In this way, we ensure that the resulting dataset covers a rich diversity of language styles, enhancing the robustness of the training data and enabling the model to generalize better across different tasks.

2. Diversity through Real-world Data Integration. We have incorporated 201K (24%) real-world collected and augmented data alongside the synthetic data (633K) into MAVIS-Instruct to diversify its distribution. Therein, 83K problems were manually collected and augmented from publicly available sources (as detailed in Appendix A.4.3), spanning diverse topics. The other 118K problems were sourced from existing mathematical datasets, predominantly covering plane geometry. Such real-world data well enriches the diversity of MAVIS-Instruct data, generalizing MLLMs across various mathematical tasks.

Q2: The author only provides the results of the 7B model. What if the model size is larger, for example, 13B?

Thanks for your advice! In the table below, we present the results of the MAVIS model using two different sizes of LLMs: Mammoth-2-7B and Mammoth-2-8x7B. The accuracy on MathVerse demonstrates that with larger LLMs, the MAVIS model showcases good scaling performance and achieves enhanced mathematical reasoning capabilities.

Q3: Whether these approaches can help a pre-trained large model (LLaVA) improve the mathematical performance?

Thanks for your suggestion! Our curated data and training techniques can well generalize to other pre-trained MLLM to enhance their mathematical performance. In the table below, on top of LLaVA-NeXT-8B, we progressively employ our MAVIS-Instruct for instruction tuning and DPO alignment, both training for one epoch with an learning rate as 1e$^{-5}$. The two continual training stages significantly boost the mathematical problem-solving skills of LLaVA-NeXT.

We sincerely appreciate your valuable comments and recognition of our work. We have provided detailed responses to your comment and updated the relevant content in the revised manuscript, hoping to address your concerns.

Q1: Although MAVIS is a mathematical visual specialist, the performance is still similar with some generalist MLLMs. For example, on the vision-only version of MathVerse, MAVIS is only 0.9% better than LLaVA-NeXT-110B

Thanks for pointing this out! We would like to clarify this observation and address it from several perspectives:

1. 7B vs. 110B. Our model is built on an LLM with only 7B parameters, whereas LLaVA-NeXT-110B utilizes a significantly larger LLM with 110 billion parameters, over 10 times the size of ours. Despite this disparity, we believe it is a noteworthy achievement that, our 7B model surpasses a 110B model in the overall results of MathVerse with significant margins (+6.8 CoT scores and +3.9 accuracy). Compared to other models of comparable sizes, e.g., LLaVA-NeXT-8B, MAVIS achieves consistently superior superior results with +18 CoT scores and +12.8 accuracy. This highlights the effectiveness of our training strategies and curated datasets in maximizing the performance of a smaller model.

2. Better CoT Performance on Vision-only Problems. While MAVIS-7B’s accuracy score on vision-only problems is slightly lower than LLaVA-NeXT-110B (-2.3), our model achieves a higher CoT evaluation score with a +0.9 margin. The CoT score is a strong indicator of step-by-step reasoning capabilities. Thus, this advantage demonstrates the effectiveness of our rationale annotation and DPO training to conduct high-quality CoT reasoning.

3. Improving Vision-only Results with OCR Data. Vision-only problems in MathVerse require the model to interpret question texts rendered directly in diagrams, relying heavily on OCR capabilities. MAVIS-7B, however, was not trained with OCR-specific datasets, limiting its performance in these tasks. In contrast, generalist models like LLaVA-NeXT include extensive OCR datasets such as OCRVQA, DocVQA, and SynDog-EN, which boosts their OCR capabilities and performance in vision-only problems. To address this gap, we also include OCR datasets (OCRVQA and DocVQA) in our third-stage instruction tuning to enhance OCR capabilities. The results, as shown in the table below, reveal a notable improvement in vision-dominant and vision-only problems. This highlights the potential of MAVIS-7B to achieve even higher results with better OCR integration.

Thanks authors for their detailed response, I'm generally satisfied and will keep my positive rating.

I appreciate the author's effort in improving the vision-only performance by incorporating OCR training, the performance, however, only improves about 3%. This might suggest that OCR ability is benificial but not the main bottleneck of their relatively weak vision-only performance. I know it is challenging to address this during this period, but it would be helpful to provide some qualitative examples to give us some intuition about what are the main bottlenecks.

We sincerely appreciate your insightful reviews and constructive advice. We have provided detailed responses to your comment and updated the relevant content in the revised manuscript, hoping to address your concerns.

Q1: The paper lacks an overview figure, which may make it challenging for readers to understand the data collection pipeline.

Thanks for your suggestion! A clear overview figure can aid readers in better understanding the methodology and contributions of our proposed data engine. Accordingly, we have included an overview figure in Figure 2 of the revised manuscript.

Q2: This work does not propose a new model architecture or training approach, with only a mathematical dataset provided for fine-tuning. (Fortunately, the engineering effort is substantial)

Thanks very much for recognizing our engineering effort. We would like to clarify the main purpose and contributions of this study, which are summarized as follows:

1. The Main Goal of this work is to provide a general and robust paradigm for training MLLMs in mathematical reasoning, including training pipelines and datasets. Our focus is not on proposing new fancy model architectures or training approaches, but on exploring a concise and effective method to target specific challenges in this field. This work provides valuable insights and a practical foundation for future research in mathematical MLLMs.
2. Our Training Pipeline is specifically designed for key shortcomings in math. Given our observation of MLLMs' three issues (visual encoding of math diagrams, diagram-language alignment, and chain-of-thought (CoT) reasoning), we respectively devise four training stages to alleviate them. The first pre-training stage of Math-CLIP bridges the semantic gap from general images to math diagrams. The second cross-modal alignment stage enhances the interpretation of LLMs on diagram embeddings from Math-CLIP. The third mathematical instruction-tuning stage endows MLLMs with fundamental problem-solving and reasoning skills. The fourth DPO training further elicit advanced reasoning capabilities and generate high-quality CoT rationales. Therefore, while the techniques themselves are not new, our work represents the first effort to integrate and order them systematically for mathematical domains, collectively optimizing MLLM performance for mathematical reasoning, which provides reference for future work in this area.
3. The Curation of Datasets is well tailored for the mathematical domain. For each training stage we designed, we respectively support a corresponding dataset to fully unleash the training effectiveness, e.g., MAVIS-Caption for the first and second stage, MAVIS-Instruct for the third stage, and DPO ranking data for the fourth stage. Unlike conventional instruction-tuning datasets, our curation process incorporates the unique characteristics of mathematical tasks. For example, to mitigate the high costs of manual data collection and annotation, we develop an automatic data engine to efficiently generate math problems, including diagrams, captions, questions, and CoT rationales. We refined these problems into text-lite versions by removing textual redundancy, ensuring the model focuses more effectively on diagram interpretation during training. These designs leverage domain-specific insights to enhance the training process for mathematical reasoning tasks.

In conclusion, our work is far from a naive application of existing techniques. Instead, it integrates thoughtful designs in training methodologies and dataset curation, specialized for the multi-modal mathematical domain.

We sincerely appreciate your insightful reviews and constructive advice. We have provided detailed responses to your comment and updated the relevant content in the revised manuscript, hoping to address your concerns.

Q1: It remains unclear how much benefit of MAVIS data over existing visual instruction data (e.g., LLaVA data) in fine-tuning Mammoth-2

Thanks for your advice! To investigate the detailed benefit of MAVIS data in Mammoth-2 (7B), we conduct an ablation study as below and updated it in the revised manuscript. As you suggested, we adopt the data from LLaVA-NeXT (558K for pre-training and 760K for fine-tuning) and compare with our MAVIS data (558K MAVIS-Caption for pre-training and 834K MAVIS-Instruct for fine-tuning). We evaluate the accuracy metric on MathVerse and don't involve the DPO training stage. Result in the first row denotes the original LLaVA-NexT with LLaMA-3 (8B).

From the results, we observe:

1. Mammoth-2 brings +2.7 score compared to LLaMA-3 with, indicating its prior knowledge of mathematical problem solving.
2. Applying MAVIS-Instruct for fine-tuning can significantly enhance the performance by +7.4, demonstrating the great advantage of our data for mathematical reasoning compared to general visual instruction data.
3. Using MAVIS-Caption for pre-training and training Math-CLIP encoder can further attain higher scores with enhanced mathematical visual perception. Therefore, our MAVIS data contributes to +9.2 score in total over Mammoth-2 with LLaVA data.

---

Q2: It is crucial to see if most of the improvements in MathVerse are mainly in the plane geometry, while solid geometry and functions still suffer

1. Sorry for the confusion caused. Our MAVIS data (including MAVIS-Caption and -Instruct) mainly consists of two subjects: plane geometry (985K, 69%) and functions (437K, 31%). They are expected to improve MLLM with their solving capabilities of both geometry and function problems.

2. We provide the detailed subject scores of MAVIS-7B on MathVerse, and compare the CoT evaluation score (the subject-level accuracy score is not released) with other models from the official leaderboard.

The results showcase that our model achieves leading performance on all three subjects. Therein, the proficiency of plane geometry and functions are primarily due to the training with our curated MAVIS data. For solid geometry, as it is similar to plane geometry in both visual appearance and reasoning process, we believe our model can well generalize its learned knowledge and reasoning capabilities to the solid geometry domain for improved performance.

---

Q3: How much of the performance comes from the synthetic data and the real data is unclear.

Thanks for your suggestion! In MAVIS-Instruct, we integrate both synthetic problems from the data engine (633K, 76%) and real-world problems augmented with GPT (201K, 24%). Synthetic data is composed of both geometry and functions, while real-world data mainly focuses on geometry. We provide the ablation of data component below and don't involve the DPO training stage for fairness.

As shown, the two sources of data exhibit complementary characteristics, and both of them are significant to the final performance. Synthetic data can better improve the results of FunctionQA and MMMU-Math, since they both contain a large portion of function problems. In contrast, real-world data contributes more to GeoQA, which are more aligned in geometry problems.