Euclid: Supercharging Multimodal LLMs with Synthetic High-Fidelity Visual Descriptions

Why is it important to solve Geoperception intermediate-free VQA, instead of methods like VDLM?

Thank you for bringing this up. We agree that alternative methods, such as VDLM, could also be applied to address geometry perception problems. However, the focus of this paper is not on identifying a specific solution for Geoperception tasks. Instead, we use these tasks as a representative testbed to investigate the design principles of MLLMs and their ability to effectively learn low-level visual perception concepts. This exploration provides insights that extend beyond geometry perception to general-purpose and domain-specific MLLM training. Furthermore, as noted in the VDLM paper, LLMs cannot directly interpret SVG code. VDLM addresses this by training a model to convert SVG code into a predefined language. However, this predefined language is highly constrained and can represent only very simple shapes (see VDLM paper, Page 24 https://arxiv.org/pdf/2404.06479#page=24 for reference—characters, for example, are not supported).

In contrast, our empirical study demonstrates that synthetic data training enables MLLMs to generalize low-level perception abilities to a wider range of geometric shapes. This approach directly contributes useful insights into designing MLLMs for low-level visual perception, both for general-purpose applications and specific domains. The Euclid testing results further validate this by showing that our model, trained on synthetic data, successfully generalizes its learned low-level visual abilities to diverse geometry shapes.

The authors state their task is "straightforward for humans" but do not provide quantitative comparison. Have they measured this?

Thanks for bringing this up. To address your concern, we have randomly chosen 20 examples from each task from Geoperception, and performed a human evaluation. The result showed that all 140 datasets are quickly and correctly solved by humans, demonstrating that the Geoperception benchmark is very straightforward for humans. We will add this human experiment in the updated version of the paper

The authors write "For our Euclid model, we include all geometry shape code used for training, along with demonstration diagrams and pseudocode for generating training questions and answers," but it does not appear that supplementary materials were submitted.

The geometry shapes and demonstrations are included in the appendix C.2, and the pseudocode for generating the training questions and answers are included in appendix C.1.

The value of curriculum learning.

As shown in Figure 6, directly training the model on Shape 3—the most complex shape—leads to non-convergence. In contrast, curriculum learning, including mixed-shape training (illustrated as spontaneous curriculum learning in Figure 5), achieves convergence with the same amount of training data. This transition from non-convergence to convergence demonstrates a significant improvement in the training process, enabling better utilization of the training data and unlocking greater potential for model performance. Furthermore, we find that adaptive curriculum is more efficient than mixed-shape training to be practically valuable. This is because: 1. For many tasks, such as real-world low-level visual perception, training datasets are often constrained, making it critical to explore and identify optimal training strategies. 2. Although our dataset generation engine can theoretically produce unlimited task-specific training samples, compute resources are typically a limiting factor—particularly when training large models or scaling the method to broader and more diverse data distributions.

It is not true that "Euclid significantly outperforms current leading MLLMs," "surpassing the leading MLLMs by a substantial margin." It is outclassed by Gemini-1.5-Pro on one of the four tasks and exceeds it by less than a percent on another.

Thanks for pointing this out. It is true that our model does not outperform the leading MLLMs across all tasks, instead our best all-in-one Euclid model surpasses Gemini-1.5-pro by 10.21% by average of four tasks. We have clarified the claims in the updated version of the paper. (Line 471-480)

Why are only four out of seven benchmark tasks used for evaluation of the proposed model (Table 4)?

In our Euclid testing, the objective is to evaluate whether the low-level visual perception capabilities learned from synthetic data training can effectively generalize to shapes in real-world geometry problems (i.e., Geoperception). We focus on the four primitive tasks because:

1. As shown in Table 3, current MLLMs struggle significantly with these tasks compared to the three annotation-based tasks.
2. These tasks do not require human-defined annotations on geometric diagrams, providing a clean and unambiguous testbed to assess the model's ability to generalize geometric shapes. In contrast, the three annotation-based tasks rely heavily on manually designed diagram annotations (e.g., arrows pointing to angles or specific styles to denote parallel lines), introducing undesirable gaps between our training datasets and the Geoperception benchmark.

Nonetheless, to address your concerns, we extended our geometry generation engine to produce diagrams for the annotation-based tasks, including segment length, angle measure, parallelism, and perpendicularity, with each task having a single consistent annotation style. Instead of training Euclid on separate tasks, both models are jointly trained on all seven tasks. We present the updated results below and will include them in the revised version of the paper:

In general, introducing geometric annotations into the training dataset improves Euclid's performance on most of the promotive tasks (except for PointLiesOnCircle). We hypothesize that this improvement is due to increased model robustness resulting from training on a more diverse dataset. However, for the three annotation-based tasks, Euclid’s performance does not surpass the current leading MLLMs. We attribute this to the annotation style gap between our synthetic training data and the Geoperception benchmark, as previously explained.

Thank you for your suggestions and insightful comments. We really appreciate your effort in reviewing the paper. We address your questions and comments below:

Concern about whether the performance on Geoperception is dominated by syntax error.

To improve clarity, in Appendix B we show the full prompts given when evaluation models on Geoperception, which include both instructions in addition to the original question. For example, in the instructions, we separately give the answer template for both single answer point and multiple answer points. We observed that all evaluated models adhered to these instructions, indicating no difficulty in understanding verbal instructions and questions. For length comparison tasks, during construction, we make sure no annotations are made on the lines in the question. Regarding label consistency, we randomly shuffle the order of the letters for segments instead of alphabetizing them. During evaluation, ‘ZV’ and ‘VZ’ are considered as one ground truth answer, we include both of them for matching purposes. Given the detailed and clear instructions that we give to the model, and the fact that most of the MLLMs achieve reasonable performance, suggest that ‘frequently enumerating all potential components’ should be considered as a visual perception weakness rather than syntax error. We will clarify the above in our updated paper.

Concern about real-world application.

We acknowledge that our designed dataset differs from real-world downstream applications; geometry perception, however, is a fundamental prerequisite for low-level visual perception in practical scenarios—a model without reliable performance on these simple and straightforward tasks could not be trusted to perform low-level visual perception in more complex settings. Moreover, geometry understanding is a focused and controllable domain where datasets can be programmatically generated and different aspects, such as task complexity, can be systematically manipulated. This enables us to better diagnose the root causes of the model's shortcomings and explore strategies for addressing them effectively.

Concern about empirical exploration.

In this study, we focus on low-level geometric visual perception, deliberately avoiding tasks that require additional reasoning. In this context, we consider the transcription of visual information into the language space to be the most critical component. Consequently, we emphasize the selection of visual encoders. Regarding the superior performance of CNN architectures, we hypothesize that CNNs extract visual features globally, effectively preserving low-level visual features. In contrast, ViT architectures split images into discrete patches, making it more challenging to retain the original low-level visual information. We will incorporate this explanation into the next version of the paper.

Following your suggestion, we conducted an experiment to study the effect of LLM size, where we train 0.5B, 1.5B and 3B models with the same other model components and the same dataset. We evaluated both the training loss and testing accuracy curves. The results indicate that scaling up the LLM size does not lead to significant improvements. This observation further supports our choice of using a 1.5B LLM. We have included these findings, along with the training loss and testing accuracy curves, in Figure 18 of the updated paper version and plan to move them to the main text in the next version.

Notably, the trends in the training and testing accuracy curves are well-aligned. We hypothesize that this alignment arises because the training and test datasets are generated by the same dataset engine and share the same distribution. Therefore, the training loss trends are representative for comparing different training strategies. That said, we fully agree that evaluating on a separate test set provides additional value. In response, we are re-running the entire empirical study, performing test set evaluations at every 50 training steps, and will include these results in the updated version of our paper.

Generalizing to Real World Images

We totally agree that gaining the low-level visual perception ability of other types of images is very important, and is a future goal in this line of work. However, we think the study of generalization ability merits its own paper so we leave it to future works. In this paper, our scope is to systematically find out the problem in today’s MLLMs, and explore optimal training architecture and data composition choices to learn such tasks effectively and efficiently.

Why not consider the multiple-choices form for question answering?

Thanks for the suggestion. In most of the questions in Geoperception, we expect a list of ground truth answers. For example, all points lying on the same line. To this end, we choose to evaluate the model’s performance smoothly (line 224), rather than using a binary evaluation metric. While we fully agree that using the format of multiple choice questions can reduce the ambiguity and make the evaluation more rigid. We observed that all evaluated models adhered to the detailed instructions (outlined in Appendix B) indicating no difficulty in understanding verbal instructions and questions.

Is it possible to transform geometric figures into SVG code, just as VDLM, to evaluate the performance of the OpenAI-o1 model? Moreover, the authors may add some evaluation on some geometry MLLMs or math MLLMs.

Thanks for the suggestion. We agree that methods like VDLM is another strategy for solving similar geometry problems. However, instead of finding alternative ways to solve these geometry problems, the goal in this paper is to use the geometry perception as a representative testbed to explore the design of MLLMs so that they can effectively learn the low-level visual perception concepts. Furthermore, in the VLDM paper, they noted that LLMs are not able to directly understand the SVG codes, so they trained a model to transform the original SVG code into a predefined language. Under this language, however, only very simple shapes can be represented, (se​​e https://arxiv.org/pdf/2404.06479#page=24 for reference, even characters are not covered)..

Following your suggestion, we run geometry MLLMs on Geoperception task to assess their performance. However, we find that they fail to follow instructions properly. Here is one example of G-LLaVA[1]: (the input image is similar to the images we present in Figure 2)

Input: <image> What is the point lying on line RB?

Output: Solution: Since angle BAC = 30° and angle B = 45°, we have angle B = 15°. Since AC = BC, we have angle C = angle BAC = 30°. Therefore, angle BAC = angle C = 30°. Thus, triangle ABC is an equilateral triangle, so AB = BC. Since AB = BC, we have AB = AC. Therefore, the answer is B. Answer:B

Note that in this example, instead of following instructions properly and identifying the point lies on a certain line, the model outputs an irrelevant geometry problem solution. We hypothesize such behavior is sourced from the limited training dataset diversity of the geometry-oriented models, which are typically only trained on geometry problem solving.

[1] Gao, J., Pi, R., Zhang, J., Ye, J., Zhong, W., Wang, Y., ... & Kong, L. (2023). G-llava: Solving geometric problem with multi-modal large language model. arXiv preprint arXiv:2312.11370.

[2] Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., ... & Xie, S. (2024). Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860.

Thank you for your suggestions and insightful comments. We really appreciate your effort in reviewing the paper. We address your questions and comments below:

Overall, this paper devotes a significant amount of space to introducing background information and other methods, while providing relatively fewer details about its own proposed approach.

Thank you for your feedback. To better clarify, we reiterate our contribution of this paper below (as we feel a lot of what you might view as background information, we view as our contributions):

1. Geoperception Benchmark: We introduce a novel benchmark specifically designed to evaluate MLLMs' low-level geometric perception, revealing critical shortcomings in current models. This benchmark offers a clear testbed for future work on precise visual perception.
2. Empirical Study on Synthetic Data Engine: We conduct a detailed design study of architectural and training strategies, supported by a high-fidelity synthetic data engine. This enables us to provide actionable insights, such as the effectiveness of curriculum-based training, with implications for general-purpose low-level visual perception.
3. Euclid Model: Leveraging these insights, we propose the Euclid model series, which significantly outperforms current leading MLLMs on many tasks and by average (e.g., up to 54.52% improvement over Gemini-1.5-Pro on certain perception tasks) despite being trained only on synthetic data and having only 1.5B-parameter LLM.

Relationship with Geometry-3K

We fully agree that the Geometry-3K dataset is inherently more complex than Geoperception. However, as the reviewer agrees, the foundational low-level visual perception of MLLMs should first be sufficiently robust, as a precursor to performing more-complex calculations or reasoning. The primary objective of proposing Geoperception is to isolate and rigorously evaluate this fundamental aspect of perception.

Our evaluation on Geoperception, as shown in Table 2, demonstrates that all leading MLLMs encounter significant difficulties in basic geometric perception tasks. The scope of this paper is specifically to tackle this critical challenge.

For baselines, why not directly fine-tune some MLLMs on Geoperception, to compare with the method of Euclid?

Thanks for bringing this up. As a benchmark dataset, Geoperception’s volume (Table 1) is not enough for a MLLM training, which typically needs at least tens of thousands of training instances. Furthermore, we were aiming to keep Geoperception data as held-out test data (so as not to "cheat" nor contaminate our training set with test data).

To further address your concern, we tested Cambrian-1[2] on our Geoperception dataset, which is reported to be trained on Geo-170K, a geometry multimodal instruction tuning dataset built on the logical annotation of Geometry-3K, having the same source with our Geoperception. The testing result below shows that Cambrian-1 also faces challenges in our Geoperception task, despite being trained on the dataset having the same images and augmented text annotations:

Thank you for your suggestions and insightful comments. We really appreciate your effort and support of the paper’s publication. We address your questions and comments below:

Concerns about Data Filtering with MLLM

Sorry for the confusion, we only use GPT-4o-mini to verify the existence of points, which is much simpler than answering the training question. To verify the reliability of GPT-4o-mini, we also manually checked 100 examples of its annotation and found only 2 mistakes that the model made (line 197-199).

Limited Comprehensive Geometric Understanding

Thanks for bringing this up. As is discussed in our error analysis (line 491-495), the images in Geoperception are highly annotated, while our training dataset has only simple geometry shapes and point characters. To make Euclid perform more robustly and address your concern, we have extended our geometry generation engine to enable producing diagrams with annotations, including segment length, angle measure, parallelism, and perpendicularity, with each task having a single consistent annotation style. We then randomly integrate annotations throughout our training dataset. Instead of training Euclid on separate tasks, both models are jointly trained on all seven tasks. We present the updated results below and will include them in the revised version of the paper:

In general, introducing geometric annotations into the training dataset improves Euclid's performance on most of the promotive tasks (except for PointLiesOnCircle) due to increased model robustness resulting from training on a more diverse dataset.

How many solutions are sampled for each question? Is any form of nucleus sampling performed i.e. sampling multiple answers per question? Please provide details for the same.

We use greedy decoding for each model to get deterministic results and a fair comparison. We only sample the answer once since it is deterministic.

Terminology of Shapes

The terminology ‘shape’ refers to a logical geometry shape (e.g., a triangle with a point connecting to the midpoint of the opposite edge, which is our first shape). During training, our geometry generation engine is able to generate many numerical instances of each logical shape (line 299-303). Specifically in this shape, the generation engine will first pick three random points from the canvas to form a triangle, and then connect the first point with the midpoint of the opposite edge. Then the generation engine will randomly pick 4 capital letters from the alphabet (A-Z). and assign the picked letters to 4 points on the diagram.

Following the above question, it might be important to increase the dataset by sampling more shapes from the geometric shape generation engine and then training on that dataset.

Following our above response, if you mean numerical instances by ‘shapes’, our model is trained on near-infinite numerical instances of the three logical shapes. If you mean logical shapes by ‘shapes’, in our Euclid section, we do indeed train our model on a larger number of shapes and tasks. We completely agree that training on more logical shapes will be beneficial, but as noted in Section 4, making the training work on several human-curated logical shapes is already challenging, therefore we leave it to future works.

Adapting Negative-hard Mining Strategy for Automatic Curriculum

We appreciate the valuable suggestion. In our paper, we demonstrated the effectiveness of curriculum learning through a human-defined curriculum order. However, in scenarios involving a broader range of tasks and image types—such as developing an automatic geometry shape generation engine—it is crucial to adopt advanced strategies like negative-hard mining to automatically determine the difficulty of training instances. While such approaches are promising, incorporating more diverse tasks requires substantial design efforts and introduces additional complexities that fall beyond the scope of our current work. Therefore, we have added a discussion of this idea to the future work section of our paper (Line 515-520) and will explore and integrate such techniques in our future work.

[1] Liu, H., Li, C., Li, Y., & Lee, Y. J. (2024). Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 26296-26306).

[2] Gao, J., Pi, R., Zhang, J., Ye, J., Zhong, W., Wang, Y., ... & Kong, L. (2023). G-llava: Solving geometric problem with multi-modal large language model. arXiv preprint arXiv:2312.11370.

Thank you for your suggestions and insightful comments. We really appreciate your effort and support of the paper’s publication. We address your questions and comments below:

Concerns About Training Dataset Volume

In this paper, we adopt the standard configuration for visual instruction tuning [1], which includes a pre-trained visual encoder, a pre-trained LLM, and a multimodal connector—a two-layer MLP—trained from scratch. Notably, the dataset size for LLaVA-1.5 (a general-purpose MLLM) is approximately 1 million instances. The intuition is that the multimodal connector, being an MLP with only a few million parameters, is the only component trained from scratch. Moreover, since the vision encoders are pre-trained with text supervision, it is easier to align it with LLM. Additionally, Geo-170K, a widely used geometry-focused multimodal training corpus proposed in G-LLaVA [2], comprises only about 9K unique images.

In our experiments, we utilize a dataset of approximately 100K instances, as we specifically focus on task-oriented scenarios rather than general-purpose multimodal capabilities. To validate the adequacy of our dataset, we observed clear patterns of convergence in both the mixed-task results in Figure 3 and the training curves in Figure 4, indicating that our dataset size is sufficient for the targeted tasks. Furthermore, during the training of our Euclid model, we incorporated a dynamic dataset generation engine that advances the training stage only when the testing accuracy on intermediate evaluations reaches a predefined threshold of 0.99, which leads to robust convergence. Leveraging the flexibility of our dataset generation engine, we ensured that all designed tasks are fully learned by the model with sufficient training data.

To further address your concern regarding dataset sufficiency, we conducted additional experiments by training the ConvNeXt-Large model on 1 million instances from a mixed-shape dataset for the PointLiesOnLine task (same setting as the last sub figure of Figure 3). We test the model on a separate test set (containing 1,500 instances) every 500 steps (i.e., 32K total training instances). The result is shown in Figure 19 of the updated paper, demonstrating that the model's test accuracy quickly approaches near 100% within fewer than 100K samples. This empirical evidence further confirms that the dataset volume used in our primary experiments is adequate for training the model effectively.

"Molmo predicts every potential answer, leading to their poor accuracy scores" – If this means, that instead of only answering the required/asked points on the line, Molmo outputs all the points on the line –– this seems less critical.

Yes, this means Molmo frequently outputs all the points on the line instead of the required ones. Thanks for pointing this out and we will clarify this in the paper in our updated version.

Clarification Regarding Sampling Multiple Answers Per Question

Our evaluation revealed that MLLMs excel at identifying all letter names in a diagram (a global OCR task) but struggle to pinpoint the required points. Consequently, we assign a score of 0 for any response that includes at least one incorrect answer. To further mitigate this issue during evaluation, we append the original questions with detailed instructions (outlined in Appendix B). These instructions include specific answer templates for single-point (line) and multiple-point (line) responses. We observed that all evaluated models adhered to these instructions, indicating no difficulty in understanding verbal instructions and questions.

Furthermore, following your suggestion, we incorporated two different evaluation metrics into the updated analysis:

1. Superset Accuracy: Accuracy is scored as 1 if the model's prediction is a superset of the required answer. The corresponding results are presented in Table 7 of the updated version of the paper.
2. Subset Accuracy: Accuracy is scored as 1 if the model's prediction is a subset of the required answer. These results are presented in Table 6 of the updated version of the paper.

The overall performance trends remain consistent, with proprietary models outperforming open-source models. Notably, under the first evaluation setting (superset accuracy), Molmo-7B-D emerges as the best-performing open-source MLLM, and GPT-4o-mini achieves performance levels comparable to other proprietary models. This phenomenon aligns with our observation that GPT-4o-mini and Molmo-7B-D frequently enumerate all potential components of an answer. Consequently, their tendency to overgenerate leads to inflated scores under the superset evaluation metric, despite not accurately targeting the required answers.

It is disappointing the authors waited until a day before the end of the discussion period to make their initial response. It is difficult in the remaining amount of time to review and discuss the large number of changes introduced. I will respond shortly today.

I am also inclined to clarify that the impressions attributed to me above are not correct. I did not previously describe the evaluation as "insightful," but as "disappointingly shallow."