We thank the reviewer for insightful questions that help refine our work further.

1. Claim 1—that GeoGLIP outperforms CLIP—is unconvincing due to missing baselines using the same VLM and data with original vision encoders, leaving it unclear whether gains come from the backbone or training data.

We have conducted an ablation study in Sec. A.4 (lines 840-848 and Tab. 6), which utilizes different visual encoders trained on the same backbone and datasets.

• We compare a single encoder setup using either CLIP or GeoGLIP, with the same backbone VLM (LLaVA) and dataset (Geo170K). GeoQA accuracy is 64.2 for CLIP and 66.1 for GeoGLIP.

• We evaluate dual encoders combining GLIP or GeoGLIP with CLIP, showing that without math-specific fine-tuning, GLIP's performance drops (67.0→65.3) due to limited geometric sensitivity, as illustrated in Fig. 10.

If the reviewer considers this ablation crucial, we are open to moving it to the main paper.

2. Additional analysis is needed to assess GeoGLIP’s robustness to visual variations (e.g., resolution, line style/width, junction size) in junction and boundary detection, and how detection errors impact final performance.

In response to this concern, we evaluated the robustness of our model using a custom set of 500 images with various distortions, as no public benchmark exists for junction and boundary detection in mathematical diagrams. We adopted standard metrics from natural image tasks:

• Junction Detection: Recall, using a confidence threshold of 0.65.

• Boundary Detection: Intersection over Union (IoU), with boundary maps binarized using a threshold of 200 (1 for boundary and 0 for non-boundary); IoU is computed via pixel-wise logical AND/OR between prediction and ground truth.

Distortions Applied:

• Gaussian Noise: Variance of 0.3.

• Resolution Change: Shortest side reduced from 800 to 400 pixels.

• Line Width Variation: Increased from 1–4 to 1–8 pixels, correspondingly impacting junction size.

• Line Style Modification: Changed from solid to dashed lines.

Our results (see table) show GeoGLIP is robust to resolution and line width changes, aided by diverse training data and augmentations (e.g., varying line widths, flipping, cropping, keep-ratio resize). It is more sensitive to Gaussian noise (−4.6 junction, −3.1 boundary) and dashed lines (−3.2 junction, due to higher false positives). These results highlight the need for improved data generation and training strategies to better handle visual distortions.

As Gaussian noise and dashed lines are the most impactful factors, we apply Gaussian noise for further reasoning evaluation due to limitations in modifying line styles without access to the original drawing codes of evaluation images. On GeoQA, top-1 accuracy dropped by −1.3 (PRIMITIVE-7B), −1.8 (PRIMITIVE-Qwen2.5-7B), and −2.7 (PRIMITIVE-Deepseek-7B). In contrast, the baseline model using only the CLIP encoder (PRIMITIVE(-)) showed larger drops of −3.1, −4.2, and −5.1, indicating that CLIP is more sensitive to distortions and provides less reliable visual input for reasoning.

3. Preservation of general ability.

While our model doesn't lead on FigureQA and VQA in Table 2, it significantly outperforms the G-LLaVA baseline with gains of +19.6 and +2.1, respectively. To address concerns about general ability, we further evaluated PRIMITIVE-7B on SEED-I and MM-Bench. On SEED-I, it maintained comparable performance to LLaVA-1.5-7B (66.2 vs. 66.9), and on MM-Bench, it achieved a +0.6 improvement. Most gains are observed in categories like instance interaction, counting, and spatial localization.

4. Inconsistent notations and formatting issue.

We apologize for the confusion. The four features transferred from GeoGLIP to the Connector include one mixed feature ($F^{1*}$) and three unmixed features. We will clarify this in Sec. 3.3, including revising the index notation in Eq. (1). We will adjust the title to meet the formatting requirements.

We thank the reviewer for insightful questions that help refine our work further.

1. The impact of visual encoders—e.g., comparing Qwen2.5-VL and PRIMITIVE-Qwen2.5-7B with fixed LLM weights.

To address the concern about visual encoders' impact on reasoning, we compare variants in controlled settings, as detailed in lines 840–847 and Tab. 6 of the appendix.

Although Qwen2.5-VL and PRIMITIVE-Qwen2.5-7B share the same LLM backbone, they differ significantly in training recipes and dataset scale (~2T vs. ~600K). For details, see Q2 (Ay8j). In Tab. 6, we control for training settings and datasets, isolating the visual encoder for direct comparison. See Q1 (sjVj) for analysis. For qualitative response differences, we analyzed model reasoning steps and found that the GLIP+CLIP dual encoder struggles with basic shape perception and interrelationship descriptions, often leading to hallucinations and incorrect answers. We will include their rationale demos in the revised paper.

To ensure a fair comparison with Qwen2.5-VL, we conducted new experiments resulting in Qwen2.5-VL-7B+ and PRIMITIVE-Qwen2.5-VL-7B (see Q2 for ARAy8j), with more detailed comparisons to be added in the revision.

2. The PRIMITIVE pipeline seems limited in contribution, as it mainly integrates existing LLMs with minor modifications.

We respectfully disagree that our contributions are merely incremental. Our core contribution is the design of a geometrically sensitive visual encoder that enhances MLLMs' fine-grained primitive perception—addressing a key bottleneck in reasoning, as shown by the analysis in Figs. 1 and 5a.

Unlike single-modality LLMs, MLLMs rely on accurate visual cue interpretation to support abstract reasoning. Misinterpretations at this stage can misguide reasoning processes. To address this, we designed a geometrically sensitive visual encoder with box- and pixel-level supervision, going beyond the image-level supervision used in CLIP (see Q3 for AR PhN5). This was enabled by a custom data engine generating diagrams with shape, junction, and boundary annotations.

Furthermore, we integrated the trained visual encoder into LLMs using two methods: hard coordinates (Sec. A.4) and soft visual prompts. For the latter, we introduced a feature router that dynamically selects features from semantic- to geometric-rich levels. Our designs were implemented in LLaMA2-7B, Deepseek-math-7B, and Qwen-math-7B, with ablations showing notable gains over baselines across three math reasoning benchmarks.

3. The paper lacks the parameter size and citation for InternVL2, limiting clarity and reproducibility; adding comparisons with recent state-of-the-art visual models would strengthen the experimental validity.

We apologize for the oversight—InternVL2 has 8B parameters, which we will clarify as InternVL2-8B. For reproducibility, we will release model weights for PRIMITIVE-7B, PRIMITIVE-Deepseek-7B, and PRIMITIVE-Qwen2.5-7B, along with training and inference code. To address comparisons with state-of-the-art models, we have integrated our design into Qwen2.5-VL (see Q2 for ARAy8j).

4. It lacks analysis of remaining reasoning errors, which may stem from LLM hallucinations rather than perception.

Our primary goal is to enhance the visual grounding of MLLMs to complement their reasoning, not to solve the full spectrum of mathematical reasoning tasks. We acknowledge that hallucinations in reasoning remain an important challenge, influenced by many factors such as data distribution, vision encoders, modality alignment, and the LLM's inherent knowledge [1–5].

Perceptual enhancement is one feasible path to reducing such hallucinations [4,5], and our work focuses on improving object- and pixel-level perception in mathematical contexts. We agree that other factors also impact reasoning accuracy and plan to explore them in future work. As suggested, analyzing caption-based descriptions could provide intuitive insights into hallucinations tied to diagram elements. However, building such datasets and benchmarks is non-trivial and beyond the scope of this rebuttal, but it remains a key direction for future research.

[1] Jaemin Cho et al., Fine-grained image captioning with clip reward.

[2] Ziwei Ji et al., Survey of hallucination in natural language generation.

[3] Lei Huang et al., A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.

[4] Shunqi Mao et al., Through the Magnifying Glass: Adaptive Perception Magnification for Hallucination-Free VLM Decoding.

[5] Hanchao Liu et al., A Survey on Hallucination in Large Vision-Language Models.

5. Additional visualization examples are needed, and Fig. 3 and Tab. 5 exceed page margins.

Thank you. We will add more qualitative comparisons between PRIMITIVE and baselines for clarity and fix the formatting issues in Fig. 3 and Tab. 5.

We thank the reviewer for insightful questions that help refine our work further.

1. The authors use existing models to extract junctions and boundaries as ground truth—could this introduce noise given the variation in geometric shapes and domains?

For junction detection, we discuss noisy cases in lines 802–807 of Appendix A.4 and show failure cases in Fig. 11. We use a CNN model (Huang et al., 2018) trained on Man-Made Environments to generate ground truth. Some label noise arises from out-of-domain (OOD) diagrams, which we mitigated using a 0.85 confidence threshold and manual correction. While 100% accuracy isn’t guaranteed across 20K+ samples, the noise level is acceptable. The test set recall is 85.6%, and GeoGLIP remains robust to distortions (see Q2 for AR sjVj). Improving junction labeling and data synthesis, as discussed in Sec. A.7, could further boost performance.

For boundary detection, we use the FoJ model (Verbin & Zickler, 2021), which applies a machine learning algorithm for generalized M-junctions. Unlike CNNs, FoJ generalizes better to OOD domains and is less dataset-biased. Its noise resilience is supported by prior work. GeoGLIP achieves an IoU of 92.3% and remains robust under various distortions (see Q2 for AR sjVj).

2. PRIMITIVE-Qwen2.5-7B shows lower MathVerse/MathVista performance compared to Qwen2.5-VL-7B (68.2%), and even earlier MLLMs.

MLLMs fall into two categories: generic (e.g., InternVL2.5, Qwen2.5-VL, GPT4o) and math-specific (e.g., G-LLaVA, Math-LLaVA, MAVIS, ours), differing significantly in training recipes and datasets when integrated with LLMs and visual encoders.

Generic models are trained on large-scale visual instruction datasets covering a wide variety of multimodal data, including OCR, academic questions, localization data, documents, and video descriptions ( e.g., Qwen2.5-VL uses 2TB+ data), while math-specific models utilize only mathematical text-diagram pairs, which require fewer training resources and provide an efficient test base for proposed math module designs. Our model is fairly compared with math-specific MLLMs, as it is trained solely on MathV360K+Geo170K. In controlled experiments, adding GeoGLIP and the soft router boosts baseline G-LLaVA by 4.6% on MathVerse and 12.3% on MathVista. We've also extended our approach to DeepSeek-Math-7B and Qwen2.5-Math-7B, achieving consistent +6% gains on MathVista (Tab. 7). Note: baselines use only CLIP visual encoders and the same training setup—not generic DeepSeek-VL or Qwen2.5-VL.

To compare with state-of-the-art 7B models, we implemented our designs on Qwen2.5-VL-7B and fine-tuned it using its official checkpoint. As the training code is not publicly available, significant effort was required to rewrite the functions, data loader, and training processes within the transformers and Hugging Face frameworks. For alignment, we trained only the projectors; Due to time limits, we trained both projectors and LLMs using LoRA to accelerate SFT stage on the MathV360K+Geo170K dataset.

Under the same training setup, we developed two versions: (1) Qwen2.5-VL-7B+, fine-tuned with math-specific visual data; and (2) PRIMITIVE-Qwen2.5-VL-7B, integrated with our GeoGLIP and soft router. Our integrated models are lightweight (<50MB, see Sec. A.6.2) and already show performance gains. We plan to further improve results by exploring full fine-tuning (beyond LoRA) and partial unfreezing of the visual encoder with lower learning rates.

3. PRIMITIVE uses a feature pyramid to extract fine-grained features—a technique common in natural images but not specifically tailored for geometric tasks.

Thank you for your observations regarding the PRIMITIVE architecture. Although the feature pyramid structure effectively extracts fine-grained visual cues in natural images, it is not central to enhancing mathematical perception in our MLLM. Instead, the key factor lies in the mathematically sensitive GeoGLIP, which incorporates visual-centric mathematical training datasets and fine-grained box- and pixel-level supervision (refer to Q3 for AR PhN5).

Our empirical evidence, detailed in lines 840-848 and Tab. 6 of the appendix, supports this. Specifically, coupling the original GLIP (trained on natural images) with the feature pyramid and soft router techniques decrease performance from 67.0 to 65.3 on GeoQA. Additionally, Fig. 10 (right panel) demonstrates GLIP's inability to accurately perceive basic geometric primitives.

These results confirm that fine-grained primitive perception ability depends on a geometrically sensitive visual encoder, and then this ability could be further enhanced by employing pyramid techniques. Our specific adaptations and innovations for GeoGLIP are deliberately designed to tackle the unique challenges of mathematical image tasks.

We thank the reviewer for insightful questions that help refine our work further.

1. The formula (1) in Section 3.3 should contain more comprehensive illustration of newly defined labels.

Thank you. Duty noted. Eq. (1) describes the soft router process. The scalar routing weight for each level is denoted as $w^i$, where $i \in \{1, 2, 3, 4\}$; the symbol $\circledcirc$ denotes the operation flow from right to left. For instance, $\mathcal{G} \circledcirc F_{\text{geo}}^i$ indicates that the different levels of GeoGLIP features $F_{\text{geo}}^i$ are resized $(\mathcal{G})$, which is essential for aligning spatial size of CLIP features. $\sigma$ denotes the normalization function, which could be either a SoftMax or a Sigmoid function, depending on the feature integration process (channel-wise or sequence-wise concatenation).

We have added a more detailed description of each component and its role in the revised manuscript to ensure that the formulation and its application are transparent to the readers.

2. In Section 4.1, authors use two checkpoints for evaluation on GeoQA and MathVerse/MathVista, which is not considered fair for performance comparison since the standard practice in this field is to test all benchmarks using a single trained model.

Thank you for highlighting this important point. GeoQA primarily evaluates geometric problem reasoning, and our baseline G-LLaVA-7B is trained exclusively on the Geo170K dataset for this benchmark. To ensure a fair comparison, we maintain consistency in the training dataset and test on GeoQA.

Regarding the MathVista and MathVerse benchmarks, which cover a diversity of subjects including scientific topics, PaperQA, and IconQA, we applied the additional MathV360K dataset for fine-tuning our model (Geo170K+MathV360K), a combination commonly used by other mathematical MLLMs like Math-LLaVA and MAVIS. In response to this concern, we test our model trained on Geo170K+MathV360K on the GeoQA benchmark. The performance improved, with PRIMITIVE-7B's top-1 accuracy increasing to 71.3 (67.0 in Tab. 3). We adopt this version as the default choice and will release the checkpoint to the public.

3. The main motivation and training approaches of this paper are similar to MAVIS, e.g., improving vision encoder and three-stage training, although with different techniques. I suggest discuss more about the relation and difference of this paper and MAVIS in the Intro and Related Work parts.

Thank you for your suggestion. We have expanded the discussion regarding our model in comparison to MAVIS in the Related Works section.

Although our high-level motivation and MLLM training strategies share similarities with MAVIS, our different training techniques for the visual encoder and MLLM design enable our model to achieve comparable reasoning performance using an 8$\times$ smaller visual instruction training dataset. For visual encoder training, MAVIS utilizes 588K caption-diagram pairs with alignment loss similar to CLIP, which provides image-level supervision. In contrast, our model employs only 40K samples with box-level and pixel-level supervision, offering more detailed local feature perception. Researchers have shown that CLIP-style encoders fail to capture fine details (e.g., Yiwu Zhong et al., "RegionCLIP: Region-based Language-Image Pretraining"; Zhe Gan et al., "Vision-language pre-training: Basics, recent advances, and future trends").

A more intuitive demonstration is shown through the visualization of attention maps. MAVIS's visual encoder, depicted in Fig. 1(a) of MAVIS's paper, primarily highlights coarse-level boundary information and often overlooks detailed features. In contrast, our model, as shown in Fig. 2, produces attention maps that are more fine-grained, clearly highlighting not only the boundaries but also detailed elements such as dotted lines and right-angle symbols. This stark contrast underscores the superior local feature detection capability of our model. Furthermore, our model design incorporates a soft router that adaptively selects visual cues ranging from semantic-rich to geometric-rich. This adaptability enhances the model's ability to accurately solve problems by leveraging relevant visual information effectively.

4. The title of the paper is a bit overclaimed. PRIMITIVE utilizes localization information to enhance the vision perception and performance, but does not enable LLMs to know where to focus within math figures.

Thank you. Duty noted. We will revise the title to more accurately reflect the model's enhanced visual perception and avoid overclaiming.