MINT-CoT: Enabling Interleaved Visual Tokens in Mathematical Chain-of-Thought Reasoning

Thank you very much for your detailed feedback and constructive suggestions. We greatly appreciate your recognition of the Interleave Token mechanism and the strong performance gains achieved by MINT-CoT. Below, we address your concerns point by point.

1. Interleave Token Structure:

the Interleave Token is effectively a formalized expression of a query-like vector in an attention mechanism, lacking structural novelty.

We would like to reclaim the novelty of our Interleave Token. While the Interleave Token superficially resembles a query vector, its design and function are distinct in several important aspects:

• The Interleave token is a special token that is specifically designed for selecting visual tokens. It will be appended to the input sequences, which are passed forward through the entire transformer. And then it will compute the similarity between text content and visual tokens.
• In a highly integrated MLLM, the Interleave Token serves as a bridge that connects high-level CoT reasoning with low-level visual structure.
• Moreover, the Interleaved token is supervised during training, which is different from the query-like vector in the attention mechanism.

2. Mapping accuracy of visual regions in the dataset

We agree that accurate alignment is important. To ensure annotation quality, we have designed our dataset curation pipeline with careful consideration and multiple safeguards:

• We used state-of-the-art OCR tools (e.g., PaddleOCR) to extract fine-grained elements such as symbols and labels.
• We employed GPT-4o, one of the strongest available MLLMs, to generate alignment annotations using specially designed prompts aimed at minimizing noise and ambiguity.

Moreover, to validate the reliability of the automatic annotations, we conducted a human evaluation: 10 annotators independently reviewed 100 randomly sampled MINT-CoT examples. They assessed whether the selected visual tokens at each reasoning step were relevant and accurate. The results showed an average step-wise accuracy of 89.49%, which demonstrates that our pipeline produces high-quality alignments suitable for training multimodal reasoning models.

3. Problem of Figure 3

with small fonts and repetitive data processing descriptions, making the methodological pipeline appear redundant and less intuitive. The coordination between textual explanation and illustration is suboptimal.

Thanks for your suggestions. Given the complexity of the pipeline, we opted for a compact layout to keep it within a single figure. We encourage readers to zoom in for better readability. And we will refine it in the final version.

4. Inplemental details of GRPO strategy

Due to space limitations in the main paper, we have provided the detailed implementation of the GRPO strategy in Sections A.4 and A.5 of the supplementary material. We will integrate these descriptions into the main text in the final version to improve clarity.

5. Limitation or sparsity constraint on the number of visual tokens inserted?

After training, the number of interleaved visual tokens does not pose a risk of inference failure, as the selected tokens generally contain minimal irrelevant information. Furthermore, as presented in Table 1 in the supplementary material, the average number of Interleave Tokens per interleaved data point is 2.80, and the average number of selected indices per interleaved data point is 19.91, which reflects a sparse and controlled selection process, rather than excessive inclusion.

6. Behavior difference between algebraic and geometric problems

Does token selection behavior differ statistically between algebraic and geometric problems? Are there task-specific biases in token filtering?

We analyzed whether token selection behavior differs between algebraic and geometric problems, using evaluation results of MINT-CoT-7B on MathVista-Math. Specifically, we compared samples from two distinct sources:

• UniGeo (geometry questions)
• FunctionQA (algebra/function questions)

We found that, on average, the proportion of visual tokens selected during inference is 3.60% for geometry questions, while for algebra/function questions, the rate is 2.48%. This suggests that the model tends to select broader visual regions for geometry problems, reflecting a task-specific preference in token filtering.

7. GPT-4o annotation accuracy

As discussed in ”2. Mapping accuracy of visual regions in the dataset”, we conduct a human evaluation on the accuracy of our annotated data generated by GPT-4o. The results show an average per-step accuracy of 89.49%, indicating that the annotation pipeline is reliable and sufficiently precise for the task.

8. If OCR or keyword extraction fails, would visual token selection break down entirely?

Overall, the annotation process handles OCR and keyword extraction failures well, and rarely leads to incorrect visual-token selection.

• Modern OCR tools (e.g., PaddleOCR) are highly accurate, and in our pipeline, OCR serves only as auxiliary input. If there are no OCR results in a image, the GPT-4o-based annotation process remains unaffected, as it does not depend on OCR outputs exclusively.
• During keyword extraction, prompts are carefully crafted to ensure that extracted keywords are both (1) highly related to the visual content, and (2) critical for reasoning. If no keyword satisfies these two conditions, no visual tokens will be interleaved for that reasoning step. As shown in Table 1 (supplementary), approximately 2,000 samples in the dataset contain no Interleave Tokens, which reflects this conservative design.

9. Limitations

The checklist claims that the limitations of the work are included in the appendix, but in fact, no such discussion could be found.

Thank you for pointing this out — we apologize for the omission. The limitations of our current work are as follows: In the dataset curation pipeline, the use of GPT-4o for dataset annotation still has some computational cost. Moreover, alternative reinforcement learning strategies beyond our method now remain underexplored. We will include a discussion of these limitations in the final version of the paper.

Thank you very much for your feedback and positive comments. You highly praised the design of our Interleave Token, our large-scale dataset, and the performance of MINT-CoT-7B. Your concerns are addressed below.

1. Visual Perception Capability

it relies on a math-domained variant of CLIP (MAVIS) for visual grounding …

We would like to clarify that our MINT-CoT framework does not rely on MAVIS or any math-domain variant of CLIP. We apply our MINT-CoT framework on Qwen2-VL-7B-Instruct model. As stated in lines 46-53, models like CLIP and SigLIP are pre-trained on natural images with general scenes, making mathematical images out-of-distribution. So attention-based token selection method to identify relevant visual tokens during mathematical reasoning is difficult.

The paper does not provide direct evidence that MINT-CoT meaningfully overcomes these perceptual limitations …

We do not aim to improve the raw perception ability of the encoder, but rather provide an alternative mechanism through the interleaving of selected visual tokens at each reasoning step, which helps the model to “watch” these important visual tokens again.

2. Necessity of Multimodal CoT for Geometric Figures

2.1. Disadvantages of text-only reasoning

Given that LLMs are already capable of understanding these concepts from large-scale text-only training …

In practice, text-only captions of mathematical figures are frequently inaccurate. As shown in MAVIS, the visual perception of models like CLIP is inaccurate on mathematical reasoning tasks, so the natural language captions generated from MLLMs are often inaccurate. As a result, we need to interleave the related visual regions during reasoning, which can significantly reduce hallucination and improve the accuracy of mathematical reasoning.

2.2. A training-free experiment

In fact, there is no direct evidence provided to show that the model understands these visual tokens better than their natural language descriptions … We have conducted a training-free multimodal CoT experiment on our base model:

• Input question: As shown in the figure, AB is tangent to circle O at point B, if angle A = 36.0, then angle C is equal to ()\nChoices:\nA:36°\nB:54°\nC:60°\nD:27°
• Output solution: Since AB is tangent to circle O at point B, we have angle ABO = 90°. Therefore, angle AOB = 90° - angle A = 54°. Since OA = OB, we have angle OAB = angle OBA = 36°. Therefore, angle C = 180° - angle AOB - angle OAB = 180° - 54° - 36° = 90°. Since angle C is an inscribed angle, we have angle C = 1/2 angle AOB = 1/2 * 54° = 27°. Therefore, the answer is D.

The output solution of the base model is wrong when it comes to “Since OA = OB, we have angle OAB = angle OBA = 36°”. So we manually concat the related visual tokens before this reasoning step, and find that the whole output solution will be:

• Since AB is tangent to circle O at point B, we have angle ABO = 90°. Therefore, angle AOB = 90° - angle A = 54°.<selected_visual_tokens> Therefore, angle C = 1/2 angle AOB = 27°. Therefore, the answer is D. So, the model outputs with correct answer when interleaving correct visual tokens during reasoning, which can proves that MLLMs benefits from Multimodal CoT when solving methematical problems.

2.3. Randomly select visual tokens

what happens if a portion of these selected tokens is randomly dropped or replaced with irrelevant ones? We conduct an experiment that randomly selects visual tokens during inference of the Interleave CoT SFT model. The results show that inference with random selections introduced irrelevant visual context in each reasoning step, which deeply hurts the original format and logic. | Qwen2-VL-7B-Instruct | Interleave CoT SFT | Interleave CoT SFT (random select) | | - | - | - | | 41.11 | 67.78 | 31.11 |

3. Precision of token selection

visual tokens are also subject to noise and overlap…

Due to the limitation of the patching method of ViT, which is not fine-grained enough, it’s hard to avoid some noise when selecting visual tokens. However, token-level grounding remains significantly more precise than box-level selection.

Furthermore, the empirical improvement in Table 4 over the box-based OCT baseline is minimal.

As shown in Table 4 in our paper, the Bounding Box CoT SFT is 2.22% lower than Interleaved CoT SFT on MathVista-Math, and even performs worse than Text-only CoT SFT on GEO and GPS domains, which demonstrates that the Bounding Box CoT SFT is worse than our Interleaved CoT SFT method.

4. Text-only CoT Baseline

Thanks for your suggestion. We conduct experiments on the text-only SFT followed by text-only RL. As shown below, MINT-CoT still outperforms this stronger baseline:

This confirms that our improvements specifically arise from the interleaved visual reasoning enabled by MINT-CoT.

5. Resolution Dependency of Token Selection

When gridding the image, we align our gridding method and image resolution with the Qwen image processing method, so the selected token indices are correct during training. For models with different patching strategies, we support token index remapping based on image resize ratios, so the selected visual tokens remain aligned.

6. Efficiency Concerns

How many visual tokens are input during reasoning on average?

• As presented in Table 1 in the supplementary material, the Average number of Interleave Tokens per interleaved data point is 2.80, and the Average number of selected indices per interleaved data point is 19.91.
• Importantly, when interleaving the selected visual tokens, we just concat the visual embeddings to the text embeddings, rather than re-encoding the image through the vision encoder (ViT). This design reduces a lot of time.
• Training time and Inference time: We provide the training time of three training stages on 8*A100 GPUs, and the average inference time per question on GeoQA: | | Text-only CoT SFT | Interleave CoT SFT | Interleave CoT RL | | - | - | - | - | | Training | 4h | 9h | 25h | | Inference | 8.15s | 8.03s | 8.24s |

7. Number of Interleave Tokens

How many Interleave Tokens are leant during training?

As presented in Table 1 in the supplementary material, the average number of Interleave Tokens per interleaved data point is 2.80.

8. Design Choice in Token Selection Loss

Thank you for the insightful suggestion. But we concern that it will not be efficient enough to select visual tokens through next-token prediction. Moreover, the order of selected visual tokens is not semantically meaningful, and attempting to predict them autoregressively may introduce unnecessary complexity and training instability. We wil explore this formulation in the future work.

9. Application on other visual reasoning tasks:

What justifies the use of selected visual tokens specifically for geometric math reasoning? I think this token election can be generalize to natural images or more general domains.

Thanks for your insight. While the current focus of our research is specifically on multi-modal math problems, our method can also be applied to other visual reasoning tasks.

• Our dataset curation pipeline can be adapted to other tasks. The image gridding strategy, auxiliary OCR tools, keyword extraction, and final visual index annotation can also be applied to general visual reasoning tasks. For math reasoning tasks, we need to align fine-grained visual indices with the reasoning steps. In contrast, general visual reasoning tasks do not require such a high level of granularity, so our pipeline remains suitable for general tasks as well.
• The introduction of an Interleave Token for grounding reasoning steps in corresponding visual content is a general mechanism that enhances multimodal alignment and interpretability across various tasks.
• Our three-stage training pipeline can be easily adapted to many other general tasks.

Thank you for your follow-up questions and for engaging deeply with our work. We appreciate the opportunity to clarify these important points. Your feedback is invaluable, and we hope our detailed responses below will fully address your remaining concerns.

Question 1:

If the visual encoder itself lacks strong perception ability, how can it effectively help the model “watch” important visual tokens again?

Sorry for making you confused. We would like to clarify that the lack of perception ability of visual encoders refers to that, the LLM backbone cannot well understand the visual embedding of the vision encoder on mathematical images. Since these images are out-of-distribution for the vision encoder, the features of significant mathematical regions, e.g., geometric characteristics, are not highlighted enough within their embeddings, which causes the LLM to fail to retrieve or associate the relevant visual tokens correctly while solving the problem. For example, when the reasoning step comes to “triangle ABC”, sometimes the LLM doesn’t know where “triangle ABC” is, which is “triangle ABC” in the image, or misreads it as “triangle BCD”.

Instead, our design using the interleave token specifically targets retrieving these important visual tokens and interleaving them in CoT, which explicitly guides the LLM to attend to the correct region and mitigates hallucinations. Such a token selection process is not dependent on the unsatisfactory capabilities of the visual encoder, but rather on our supervised fine-tuning with MINT-CoT datasets.

So we do not try to improve the perception ability, but to design a method that helps to provide the LLM with the related visual embeddings in a specific reasoning step.

Additionally, how can you ensure that the interleaved visual cues are faithful and accurately reflect the relevant content?

We ensure the accuracy both through dataset curation and the training process.

1. To ensure annotation quality, we have designed our dataset curation pipeline with careful consideration and multiple safeguards:

• We used state-of-the-art OCR tools (e.g., PaddleOCR) to extract fine-grained elements such as symbols and labels.
• We employed GPT-4o, one of the strongest available MLLMs, to generate alignment annotations using specially designed prompts aimed at minimizing noise and ambiguity.

Moreover, to validate the reliability of the automatic annotations, we conducted a human evaluation: 10 annotators independently reviewed 100 randomly sampled MINT-CoT examples. They assessed whether the selected visual tokens at each reasoning step were relevant and accurate. The results showed an average step-wise accuracy of 89.49%, which demonstrates that our pipeline produces high-quality alignments suitable for training multimodal reasoning models.

2. We conduct Interleaved CoT SFT to help the model learn how to select correct visual tokens. As shown in Figure 4 in the paper, the F1 score exhibits an upward trend during training, which makes the interleaved tokens faithful and accurate as much as possible

Question 2:

how do you ensure that the model understands these abstracted visual tokens better than their natural language descriptions? For example, how do visual selection tokens for triangle ABC outperform textual tokens like “triangle ABC”?

Sorry for making you confused. We think the natural language descriptions and our interleaved visual tokens are not supposed to compare in this task and setting, for the following two reasons:

1. MLLM cannot well understand the original visual embeddings, so it's hard for it to generate accurate natural language descriptions during reasoning. The inaccurate math descriptions would harm the reasoning quality. Instead, our interleave token selection method overcomes it by selecting visual tokens using the similarity between the hidden states of the interleave token and visual tokens. This approach helps the model to locate important regions, but does not enforce it to describe them using language.

2. Further, these two settings (the natural language descriptions and our interleaved visual tokens) are not comparable alternatives, but rather a sequential relationship. Only when the model accurately locates the visual tokens (like what we do), can the model generate accurate descriptions. Therefore, our method is more like a foundation for accurate natural language descriptions because reasoning only with textual description is hard. This will be proved in our ablation study below.

You also mentioned in your answer that the visual perception of models is inaccurate on mathematical reasoning tasks—so why would the selected visual tokens be better understood than textual tokens?

As we clarified in Question 1, the visual tokens can help the model to locate the visual regions where text-only inference mode has hallucinations. It is the structured interleaving that helps the model associate visual and textual modalities more reliably, which improves reasoning accuracy.

A more controlled comparison would be to train models on the same scale of data, but replace the interleaved visual tokens with equivalent textual descriptions.

Thanks for your suggestion. Following your suggestion, we conducted an ablation study where we replaced the interleaved visual tokens with their corresponding textual keywords (e.g., "angle AOB", "triangle ABC") and add a prefix like: "This step will reason with …". In this modified setting, the model was trained using only textual descriptions of key visual regions rather than interleaved visual tokens.

When trained with identical parameters to our Interleaved CoT SFT stage, the model demonstrated inferior performance to Text-only CoT SFT, which may be due to the redundancy of the keywords. It is also well below our Interleaved CoT SFT. This result confirms that accurate visual grounding through our token selection method is helpful for generating correct textual descriptions during reasoning.

A concrete failure case illustrates this point clearly: “... Since E is the midpoint of angle FEB, angle FEB = 2 times angle AEF = 40 circ …” In fact, E is the midpoint of arc FEB, rather than angle FEB, so angle FEB doesn’t eual to 2 times angle AEF. This wrong textual description leads the model to reason in the wrong direction. In contrast, our MINT-CoT method avoided this error, which shows that our method is more like a foundation for accurate natural language descriptions, helping the MLLM for better reasoning.

Moreover, authors could train the model using the token selection loss, but without interleaving visual tokens during reasoning.

As you suggested, we conduct an experiment that trains with the interleave tokens, while without visual tokens inserted during reasoning.

As shown in the table, the performance drops when the interleaved visual tokens are removed during inference. This experiment demonstrates that the performance gain comes from the model truly understanding and utilizing the interleaved visual tokens within the CoT reasoning process, not just from the additional visual supervision during training.

Thank you for your review and feedback. We sincerely appreciate your recognition of the motivation and simplicity of our approach. We address your concerns in detail below.

1. Application on other visual reasoning tasks:

The method is task or dataset specific. I am not convinced that the method can be easily applied to other visual reasoning tasks.

While the current focus of our research is specifically on multi-modal math problems, our method can also be applied to other visual reasoning tasks.

• Our dataset curation pipeline can be adapted to other tasks. The image gridding strategy, auxiliary OCR tools, keyword extraction, and final visual index annotation can also be applied to general visual reasoning tasks. For math reasoning tasks, we need to align fine-grained visual indices with the reasoning steps. In contrast, general visual reasoning tasks do not require such a high level of granularity, so our pipeline remains suitable for general tasks as well.
• The introduction of an Interleave Token for grounding reasoning steps in corresponding visual content is a general mechanism that enhances multimodal alignment and interpretability across various tasks.
• Our three-stage training pipeline can be easily adapted to many other general tasks.

2. Evaluation on other benchmarks

More different benchmarks need to be used to validate how general the method can be applied.

We appreciate this suggestion and have extended our evaluation to cover additional datasets. Specifically:

• In our paper, the results of GeoQA have been presented.
• In our supplementary materials, we have added the evaluation results on the Mathematics section of MMStar benchmark (Table 2 in Suppl.)
• Due to the time limit, we add evaluation experiments on the Vision Only domain of Mathverse, which can well represent the improvement of our method.
• Moreover, we add experiments on the Mathematics section of the MMMU-Pro-V. Although this subset contains only 60 questions, our method still achieved the best performance.

The combined results of those benchmarks are shown below:

3. Clarification of the use of "interleave tokens"

Note that an interleave token is the output of a reasoning step. (without visual tokens) …

Sorry for the confusion caused. We need to clarify that Interleave Token is not ”the output of a reasoning step”, but a special token concated prior to each reasoning step. It is in the token sequences, so it is not “without visual tokens”. Because transformers allow all tokens to attend to prior context, the Interleave Token can access:

• Previously selected visual tokens;
• The input image tokens;
• All prior reasoning steps.

Therefore, the Interleave Token is not blind to visual input, but can “see” both the text reasoning steps and visual tokens before that step, to choose the related visual tokens. This process is visualized in Figure 2, where the red token marks the Interleave Token.

Thank you for your thoughtful review and constructive feedback. We greatly appreciate your recognition of our dataset design, the novelty of the Interleave Token, and the effectiveness of our three-stage training pipeline. We address your concerns below.

1. Choice of Qwen-2-VL-7B as Base Model

We choose Qwen-2-VL-7B as the base model to more clearly demonstrate the effectiveness of our method, as it leaves more room for improvement and highlights relative gains. To address your concern, we also conduct additional experiments using the stronger Qwen-2.5-VL-7B model. As shown in the table below, our method still brings consistent improvements on MathVista-Math, confirming its generality across different base models.

Due to the limited time available during the rebuttal period, we did not perform any hyperparameter tuning and directly adopted those from Qwen-2-VL-7B. Despite this, the MINT-CoT framework on Qwen-2.5-VL-7B still shows significant potential for improvement and demonstrates the robustness of our approach.

2. Evaluation on other benchmarks

The evaluation is quite narrow – just MathVista and GeoQA.

We appreciate this suggestion and have extended our evaluation to cover additional datasets. Specifically:

• In our supplementary materials, we have added the evaluation results on the Mathematics section of MMStar benchmark (Table 2 in Suppl.)
• We did not evaluate on MathVision, since it was used in constructing our training data (aligned with the training data of Mulberry), which would make evaluation results biased.
• Due to the time limit, we add evaluation experiments on the Vision Only domain of Mathverse, which can well represent the improvement of our method.
• Moreover, we add experiments on the Mathematics section of the MMMU-Pro-V. Although this subset contains only 60 questions, our method still achieved the best performance.

The combined results of those benchmarks are shown below:

There should be some general-purpose evals too like VQAv2, GQA, Blink, ChartQA …

The focus of our research is specifically for multi-modal math problems, including the interleave token for finer-grained selection and a dataset construction pipeline. These designs target math-specific challenges, such as geometric structures and compositional diagrams. Whereas our methodology can also generalize to general-purpose scenarios:

• Our dataset curation pipeline can be adapted to other tasks. The image gridding strategy, auxiliary OCR tools, keyword extraction, and final visual index annotation can also be applied to general visual reasoning tasks. For math reasoning tasks, we need to align fine-grained visual indices with the reasoning steps. In contrast, general visual reasoning tasks do not require such a high level of granularity, so our pipeline remains suitable for general tasks as well.
• The introduction of an Interleave Token for grounding reasoning steps in corresponding visual content is a general mechanism that enhances multimodal alignment and interpretability across various tasks.
• Our three-stage training pipeline can be easily adapted to many other general tasks.

While our current work focuses on math-specific challenges, we acknowledge the importance of evaluating our method on broader vision-language tasks. Due to the domain gap between our mathematical reasoning dataset and benchmarks like VQAv2, GQA, BLINK, and ChartQA, we did not include them in our current evaluation. We plan to explore this in future work.

In addition, it would be better to include the numbers for the entire mathvista data …

We chose to evaluate only the MathVista-Math subset because our method specifically targets mathematical reasoning tasks. The full MathVista dataset includes general visual question answering tasks (e.g., VQA, TQA), which fall outside the scope of our training data and method design.

3. Text-only CoT Baseline

Thanks for your suggestion. We conduct experiments on the text-only SFT followed by text-only RL. As shown below, MINT-CoT still outperforms this stronger baseline:

This confirms that our improvements specifically arise from the interleaved visual reasoning enabled by MINT-CoT.