GRIT: Teaching MLLMs to Think with Images

Response to scalability to larger models

We appreciate the concern. This work aims to show that GRIT efficiently teaches pretrained MLLMs a unified grounding‑and‑reasoning ability with minimal training data. We therefore used smaller backbones to validate the method—especially its training‑data efficiency—and to isolate the effect of the grounded‑reasoning paradigm. Because smaller models are generally less capable than larger ones, we expect the observed gains to transfer on larger backbones. GRIT is model‑agnostic, specifying an interleaved text+box format and an RL objective without changing network internals, so it can be applied directly to larger models. A broader large‑model evaluation is planned for follow‑up work when additional computing resources are available, and we appreciate the reviewer’s understanding regarding typical academic compute budgets. We believe a proof‑of‑concept on compact models is crucial evidence that the paradigm itself drives the grounded‑reasoning gains.

Response to adequacy of accuracy for practical deployment

We thank the reviewer for raising this concern. Fig. 6 shows a sub‑linear performance improvements of the model with more data—an expected pattern in post‑training, however, the 7k‑sample result only reflects results under our intentionally constrained setting (eg. 3B backbone model, no domain‑specific warm‑start). It mainly underscores that besides the training data’s quantity, the in‑domain relevance plays an important role in model performance improvement. Real‑world deployment entails greater robustness requirements and design choices such as model size and cold‑start initialization; we leave a full deployment study to future work.

Response to question 1 on compute scaling with data size

We thank the reviewer for the detailed question. No—the cost does not scale linearly with data. Our 200 training steps are fixed across data sizes (RL update steps, not epochs). This is because we observe that with either 20 samples or 7k samples, performance saturates by ~150–200 steps. Although the 20‑sample training cycles through each data many more times than the 7k‑sample case (each example only ~3 effective epochs, ≈ 200×128/7,000), the GPU‑hours stay the same. We will further clarify this in our next paper revision.

Response to question 2 on reducing batch size or training steps

We thank the reviewer for the thoughtful question. Fewer training steps are harmful: with fixed batch size, performance vs. steps is non‑linear and exhibits an “aha” jump around ~50–150 steps (akin to the transitions reported in DeepSeekMath [1]). Cutting steps before this window often misses the jump and yields markedly worse unified grounding‑plus‑reasoning performance. For smaller batch sizes, effects are intertwined with other factors such as the learning rate: in our experience, training with smaller batch sizes can reach similar end performance but requires more steps to saturate.

Reference:

[1] Shao, Zhihong, et al. "Deepseekmath: Pushing the limits of mathematical reasoning in open language models." arXiv preprint arXiv:2402.03300 (2024).

Response to question 3 on a typo

We thank the reviewer for catching this. It is a typo: “Qwen2.5‑VL 2B” should be “Qwen2.5‑VL‑3B,” the same base model reported in Section 4.1 Setup. We will correct this in the next revision and ensure consistency throughout the paper and appendix.

Response to contribution of GRPO‑GR

We thank the reviewer for this perspective. In this work, we mainly propose the GRIT method and demonstrate that MLLMs trained with GRIT successfully unify grounding and reasoning abilities. Within GRIT, GRPO‑GR is one component of the contribution alongside the other one, grounded‑reasoning paradigm, leading to model outputs with interleave natural language with explicit bounding‑box coordinates. GRPO-GR is not a brand‑new optimization method, but rather an algorithm with new optimization target that is tailored to this grounded‑reasoning paradigm, i.e., rewards that enforce valid interleaved text+box structure and tie rationales to evidence, thereby enabling small MLLMs to “think with images” from answer‑level supervision alone.

Response to adequacy of accuracy for practical deployment

We thank the reviewer for raising this concern. Our paper’s focus is on unifying grounding and reasoning with training‑data efficiency for MLLMs. The in‑domain accuracy in Fig. 6 only reflects results under our intentionally constrained setting (3B backbone model, no domain‑specific warm‑start, 7k data), where the experiment aims at showing the importance of in-domain data and data quantity. Real‑world deployment entails greater robustness requirements and design choices such as model size and cold‑start initialization; we leave a full deployment study to future work.

Response to question 1 on undefined acronyms and terminology

We thank the reviewer for the clarity suggestion. We will define MLLM (Multimodal Large Language Model), GRPO‑GR (Group Relative Policy Optimization-Grounded Reasoning), and core terms such as grounding (the task of localizing specific objects or regions within an image based on a given textual description), at first mention and ensure consistent use throughout the paper and figures for accessibility.

Response to question 2&3 on experiment details and appendix

We appreciate the request for specifics and apologize for the missing information. For our evaluation, we curated testing data derived from six public open‑source datasets covering a range of visual reasoning and grounding tasks. The statistic for the testing data is shown in the following table. The appendix with these information was inadvertently omitted in the initial PDF export and we will further update our paper with these information.

• VSR tests spatial relation verification. For our VSR evaluation set, we source question‑image‑answer triplets from the VSR subset of the Visual CoT benchmark [1] and manually filter out those with ambiguous answers, reducing the size from 404 to 288.

• TallyQA focuses on counting; we uniformly sample evaluation questions where the target object counts range from 0 to 9 to create our TallyQA evaluation set.

• GQA offers scene‑graph‑grounded, compositional object spatial questions. We first take the GQA subset from the Visual CoT benchmark [1] and then manually filter these to retain high‑quality instances for our GQA evaluation set.

• From MME, we use only the counting, position, and existence subsets to broaden our evaluation scope.

• MathVista evaluates mathematical reasoning in visual contexts. Following prior works, we adopt its TestMini split.

• Finally, OVDEval is an open‑vocabulary detection (OVD) testing set that requires the model to ground fine‑grained semantics from the language query to the coordinates of visual features. We use its position subset and simplify it to object detection tasks with a single target existed in the target image.

Reference: [1] Shao, Hao, et al. "Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning." Advances in Neural Information Processing Systems 37 (2024): 8612-8642.

Response to the limitation suggestions

We appreciate the reviewer’s insightful suggestions. We would like to point out that while our approach uses a GPT-based model as a reward judge, it is only employed to compare a candidate answer against a ground truth answer, which limits (though does not fully eliminate) potential bias; we will further discuss and clarify this in the revised manuscript. We also acknowledge that our current focus on teaching MLLMs to think with images leaves open questions about scaling to more complex reasoning tasks, which we will explicitly highlight as an important direction for future work.

Response to generalizability of GRIT to visual grounding tasks

We thank the author for raising this point. Our paper’s primary objective is to show that GRPO‑GR can teach small MLLMs to “think with images” using tiny data, requiring both reasoning and visual grounding abilities. In the Experiments section, we also evaluated models in a visual-grounding‑oriented setting: OVDEval, which requires detecting locations of specified targets. As noted in the paper (line 252), GRIT‑trained models achieve even more accurate results on OVDEval than zero‑shot MLLMs. Crucially, these gains are achieved with only VQA‑style answer supervision—no ground‑truth grounding labels—indicating emergent improvements in visual grounding. That said, we believe that with more specific training data in the visual grounding domain, GRIT is well‑suited to further benefit models in visual grounding tasks as the proposed optimization target can be directly applied. We are making plans to further explore along this direction.

Response to missing related work

We thank the reviewer for pointing out these references. We will add and discuss CogVLM (NeurIPS’24) and CogCoM (ICLR’25), and clarify how GRIT differs: our method learns interleaved text‑and‑bounding-box reasoning traces via RL, without requiring chain or box supervision, directly targeting the capability to “think with images.”

Response to the question of benefit on compositional visual grounding

As GRIT is designed to teach MLLMs with interleaving language with explicit, self‑selected boxes, we believe it encourages tighter text‑region coupling, which is conducive to compositional grounding. Our experiments on the models’ grounding performance (the GIoU metrics in Table 1) also suggest this transfer, where GRIT‑trained models achieve stronger grounding performance than zero‑shot MLLMs even though training uses only visual-question answering data. In the future revision, we are planning to include demonstrations on how GRIT can be applied to training models for compositional grounding tasks with more in-domain training data.