Authors' Response to Reviewer NF8o

We thank the reviewer for their time and effort to review our paper. Please find responses to your comments and questions below.

Questions

Experiments are restricted to Synthetic Tasks

We use synthetic tasks to enable controlled studies of modality imbalance and simple-to-hard generalization in VLMs. These tasks are carefully designed to require diverse skills such as OCR, spatial navigation, and visual reasoning, which are essential to real-world VLM applications. As shown in Table 7 (Appendix I.1) (referenced in lines 048–049), frontier VLMs struggle with these tasks, suggesting that even simplified settings can reveal gaps in current models. Importantly, incorporating these synthetic tasks into pre-training improves performance on a broad range of real-world benchmarks, show in Table 6 (Appendix G.6) (referenced in lines 427–428). Thus, the synthetic tasks that we use are relevant to the real-world applications and could be utilized to motivate training choices to boost VLM’s general abilities.

What are the challenges for the remaining modality gap?

Thank you for the interesting question. Our work highlights a modality gap in simple-to-hard (S2H) generalization between text and image inputs, which we attribute in part to current VLM training paradigms. Current VLMs rely on adapter-based architectures that loosely integrate small visual encoders with large pre-trained LLMs, which have been pre-trained separately on image and text pre-training data.

Our approach uses image-to-text conversion to better align image inputs with the model’s strong capabilities in language, partially mitigating this gap. However, fully closing it likely requires rethinking the model design itself. Moving beyond adapter-based setups towards early-fusion architectures or joint multi-modal pretraining could help models learn more unified representations and improve generalization across modalities.

How do the authors propose to mitigate additional inference cost with image-to-text conversion

We would like to clarify that our proposed approaches, Mix, Mix+, and Align-Mix+ supervision types, use image-to-text conversion only during training the model (please see lines 151-152, 245-247, 256-257). During inference, the models trained with these supervision types don’t explicitly perform image-to-text conversion. Please see Figures 3 and 6 in the main paper and lines 205-209, 257-264 for discussions on lengths of inference-time generations for Mix, Mix+, and Align-Mix+ supervisions.

Further analysis of gradient alignment for more models and datasets

Thanks for the question! We evaluated gradient alignment on the Consecutive Table Readout and Table Readout datasets. Due to the high cost of saving frequent checkpoints, we did not extend these studies to other settings. Instead, we focused on fine-grained alignment measures (see Figures 17–20) and provided theoretical support in Theorems H.1 and H.2. Together, these offer strong initial evidence for the robustness of our alignment metric as a generalization indicator.

More details on CoT Generation

All of our data (image, its text description, the CoT trace, and the solution) is generated with a Python script. When we generate an image (e.g., using matplotlib), we store relevant metadata (e.g., sequence of (row index, col index) of highlighted cells in Table Readout) and insert it in a fixed chat template. We give examples of our CoT data in Figures 32-37 in the appendix. See Appendix D.6.2 (mentioned in footnote 1 of page 1) for alternate versions of CoT templates.

Authors' Response to Reviewer d6eT

We thank the reviewer for their comments and suggestions. Please find our responses to your comments below.

Is the paper related to knowledge distillation or prompt engineering?

We would like to clarify that we do not employ any knowledge distillation. All of our data (image, its text description, the CoT trace, and the solution) is generated with a Python script. When we say the reasoning transfers from the LLM to a VLM, we mean that the innate (or learned) reasoning capability of the LLM backbone (i.e., Llama-3-8B-Instruct) of an adapter-based VLM helps the entire VLM learn to perform the same task in the image modality. There is no external LLM/VLM here.

The datasets are self-created without detailed descriptions, and public benchmarks are not used

We propose to measure the modality imbalance in VLMs by looking at the different simple-to-hard generalization behaviors (e.g., length generalization, compositional generalization). For this, we need a dataset where 1) there is a clear level of difficulty and 2) each image has an equivalent text description. There is no public dataset that fits this description. We also mention in the Related Works paragraph that “current VLM benchmarks are often solvable without the visual input,” which makes our analysis impossible. Additionally, we provide details in generating the datasets in Appendix D and provide the example data points in Figure 32-37. We were planning on releasing the entire codebase with the final version of the paper, but here is an anonymous link to the code to generate the data: https://github.com/asjdifpjadsi/VLM_S2H. Also see our response to Q1 of Reviewer NF8o.

Please provide Comparisons with “Multimodal chain-of-thought in Language Models”

We thank the reviewer for the suggestion. We would like to clarify that our paper focuses on modality imbalance and its measurement through simple-to-hard generalization, aiming to improve a VLM’s reasoning ability on images to a comparable reasoning performance on equivalent text data. We would like to reiterate that we do not employ any knowledge distillation.

• Firstly, although both Multimodal-CoT and our paper use CoT to boost the model’s reasoning capability, we would like to point out that it is a very common technique in practice.
• Multimodal-CoT would be mostly comparable to our Image-via-Text Supervision as both attempt to leverage extra text generation to assist reasoning. However, a significant difference is that Multimodal-CoT optimizes for CoT while ours only relies on CoT with a fixed template to assist long-chain reasoning.
• Furthermore, our method converts an image to an equivalent text – a step that is later internalized – which is not equivalent to (either human-annotated or AI-generated) CoT. This conversion introduces no new information, so the help of text conversion in the model’s reasoning, if there is any, is entirely from a difference in modality, as opposed to a more optimal reasoning trajectory. Our work is more aligned with the idea of vision-depicting-prompting in Zhang et al. (2023) and many more we cited in the Related Works section (Appendix B, mentioned in page 8). Instead of prompting, we perform SFT and propose a testbed to quantify the benefit of an extra step of text conversion in mitigating modality imbalance in terms of S2H generalization.

Comments on Presentation / Figures

English needs further improvement and a thorough grammar check

We thank the reviewer for the comment. We have taken a lot of care when writing the paper. We will modify any remaining grammatical mistakes, if any, in our final version.

[1] Zhang et al., Lost in Translation: When GPT-4V(ision) Can’t See Eye to Eye with Text A Vision-Language-Consistency Analysis of VLLMs and Beyond, 2023.

[2] Zhang et al., Multimodal Chain-of-Thought Reasoning in Language Models, 2023.

Authors' Response to Reviewer pko5

We thank the reviewer for their thoughtful comments and suggestions regarding the paper. Please find our responses to your comments below.

Weaknesses

Experiments are limited to one base model

Please see our reply to Reviewer fXux. We observe the same conclusion from Qwen 2.5 VL 3B and 7B.

The simple and hard table readout tasks are more like scaling up the recognition and counting complexity, rather than requiring significantly better reasoning capabilities, e.g., chess puzzles.

We agree that the reasoning in Table Readout is simpler compared to e.g., solving chess puzzles. However, such simple synthetic tasks allow us to clearly define SIMPLE and HARD examples and enable a more controlled analysis of model behavior. That being said, Table Readout does not only involve recognition and counting, but is analogous to Grid Navigation. At any cell, the model needs to know where to read the next number from. E.g., to realize the next number is on the left, the model needs to check that up/right/down is not highlighted but left is. This decision process mirrors the one in Grid Navigation, where all non-highlighted cells are “obstacles” to avoid.

Please discuss IsoBench

Thank you for suggesting the reference. We will include the citation in lines 020-021 (page 1, right).

Questions

How would ‘Hardness’ impact the performance? The choice of threshold to split SIMPLE and HARD needs to be clarified.

As S2H generalization measures OOD performance, “hardness” is relative with respect to the difference between training and test distributions and is model-specific. On Consecutive Table Readout, we consider two hardness levels: HARD (15-20) and (25–30), each reading 15-20 or 25–30 consecutive numbers (line 150) and find modality gap gets wider as the task becomes more challenging (line 210-212, Fig. 2). For the other three tasks that also require compositional generalization, we decided the hardness by either increasing the number of components to be composed (Table Readout and Grid Navigation) or introducing held-out compositions (Visual Analogy).

We agree that ultimately the SIMPLE-HARD split is a matter of judgment. We tried to choose the most intuitive splits between SIMPLE and HARD while being aligned with existing studies (e.g., Hill et al., 2019 and Barrett et al., 2018). Further justifications for these decisions are in Appendix D and the specific thresholds in Table 1.

How did previous works approach S2H generalization in general? Any of them worth being used as a baseline (only for the text supervision)?

In Appendix B, we discuss several prior works on S2H generalization that explore ICL with scratchpads (Anil et al., 2022), positional encodings (Kazemnejad et al., 2024), train set priming (Jelassi et al., 2023), curriculum learning (Abbe et al., 2024), and looped transformers (Fan et al., 2024). However, as we focus on modality imbalance, we do not compare with these methods. The aforementioned works (1) focus only on LLMs and are not directly adaptable to a multimodal setup and (2) study only length generalization, but not compositional generalization. More recent efforts such as self-improvement (Lee et al., 2025) and generalizable verifiers (Sun et al., 2024) examine S2H generalization in different setups that rely on curriculum learning with progressively harder tasks or require reward models. In contrast, our study is restricted to supervised fine-tuning (SFT) approaches. While we acknowledge that Mix or Mix+ may not be the optimal strategies for improving image generalization on our proposed tasks (line 429), it is still notable that reasoning transfer from the text to image modality—enabling S2H generalization on images—emerges naturally through autoregressive SFT. This, in our view, is both non-trivial and interesting. We will include an explicit Related Works section in the main paper to clearly differentiate our contributions from prior work.

Comments on Presentation / Figures

We appreciate the reviewer’s detailed feedback on this matter. We hope to address these issues in the final version of the paper.

The authors interchangeably use “image” and “text”, which might create confusion

To reduce the number of new terms introduced, we used “Text/Image” (capitalized) for the supervision that trains on the corresponding modality. However, we understand the potential for confusion.

The authors can do a better job to help the audience focus on the main storyline

In Section 1.1, we presented a compressed overview of the paper to direct the readers to relevant sections. In the final version, we plan to trim some of the ablations but instead expand on the discussions that better highlight the main storyline.

[1] Lee, et al., Self-Improving Transformers Overcome Easy-to-Hard and Length Generalization Challenges, 2025

all other citations in Related Works / References of the paper

Authors' Response to Reviewer fXux

We thank the reviewer for their careful review of our paper. Please find responses to your concerns and questions below.

Weaknesses

Experiments are limited to one architecture

We used EAGLE because it was the best performing open-source model which also released all details on its training data and the set of hyperparameters.

To test the generalizability of our results, we ran additional experiments by continual finetuning 3B, 7B models from the Qwen2.5-VL family. See the accuracy on HARD-image below. CTR, TR, GN, VA are respectively short for Consecutive Table Readout, Table Readout, Grid Navigation, and Visual Analogy. On the text version of CTR, we notice that the 3B model doesn’t show S2H generalization (which is expected since task difficulty can be model specific). However, across TR, GN, and VA, we generally observe the same conclusion - Mix+ achieves S2H generalization on the image modality and Align-Mix+ improves the generalization.

Qwen2.5-VL-3B-Instruct

Qwen2.5-VL-7B-Instruct

Experiments are limited to synthetic settings

Please see our reply to the first question from Reviewer NF8o.

Experiments are only on vision-text modalities

Our motivation came from works that identify the modality gap in VLMs. Additionally, we require an open-weights model that has strong enough reasoning capability in each modality, which is available in a VLM. We would love to explore more modalities in the future when open models can also incorporate multiple modalities.

Task-specific chain-of-thought templates might limit generality of observations

By keeping the chain-of-thought templates fixed throughout the task, we were able to accurately probe the existing modality imbalance. In the future, we would like to explore more on how the results would generalize to a more flexibly generated CoT trace.

Suggested Questions

Can gradient alignment be used to measure cross-modal alignment?

We found that different supervision strategies could be distinguished only when analyzing gradient alignment between SIMPLE and HARD examples within each modality independently. This may be attributed to our use of a local definition of gradient alignment—examining gradients at each step in isolation. Capturing cross-modal alignment likely requires a more global perspective, as gradients at earlier time steps (e.g., on text) can influence gradients in later steps (e.g., on image). We consider the development of such alignment measures an important direction for future work.

Consecutive Table Readout vs. Table Readout

Both tasks involve sequentially reading highlighted cells in a table. In Consecutive Table Readout, the path is always consecutive row-wise, and the model knows where the next cell should be. In Table Readout, the path can take turns in arbitrary directions at arbitrary locations, so the model needs to additionally check which adjacent cell is highlighted at every location.

In the final version, we will list all 4 tasks in the abstract or rename Table Readout to e.g., Path Table Readout and use Table Readout as an umbrella term for both Consecutive and Path Table Readout.

Why does internalizing CoT fail

We observed that CoT is crucial for the model to transfer its reasoning from text to image input by repeating the same reasoning steps. Our hypothesis is that during SFT, the model establishes an implicit equivalence between reasoning on text and image inputs via the common CoT tokens.

There may exist multiple ways, e.g., a more fine-grained curriculum, to internalize CoT. For the scope of this paper, we limit our exploration to simple settings with randomly shuffled data and progressive internalization, and leave a complete study for future.

Can you include results on Grid Navigation in Figure 6

The model already performs nearly perfectly on Grid Navigation with 60k examples of Mix+, so we did not compare with Image-via-Text+.

What is the meaning of Mix$\to$Mix+ in Figure 8

Mix$\to$Mix+ means we took an intermediate checkpoint of Mix and resumed training with Mix+. This setup allows us to assess the effect of introducing Hard text examples on the loss curves of Mix training, enabling a fine-grained analysis of the differences between Mix and Mix+ (lines 334-341).