Caption This, Reason That: VLMs Caught in the Middle

We would like to first thank the reviewer for their thoughtful feedback and suggestions. Our responses to your comments are as follows:

The motivation is not clearly presented. In the introduction, after the background section (Lines 25–41), the authors claim that "VLMs struggle with multi-object reasoning tasks such as counting or identifying objects...". However, the rest of the paper does not clearly explain how their method addresses the counting problem.

To clarify, our aim for that section of the introduction was to introduce weaknesses in VLM reasoning that other studies have found. Reasoning is a broad term that encompasses many cognitive abilities, including counting objects. We did not intend to imply that counting would be directly addressed in our study, as this work focused on evaluating core cognitive abilities (PAM) as well as high-level decision making where those core abilities are used (CVR). We will be sure to update the introduction section to make this clearer to the reader.

The paper lacks a clear and detailed description of the PAM and CVR datasets in Section 3.2. There is no information available about the dataset statistics, the creation process of the datasets, or example cases. It’s also unclear why two separate datasets are needed.

Thank you for pointing this out. We agree that Section 3.2 would benefit from a more detailed presentation of both the PAM and CVR datasets. In the revised manuscript, we will add dataset statistics and clarify the creation process. We had already included examples for all PAM and CVR task types in Figures A.1.1-5. We also address the differences between PAM and CVR and why they are both necessary below in response to the reviewer's question of The PAM and CVR datasets are built using iWISDOM….

The description of the CVR dataset (Lines 139–145) is too brief.

In the revised manuscript we will provide further details on the description of the CVR dataset, similar to how we describe it below in response to the reviewer's question of "The PAM and CVR datasets are built using iWISDM…". However, as CVR is a replication of the data used in the benchmark section of the iWISDM paper, we feel it is sufficient to have a brief explanation and list of details, and point to the iWISDM paper, if the readers are interested in further details.

The paper claims it can "identify specific weaknesses" of VLMs (Line 60), but this is not clearly described.

We already mention the two specific weaknesses revealed by our analyses (spatial localization and selective attention) in the list of contributions at the end of the Introduction (lines 60-61). Furthermore, sections 4.1 and 4.2 of our manuscript provide detailed descriptions of the VLM weaknesses exposed by the PAM and CVR datasets.

The claim that the paper reveals the reasoning bottleneck of VLMs (Line 62) is also not clearly described.

We appreciate the reviewer’s feedback, and we will update the manuscript to include a clearer description of the reasoning bottleneck we find. We will amend the end of Section 4.3 (lines 275-292), where we have our current descriptions, to more directly describe that the reasoning bottleneck is caused by an inability of SOTA VLMs to effectively use CoT to first caption images and then subsequently reason with the extracted information.

Fine-tuning on a small dataset does not necessarily address cognitive reasoning issues if the training and testing follow the same distribution. To prove that fine-tuning improves general reasoning rather than just helping the model memorize the data, the train/test should be out-of-distribution data.

We agree entirely, and this is precisely why we made sure to evaluate the finetuned models on other cognitive benchmarks like MMMU-Pro and MMBench. You can see in the Appendix Table A.3.2 that slight improvements to these benchmarks were made. Moreover, following suggestions from reviewers o9hF, Dy9Q, and ptot, we further tested the models on the VQAv2 dataset consisting of diverse questions about natural images, completely different from those used during fine-tuning. A gist of these results can be seen in our response to reviewer Dy9Q’s comment on "Generalisation. Has SC/LoRA been tried on natural photographs (e.g., VQAv2) to confirm transfer beyond synthetic scenes?". Altogether, these results confirm that fine-tuning improves the model’s performance on multiple out-of-distribution datasets. We will be sure to make these results more evident in the main text.

Questions:

The PAM and CVR datasets are built using iWISDOM, but the paper does not explain how they differ from the original iWISDOM dataset. It remains unclear why the existing dataset cannot be used directly. More explanation is needed to justify the creation of these new datasets.

To clarify, iWISDM is an environment where datasets for visual decision-making tasks can be generated randomly or constructed manually, rather than a set of fixed datasets. To perform the experiments and analyses that our paper focuses on, we created PAM and CVR using iWISDM:

• PAM is manually designed, explicitly targeting core cognitive abilities: Perception, Attention, and Memory. The Attention dataset also included the use of distractor objects, a feature of iWISDM that was not explicitly used in the iWISDM paper.

• CVR employs the iWISDM AutoTask environment to dynamically generate complex tasks via random compositions of basic task operators. These tasks are specifically generated to assess the general visual reasoning capacity of different models in our work. The tasks in CVR were randomly generated following the same parameters as used in the iWISDM paper for its section on benchmarking.

While necessary in achieving the main contributions of this study, PAM and CVR were not the main contributions in themselves. We will clarify this and provide further details on the dataset distinctions in the revised manuscript.

The perception ability of VLMs has already been extensively studied. The PAM and CVR datasets do not seem to provide significant new insights in this area. As for attention and memory, the tasks mostly involve referring to objects across single or multiple frames, which are still closely tied to perception rather than true reasoning. Therefore, the connection to reasoning ability is weak.

We acknowledge that the perceptual abilities of VLMs have been studied previously. However, our work offers a new perspective in several key ways: 1) it reveals substantial failures in object localization, even in simple scenes with few objects and no background; 2) it demonstrates that a simple self-captioning strategy can meaningfully address this limitation across multiple visual reasoning tasks; and 3) it grounds evaluation in a cognitively inspired framework, using tasks from classic studies of perception, memory, and attention. Together, we believe these contributions offer a fresh lens on the cognitive capabilities of VLMs that, to our knowledge, has not been explored in prior work.

There are already several works that explore the reasoning capabilities of VLMs, such as [1] and [2]. The paper should discuss these related works, clarify how its approach differs, and explain why it is necessary to create new datasets or tasks focused on reasoning.

We thank the reviewer for pointing us to these papers and will include them in our discussion of reasoning in VLMs in the revised manuscript. To clarify the differences: both papers focus on visual abductive reasoning, which differs from our work in key ways. (1) Abductive reasoning involves inferring the most plausible explanation—often a hidden cause—for a given observation. In contrast, our work focuses on logical reasoning, where VLMs are required to follow explicit instructions with unambiguous answers, leaving no room for alternative interpretations. (2) Unlike those works, our tasks involve multiple images and require explicit relational reasoning across them. Existing multi-image datasets are limited in number and typically involve only two images per task, whereas our tasks operate over larger sets.

In Table 1, the performance of GPT-4o and human participants on CVR-Cat is confusing. Despite being labeled as a low-difficulty task, the results show performance worse than the medium-difficulty tasks. The paper lacks a detailed analysis and fails to provide convincing explanations for this inconsistency.

We acknowledge the reviewer's observation regarding the unexpected performance results in Table 1. We had briefly noted this anomaly in the original manuscript (line 239) with a possible explanation related to GPT-4o. On further reflection, we believe this counterintuitive result for human participants may also stem from the presence of the if-then-else operator in Medium complexity CVR-Cat tasks. Although Medium tasks involve more logical operations overall, the structured nature of if-then-else may restrict the number of objects and comparisons that need to be considered at a time, helping participants effectively segment instructions into manageable substeps.

Relatedly, we also found that despite the apparent trend in accuracy across the three levels, the response times for human subjects consistently ramped up with higher complexity (averages: L=32.6s, M=39.1s, H=59.3s). These response times provide a more nuanced outlook on task difficulty. Despite having a higher response accuracy on medium CVR-Cat tasks than on the equivalent low-complexity tasks, subjects required more time to solve them. We will ensure that all response time data is included in the updated manuscript, along with a more detailed discussion of the specific result. We are also open to any explanations or approaches that the reviewer may have for further investigation.

Formatting Concerns: Thank you for pointing out these issues. They have been corrected in the updated manuscript!

We would first like to thank the reviewer for their thoughtful feedback and suggestions! Our responses to your comments are as follows:

Limited Evaluation Data Diversity and Domain-Specific Testing Setup:

We chose iWISDM and synthetic objects because of their strong controllability, which allowed us to precisely control the object properties in the context of diverse cognitive tasks. With regard to the reviewer’s concern about ecological validity, we had already shown that the performance of different models on our PAM and CVR task sets is strongly correlated with those on benchmarks with natural images (MMMU-Pro). This suggests that the low performance of these models on our tasks is not incidental nor a result of poor data diversity.

Regardless, to directly address the reviewer’s concern, we ran new experiments in which we tested our methods on the VQAv2 dataset. The results are as follows:

As evident, an almost 5-6% improvement on VQAv2 from both our simple SC method and LoRA finetuning for the 7B models, and 1% improvement for the larger 72B model, which suggests that both of these approaches lead to general improvements in the VLMs’ ability to perform general visual reasoning. We will include these results along with other benchmarks in Table A3.2.

Interpretation of Fine-tuning Gains:

We included this claim based on two key results in the paper: 1) In Table 3, we showed that the fine-tuned QWEN2.5-VL-7b model performed substantially better on unseen tasks from PAM; 2) In Table 3 (reported on lines 312-315), we show that fine-tuning the QWEN2.5-VL-7b model led to slight improvements on MMBench and MMUPro benchmarks, both of which contain realistic images from different domains. In addition, the newly performed VQAv2 experiments and results show significant improvements from fine-tuning. Together, these results indicate that the gain in performance is large in tasks that are conceptually close to the training distribution, but somewhat remains in natural domains with relatively little overlap with the training distribution. To address the reviewer’s concern we will revise the claims (and related statements) to the following: “Demonstrating that targeted fine-tuning on diverse cognitive reasoning tasks can moderately enhance core cognitive abilities in VLMs when tested with paradigms similar to those used to test humans, while also generally validating that improvements on PAM and CVR tasks translate to broader gains in visual reasoning performance.”

Finally, we will revise the abstract and the text and tone down our claim about the effect of finetuning by changing the claim from “substantially improving core cognitive abilities” to “moderately improving core cognitive abilities”.

The generalization of the fine-tuning of LoRA is limited to Qwen2.5-VL-7B., why not other larger models?

We performed new fine-tuning experiments on Qwen2.5-VL-32b to address this request. The table below shows how fine-tuning on 1k and 10k datasets affected PAM and CVR performance.

Apart from a few outliers like CVR-Cat-M/H and Feature Attention, fine-tuning improved the model’s performance. Compared to the 7b variant, fine-tuning 32b had a similar but lesser effect. Unfortunately, due to time and computational constraints, we were unable to fully test our 32b models. The results for their performance on held-out iWISDM tasks, MMMU-Pro, MMBench, as well as a 100k LoRA model, will be added to the manuscript.

One of the major questions is relying on a single chain of thought template, the authors might explore few-shot prompting etc.

We had not included few- shot prompting in our experiments because of two reasons: 1) The context window of VLMs, as well as computational resource limits, severely restricted the number of images that could be included in the prompt. For this reason, we were unable to include more images as examples, especially for the complex tasks which already included up to 9 images; 2) Prior work had reported limited improvements in VLM reasoning capacity in few-shot settings [1,2].

Nevertheless, to address the reviewer’s request more directly, we performed new one-shot experiments with Qwen2.5-VL-7B.

Compared with SC, one-shot didn't provide a strong performance boost. We will include these results as an additional table in the appendix.

[1] Zong, Y., Bohdal, O., & Hospedales, T. (2025). VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning. ICLR 2025

[2] Shukor, M., Rame, A., Dancette, C., & Cord, M. (2024). Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning. ICLR 2025

The authors could have tested on unseen real-image dataset VQAv2, as there is marginal change on MMBench and MMMU-Pro for LoRA finetuned model.

That’s an excellent suggestion! Following the reviewer’s advice, we performed new experiments where we additionally tested the fine-tuned Qwen2.5-VL-7B and Qwen2.5-VL-72B on the VQAv2 dataset. The results for these tests:

This indicates an almost 5-6% improvement on VQAv2 from both our simple SC method and LoRA finetuning for the 7B models, and 1% improvement for the larger 72B model. These results further confirm that the observed improvements are robust and generalize to broader settings involving natural images.

We will include these results along other benchmarks in Table A3.2.

The authors do mention of LoRA overfitting and add dropout but might add a detailed ablation such as different dropout rates to quantify better.

To find the best fine-tuning procedure, we ran a number of experiments during which we changed several hyperparameters, including the LR, LR scheduler decay ratio, dropout ratio, and optimizer parameters. Due to the sheer number of possible combinations, the high computational cost, and our limited academic-level resources, we were unable to try all possible combinations and therefore adopted the best setting for our smallest-scale experiments. Regarding dropout, we had briefly tried 0.0 and 0.1 before settling on 0.2. Our training runs with lower dropout ratios (0.0 and 0.1) experienced overfitting, resulting in a rise in the validation loss during training. These performance values will be added to the appendix to address the reviewer’s concern. Finally, we would like to note that the efficiency of the fine-tuning experiments was not directly tied to any of our main contributions.

The paper could have presented attention heat maps for attention capacity issues.

We thank the reviewer for their thoughtful suggestion. We believe examining the attention scores would be an effective way to probe the role of attention capacity in our observations. To address the reviewer’s suggestion, we will compare the total attention scores on the relevant caption tokens between SC and SC-I experiments, visualize individual samples and also present the average values across many samples. We hypothesize that if the textual attention scores in SC-I are significantly lower than those in SC, that would confirm our hypothesis that the inclusion of images in the context distracts the model from the more reliable textual information. Our results show that the caption text tokens received a total of 0.022 attention averaged over the model's output when the SC method is used, and 0.017 when SC-I is used. This difference in attention provides strong evidence in support of our attention capacity hypothesis.

Note: we cannot provide visuals of these attention scores due to Neurips guidelines; however, we will add the results of these and further analyses to the Appendix.

We would first like to thank the reviewer for their thoughtful feedback and suggestions! Our responses to your comments are as follows:

Questions:

Human study details. How many annotators? Were images shuffled to reduce practice effects?

The details of the human study were included in Appendix A.2. To answer your question briefly, the study included 8 subjects and trials were all randomly generated to include random selection of object images.

Statistical significance. Are accuracy gaps tested (e.g., bootstrap CIs) or visually inspected only?

The sample size in our studies was limited (N=3 per model per subtask), resulting in low statistical power for some tasks, especially CVR and Perception tasks, they have 3 or 6 samples in total per task. However, in response to the reviewer’s concern, we performed significance tests for all of our main results.

Briefly, we found that for Qwen 2.5 VL 7B, SC, SC-I PC contrasts with Base were 5/6 tasks, 2/6 tasks, 4/6 significant with at least p<0.05 on PAM, both SC and PC has a significant different (p < 0.05) on CVR-Loc-H with Base.

For larger Qwen 2.5 VL 72B SC and PC contrast with Base in all Memory and Attention tasks with at least p < 0.05 significant.

For closed model, GPT 4o, SC and PC contrast with base in 3/6 and 5/6 tasks, with p < 0.05 in PAM and both in CVR-Loc-H.

Finally, the LoRA fine-tuned Qwen 2.5 VL 7B models, LoRA 1k, 10k, 100k demonstrated p < 0.05 differences in 4/6, 4/6, 6/6 tasks in PAM with the base model and all LoRA models have a p < 0.05 difference in CVR-CAR-H, 10k and 100k in CVR-Loc-H too.

We will include the results in all tables in the revised manuscript.

Caption quality. For self-captioning, do hallucinated captions ever hurt reasoning, and how is that measured?

The answer is yes, and some results in the manuscript already address this. Specifically, the differences between PC and SC values reported in Tables 2 and A.5 capture the accuracy of VLMs when using ground-truth image captions (PC) versus VLM-generated captions (SC). We interpret the drop in performance between these conditions as an effect of captioning hallucinations. This is an important point that was not clearly stated, and we will clarify it in the revised manuscript.

Generalisation. Has SC/LoRA been tried on natural photographs (e.g., VQAv2) to confirm transfer beyond synthetic scenes?

That’s an excellent suggestion! Following the reviewer’s advice, we performed new experiments where we additionally tested the fine-tuned Qwen2.5-VL-7B and Qwen2.5-VL-72B on the VQAv2 dataset. The results for these tests:

As evident, an almost 5-6% improvement on VQAv2 from both our simple SC method and LoRA finetuning for the 7B models, and 1% improvement for the larger 72B model, which suggests that both of these approaches lead to general improvements in the VLMs’ ability to perform general visual reasoning. We will include these results along with other benchmarks in Table A3.2.

Limitations

Synthetic bias. Only eight ShapeNet categories on plain backgrounds limit ecological validity.

We chose iWISDM and synthetic objects because of their strong controllability, which allowed us to precisely control the object properties in the context of diverse cognitive tasks. With regard to the reviewer’s concern about ecological validity, we had already shown that the performance of different models on our PAM and CVR task sets is strongly correlated with those on benchmarks with natural images (MMMU-Pro). This suggests that the low performance of these models on our tasks is not incidental nor a result of poor data diversity.

Further, as shown previously in response to the reviewer’s question about generalization to natural photographs, we validated our methods on the naturalistic benchmark VQAv2, and found a 5-6% gain in performance from both SC and LoRA.

Single-frame captioning. SC assumes each frame can be described independently; coherence across complex videos remains untested.

We acknowledge that this is a limitation of our current tests, as our experiments only involve video and do not consider more complex media formats such as videos. We will add this point to the limitations section of our paper to address the reviewer’s comment.

Overfitting risk. LoRA boosts CVR but slightly hurts MMMU-Pro at small data sizes; larger sets might magnify this trade-off.

We agree that LoRA fine-tuning comes with the risk of overfitting. However, on our largest training dataset size of 100k, LoRA actually led to marginal improvements on both MMBench and MMMU-Pro. Even more substantial were our additional results on VQAv2, probed from the reviewer’s comments on generalization, which show a ~5.8% improvement. With that said, fine-tuning via other methods, such as reinforcement learning or full SFT, would be interesting avenues for future experiments.

Attention-capacity hypothesis unproven. The paper infers attention interference from SC-I drop, but no probing of cross-modal token limits is shown.

We agree with the reviewer that we did not offer any evidence to support this statement and only speculated that this is likely due to this hypothesis. Following reviewer o9hF’s suggestion we examined the attention scores to probe the role of attention capacity in our observations. To address the reviewer’s suggestion, we compared the total attention scores on the relevant caption tokens between SC and SC-I experiments, visualized individual samples and also calculated the average values across many samples. We hypothesize that if the textual attention scores in SC-I are significantly lower than those in SC, that would confirm our hypothesis that the inclusion of images in the context distracts the model from the more reliable textual information. Our results show that the caption text tokens received a total of 0.022 attention averaged over the model's output when the SC method is used, and 0.017 when SC-I is used. This difference in attention supports our attention capacity hypothesis.

Note: Unfortunately, we cannot provide visuals of these attention scores due to Neurips guidelines; however, we will add the results of these and further analyses to the Appendix.

Missing energy/runtime analysis. Procedural tasks are cheap, yet large-scale SC + LoRA incurs extra inference cost.

We thank the reviewer for suggesting this critical analysis we missed. We have now performed runtime analyses on 10 trials of a 1-frame task (Perc-Loc-R) and 10 trials of a 9-frame task (CVR-Loc-H) given to Qwen2.5-VL-7b. We find that the SC method is indeed slower than the standard method of input on average for both trial lengths. SC performed the 1-frame trials in an average of 5.731 seconds and the 9-frame trials in an average of 35.259 seconds. The base method performed the 1-frame trials in an average of 2.841 seconds and the 9-frame trials in an average of 13.410 seconds. These analyses revealed an important limitation to our method; despite the significant performance gains of SC, it comes at the trade-off of roughly a 2x slower run-time. We will be sure to include this limitation and the details of our run-time analysis in the manuscript.

Formatting Concerns

Thank you for pointing out these issues. They have been corrected in the updated manuscript!