CogCoM: Train Large Vision-Language Models Diving into Details through Chain of Manipulations

Dear Reviewer, we are very grateful for the valuable time you have spent reviewing our paper and for your recognition of our work, which is of great significant to us. Concerning the issues your have mentioned in our paper, we will make the following improvements:

• Design of Figures: Thank you very much for pointing out this issue. Following your suggestion, we have adjusted the Figure 1 by adding dark background to reduce the overall brightness of the image. Compared to the Figure 1 that introduces different capabilities of the model, the Figure 2 is primarily intended to depict the example mentioned in the introduction (VLMs could not answer correctly or even output hallucinations for the detailed recognition problems without reasoning). We have increased the font size for the Figure 2 and will add a diagram of the image decoder structure to depict content that differs from Figure 1.
• Design of the Tables: We have adjusted the tables in our paper to increase the spaces between tables and corresponding captions. Please refer to the tables in the supplemented PDF file to view the effects of these adjustments.
• Discussion to the related work: Thank you very much for your reminder. We have added the following description for this related work to our Related Works section: The authors from LLaVA-PLUS [1] have contributed efforts to train VLMs to develop the capability of invoking external tools. They constructed an instruction-tuning dataset incorporating tool use examples and trained VLMs to call external tools to solve challenging tasks. In comparison to their efforts, this work focus on stimulating the model’s inherent abilities to solve problems in an end-to-end manner through active reasoning. Though disadvantage in producing pixel-level masks, it offers advantages and potential for enhancing the model’s inherent reasoning abilities and reducing the time complexity of reasoning.

[1] https://arxiv.org/abs/2311.05437

Dear reviewer, thank you for taking your valuable time to review our paper and the insightful suggestions. We have added experiments (in the supplementary PDF) and supplemented content (in this page) of our paper, as detailed below:

• We have evaluated our model on the suggested new benchmarks, MMBench, SEDD-Bench, and MathVista, and the experimental results are listed in the Table 2 of the supplementary PDF file. In conclusion, our model achieved stronger performance in comparison with concurrent and the suggested new baseline models, outperforming LLaVA-1.5 by 10.2, 4.9 and 8.1 percentage points on the three benchmarks, respectively.

• We compared our model with the suggested up-to-date baseline models, and the results are listed in the Table 1 of the supplementary PDF file. In conclusion, our model consistently outperforms these baseline models across the VQA and general multimodal benchmarks, demonstrating effective multimodal capabilities.

• Discussion with closely related works: (1) The efforts of ViperGPT shares the same basic idea with our work that decompose complex visual problems into reasoning steps. In comparison with their concentration that aims to build a training-free framework combining external VLMs using a code LLM, we focus on training an end-to-end VLM to enhance its multimodal reasoning ability to solve complex visual problems. (2) V* is a concurrent work who also aims to solve VQA problems by progressively acquiring visual cues. Their two-parts framework first utilize a VQALLM model to list visual objects needed for answering, followed by a dedicated search model to accurately acquire the objects. On the other hand, our approach focuses on using one model to actively performing reasoning and to identify or mark the most useful visual information, which may offer the potential for solving more complex reasoning tasks in addition to the detailed identification, such as the challenging geometric math problems.

• Discussion for the differences with other related works: (1) CogVLM is a multimodal foundation model which aims to achieve reliable performance on broad multimodal tasks (e.g., VQA, Grounding, Captioning). This work is motivated by the observation that CogVLM produces hallucinating answers for the detailed reasoning questions (i.e., the Figure 2), and aims to study an effective methodology to enhance the multimodal reasoning capabilities of VLMs while reducing the hallucinations by visual evidences. (2) The DualFocus was released around the same time as ours. They also made efforts to construct a training dataset that includes intermediate cues (bounding boxes) and trained the model with two stages based on accurate localization. Compared to their work, our CoM training places more emphasis on answering questions in a single reasoning process and marking image to assist in solving complex problems. We will include the above works and discussions into the related works section.

• The insignificant improvements: Due to the focus of this paper on enhancing the reasoning capabilities of VLMs, the improvements of our model over CogVLM on Grounding and General Multimodal benchmarks is relatively limited. However, on the evaluation sets that require logical reasoning (i.e., GQA), complex counting (i.e., TallyVQA), and detailed recognition (i.e., TxtVQA, ST-VQA), our model achieves improvements of 6.5, 10.9, 1.04 and 9.0 percentage points relatively. Additionally, we found that mistakes in reasoning paths are indeed a major factor affecting performance on the general multimodal benchmarks.

Thank you for your efforts in solving my concerns. One additional question: since mistakes in reasoning paths are indeed a major factor affecting performance on the general multimodal benchmarks, the author may have some discussions (experiments are not required) about how to compensate for the negative effects of planned pathway failure, as in the concurrent work DualFocus.

Overall, most concerns are solved. I will raise my rating to 6.

Dear reviewer, thank you for taking your valuable time to review our paper and the insightful review. In response to your suggestions and questions, we have conducted experiments and provided explanations, as follows:

• The filtering strategy for quality control in data collection: Thank you for your carefully reading of our analysis on tool recognition errors during data generation as stated in Appendix C.5. During the collection of our 70K CoM training data, we discard the wrongly recognized data (i.e., we refer to these data as the negative paths in our paper), as these data can not terminate to the golden answer node during the DFS traversal. Therefore, we used this filtering strategy to ensure that only the correct data capable of reaching the golden answer (i.e., positive paths) was included in the 70K training data. We look forward to utilizing these negative paths as negative rewards in future work.
• The ablation experiment to validate the effectiveness of CoM data: Thanks for your suggestion. We have conducted ablation experiments on training our model with and without the incorporation of the collected CoM training data to validate the impact, and the comparison results on three typical categories of benchmarks, TextVQA, MMVet, and MathVista are shown in Table 3 of the supplementary PDF file. In conclusion, our model benefits from the CoM corpus that contains informative visual reasoning evidence, achieving improvements of 6.6, 0.2 and 0.9 percentage points on the three benchmarks, respectively.
• The experiments to validate the effectiveness of manually annotated CoM-math data: Motivated by the effectiveness of AlphaGeometry, we annotated 6K reasoning chains for the purpose of advancing the development of VLMs on this specific task. Solving geometry math problems is highly challenging for current VLMs. We conducted experiments on MathVista, and the results are shown in Table 3 of the supplementary PDF file. In conclusion, our model, trained on the 6K annotated samples, achieved an improvement of 0.90 percentage points on MathVista. Though the relatively limited improvement in answer’s matching metrics, we believe that the annotated data involving the multimodal reasoning processes like humans (such as drawing auxiliary lines) may contribute the potential to the development of this field.
• The clarification of the multi-turn multi-image capability: CogCoM can solve visual reasoning problems through multiple rounds of interaction with an input image (e.g., marking auxiliary lines), re-inputting the resulting image at each new round (e.g., a marked image with auxiliary lines) in an end-to-end manner. Taking the geometry math problem-solving as an example (e.g., the last case in Figure 1), given the initial inputs of an image $I_0$ and a question $Q$, our model undergoes the first round of reasoning (outputting multiple reasoning steps) and then draws an auxiliary line on the image to obtain a new image $I_1$, and then re-input this marked new image to continue the next round of reasoning (may draw additional lines or crop and zoom in on a specific region). Compared to most of existing methods, we refer to this approach, which is similar to the human thinking process, as a multi-turn, multi-image process.
• The maximum number of the multi-image turns: Thanks for you suggestion. We have detailed the statistics for the training and testing data in the Appendix C.3, including the total number of reasoning chains, the average number of reasoning steps, and the average number of manipulation types. In accordance with your suggestion, we have also conducted statistics on the number of turns divided by multiple images. The results are as follows: in the training data source from TextVQA and ST-VQA which may involve generating new images such as through zooming, the average number of turns is 1.42, and the maximum number of turns is 7 (we restrict the maximum number of turns to 4 during training to prevent OOM). In the test set of TextVQA, our model produced an average of 1.54 turns involving multiple images. It is worth noting that not every image requires manipulations such as zooming, and some can be answered through reasoning with evidence or direct observation. We will add this statistics to the paper content.
• Thank you for pointing out the typos. We will thoroughly check the paper again to ensure there are not ambiguities and mistakes.