R-CoT: Reverse Chain-of-Thought Problem Generation for Geometric Reasoning in Large Multimodal Models

1. The reverse chain-of-thought concept is essentially a reconfiguration of existing Chain-of-Thought (CoT) techniques applied to geometric data generation. The novelty is thus limited, as R-CoT primarily adapts and combines established ideas without introducing a fundamentally new approach.
2. The effectiveness of R-CoT heavily relies on the fidelity of the generated images and descriptions. However, the process for validating the quality and accuracy of these generated descriptions seems insufficiently detailed, especially given the complexity of geometric reasoning.
3. The R-CoT pipeline involves a multi-stage generation and reasoning process, which could increase computational cost. It is unclear whether this approach offers efficiency or if the computational demands outweigh its benefits.

1. Unclear Necessity of the Vision Modality: The integration of a vision component within this framework raises concerns regarding its necessity and effectiveness. The core of the geometric problem-solving process, as presented, appears to rely on CoT reasoning based solely on textual descriptions. Given that the problems are fully described in the text, the role of the generated geometric images in augmenting reasoning is ambiguous. Specifically, the added visual modality does not clearly offer unique insights or support that could not otherwise be inferred through text alone. Without a compelling demonstration of how the visual component uniquely facilitates problem-solving, its inclusion seems unnecessary and potentially redundant.

2. Ambiguity in Difficulty Level Classification: The criteria used for categorizing the problems into three difficulty levels—easy, medium, and hard—are insufficiently defined and appear somewhat arbitrary. For instance:

• Inconsistent Alignment with Difficulty Criteria: The difficulty classification seems to focus on the number of reasoning steps, but without concrete justifications, this metric alone may not adequately capture the true complexity. The nature of geometric reasoning can vary widely, and simply having more reasoning steps does not necessarily equate to a harder problem.

• Spatial Reasoning as a Key Factor: For certain problems that require interpreting spatial relationships or auxiliary constructs (e.g., adding segments for further reasoning), a visual component might be essential. These types of problems could then reasonably fall under higher difficulty levels if they require the vision modality to infer spatial arrangements. However, the paper does not fully clarify when and why visual input would be required in these cases.

Additionally, according to [Ref. 1], those data correctly generated by CoT of LLMs can be considered within a Completely Feasible Reasoning Boundary (CFRB), which characterizes the simple reasoning tasks. Therefore, the difficulty criteria should be re-evaluated to better suit the geometry task, accounting for situations where visual input may or may not be essential.

[Ref. 1] Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought. NeurIPS 2024.

1. Overclaim: The statement, "Notably, R-CoT-8B significantly outperforms previous state-of-the-art open-source mathematical models by 16.6% on MathVista and 9.2% on GeoQA, while also surpassing the closed-source model GPT-4o by an average of 13% across both datasets," appears overclaim. Given that this work primarily focuses on data generation, fair assessment would require consistent model comparison settings. The claim in the abstract is misleading due to differences in model size and backbone. For instance, the authors use an 8B model as the backbone, whereas prior methods rely on a 7B model. As shown in Table 1, improvements under comparable settings are marginal.
2. Concern of Entire Data Quality: In Table 1, combining your data with Geo170K yields only marginal improvements, which raises questions about the efficacy of the GeoMM dataset. Can the authors provide results that reflect the performance of using only GeoMM?
3. Image Diversity: Clarification is needed on how images generated by GeoChain differ from those in existing datasets. For example, the images shown in Figure 12 (Examples of GeoMM dataset) appear similar to datasets like GeoQA and Geo3K. The marginal improvement observed further suggests potential alignment or overlap with existing data domains, calling into question the added diversity. Moreover, how is diversity measured?
4. Concern of QA Quality: The synthetic questions are annotated using ERNIE Bot 4.0. After generating the QA pairs, what steps were taken to ensure their correctness, and has this been verified?
5. Question about Method: When segmenting descriptions into patches for reasoning, it can be challenging to assess the difficulty level of each condition in relation to others. For any given description, how is the hierarchy of condition difficulty determined? Additionally, how do you ensure that each subsequent condition is logically derived from previous ones to yield varied difficulty levels in the answers?
6. Writing: In Table 4, it’s noted that the baseline models vary across R-CoT models of different sizes, yet comparisons mention only the model sizes, which creates an unfair comparison. Stating specific settings for both your models and baseline models (LLMs/VLMs) in the table is necessary to give readers an accurate understanding of performance.