Do Large Language Models Truly Understand Geometric Structures?

Thanks for your constructive feedback. We respond to your questions one by one:

Q1: While the advanced split of GeomRel does require managing multiple geometric relationships, it still treats each relationship as relatively independent. Real-world geometric problems, however, often demand that LLMs execute step-by-step calculations in sequence, where each conclusion logically affects the next.

Thank you for your thorough reading and deep reflection on our paper. Our evaluation separates relationships into distinct categories: line-based relations focus on fine-grained interactions between points and lines, angle-based relations involve directional and symmetry-related properties, and shape-based relations assess holistic understanding of geometric structures like circles and polygons.

While complex problems often involve interactions between relationships, this decomposition helps analyze specific skills like local understanding and overall positional reasoning. We also limit reasoning chains to 2–3 steps for advanced tasks to better evaluate core reasoning capabilities.

Q2: Some analyses in Appendix F are particularly insightful, demonstrating the reasoning path and showing how the vanilla model fails while reverse reasoning aids understanding. It would be valuable to provide experimental results to support these explanations.

We appreciate your positive feedback on the analyses in Appendix F. To further support our explanations, we introduced a new metric, Necessary Condition Coverage (NCC), which measures the proportion of necessary intermediate conditions covered during the reasoning process that lead to the correct answer. We evaluated this on 24 random selected questions using GPT-3.5-turbo with Few-Shot CoT and Few-Shot GeoCoT methods. The results are as follows:

The results demonstrate that our two-stage GeoCoT method improves NCC, particularly for advanced problems. This highlights the effectiveness of decomposing geometry problems and incorporating reverse reasoning to enhance coverage and reasoning quality.

Q3: the paper show marginal gains from fine-tuning. It would be valuable to explore the reasons behind this limited improvement, possibly factors like dataset size, or the nature of the fine-tuning data.

Thank you for your suggestion. We conducted further analysis on the fine-tuning results of LLaMA3-8B-Instruct and found that its marginal gains were due to the model frequently outputting the uncertain answer, "cannot be inferred." Additionally, we fine-tuned the LLaMA3-8B-Base model, which showed substantial improvement, achieving 81.26% and 62.75% accuracy on the Basic and Advanced subsets, respectively, surpassing GPT-4o.

We analyzed the proportion of "cannot be inferred" responses generated by the models, as shown below (with Truth representing the actual proportion in the dataset):

The fine-tuned LLaMA3-8B-Instruct model heavily leaned toward uncertain answers, whereas the Fine-tuned LLaMA3-8B-Base model closely aligned with the actual data distribution. This discrepancy highlights the impact of fine-tuning data and the model's initial instruction-tuning bias.

Q4: In explaining the performance improvements from point relabeling, the authors hypothesize that "using more complex descriptions may stimulate LLMs’ reasoning abilities." This point could benefit from further explanation.

Thank you for raising this point. Initially, point representations used simpler labels such as "triangle ABC." Through Re-labeling Points (RP), we introduced labels like "triangle MGQ," which, while preserving the geometric structure, add complexity to the description. We evaluated this approach on the base subset of the dataset, and the results are as follows:

The improvement suggests that Re-labeling Points stimulates the model to adopt more complex reasoning paths for problems that initially appeared simpler. However, this effect diminishes for the advanced subset, where descriptions are already sufficiently complex. This indicates that while RP enhances performance by unlocking latent reasoning potential, it does not fundamentally improve the model's reasoning ability.

W1: Although fine-tuning on GeomRel was attempted, the results only report results of a single model. More experiments exploring different models could clarify which models has performance improvements with fine-tuning.

First, thanks very much for your recognition of our work and the suggestions. We have added the fine-tuning results for the LLaMA3-8B-Base model, as shown below:

The results show that fine-tuning on the LLaMA3-8B-Base model is highly effective, significantly improving accuracy across all subsets. It also outperforms the best-performing model (GPT-4o) in most categories. Comparing the two fine-tuned models, we found that the Finetuned LLaMA3-8B-Instruct model underperforms, likely due to its tendency to select the uncertain option "cannot be inferred." This additional experiment highlights the importance of model selection for fine-tuning on GeomRel. Additionally, fine-tuning experiments on other models (e.g., the Qwen series) are ongoing, and we will provide updated results later.

W2: It is difficult to truly understand the difficulty of the dataset for human intelligence. This can be understood by sampling the dataset and carrying out a systematic human evaluation to report human baselines on these geometrics reasoning problems.

Thank you for this valuable suggestion. To address this, we conducted an evaluation with five graduate students in computer science. The results, along with comparisons to GPT-4o and random baselines, are shown below:

We found that human performance is overall similar to GPT-4o, but with notable differences across subsets. GPT-4o outperforms humans on Line-based and Angle-based subsets, indicating stronger capabilities in fine-grained reasoning. However, humans significantly outperform GPT-4o on Shape-based tasks, suggesting that the model may struggle with holistic perception of geometric structures compared to humans.

W3: GeomRel, while valuable for evaluating LLMs on basic geometric relationships, covers a limited range of geometric concepts, primarily focusing on 2D relationships involving lines, angles, and shapes. This narrow scope may not fully represent the complexity of real-world geometric tasks and it will further help evaluate reasoning capabilities particularly those that involve 3D relationships, transformations (like rotations and reflections), or coordinate-based reasoning.

Thank you for highlighting this limitation. Our work intentionally focuses on 2D geometric relationships to ensure thorough coverage within this scope, as well as to align with the current reasoning capabilities of LLMs. While 3D relationships and transformations were considered during dataset design, we chose to prioritize foundational 2D concepts. Expanding GeomRel to include 3D reasoning and transformations is an important direction for future work as model capabilities continue to improve.

Q1: It would help the readers better, if the authors can provide the exact versions of proprietary LLMs that was used as well as other hyper parameters set for these models for inferences as well as for the fine tuning experiments.

Thank you for your suggestion. We have added the exact versions of the proprietary LLMs and the details of the hyperparameters used for inference and fine-tuning experiments to the appendix of the paper.

Thanks for your constructive feedback. We respond to your questions one by one:

W1 & Q1: Clarification on how the benchmark was constructed, including relationships, difficulty levels, and augmentation methods, and whether LLMs were involved.

Thank you for your insightful question. The benchmark construction begins with fundamental geometric elements: points, lines, angles, and shapes, where points serve as the most basic, indivisible concept underlying all geometric relationships.

We first identified key geometric relationships, including line-based (e.g., interactions between points and lines), angle-based (e.g., angular orientation and symmetry), and shape-based (e.g., positions of points or lines relative to a central shape). Using these, we manually created a base dataset of simple problems, expanding it slightly with LLM-generated examples. These problems typically require no additional reasoning, forming the basic difficulty level.

To construct the advanced difficulty level, we combined base relationships into reasoning chains (2–3 steps) to create more complex problems. This was primarily done using rule-based generation, with manual validation to ensure quality.

Finally, we applied two augmentation methods for diversity: (1) Adding unrelated information, where non-essential details were added (some generated using LLMs); and (2) Re-labeling points, which used rules to alter point labels without changing problem logic. Both methods preserved the original difficulty level since reasoning chain complexity remained unchanged.

W2 & Q2: Using the LLaMA3-8B-Instruct model for fine-tuning is unconventional and suggesting providing results from fine-tuning the LLaMA3-8B base model.

Thank you for pointing this out. Since our former evaluation focused on instruction-tuned models, we fine-tuned the LLaMA3-8B-Instruct model for consistency. We have now added results from fine-tuning on the LLaMA-3-8B base model, as shown below:

Fine-tuning on the base model proves highly effective, significantly improving accuracy across all subsets, and surpassing the best-performing model (GPT-4o) in most categories. Additionally, comparing the two fine-tuned models reveals that the Finetuned LLaMA3-8B-Instruct model underperforms due to frequently selecting the uncertain option "cannot be inferred."

W3: Including additional experimental metrics commonly used in math domains, such as pass@k, majority voting, or best-of-n, could improve the depth of analysis and provide a more comprehensive evaluation of model performance in this paper.

Thank you for the suggestion. In our main experiments, we used single greedy decoding (temperature set to 0) for all large models. To address your point, we conducted additional experiments on the GPT-3.5-turbo model using multiple sampling runs (temperature set to 1). We evaluated its performance using pass@k and self-consistency with majority voting over k generations (denoted as acc(SC=k)). The results are as follows:

We observed that pass@k increases rapidly with larger k, indicating a growing likelihood of generating the correct answer through multiple attempts. However, the majority voting accuracy (acc(SC=k)) does not improve significantly, suggesting that the model's probability of generating correct answers remains low and that its reasoning over geometric tasks is inconsistent, resulting in dispersed outputs.