LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts

1. Although LogicVista covers various logical reasoning tasks, the sample size of 448 may be insufficient to fully capture the performance of MLLMs in real-world complex scenarios.
2. The tasks in LogicVista mainly focus on basic logical reasoning, such as mechanical and inductive reasoning. There is a lack of sufficient coverage for higher-level complex reasoning tasks, such as multi-step reasoning or continuous multimodal reasoning.
3. The paper mentions the risk of data leakage in many benchmarks, indicating that MLLMs might have encountered some test data during training. Although LogicVista avoids publicly available internet data in its dataset selection, it does not provide detailed mechanisms to verify the complete independence of all samples.

Relatively small dataset — 448 samples

In my assessment, a high-quality benchmark typically excels in at least one of three areas:

1. Data Quality: The benchmark offers exceptionally high-quality data that often serves as an evaluation standard within its field, as exemplified by VQAv2 and MMMU.
2. Tooling and Resources: It provides innovative tools or resources, such as novel code architectures, metrics, or other practical assets that advance usability and applicability.
3. Research Insight: The benchmark highlights a previously overlooked problem or dimension, encouraging new avenues for research and fostering deeper understanding.

While the quality of this work is solid, it falls short of providing substantial new insights or advancements in these areas. As a result, its impact on the field may be limited.

1. LogicVista presents significant difficulty differences across reasoning tasks, leading to unbalanced performance. For example, while models perform relatively well in deductive and mechanical reasoning, they perform less effectively in inductive, numerical, and spatial reasoning tasks. This variation may affect the fairness of overall performance evaluations.
2. The paper notes that current visual encoders face substantial limitations in recognizing spatial and abstract relationships, particularly in complex spatial reasoning and 3D pattern recognition tasks, where models tend to underperform. However, the paper lacks specific improvement suggestions, such as ways to enhance visual encoders or improve training data to boost reasoning ability, which somewhat constrains future developmental direction.
3. There is a lack of ablation studies to deeply analyze the specific contributions of each component to overall performance. Ablation studies can reveal the independent impact of each module, helping to better understand model composition and optimization paths.

Although I believe it will be a valuable data resource (if the authors agree to open-source), my main concern is the necessity of this dataset in these aspects:

(1) There are already many datasets in the field of reasoning, such as MathVista [1], MMMU [2] and ScienceQA [3], with some subsets even overlapping with LogicVista (such as IQTest in MathVista).

(2) As the dataset is sourced from 15 private IQ tests, why are IQ tests used in this dataset specifically designed for logical reasoning? Need some citations to support this claim.

(3) The categorisation of skills, broad and specific capabilities need further explanations or authorized references. Did you refer to previous MLLM evaluation research? For example, FLASK [4] refers to this QA taxonomy [5] for the definition of skills.

(4) Regarding the quantity of this dataset, there are only 448 questions, much smaller than datasets in this category, such as ScienceQA (21208 questions) and MathVista (6141 questions).

[1] MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts https://arxiv.org/abs/2310.02255

[2] MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI https://arxiv.org/abs/2311.16502

[3] Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering https://scienceqa.github.io/

[4] FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets https://arxiv.org/abs/2307.10928

[5] QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension https://arxiv.org/abs/2107.12708

Some minor weaknesses:

• Lack of human annotation details: As the authors state there is cross-validation during annotation, what is the inter-annotator agreement? How much payment is given to each annotator? How many hours does it take to finish annotation?

• Reliability of LLM-based evaluator on the chain-of-thought: What is the agreement rate between human-based and LLM-based evaluators on the rationales? Only when the agreement is acceptable can the LLM-based evaluation be convincing.

1. Limited Data Source: Although the IQ test data is rigorous, it is relatively concentrated in scope, lacking coverage of other fields (e.g., scientific reasoning, language understanding). The dataset size is also limited, potentially hindering the ability to fully capture the model’s performance across a broader range of tasks.
2. Lack of Depth in Benchmark Design and Ablation Studies: The paper primarily uses random and frequentist baselines, which provide limited insights into the deeper aspects of reasoning. The absence of ablation studies restricts the analysis of how different model components contribute to performance. Introducing more complex baselines and conducting ablation studies could enhance the interpretability of the results.
3. Redundant Logical Expressions: The distinction between the “Broad Capabilities” and “Specific Capabilities” sections in 4.2 is somewhat redundant, particularly in the discussion of OCR and diagram tasks. Simplifying these sections and clearly distinguishing between the two could improve the clarity of the paper’s logic.