The Curious Case of Nonverbal Abstract Reasoning with Multi-Modal Large Language Models

We thank the reviewer for their insightful feedback. In the following, we address their questions and concerns:

Q1: At the time of submission, we used most of the available state-of-the-art models, including Instruct-BLIP, for our evaluations; however, as you mentioned, keeping up with the rapid pace of changes is costly.

Q2: Similar to the previous questions, most of these models were not released during our evaluations. Our only examples of high-res models are Gemini and GPT-4v, which are closed-source. However, it’s important to note that the puzzles we used in our evaluations involve extremely simple shapes, such as squares or circles, with simple features like blue or yellow colors that do not require heavy reliance on detecting small details in a high-resolution image. Moreover, many models performed well on our visual awareness tests (Section 5, Figure 4), showcasing their understanding of the puzzles' details from a purely visual standpoint. At the same time, many of them performed extremely poorly when provided with all the details in a textual format (Section 5, Figure 3), which showcases a gap in reasoning abilities rather than perception abilities.

Q3: We have done experiments on both vision-only (Section 5, Figure 4) and text-only variations (Section 5, Figure 3) as detailed in Section 5. However, given the focus on MLLM, we kept the model pool to existing MLLMs so as not to diverge from the paper's goal.

We thank the reviewer for their insightful feedback. In the following, we address their questions and concerns:

Q1 & Q2: We used the term "nonverbal abstract reasoning," as defined by Huang et al. [1] (Section 4.2), referring to the task of providing IQ-style puzzles as images paired with short textual instructions of desired output. Moreover, Huang et al. [1] and Zhao et al. [2] provide evidence of non-trivial performance for such models. Due to our skepticism about the correctness of the underlying reasonings, we investigated this task using a much broader set of models and experimental questions in this work. Moreover, we used labor-intensive manual assessment to provide more grounded insights compared to automatic scores that don't capture the nuances of LLM-generated responses.

To solve these puzzles, a model must first correctly detect the relations between the presented objects (e.g., a red circle turning into a blue circle in the first row) and then apply them to determine the missing piece (e.g., what would a red square turn into in the second row?), which requires a certain level of analogical reasoning. To the best of our knowledge, this is the first analysis of MLLMs on this task. We will publicly release our codes, datasets, annotations, and assessments to mitigate any concern over reproducibility or annotation quality.

[1] Huang, Shaohan, et al. "Language is not all you need: Aligning perception with language models," NeurIPS 2023

[2] Zhao, Haozhe, et al. "MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning," ICLR 2024

Q3: All evaluators and annotators were mid-level to senior Computer Science PhD students specializing in NLP with extensive experience evaluating LLMs. Moreover, each sample was annotated by one person, and any uncertain case was flagged and shared among all three evaluators for discussion. After discussions, the final label was determined by a majority vote.

Q4: We ran a study with 25 college-level participants with diverse backgrounds. Each participant was given ten randomly selected samples from the IQ50 dataset (20% of the dataset) and instructed to provide a short reason for their answer. The average performance of this group was 95.9%, with a 6.92% standard deviation. We will add these results to the paper to emphasize the performance disparity.

Q5 & Q6: Due to the rebuttal's limited space (2500 characters), we will provide more details once the discussion period begins.

Thank you once again for your insightful review. In a general comment, we have provided more details regarding human performance and the manual evaluation process. If you have any other questions or concerns, please feel free to let us know.

We thank the reviewer for their insightful feedback. In the following, we address their questions and concerns:

Q1: While we acknowledge that RPM and CoT are not new methods, our paper is more geared toward providing novel analysis and insight into MLLMs and their potential shortcomings rather than introducing novel approaches.

Q2 & Q3: We agree that there is a need for further investigations on the effect of model architectures, training data, training regimens, fine-tuning, etc.; however, given the space of possibilities and cost of training these large models, such a study is prohibitively costly and broad for one article. Moreover, many of these details are often not disclosed publicly for closed-source models. As a result, we focused on testing a wide range of models (implicitly diversifying the approaches) in common zero/few-shot scenarios, providing novel insights into their strengths and limitations. We hope this work emphasizes the importance of these open challenges in future works.

Q4: We ran a study with 25 college-level participants with diverse backgrounds. Each participant was given ten randomly selected samples from the IQ50 dataset (20% of the dataset) and instructed to provide a short reason for their answer. The average performance of this group was 95.9%, with a 6.92% standard deviation. We will add these results to the paper to emphasize the performance disparity.

Q5: Regarding the rubric and the inter-annotator agreement, our main challenge was to overcome the nuances that appear in generated reasonings. As such, the evaluators first met to 1) determine the correct reasoning paths for the samples and 2) review examples of generated responses (e.g., GPT-4v, Gemini, etc.) to determine the evaluation strategy. Based on the observations in this initial meeting, the evaluators decided to allow for extra/wrong details in the generated responses as long as they did not affect or interfere with the alignment of the generated response to correct reasoning paths.

For example, in some instances, the models perceived shadows (or incorrect colors) in the shapes that were not present in the puzzle; however, as long as they correctly detected the row-wandd/or column-wise change of patterns (e.g., square turning to circle) and grounded their reasoning on them, the evaluators marked the response as correct.

Due to the rebuttal's limited space (2500 characters), we will provide more details once the discussion period begins.

Thank you once again for your insightful review. In a general comment, we have provided more details regarding human performance and the manual evaluation process. If you have any other questions or concerns, please feel free to let us know.

We thank the reviewer for their insightful feedback. In the following, we address their questions and concerns:

Q1: Regarding resolution, all samples were high-resolution (> 1000x600 pixels) to ensure all details were adequately presented. We will add this information to the paper for clarification. Regarding the objects' position, while it is true that each model has a slightly different preference, we tested them on the widespread use case of one picture with details centered from both directions. It is important to note that many models performed well on our visual awareness tests (Section 5, Figure 4), showcasing their understanding of the puzzles' details from a purely visual standpoint. At the same time, many performed exceptionally poorly when provided with all the details in a textual format (Section 5, Figure 3), showcasing a gap in reasoning abilities rather than perception abilities.

Regarding the effect of textual description, our experimental goal was to test the current state of the models in a natural scenario where the user does not need to guide the model. In parallel, we have a set of experiments in Section 6.1 (Figure 5) to show how much we can boost their performance through targeted textual guidance.

In general, we assumed that any well-trained model must work well in typical scenarios, as it would be challenging and costly for a user to find the optimal setting.

Q2: Regarding the demographic and expertise of the evaluators and annotators, all were mid-level to senior Computer Science PhD students specializing in NLP with extensive experience working with and evaluating LLMs.

Regarding the rubric and the inter-annotator agreement, our main challenge was to overcome the nuances that appear in generated reasonings. As such, the evaluators first met to 1) determine the correct reasoning paths for the samples and 2) review examples of generated responses (e.g., GPT-4v, Gemini, etc.) to determine the evaluation strategy. Based on the observations in this initial meeting, the evaluators decided to allow for extra/wrong details in the generated responses as long as they did not affect or interfere with the alignment of the generated response to correct reasoning paths.

Due to the rebuttal's limited space (2500 characters), we will provide more details once the discussion period begins.

Q3: No, the non-annotated version of the datasets has already been used in previous publications and is publicly available.

Thank you once again for your insightful review. In a general comment, we have provided more details regarding human performance and the manual evaluation process. If you have any other questions or concerns, please feel free to let us know.