Large Language Models Are Not Strong Abstract Reasoners

Thank you for your constructive feedback!

Similar conclusion has been explored by previous studies [1][2]. [1] "Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks." arXiv preprint arXiv:2307.02477 (2023). [2] "Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners." arXiv preprint arXiv:2305.14825 (2023)

Yes, some papers have recently attempted to tackle similar problems than ours. Our study has been independently conducted and released at the same time as [1,2].

Regarding [1], the authors measure counterfactual tasks, defined as tasks close from one in the training data but with one uncommon setting (e.g. arithmetic in base 9). While this is related to our work, this is different from a measure of the ability of LLMs to extract and reason on abstract patterns. We also provide an additional analysis of how using different prompting strategies, text/symbolic formats, open-ended or multiple-choices affect the models' performance, and what kind of tasks are the most affected.

Regarding [2], the authors measure the logical reasoning abilities of LLMs via inductive, deductive and abductive reasoning tasks. While this is related to our work, the paper measures an ability different from abstract reasoning. The complexity in logical reasoning tasks lies in the correct application of logical rules. In constrast, the challenge in abstract reasoning is to identify a high-level unknown pattern from a few examples. In our work, we also provide an additional analysis using different prompting strategies and explain what kind of tasks are the most affected.

We will add the discussion on these two papers and the differences with our approach to our related work.

Lack of experiment with larger models or advanced models. Fine-tuned on smaller models cannot sufficiently draw the conclusion. Can you experiment with more advanced models Llama-2, with better performance than Llama1, Alpaca, or fine-tune with larger models (13B, 70B)?

Yes this is a good remark, we plan to add results with the latest LLaMA2 to our paper. In our initial experiments, we used the models that were the state-of-the-art at the time and we also included in the Table 9 of appendix B.3 results for LLaMA models of larger size. We have now performed new experiments using the latest LLaMA2. Here are the results we obtained:

Fine-tuning models is a good direction but the abstract reasoning abilities of fine-tuned models are challenging to evaluate due to their memorisation ability. This is why we focused on in-context learning tasks that have not been seen during training. Per your request, we have performed additional experiments on fine-tuned LLaMA and LLaMA2 models. This remark was also raised by reviewer LbQe so we put the results in a general comment at the top. For cost reasons, we have not performed fine-tuning experiments on bigger versions of LLaMA.

The details of fine-tuning experiments, such as training data, training steps.

For our fine-tuning experiments, we indicate the training set in the corresponding tables. Unless indicated otherwise, we always use a training split from the same task that is evaluated. We also evaluate the fine-tuned models on a different format (symbolic or text) than the one they have been trained on to assess if they learned abstract patterns. The training split always contains different samples than the ones used for evaluation. We fine-tune for 3 epochs on batches of size 64 with learning rate 0.0005.

Do you consider incorporating the instruction about “how to induce”, “how to deduce” into supervision?

This is an interesting direction! The main challenge in our tasks is to recognise an unknown general pattern from a few examples, which is closer to abductive reasoning, so our instructions focus on the generation of this abstract pattern. We performed a few experiments in this direction in Appendix B.3 with MERIt, a model fine-tuned on logical reasoning tasks. The question of whether a model tailored for deductive or inductive reasoning can perform abstract reasoning is interesting but it is not the focus of this paper. We will consider it in our future work.

Thank you for your constructive feedback!

This task will be automatically solved when models of better reasoning capabilities become available.

Our aim in this paper is to highlight current important limitations of Large Language Models to raise awareness in the research community and explore leads to mitigate them. We believe our benchmark can be used to monitor the progress of LLMs on reasoning tasks.

The authors frame abstract reasoning as "a potential task for effective measurement of the cognitive abilities of neural models", so the utility of this benchmark is mostly evaluation of LLMs. One concern is that there isn't an actual application that would benefit from studying this kind of reasoning capabilities.

We aim to measure the abilities of LLMs to reason abstractly and causally, i.e. perform reasoning on symbols invariant or robust to distribution changes. This is critical for strong generalisation and use in the real-world. Although this is not tailored to a specific application, this ability is crucial for many downstream tasks.

Have authors considered fine-tuning? It would be nice to show even fine-tuning Llama2 is not enough for solving the abstract reasoning tasks.

This is an interesting direction! We aim to measure the ability of a model to build abstraction and reason on top of it but this ability is hard to measure due to the memorisation ability of LLMs. This is why we focused on in-context learning tasks that have not been seen during training. Per your request, we have performed additional experiments on fine-tuned LLaMA and LLaMA2 models. This remark was also raised by reviewer As6m so we put the results in a general comment at the top.

Curious to see how zephyr-7b-beta (https://huggingface.co/HuggingFaceH4/zephyr-7b-beta) performs on the proposed benchmark.

This is an interesting question, the Zephyr model was not yet released when we conducted our initial experiments. Per your request, we tested it on our benchmark. Here are the results we obtained:

How is open-ended QA evaluated?

For the open-ended QA tasks, we ask the LLMs to generate the expected output pattern. We use regular expressions to catch the pattern in the model's answer as it may not follow our instructions and wrap its answer in some text (e.g. reasoning justifications or "The answer is ..."). We consider the answer correct even if the format of the answer diverges from the instructions. Before the evaluation, we performed multiple trials with each model to capture all the possible formats they used and we included them into our regular expressions. We did sanity checks to ensure we did not miss any formats they could use.

Do the authors have plans to maintain a leaderboard for this task? Will there be a held out test set? What is the data releasing plan for this benchmark?

This is a good suggestion! We did not plan to create a leaderboard but we will look into the idea and see if its is feasible. We currently have no heldout test set but we will consider creating one. We currently plan to release our code and data in open-source along with the paper to ease reproducibility and evaluation with new models.

Also curious about human performance on this benchmark. For example, I couldn't figure out the example in Figure 6.

Sorry for the confusion, Figures 5 and 6 are the same example in text and symbolic formats. We will clarify this point in the paper. This example illustrates a backward-blocking case. From the input examples, we aim to determine if the red sphere causes the light activation. The red sphere is never observed alone in the examples, it is only seen with the yellow cube, so we cannot determine which one causes the light activation. So from the data, we cannot determine if the light should be on or off in the test case.

Thank you for your constructive feedback!

This new benchmark introduced are largely existing datasets thus with limited novelties. There are also existing works on evaluating the inductive reasoning ability of LLMs such as https://arxiv.org/pdf/2306.09841.pdf. ([1])

Our study has been independently conducted and released at the same time as [1]. The authors measure the logical reasoning abilities of LLMs on standard benchmarks. While this is related to our work, the paper and the datasets used measure an ability different from abstract reasoning. The complexity in logical reasoning tasks lies in the correct application and extraction of logical rules. In constrast, the challenge in abstract reasoning is to identify a high-level unknown pattern from a few examples. In our work, we also provide an additional analysis using different prompting strategies and explain what kind of tasks are the most affected based on multiple factors (e.g. input dimensionality, causal reasoning needed). We also include in our experiments GPT-4, LLaMA, and LLaMA2 in the latest version. We will add the discussion on this paper and the differences with our approach to our related work.

[1] Xu, Fangzhi, et al. "Are Large Language Models Really Good Logical Reasoners? A Comprehensive Evaluation From Deductive, Inductive and Abductive Views." arXiv preprint arXiv:2306.09841 (2023).

This paper does not evaluate slightly more complicated prompting methods, such as simply generating more samples of code and filter by number of training examples passed.

In our early experiments, we found that LLMs struggle to extract the correct abstract pattern and refinement techniques that ask the model to correct its answer did not solve this initial issue. Therefore, we focused our work on prompting techniques that have been shown to significantly improve the performance of LLMs, from their first response onwards, such as chain-of-thought and program-of-thought. Per your request, we will perform new experiments with code refinement. Given the short amount of time, we will do our best to post our results in a comment before the end of the discussion period. In all cases, we will integrate the results to the paper.

Please, let us know if this answers your concerns.