Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models - A Survey

We express our gratitude to the reviewer for the thoughtful and constructive feedback on our survey which can “help the researchers [to] quickly go through the progress” in the domain of evaluating LLM-based reasoning behavior. We are especially grateful for the reviewer’s notion that “the practitioners will be interested in this paper and get insights from this paper about the research progress of LLM reasoning”.

We would like to address the reviewer’s main concern regarding the presentation of our work:

We appreciate the reviewer’s suggestion. We have made significant efforts to organize our survey well and present the key findings of the papers we review clearly. Additionally, we devoted considerable effort to the creation of additional supporting figures (see Figure 1 and Figure 2). We will add an additional table outlining the advantages and disadvantages of current evaluation methods (cf. rebuttal to reviewer bLrd), which is a suggestion by several reviewers and we agree will help enhance the presentation of our work.

We thank the reviewer for the thoughtful and constructive feedback.

We are grateful for the reviewer's comments for a “comprehensive literature review” that is “thorough and encompasses most aspects of reasoning behavior evaluation in the context of LLMs”. Furthermore, we are pleased that the reviewer considers our survey “a valuable reference for researchers entering the field or looking to understand the current landscape of reasoning evaluation”.

In the following, we would like to address the reviewer’s main concerns:

1. “The framework for categorizing reasoning tasks is not particularly novel” and “risks overlooking these cross-task similarities, which are crucial for deeper analytical insights.”

• Our taxonomy is motivated by the literature we have reviewed, wherein most works employ tasks designed to capture specific types of reasoning. While similar categorizations might have been employed by existing works (e.g., the HELM paper), we believe that it remains helpful to systematically review literature that evaluates the reasoning behavior of LLMs. We acknowledge the reviewer’s point that similar behaviors might manifest across different tasks. However, we maintain that it is still possible to identify trends with the current taxonomy. For instance, across various task types, “slight alterations in the task context can markedly impair the models’ reasoning, multi-step reasoning is often reduced to shortcut pattern matching, and scenarios that deviate from the training distribution pose substantial challenges for current models” (see Section 3.3).

2. “Key findings from the literature could be more effectively highlighted or presented in a more streamlined manner.”

• We tried our best to review the literature in a coherent manner, aiming not only to list the studies we surveyed but also to connect them through their findings. Nevertheless, acknowledging the diverse backgrounds of our readership, we appreciate your suggestion. We will try to present key findings more effectively in the final version of our paper.

We express our gratitude for the reviewer’s feedback.

We would like to address the reviewer’s main concerns:

We agree that our discussion might be too succinct, and we intend to expand it in the final version of our paper (please refer to our rebuttal to reviewer AG6y). However, we would like to emphasize that our comprehensive review does indeed provide various insights into the models' reasoning processes. For example, slight alterations in the task context can significantly affect the models’ reasoning, multi-step reasoning is often reduced to shortcut pattern matching, and scenarios that differ from the training distribution pose substantial challenges for current models (see Section 3.3). In addition, we find that fundamental conceptual errors have been identified across various studies, which highlight the models’ deficiency in understanding basic principles in logic, mathematics and causal reasoning, especially in counterfactual setups.

Moreover, we advocate for more sophisticated reasoning assessments as presented in Section 4, to better discern and comprehend the essential components missing in LLMs that are crucial for robust and systematic reasoning (see Section 5).

We believe that these insights and recommendations are valuable contributions to the advancement of the research community.

We thank the reviewer for the insightful feedback.

We appreciate the recognition of the “impressive number of recent papers” covered in our survey and are encouraged that these papers are summarized “clearly.” Additionally, we are pleased the reviewer highlights some of our paper’s “insights on top of presenting a summary.”

Due to character limitations, we will address the reviewer's main concerns and respond to specific comments in a separate rebuttal.

1. Reasoning behavior vs. task performance:

• We agree that for more sophisticated task designs, conclusions about the model’s reasoning behavior can be drawn based on the model’s task performance, particularly when tasks are designed to track specific types of errors. In Section 4.1, we note that this: “outcome-oriented approach […] can nonetheless provide insights into the model’s reasoning behavior. For instance, a thorough error analysis can unveil conceptual errors”. We would like to clarify that our critique is directed at benchmarks with less nuanced task designs where accuracy metrics provide a less informative or even misleading assessment of the models’ reasoning [1]. Thanks for raising this point as it helps us to clarify our stance.

2. Categorization of reasoning tasks:

• First, we would like to emphasize that the tasks and abilities listed in our paper are not exhaustive, as noted in the caption of Figure 1. Our taxonomy is motivated by the literature we have reviewed, wherein most works employ tasks designed to capture a particular type of reasoning. Although these tasks may invoke more than one reasoning skill, this taxonomy helps us to systematically review the reasoning behavior of LLMs within the context of such tasks.

3. Robustness of LLM-based reasoning:

• We agree with the reviewer that our discussion is concise, and we plan to extend it in the final version of our paper. One significant challenge in evaluating the reasoning capacities of LLMs is the closed-source nature of their training data. Various studies indicate that current LLMs are significantly affected by benchmark leakage [1,2]. This underscores the need for more nuanced evaluations that move beyond potentially misleading accuracy values. While LLMs can generalize to unseen samples, our review highlights several studies that reveal conceptual errors and limitations in novel reasoning scenarios, which account for data leakage

[1]: https://aclanthology.org/2024.eacl-long.5

[2]: https://doi.org/10.48550/arXiv.2311.01964

Thanks for the response!

The reversal curse indicates that models struggle to capture a basic symmetry property of the identity relation, instead learning a more superficial, unidirectional relationship. As noted by the authors, this is “a basic failure of logical deduction in the LLM’s training process”, which could extend to other relationships that are essential for effective reasoning such as “logical implications [...], spatial relationships [...], or n-place relations.” (see Section 1 and 4.1 of Berglund et al. 2024)

I disagree. The reversal curse does not happen in context: models are perfectly capably of inferring "Joe Biden is US president" when prompted with "US president is Joe Biden"; reversing relations like this is not a problem. They just struggle to do that during in-weights learning. This suggests it's a failure of knowledge storage/retrieval (when a "A is B" is stored in one direction, models fail to retrieve A when conditioned on B) not of logical deduction.

The first passage you cited is a hypothetical that is debunked two sentences later: "Moreover, this is not explained by the LLM not understanding logical deduction." The second cited passage is also a question, not a claim.

The limited number of tasks in McKenzie et al. (2023) does not invalidate the authors' finding that LLMs replicate certain types of human-like logical errors present in their training data.

They sometimes replicate human errors. I think asserting this without a "sometimes" is a claim the McKenzie et al. don't provide sufficient evidence for.

The term "goal misgeneralization" is used by McKenzie et al. (2023) and by us in the same context as defined in reference [4]. Could the reviewer please provide further clarification?

You're right!

We thank the reviewer for the constructive and favorable feedback.

We highly appreciate that the reviewer acknowledges our efforts to present a “comprehensive review of literature that evaluates LLM-based reasoning beyond mere task accuracy”, which “provides a clear picture” and is “helpful to the researchers in this community”. Furthermore, we are encouraged that the reviewer finds that “the two figures in the paper are of great help to understand the situation”.

We would like to address the reviewer’s main concerns:

1. Advantages and disadvantages of each evaluation framework:

• We agree that an overview of the advantages and disadvantages for each evaluation framework would benefit the research community, and acknowledge that this is currently missing in our survey. Therefore, we have created a respective overview table (please refer to our rebuttal to reviewer bLrd), and are happy to include it in the final version of our paper. If you have further suggestions on the summary above please let us know.

2. Scope of the conference:

• The conference's FAQ page indicates: "We don't have a special track for survey papers. If you wish, you can submit a survey paper and we will route it to the appropriate area chairs and reviewers and based on their judgment, you will get a decision". Based on this information, along with various announcements on Twitter/X (cf. this post), we are confident that COLM welcomes survey papers.

In response to the reviewer's question, could the reviewer kindly provide further clarification? Is the question targeted at how LLMs behave in reasoning scenarios when prompted via in-context learning?

We thank the reviewer for the thoughtful feedback.

Firstly, we wish to express our gratitude for acknowledging our efforts in providing a “well-researched summary”, which captures “a good breadth” of literature, “and serves as a good overview of the reasoning abilities and deficiencies" of current LLMs. We are encouraged that the reviewer finds our writing “clear and easy to follow”, and are grateful for the recognition of our work as a “noteworthy contribution to a community that is contending with an open question of how much reasoning abilities truly emerge at scale”.

In response to the reviewer’s questions

1. How reasoning behavior is induced:

• Our survey focuses on studies that evaluate the reasoning behavior of LLMs beyond mere task accuracy. We agree with the reviewer on the importance of linking observed reasoning behavior with the employed reasoning techniques. Consequently, we always indicate when a technique other than standard prompting is used. However, we believe that providing an overview of reasoning techniques and their effects on model reasoning would expand the scope of our survey. Additionally, existing surveys already offer such overviews. For reference, please see [1,2] as mentioned in Section 1 of our paper.

2. Trade-offs for each evaluation method:

• We agree and thank you for this suggestion. Below, we summarize the main trade-offs for each evaluation method. We intend to include an elaborate comparison in the final version of our paper.