LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models

Thanks for acknowledging that our new evaluation reveals certain limitations of existing LLMs!

Task-specific training

There are various open-source libraries to train reward models (RMs), e.g., Transformers, TRL, etc. Our library focuses on inference-time algorithms and is orthogonal to them. LLM Reasoners allows for flexible implementation of the reward function, including using a trained RM, and users can easily call any algorithms to search for high-reward reasoning chains. In the future, we plan to support RM training natively with our library.

70% incorrect reasoning chains

AutoRace recognizes 70% of the false positive reasoning chains. These are more challenging to identify than usual incorrect chains, because their errors are often subtle, and don’t affect the final answer. With traditional answer-based evaluation, NONE of these chains would be detected. Discovering 70% of them already led to multiple novel findings (Section 5.2)

To further clarify, AutoRace can detect ~90% of all incorrect reasoning chains across 6 datasets (Figure 3), and we have shown it has higher accuracy and better robustness compared to various baselines with a comprehensive evaluation in Section 3.

World model

A world model $P_\theta$ predicts state transition: $s_t \sim P_\theta (s_t | s_{t−1}, a_t)$. LLM Reasoners allows users to flexibly implement the world model, e.g., For RAP on the embodied task Blocksworld, the world model is implemented as a function that calls an LLM to predict the new arrangement of blocks after an action.

Generalization

We test AutoRace with the criterion lists of GSM8k on all tasks. We also test SocREval on all tasks with its prompt for GSM8k. AutoRace shows a high degree of generalization. We found that it’s because many of the criteria discovered by AutoRace are general among different reasoning tasks.

Error collection and example set

Due to the word limit, please refer to our response to reviewer SuhS for additional experiment results.

Thanks again for your thoughtful review! We will update our paper to reflect the additional results and discussion.

Thank you for recognizing that AutoRace is fully automatic and performant, LLM reasoning is a good principle, and the experiments suggest intriguing observations about LLM reasoning!

Novelty

We’d like to highlight 3 novel contributions of this work:

• We propose a novel reasoning chain evaluation method and show the promise to be an important metric complementing answer-based evaluation.
• We present a novel perspective to unify numerous current reasoning methods, and build a library for developing, deploying, and analyzing reasoning algorithms.
• Our comprehensive experiments lead to several novel findings about LLM reasoning:
  • Reward-guided search reduces false positives
  • Search breadth matters more than depth
  • World model is crucial for embodied tasks
  • Inappropriate prompt format may lead to false positives

Reliance on training set

While we understand this is a reasonable concern, we argue that most reasoning datasets already include a training set, which eliminates the need for specific construction for AutoRace. Furthermore, as AutoRace can be effective with a very small demonstration set, which we will show below, even if the training set is not available, it’s easy to find or create a few demonstrations and apply AutoRace.

For each task in Table 1, we use only 4 wrong reasoning chains to create a criterion list. Here, we further explored different sample sizes (2, 4, 10).

The results indicate that AutoRace is relatively robust to the sample set size. We found that it’s because:

• Besides concrete errors in examples, GPT-4 can also infer some evaluation criteria from the problem domain. E.g., For math word problems, even if there is no calculation error in the examples, GPT-4 may supplement a criterion about “accurate calculation”.
• GPT-4 has certain prior knowledge on how to evaluate a reasoning chain. Thus, even if a criterion list misses some items, GPT-4 still has a chance to correctly evaluate a reasoning chain with that type of error.

Additionally, our tests confirm that the criteria list generalizes well across different reasoning tasks (See new results in our response to reviewer bv3g). This further reduces the dependency on task-specific demonstration data.

Thanks for acknowledging that AutoRace is an interesting evaluation framework, and LLM Reasoners nicely formulated reasoning problems under a common framework.

Steps taken in Autorace

We have described the steps of implementing AutoRace in Section 3.1, along with a concrete running example for illustration (Figure 2). Briefly, it first creates a criterion list for a reasoning task automatically, by (1) collecting a list of wrong reasoning chains, (2) asking GPT-4 to elaborate the errors in them (3) summarizing the concrete errors into a criterion list. Finally, we apply the criterion list to evaluate new reasoning chains with GPT-4.

Empirical findings most known

We respectfully disagree with this claim. In Section 5, we have presented several novel findings:

1. Reasoning as reward-guided search helps not only improve final accuracy, but also effectively alleviate false-positive reasoning chains;
2. For efficient search in the reasoning space, the breadth of search is generally more important than the depth for most tasks;
3. incorporating a world model that explicitly infers reasoning state would effectively improve the LLM reasoning ability, particularly for tasks in embodied environments;
4. inappropriate prompt format design might inadvertently lead to false positive reasoning chains.

These findings are from our comprehensive analysis of reasoning methods, which rely on our new reasoning chain evaluation method and the unified library. To the best of our knowledge, these findings haven’t been discussed in previous studies.

No comparison & Accuracy of AutoRace

In Section 3, we have compared AutoRace to four baseline methods for reasoning chain evaluation. In Table 1, we show that AutoRace can achieve 86% accuracy averaged on 6 human-annotated datasets, while the best baseline can only achieve 82%. It’s also the only method that performs within 5% of the best results across all six datasets, showing outstanding robustness. We have made a more detailed comparison with Socreval [1], the strongest baseline, by visualizing the confusion matrix (Figure 3), conducting case studies (Figure 6), and discussing the source of performance improvement (Section 3.2).

We hope this response adequately addresses your concerns. We are eager to engage in discussions and welcome any additional questions you may have.

[1] He et al., 2023, “Large language models with the Socratic method for reference-free reasoning evaluation”

Thanks for your encouraging comments that AutoRace is performant and can be a key to standardizing LLM evaluation!

Pseudocode

We attach the pseudocode of the criterion list construction to this link due to the formatting issue of OpenReivew. After the criterion list is constructed, it's straightforward to prompt GPT-4 with it to evaluate new reasoning chains.

Consistency of Criteria list

The criteria list is generated by GPT-4. To ensure the consistency of the criterion list, we divide criteria list construction into three easier subtasks for GPT-4: (1) Collecting wrong reasoning chains (2) Detecting the errors (3) Summarizing the criteria list.

Besides, our manual examination reveals that the constructed criterion lists for all tasks look reasonable (See an example in Appendix D.2). Finally, the comparative experiments in Section 3 demonstrate that AutoRace consistently outperforms established baselines, affirming the reliability of the generated criterion lists.

Case study

We have listed several successful cases of AutoRace in Table 6 and Appendix G.We conduct additional case studies and collect several failure examples (link). Errors made by AutoRace can be mainly attributed to (1) Incorrect understanding of questions (2) Wrong calculation (3) Being too strict on some corner cases.

Bias of LLM evaluator

In our experiments, we observed that while LLM evaluators are effective at identifying reasoning errors, they sometimes tend to misclassify correct reasoning chains as erroneous (refer to the confusion matrix in Figure 3). Compared to the previous LLM evaluation method SocREval [1], AutoRace exhibits less bias. This improvement is attributed to the fundamental difference in their evaluation processes. SocREval relies on the evaluator LLM to first correctly solve the problem. Once the evaluator LLM itself fails to do so, it will misclassify correct reasoning chains as wrong. In contrast, AutoRace is not constrained by the evaluator’s problem-solving capabilities. Instead, it assesses the reasoning chain based on established criteria, simplifying the evaluation task.

Thanks again for your review and suggestion. We will update our paper with the pseudo-code and case study.

[1] He et al., 2023, “Large language models with the Socratic method for reference-free reasoning evaluation”