Make LLMs better zero-shot reasoners: structure-oriented autonomous reasoning

We thank the reviewer for reviewing our manuscript and providing useful suggestions. Below are our responses:

1. [W1 and Q1] PGM-based analysis

We appreciate the reviewer sharing the comments about Section 3.2. As mentioned in the global response, since we have included many new contents in the revision (e.g., related works, new experiment results, new remarks), we have together compressed Section 3.2 in the main content and moved its main parts to Appendix A. We hope this adjustment can better highlight our main focus on the structure-oriented analysis and improve the reading experience of our paper.

In terms of the target of Section 3.2, we agree that the theory does not exactly explain the benefit of the structure-oriented analysis (i.e., how the analysis helps the LLM catch the critical states from a transformer mechanism perspective). But rather, it explains the benefit if an algorithm can successfully identify and reach the correct reasoning path. It is applicable to any prompting/reasoning techniques and is the upper bound of the potential benefit for any method.

2. [W2.1] Additional experiments on diverse tasks and datasets

Thanks for your comment, we have updated some additional experiments for different tasks and datasets, and the results are as follows (copied from the global response):

In short, our method performs the best in GSM8K and StrategyQA compared to other methods and achieves a good performance in MATH. We conjecture that the structure-oriented analysis by its nature focuses more on the language, thus its performance in MATH is not as superior as in the other tasks. It is an interesting direction to improve the LLM’s ability to understand symbol representations.

3. [W2.2] Compare without retrieval (elaborate more on experimental setting)

We would like to clarify our experiment settings and baseline methods. Both ReAct and CoK leverage external knowledge to enhance reasoning. In Section 5.1, when we introduce ReAct and CoK in the paragraph of Baseline, we clearly mention that they adopt Wiki API and knowledge from different domains during reasoning. Therefore, it is a fair comparison with ReAct and CoK.

4. [Q4] Elaborate on novelty

We thank the reviewer for raising this concern. We would like to clarify our major contributions and emphasize why we are different from existing works.

The main scientific contribution of this paper is our observation that the zero-shot reasoning ability of LLMs is not fully explored (as shown in Section 3.1 in the main text). We propose a principal strategy, ‘structure-oriented analysis,’ to activate the self-thinking ability of LLMs and significantly improve the zero-shot reasoning ability, which can reduce the gap between zero-shot reasoning and few-shot reasoning to a large extent. This is different from [1] and [2] mentioned by the reviewer, in which they focus on the design of the agent system rather than improving the capability of the individual LLM.

We further provide theoretical analysis in Section 3.2 to support our strategy. Our findings are different from previous works, as mentioned by the reviewer, which mainly rely on post-reasoning feedback, including internal feedback, such as self-correction, and external feedback, such as multi-agent debate.

Given the observations in Section 3, the proposed agent system in Section 4 is then an additional contribution that provides a more comprehensive and practical solution for the structure-oriented analysis. As mentioned in Section 4.1, Refine Agent is designed to ensure the guidance of structure analysis is well followed; Retrieve Agent is designed to obtain external knowledge to better solve sub-questions from Reason Agent; Memory Agent is designed to track structure analysis and reasoning trajectory. Therefore, the whole design of our agent system is based on our principal strategy, which is different from that of existing agents. The components or architectures of the agent may be similar to existing works, but the main purpose and logic of design is to serve our principal strategy better. We believe our findings and proposed strategy are novel and necessary to understand and unleash the LLM’s reasoning capability.

We have revised the introduction to clarify our main contribution.

Besides, we include a discussion on the agent works mentioned by the reviewer and compare them with ours. [1] introduces an LLM multi-agent framework including an LLM Decomposer, LLM planner, and LLM interface to conduct tasks and interact with the environment in Minecraft. [2] categorizes most of existing LLM agents via information storage, action space and decision-making procedure. [3] focuses on tool-use of LLMs and trains a series of models with enhanced ability of tool-use. [4] proposes an agent system to implement Monte Carlo Tree Search with the help of few-shot examples. We acknowledge that existing works have adopted components like reasoning core, refinement, memory, external knowledge, etc in the agent system.

We do not claim that we develop novel components or structures in agent design. In fact, we adopt the basic components and structures to build a multi-agent system, and our goal is to utilize agents to fully illustrate the effectiveness of the proposed strategy, and structure analysis, rather than proposing a novel agent system.

[1] Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory

[2] Cognitive Architectures for Language Agents

[3] ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving

[4] Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models

We appreciate the reviewer in providing the constructive suggestions. Below are the responses for your comments:

1. [W1, Q1, Q2] Additional experiments on diverse tasks and datasets

The additional experiments as in the following table (copied from the global response):

Our primary goal is to explore the limit of zero-shot reasoning capability of LLMs, so we aim to improve the general reasoning performance. According to our results, our method indeed generalizes well on diverse reasoning tasks and datasets, not limited to knowledge-intensive tasks. We also identify some limitations such as symbolic reasoning, which we aim to improve as a future direction.

2. [W2, Q3] Discussion of computation cost

The computational cost is summarized as follows (copied from the global response):

In short, our method is affordable compared with baselines and achieves a good balance between cost of tokens and effectiveness

3. [W3] Discussion of PGM

Thank you for the comment. Our aim for Section 3.2 is to provide a rigorous quantification of the potential benefit when identifying and reaching the correct reasoning steps. While this is an intuitive idea, our results in Section 3.2 provide a detailed theory. Nonetheless, following the comment of Reviewer uEkW and the page limit in revision, we have reorganized Section 3.2 to shorten it in the main content and move most of it to Appendix A. We hope this can better clarify the main logic of the paper and put Section 3.2 as a supplementary result to the new methodology of the structure-oriented analysis.

We would like to clarify that existing works theoretically analyze how few-shot examples benefit the reasoning process assuming the correct reasoning path is achieved (mentioned in Section 3.2). Different from them, we focus on the benefit of the correct reasoning path itself: We formally show how critical steps extracted by structure analysis can benefit LLM to reach the correct reasoning result. This difference distinguishes our unique contribution from other theoretical and empirical works.

We appreciate the reviewer for sharing useful comments to improve our manuscript. We provide the response to your comments as follows:

1. [W1] Additional experiments on diverse tasks and datasets.

The additional experiments as in the following table (copied from the global response):

We notice that our method works well on GSM8K. Since these questions in GSM8K are described in natural language, our strategy can still provide a deeper understanding of the question and lead to the correct solution. On the other hand, MATH dataset contains symbolic reasoning problems and we do notice that SARA does not outperform baselines on this dataset. This may suggest that our strategy does not work well on questions fully expressed with symbols. We highlight this limitation in the revision Section 7, and consider it our future direction to improve.

2. [W2] Elaborate on the evaluation of structure analysis

We thank the reviewer for raising this valuable comment. Since the major goal of this work is to improve the zero-shot reasoning ability of LLMs, we use prediction correctness as the evaluation metric. To confirm that structure analysis indeed helps, we conduct ablations, as shown in Figure 2. While we agree that evaluating how correct the structure-oriented analysis is can provide more details about the proposed method, this evaluation does not directly measure whether the proposed method can benefit reasoning or not. Besides, it is also expensive to develop a dataset for this purpose as it may require extensive human annotation efforts to manually analyze the structure. As a result, we directly test the reasoning performance to align with the main purpose of our work.

3. [Q1] Discussion on the mentioned work.

We thank the reviewer for mentioning this work [1]. This work proposes prompting the LLM to select proper reasoning modules first and then adapting them to each question for better solving. This method combines various reasoning strategies and few-shot examples to improve performance. Our work focuses on the zero-shot reasoning capability of LLMs and the proposed structure analysis is not grounded on reasoning modules. We include this work in the revision (Section 2).

4. [W3, Q2] Elaborate on robustness

We include the section on robustness to show that our method is robust on potential attacks and to support our claim that the structure analysis can extract key concepts and relevant information and filter out irrelevant information. Existing attacks on reasoning usually insert irrelevant and malicious information in the problem or reasoning steps (in few-shot examples) to corrupt the performance. The robustness experiments show that our method can identify the semantic and grammar structures and ignore irrelevant information. These results demonstrate the advantage of the proposed method.

In our experiments, one of the attacks targets zero-shot reasoning (Preemptive attack) while another (Badchain) target few-shot reasoning. We include them for a more comprehensive evaluation, and this does not mean our method is a few-shot method.To adopt Badchain on our method, we simply replace the original problem with the problem attached with the trigger. We have revised our manuscript and updated the clarification in Section 5.6.

In terms of how these attacks work, they assume that the potential attacker provides third-party prompt engineering services and has access to the victim’s prompts. We directly utilize their implementation in our experiments.

5. [W4] Limitation paragraph

Based on your comment, we have updated a new Section 7 to discuss the limitations of the proposed method. Briefly speaking, there are two major limitations. (1) Due to the nature of the structure-oriented analysis, our current method works better on problems that are clearly described in natural languages (e.g., GSM8K) than tasks with pure symbol expressions (e.g., MATH). It can be overcome by extracting the symbolic expressions. (2) Our current agent framework only uses the Retrieve Agent instead of other possible agents with additional tools. Adding more agents can enable the framework doing more tasks, e.g., executing code or a calculator.

(Continued from the previous response)

In short, our method performs the best in GSM8K and StrategyQA compared to other methods and achieves a comparably good performance in MATH. We conjecture that the structure-oriented analysis by its nature focuses more on the language, thus it can better understand the question in GSM8K than the other methods, but its performance in MATH is not as superior as in the other tasks.

For MATH dataset, to explain why SARA still improves performance, first, there are also some problems described in natural language, and SARA can understand the language better. Second, the combination of step-wise reasoning and refinement in SARA also guarantees its effectiveness. Therefore, SARA can still work better than 0-shot methods, and it is an interesting future direction to study how to better handle symbolic representations in math problems.

3. [W3, 4] discussion of computation cost.

The statistics of the computational cost are summarized in the following table (copied from the global response):

In short, our method is affordable compared with baselines, and achieves a good balance between the cost of tokens and effectiveness. We also include zero-shot CoT with self-consistency as an additional zero-shot baseline on all tasks and datasets. Results are updated in Table 1, 2 and Table 8 in Appendix G.

4. [Q2] Clarification on the structure analysis

As mentioned in Section 1, existing works have shown that LLMs show a strong ability to understand and parse semantic and grammar structures in the text. Therefore, we directly leverage this ability in our work. As demonstrated in the new results, this ability generalizes well to math problems that are written in natural language, such as GSM8K dataset, where SARA shows superior performance. However, for some problems written with math symbols, such as some equation-solving problems in MATH dataset, there are few grammar or semantic structures to capture, thus the performance of SARA is not as superior as in other tasks. These results together with our original results indicate that our method generalizes well on problems that are described in natural language, and we agree that it may need some modifications such as additional training to generalize to symbolic problems. We mention this limitation in the revised Section 7 and take it as an interesting future direction.

5. [Q3] Elaborate on memory

Our memory is designed to be instance-specific and store everything in the reasoning process, including structure analysis, reasoning trajectory, and retrieved knowledge. The difference between our method and ReAct is that we have multiple agents, so we need this memory module as a shared message pool with which all agents can interact. Besides, different from ReAct where all previous thoughts and observations are directly fed into the model, we only send part of the memory into the agent. For example, the Refine Agent in SARA only needs one previous step and corresponding knowledge. These highlight the necessity of the memory module and the difference between our design and ReAct.

6. [Q4] Elaborate on metric EM

We notice that, unlike Fever and MMLU, whose answer is covered by some options (like True/False, A/B/C/D), the model-generated answer for HotpotQA is usually in free-form, such as ‘George Archainbaud died first than Ralph Murphy’ and ‘Pirate's Cove was published more recently’ and hard to extract the answer from them using some general hard rules. Thus, using EM can cause false positives and false negatives, especially for zero-shot methods (Vanilla, SARA), where no examples are provided to regularize the form of the final answer. Therefore, we leverage LLMs to compare the generated and ground-truth answers.

We thank the reviewer for reviewing our manuscript. Below are the responses and clarifications towards your comments:

1. [W1] Discussion on related works

Thanks for sharing the list of related works. We have checked these works. Below is a summary of the comparisons with them.

[1] proposes SOCRATIC COT, which trains a generator to decompose the question and a question-answering model to solve these sub-questions. Their method is designed to utilize CoT examples generated by the large models to improve the reasoning ability of small models. Therefore, their method has a different scope as we focus on a general strategy for zero-shot reasoning capability. Besides, SOCRATIC COT requires additional training.

[2] proposes the least-to-most prompting technique to query the model to decompose the problem first and then solve sub-questions. However, they rely on few-shot examples and could be considered as a few-shot reasoning method, while we focus on the zero-shot method and leverage the model’s ability to analyze semantic and grammar structures.

[3] has a similar strategy to [1] and proposes a decomposer to decompose the problem into sub-tasks. However, it also relies on few-shot examples in decomposing and sub-task solving, which differs from this work’s focus.

Similarly, [4] also decomposes the task first using a planner, which is powered by an LLM, and then utilizes executors to solve these tasks. This framework, ADAPT, also requires task-specific examples to help decompose and execute, as shown in its appendix.

[5] proposes a modular framework, SCREWS, to enhance reasoning with revisions. SCREWS will first generate an initial output and then a refined output. Two outputs will be sent to the selection module to choose the correct one. Their module design can accommodate existing methods like CoT and self-consistency. However, our primary goal is not to propose a reasoning system but to explore the limit of the zero-shot reasoning capability of LLMs. So, we observe and propose the structure analysis to activate the self-thinking ability of LLMs and enable them to automatically reason on questions without task-specific prompting or examples.

[6] proposes a refinement strategy (ask, refine, trust) to reduce the potential error during the reasoning process. While we adopt a similar strategy by leveraging a Refine Agent to evaluate and refine the reasoning step, we aim to ensure the reasoning trajectory follows the structure analysis as we mentioned in Section 4.1. We do not claim the design of Refine Agent to be our novel contribution, but it is a necessary strategy to unleash the power of structure analysis fully, which is the core of this work.

[7] proposes prompting the LLM to select proper reasoning modules first and then adapting them to each question for better solving. This method combines various existing reasoning strategies and few-shot examples to improve performance. However, we primarily focus on exploring the limit of zero-shot reasoning capability rather than proposing a reasoning system.

Due to page limit, we summarized the above and updated a shorter version in Section 2.

[1] Distilling Reasoning Capabilities into Smaller Language Models.

[2] Least-to-Most Prompting Enables Complex Reasoning in Large Language Models

[3] Decomposed Prompting: A Modular Approach for Solving Complex Tasks

[4] ADaPT: As-Needed Decomposition and Planning with Language Models

[5] SCREWS: A Modular Framework for Reasoning with Revisions

[6] The ART of LLM Refinement: Ask, Refine, and Trust

[7] Self-Discover: Large Language Models Self-Compose Reasoning Structures

2. [W2, Q1] Diverse tasks and datasets

Thanks for your comment, we have updated some additional experiments for different tasks and datasets as in the following table (copied from the global response):

(To be continued in the following response)

We appreciate the reviewer in providing constructive suggestions and useful comments. In addition to the new results and updates in the revision, below are the itemized responses to your comments:

1. [W1] Additional ablation experiments

We thank the reviewer for this suggestion. In our paper, we presented the ablation study regarding components in the structure-oriented analysis and the combinations of agents in SARA. Those experiments can be considered as extreme cases where the LLMs are entirely removed from the corresponding parts (structure-oriented analysis components and agents in SARA). We conjecture that replacing the LLM in those parts should lead to performance between the full settings (best case) and the one of which the corresponding part is removed in the ablation study (worst case).

To verify our conjecture, we are conducting experiments that a stronger model (GPT-4) is replaced by a smaller model (Qwen2-57B) for agents to illustrate the influence of models. We will update the results once finished.

2. [W2] Elaborate on novelty

The key contributions of our work are around the theme of structure-oriented autonomous (zero-shot) reasoning, which provides insights into general mechanism design and theoretical support for LLM-based reasoning, and guides our multi-agent system design. We would like to clarify our major contributions and emphasize why we are different from existing works.

• We observe that the zero-shot reasoning ability of LLMs is not fully explored (as shown in Section 3.1 in the main text). Therefore, the first contribution is a principal strategy, ‘structure-oriented analysis,’ to activate the self-thinking ability of LLMs and significantly improve the zero-shot reasoning ability, which can reduce the gap between zero-shot reasoning and few-shot reasoning to a large extent.
• In Section 3, a second contribution in our work is that we further provide theoretical analysis in Section 3.2 to support our strategy. Our findings are different from previous works, as mentioned by the reviewer, which mainly rely on post-reasoning feedback, including internal feedback, such as self-correction, and external feedback, such as multi-agent debate.
• Given the observations in Section 3, the proposed agent system in Section 4 is then a third contribution that provides a more comprehensive and practical solution for the structure-oriented analysis. As mentioned in Section 4.1, Refine Agent is designed to ensure the guidance of structure analysis is well followed; Retrieve Agent is designed to obtain external knowledge to better solve sub-questions from Reason Agent; Memory Agent is designed to track structure analysis and reasoning trajectory. Therefore, the whole design of our agent system is based on our principal strategy, which is different from that of existing agents. The components or architectures of the agent may be similar to existing works, but the main purpose and logic of design is to serve our principal strategy better. We believe our findings and proposed strategy are novel and necessary to understand and unleash the LLM’s reasoning capability.

To clarify the novelty and major contribution of this paper, we revised the last paragraph of Section 1 on Page 2 to highlight our scientific contribution of the structure-oriented analysis.

We are grateful for the reviewer's valuable comments. Since the rest time of the discussion period is limited now, please let us know if you have other concerns. We are looking forward to hearing back from you.

We thank the reviewer for the valuable comments and suggestions. We would like to address them as follows:

1. [W1] Additional experiments on more models

We include two open-source models: Mixtral-8*7B and GLM-4-9B-chat, and test them on three datasets from three types of reasoning tasks. Results are shown in the following table.

According to these results, our proposed method consistently outperforms baselines, suggesting a good generalization among models. We also include these results in the revision Appendix F.2.

2. [W2,3,4] Additional experiments on diverse tasks and datasets, discussion about computation cost

Thanks for your comment. We have updated some additional experiments for different tasks and datasets as well as the computation cost as follows (same tables as the global response).

In short, our method performs the best in GSM8K and StrategyQA compared to other methods and achieves a comparably good performance in MATH. We conjecture that the structure-oriented analysis by its nature focuses more on the language, thus its performance in MATH is not as superior as in the other tasks. For the computational cost, our method has fewer input tokens than few-shot methods and generates fewer tokens than 0-shot methods, and is affordable considering the overall cost.

3. [W5] Discussion on related works

We have updated Section 2 (Related Work) in the revision to add more literature about zero-shot reasoning and multi-agent systems. To name a few, we added [1,2,3,4]. We have also discussed OpenAI-o1 [5] in the revision.

[1] ​​Distilling Reasoning Capabilities into Smaller Language Models.

[2] Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory

[3] ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving

[4] Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models

[5] Evaluation of OpenAI o1: Opportunities and Challenges of AGI

4. [W6] Discussion on limitation

We have updated Section 7 to discuss the limitations of the proposed method. Briefly speaking, due to the nature of the structure-oriented analysis, our current method works better on problems that are clearly described in natural languages (e.g., GSM8K) than problems with pure symbol expressions (e.g., some problems in MATH). It is an interesting direction to enable models to effectively extract structures within the symbolic expressions. Besides, some specific components such as tool-agent involving tools like code executor or calculator can also be included to overcome such limitations to further improve the symbolic reasoning performance.

Thank you to the authors for thoroughly addressing my concerns. Based on your detailed responses, I am raising my score to 6.

Below is a summary of the major changes in our revision:

• Experiments (Reviewer 4YQ2, 57Cm, i9ee, 8je1, N64Z, uEkW):

  • The news results on GSM8K, MATH, and strategyQA have been updated in Table 1 and 2.
  • We added the Zero-shot CoT-SC@10 as an additional baseline in our experiments.
  • The computation overhead has been updated in Appendix G.
  • New results in additional open-source models (Mixtral and GLM), in Table 7, Appendix F.2.

• Limitation (Reviewer 4YQ2, 57Cm, 8je1, N64Z): We have updated Section 7 to discuss the limitations of the proposed method. In summary, due to the nature of the structure-oriented analysis, SARA works better on problems that are clearly described in natural languages (e.g., GSM8K) than problems with symbolic expressions (e.g., some problems in MATH). An interesting future direction is to explore ways to enable models to effectively extract structures within the symbolic expressions. Besides, some specific components such as tool-agent involving tools like code executor or calculator can also be included to overcome such limitations to further improve the symbolic reasoning performance.

• Theoretical results in Section 3.2: Since we have included additional experimental results in our revision and Reviewer uEkW also commented that our theoretical insights in Section 3.2 can be improved in terms of presentation, we have re-arranged Section 3.2 and moved its detailed and formal theoretical analysis to Appendix A. In the revision, Section 3.2 is only 1 page and provides the basic intuition and an informal theoretical statement. We hope this revision improves the clarity and logical flow of this paper.

• Additional related works (Reviewer 4YQ2, i9ee): We updated Section 2 (Related Work) to incorporate the suggestions of the reviewers. A more detailed related work comparison can be found in the response to Reviewer 4YQ2 and i9ee.

• We also added clarifications in the revision to clarify some comments of reviewers, e.g., the novelty of this paper (Reviewer 57Cm) in Section 1 and knowledge conflict of the Retrieval Agent (Reviewer 4YQ2) in Section 4.2.