Thank you for reviewing our work. Here are our responses to your concerns:

For Weakness 1 and Question 1

why do the LLM gets two different “New condition” (one correct, one incorrect) under the same prompt (prompt 2)?

Thanks for the question! In MACM, the conversation takes place in multiple rounds, with previous dialogue information added to the conversation history. This means that in Figure 3, after the first prompt 2 generates an incorrect new condition, the judge's evaluation of this new condition will also be recorded in the conversation history (the leftmost prompt 3 and prompt 4 in Figure 3). This reduces the likelihood of the model considering incorrect conditions in subsequent sampling processes.

Does this mean that thinker in this paper is more likely to take advantage of the randomness of large models in repeated condition generation, rather than really improving the ability of LLMs to discover new conditions?

MACM includes various thought-guiding processes that help the LLM continuously correct potential past errors and guide the large model to gradually use existing conditions to obtain the correct new conditions, ultimately finding the correct result. In addition, In the process of the judge evaluating each new condition, we have configured the use of a code interpreter. This can significantly reduce potential errors and gradually guide the model towards producing correct answers.

For Weakness 2 and Question 2

In the theoretical analysis section, our aim is to highlight the advantages of MACM over ToT and GoT in ideal situations. Specifically, each condition search in MACM can be viewed as a new start, meaning it doesn't need to begin its search from a few initial paths like ToT and GoT. Therefore, theoretically, as the search time increases, MACM's exploration space will gradually approach the thought space $T$

In contrast, ToT and GoT are constrained to specific trajectories, making it difficult for them to consider alternative perspectives. Thus, MACM offers greater flexibility and potential in exploring and traversing the thought space, which can enhance problem-solving effectiveness.

For Question 3

Thank you for your suggestion. We have standardized our testing by using GPT-4-1106-preview to re-evaluate the results for the 24-point game, with all code-interpreter functionalities disabled during the testing process. The testing code for IO/CoT/ToT is sourced from ToT's official GitHub repository.

Experimental results show that MACM still outperforms ToT. By analyzing the error cases in ToT, we found that many problems that MACM can solve but ToT cannot are due to MACM's judge correcting these errors.

For Question 4

Thanks for your suggestion!

Regarding the average number of responses $n$ generated by LLM, we have summarized the results in the following table.

For Table.1

†: They did not provide official code implementation and the related data is not involved in their paper. The data come from our own implementation.

For Table.2 24-point problem:

For Table.2 sequence sorting:

*: The related data is not involved in their paper, we obtained the data from their official GitHub repository.

Experimental results show that, compared to baselines such as CSV, ToT, and GoT, MACM requires fewer LLM responses even without a code interpreter function, while achieving higher accuracy.

Thank you again for your feedback! We hope our response can address your concerns.

Thank you for reviewing our work. Here are our responses to your concerns:

For Weakness 1:

MACM and Cumulative Reasoning differ significantly, especially in the rechecking and voting processes. Intuitively, Cumulative Reasoning has an accuracy of only $72.2$% on MATH. According to OpenAI's official data [1] , GPT-4 Turbo itself can achieve an accuracy of $72.6$% on math problems. Meanwhile, MACM achieves an accuracy of $87.92$% on MATH, which is significantly higher than Cumulative Reasoning.

For Weakness 2 (a):

In section 4.2, It is unreasonable to state that ToT and GoT are incompatible with MATH and quit comparing with them.

ToT and GoT have significant limitations in practical applications. They can only be applied to data with relatively fixed formats and low information content, as stated in our Appendix A. In contrast, MACM has greater versatility and applicability. In LLM reasoners, ToT has only been applied to relatively simple tasks like GSM8K, AQuA, and Game24.

To our knowledge, there has not yet been any work that applies ToT to an entire MATH dataset.

For Weakness 2 (b):

for GPT-4 seems to always use CoT on reasoning tasks

Currently, there is no evidence to suggest that GPT-4 always uses CoT on reasoning tasks. To ensure a comprehensive and accurate comparison, we adopted the same experimental settings as previous baselines, including CSV, ToT, and GoT prompting methods. This involved thoroughly testing both IO and CoT methods.

For Weakness 2 (c):

experimental setups described in Appendix B seem rather unreasonable

1. The specific process for performing SC-CoT using greedy decoding is as follows: First, let LLM generate several plans using a prompt such as, Please help me to list several different plans for solving this problem. Then, add each of these plans to different execution paths and continue the subsequent reasoning separately. Finally, aggregate the results and conduct a vote. We use greedy decoding here to reduce the randomness in the experiments.

2. The length of the chain $l$ refers to the number of reasoning steps the model performs in CoT. We set this parameter to fix the number of reasoning steps to facilitate the evaluation of efficiency.

3. For MACM, we set the thinker's max_tokens to $512$, the judge's max_tokens to $4$, and the executor's max_tokens to $256$. These max_tokens values are for each step of the process. MACM decomposes each problem in the MATH dataset into multiple reasoning and judgment steps, making these max_tokens settings reasonable and feasible for each step.

For Weakness 3 (a):

The paper tends to list many settings that are different in multiple dimensions and their results in a line (e.g. Figure4,7, Table2,3), making attribution of the results difficult.

Different models and prompting methods can affect the final results.

In Figure 4, we tested various LLaMA models using different prompting methods on the MATH dataset. Different model versions are distinguished by different colors, with the model parameters labeled on the first row of the x-axis and the prompting methods labeled on the second row of the x-axis.

In Figure 7, we tested the contribution of four different components in MACM to the improvement in model accuracy. Different components are indicated by different colors on the x-axis, and combinations of different numbers of components are represented by bars of various colors.

In Table 2, we have 5 columns: the first column lists the tasks tested, the second column indicates whether a code interpreter was used for verification, the third column specifies the model used, the fourth column details the prompting method employed, and the fifth column shows the resulting accuracy.

In Table 3, we have 4 columns: the first column lists the methods used, which include information about the model, prompting method, and whether Python was used. The second to fourth columns correspond to three different subsets of SciBench.

For Weakness 3 (b):

The paper spends much space on specific examples (~2 pages across pages 4-6) but seems to show few special things.

As with previous prompting papers, we have listed specific examples to help readers better understand the details of our method. For instance, in the Graph of Thought [2], refer to Figures 2, 3, 4, and pages 3, 5, 6. Similarly, in the CSV paper [3], refer to Figures 1, 3, 4, and pages 3, 5, 6, 8.

Reference

[1] GPT-4o text evaluation report.

[2] Graph of Thoughts: Solving Elaborate Problems with Large Language Models, AAAI 2024

[3] Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification, ICLR 2024

Thank you again for your feedback! We hope our response can address your concerns.

Thanks for your reply! You've resolved almost all my concerns. I've updated my final comments in the official review.

Thank you for your valuable suggestions regarding our work. Here are our responses to the comments you raised:

For Weaknesse 1 & Question 1

What is the number of responses for each model (including baselines and the proposed MACM) in Table 1 and Table 2?

Thanks for the question! In Table 1, for the methods we self-implemented, we standardized the number of responses $n$ generated by GPT-4 Turbo to $1$ for all prompting methods (as stated in line 375 of our manuscript). For SC-CoT, we set the number of voters $v = 5$. The specific implementation method involves first asking the model to generate $v = 5$ different solution approaches for the problem, and then instructing the model to solve the problem according to each of these approaches. In Baseline CSV + Voting, their setting is: the number of sampled paths $k = 16$.

Regarding the average number of responses $n$, we have summarized the results in the following table.

†: They did not provide official code implementation and the related data is not involved in their paper. The data come from our own implementation.

In Table 2, for the 24-point experiment, the data for ToT are sourced from their original paper, where the breadth b of the corresponding tree for ToT was set to $b=5$. In SC-CoT, the best of k samples parameter was set to $k=100$. For the sequence sorting experiment, the data for GoT also come from their original paper. They did not provide specific settings for this. For the MACM, we standardized the number of responses $n$ generated by GPT to $1$.

Regarding the average number of responses $n$, we have summarized the results in the following table.

For 24-point problem:

For sequence sorting:

*: The related data is not involved in their paper, we obtained the data from their official GitHub repository.

For Question 2

Thank you for your question! As stated in lines 237-239, we performed these two experiments on 200 randomly selected questions from the MATH dataset that the original GPT-4 Turbo model answered incorrectly. Figure 6 demonstrates the corrective capabilities of different prompting methods under various parameter settings for these questions that GPT-4 originally answered incorrectly.

Thank you again for all your feedback! We hope our answers can address all your concerns.

Thank you for reviewing our work. Here are our responses to your concerns:

For weakness (a)

MATH has its own evaluation protocol, and the Minerva paper [1] also gave a good evaluation protocol. The reliance on GPT4 Turbo as a judge seems unjustifiable.

Thank you for your question! GPT-4 serves as a judge for voting/judging intermediate steps in the reasoning process, thereby helping the overall system achieve higher accuracy. It does NOT determine the correctness of the final result.

In fact, our evaluation protocol follows the standards set in [1], Section D.1 of their paper. Initially, results are output in an easily extractable box and then verified using SymPy, as shown in line 450 of our code in prompt/prompt.py.

For weakness (b)

only tested on a rather randomly selected set of MATH. This is very worrying

The MATH dataset contains tens of thousands of questions, and testing multiple methods on the entire dataset is often a costly endeavor. Even if testing one question with different prompting methods only costs 1 dollar, testing the entire dataset just once would require more than 12500 dollars, which is generally unaffordable. If a code interpreter is used, the cost would be even higher.

We referred to the evaluation methods of other top-performing methods on the MATH dataset leaderboard and selected a subset of the data for testing. For example:

1. [2] includes the code implementation for the methods ranked 2nd and 3rd in accuracy on the leaderboard for the MATH dataset. Their results are based on a self-selected 1000 test cases. In their paper, Section 4.1 states: We report results on the 1K-sized validation subsets for MATH and GSM8K created by us.

2. [3] includes the code implementation for the method ranked 4th in accuracy on the leaderboard for the MATH dataset. Their results are based on a 500 test cases. In their paper, the first line of Table 5 states: * denotes using 500 test examples subset.

3. In OpenAI's prompting-related paper [4], their evaluation results for the MATH dataset are also based on a self-selected 500 test cases. In their paper, Appendix C states: We therefore evaluate our models only on the remaining 500 held-out problems. We selected these 500 test problems uniformly at random.

We randomly selected one-third of the data from the MATH dataset and ensured that the distribution of level 1-5 questions is consistent with the original dataset. To ensure a fair comparison, all self-implemented experiments were conducted on this one-third subset. The size of this subset already exceeds the 500 or 1000 cases chosen in previous work.

For Question

According to Figure 7, the four components of MACM each contribute to logical reasoning and solving complex problems. To adapt to problems of varying difficulty, we have set some external parameters, such as the number of voters and the number of Condition Mining iterations. For simpler problems that do not require complex logical reasoning, users can lower these parameters to effectively increase problem-solving speed.

Reference

[1] Solving quantitative reasoning problems with language models

[2] OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset

[3] Cumulative Reasoning With Large Language Models

[4] Let's Verify Step by Step

Thank you again for your feedback! We hope our response can address your concerns.