Modeling Complex Mathematical Reasoning via Large Language Model based MathAgent

Thanks for your valuable comments and suggestions about the motivation and limitations of the work. We summarized and replyed the frequent key issues among reviewers in "For All Reviewers" (FAR). Here are our specific replies:

C1: Why avoiding the use of theorem-provers.
R1: We reasserted our contributions and design principles in FAR1 and FAR2. Despite PRER can be combined with external tools, it is costly to provide additional implementation and the tool is not our contribution. Moreover, directly using theorem provers like lean wastes the textual ability of powerful LLMs. We will re-consider this trade-off in the future.

C2: There are a few pages on decomposition for mathematical reasoning.
R2: We addressed the comparison with previous works in FAR1. In short, we introduced more definitions and designs to tell the model how to decompose, and provided an agents-based framework with other concepts like logical reasoning to model the reason process.

C3: Equation 1 is not entirely clear. The framework is presented rather in a hurry.
R3: We apologize for the issues in writing. The formulas of the PRER are reorganized in FAR4. We will improve our writing and supplement required descriptions in latter paper revisions.

C4: The current prompts are manually crafted, might not be the most promising way forward.
R4: We acknowledge the limitations. Prompts have many advantages (interpretability, generalization, low cost, etc) and many drawbacks (see FAR5 and FAR6). We plan to explore more methods like SFT in the future. Besides, recent work proposed some methods for automatic generation [1,2] and discrete search [3,4] of prompts, which may alleviate the concerns. Identify the intentions of non-experts to generate prompts is also an interesting direction to consider. The recent released "agents" by OpenAI imply the possibility.

C5: What is the rationale for choosing two MathAgents?
R5: We describe the how we define MAs in FAR2 and explain why we provided two MAs and what's their diffence in FAR3.

[1] Wang, Yizhong et al. “Self-Instruct: Aligning Language Models with Self-Generated Instructions.” Annual Meeting of the Association for Computational Linguistics (2022).
[2] Honovich, Or et al. “Instruction Induction: From Few Examples to Natural Language Task Descriptions.” Annual Meeting of the Association for Computational Linguistics (2022).
[3] Zhou, Yongchao et al. “Large Language Models Are Human-Level Prompt Engineers.” ArXiv abs/2211.01910 (2022): n. pag.
[4] Sordoni, Alessandro et al. “Deep Language Networks: Joint Prompt Training of Stacked LLMs using Variational Inference.” ArXiv abs/2306.12509 (2023): n. pag.

Dear reviewer DbPQ:

Thanks again for your great effort in reviewing our paper! Since there are only less than 3 days to the deadline for discussions, we are really looking forward to having a discussion with you about all technical details and concerns. We sincerely hope to get your further feedback. Would you mind checking our response and letting us know if you have further questions?

Best regards, Authors of #1727

Thanks for your valuable comments and agreement with the concepts. We sincerely apologize for the lack of clarity in our writing. We summarized and replyed the frequent key issues among reviewers in "For All Reviewers" (FAR). Here are our specific replies:

C1: How do the authors define the actions of different modules?
R1: We supplemented the relevant theories in FAR2 to explain how we define the modules and actions, including the introduction of logical reasoning and several empirical considerations.

C2: What is the necessity of proposing MathAgent-M independently?
R2: We explained the rationales for providing two MAs in FAR3, and clarified the motivation and contributions of this paper in FAR1. Although MA-H might be better, the comparision of two MAs can yield some interesting conclusions.

C3: Why the authors state the MA-M is more aligned with LLMs, while the description of "infer" in MA can also be viewed as an action in human cognition
R3: We primarily distinguish them by whether the LLM has the freedom to make decisions and self-evolve (see FAR3). The actions in MA-H is more specific and we contrain the LLM with program. Meanwhile, the actions in MA-M are more general. These actions like observe and infer are defined emperically, thus can contain human experiences. After all, language modeling based LLMs natually contains human knowledge. But overall, these actions allow the LLMs to execute freely.

C4: What is the meaning of $m^1_n$, $m^2_n$ in Eqs.(2),(3) in section 2.2. In Eq.(3), why is $t_n$ an output, not an input? What is it useful for subsequent reasoning?
R4: We re-organized the formulas in FAR4. For instance, when the LLM determines to classify, the LLM then need to seperate the context into sub-topics, that is $t_n$. This topology plays many roles in subsequent reasoning, as filtering memory to retain context only related to the sub-topic, and tell the LLM the relationships between sub-structures when finish discussion and integrat the memories.

C5: The calculate intuitively should not have such a high frequency.
R5: We elaborated what we observed about the actions in FAR5, analyzing why it is reasonable for calculate to have a higher proportion. However, we indeed observed some inconsistencies, which might make it abnormal. We provide further discussion in both FAR5 and FAR6, explaining the possible reasons and solutions. It is also worth noting that as the difficulty increases, the proportion of other actions rises.

C6: There exist some other works that also decompose the mathematical reasoning into several steps.
R6: We provided some comparisons with existing work in FAR1, supplemented some unique theoretical foundations in FAR2, and analyzed the scalability of PRER in combination with these methods in FAR6. We will further improve the writing and supplement the required descriptions and comparisons in future revisions of this paper.

Dear reviewer M3Nq:

Thanks very much for your attention and great effort in reviewing our paper. As the deadline is approaching and there are only less than 3 days for discussion, we would like to have a further discussion with your feedback. We understand you may have a busy schedule, but we believe that we have addressed all your concerns. If you still have further concerns or feel unclear after reading our responses, please kindly let us know and we are willing to make clarification and discussion. If you are satisfied with our responses so far, we sincerely hope you could consider your score. Thanks very much!

Best regards, Authors of #1727

Thanks for your valuable comments and suggestions about the soundness of this paper. We summarized and replyed the frequent key issues among reviewers in "For All Reviewers" (FAR). Here are our specific replies:

C1: More clarifications should be provided.
R1: We apologize for the deficiencies and omissions of writing. We supplemented the theoretical basis of MathAgents (MAs) in FAR2, and organized the key formulas of PRER in FAR4. We will further improve the writing.

C2: What actions are truly necessary?
R2: First it should be clarified that the core principle of our designing is the completeness of actions, rather than which one is important on MATH or MiniF2F (see FAR2). We provided more details of the action invocation in FAR4. For mathematical tasks, it is reasonable that the most important action is infer/calculate. Besides, we analyzed why the ratio of calculate is so high in FAR4, as well as the reasons and possible solutions. It's worth noting that when the difficulty increases, the frequencies of other actions also increase (Fig.3). We conducted the ablation experiments (Table 4). Considering the cost, more combinations of ablation might be hard.

C3: What does this sentence mean: "whereas the MATH dataset does not offer final answers for reasoning"
R3: We apologize for this error. The actual meaning is that when input to LLM, MATH does not provide the final answer, while MiniF2F informs the LLM of what the final goal to be proven is.

C4: Figure A1: where is (c)?
R4: Sorry for the typo error, the (c) here refers to the bottom "MathAgent" in the figure. We will correct the error later.

C5: Could the authors provide examples on IMO problems and more details of prompts on MiniF2F.
R5: An example is as follow:
"Consider the system of equations

Dear reviewer CLmw:

Thanks very much for your dedication in reviewing our paper. Your comments about some unclear details have been taken into account and prompted us to improve them in a revision of manuscript. Since there are only less than 3 days to the deadline, we would appreciate it if you could check our replies and post further valuable comments about the replies and the presented example. We sincerely hope to continue the discussion with you and address the concerns.

Best regards, Authors of #1727

Thanks for your valuable comments and meaningful concerns about the costs, theories and soundness of this work. We summarized and replyed the frequent key issues among reviewers in "For All Reviewers" (FAR). Here are our specific replies:

C1: It raises concerns about potential high costs and trade-off.
R1: We explained and discussed the scalability and cost of PRER in FAR6, hoping these explanation can alleviate the concerns. Our main focus in this paper is to explore the potential of PRER, as stated in FAR1. We acknowledge the limitations and discussed how to reduce the costs as future research.

C2: Why not evaluating datasets like GSM8k.
R2: The framework we designed contains a variety of mathematical actions. Thus it may be meaningless to evaluate on GSM8k, which primarily requires computational operations, considering the cost. Previous methods have achieved very high acc on GSM8k, thus we chose more challenging datasets like MATH. Besides, MATH includes the topic of "algebra", which can reflect the generalizability of proposed MAs.

C3: It seems unsubstantiated with neither theoretical nor empirical backing. What's the significance or need for Executors.
R3: We apologize for the missing descriptions. We supplemented the theories in FAR2 to explain how we define the actions and re-organized important formulas of PRER in FAR4. These contents will be added to the paper in the next version.

C4: How about the method's adaptability and comparisons with alternative techniques?
R4: We apologize for any confusion caused by our writing. We explained the scalability of PRER in FAR6. Theoretically, most prompting methods can be integrated as a type of agent. They are not alternatives, but rather compatible with PRER.

Dear reviewer WFXS:

Thanks again for your constructive and valuable comments, which have helped us improve the paper considerably. We believe that your concerns have been addressed somewhat satisfactorily. There are only less than 3 days to the deadline for discussions. Sincerely hope to gain further valuable feedback from you and continue to improve this work. We are also willing to clarify any additional concerns.

Best regards, Authors of #1727

FAR4: The core formulas of proposed PRER and MAs.

MA consists of four modules that are executed iteratively, solving the problem step by step. In each step, we infer the conditions, objectives, and memories {$x_{n+1}$, $y_{n+1}$, $m_{n+1}$} of step n+1 by inputing the corresponding {$x_n$, $y_n$, $m_n$}.

Planner is to choose the current action $a_n$ by $a_n = f_{PL}(x_n, y_n, m_n).$ Then the Reasoner selects the current conditions $x_n'$, objectives $y_n'$, and topology $t_n$ to find proper context as the input of executor: $x_n', y_n', t_n = f_{RS}(a_n, x_n, y_n, m_n, M),$ where $M$ denote the domain knowledge provided via prompts. The executor performs the action to generate new memories: $m_n' = f_{EX}(x_n', y_n', a_n, m_n, M)$, and finally Reflector update the conditions, objectives, and memories via memories: $x_{n+1}, y_{n+1}, i_n, m_{n+1} = f_{RF}(x_n', y_n', t_n, m_n')$, where $i_n$ is used to determine whether the reasoning should be re-executing or early-stop.

FAR5: More details and analysis of action execution/importances.

The invocation/importance of actions is a common concern among reviewers. It should be first clarified that these actions are designed in theory, aiming to ensure MAs could tackle most mathematical problems, rather than designing specifically for MATH and MiniF2F. In experiments, the most important action is actually infer/calculate. Considering that MATH and MiniF2F are mostly multi-step calculations, a higher proportion of infer/calculate might be reasonable, also corroborated by SKiC [1]. It is notable that as the difficulty increases, the proportion of other actions gradually increases.

In addition, there are indeed inconsistencies in the experiments as MAs are implemented via zero-shot prompts. We allowed powerful textual reasoning [2] of LLMs for calculate, thus expanding the meaning of such actions. Meanwhile, LLMs might not fully comply with the instructions and perform tasks outside the definition sometimes, as they are not pre-trained as collaborated agents. This interesting inconsistency led to a decrease in the invocation of other actions and made weaker models fail to understand and execute the actions. We will improve these issues by making agents more professional and independent in the future, using ICL, SFT, etc. In this paper, we focus on demonstrating the overall performances and potential and state the PRER and proposed MAs contribute.

FAR6: Scalability and cost of proposed framework.

The scalability of PRER is theoretically good. It can adopt different prompting methods (therefore they are compatible rather than alternative) as the strategies via instructions or workflow of agents via program. External tools and diverse feedback signals can further enhance LLMs.

One primary concern about the provided MAs is that the presented instructions can only work well with GPT4. As analyzed in FAR5, the reason could be that weaker models fail to follow instructions since they were not pre-trained as agents. Methods like ICL, SFT, and combining external tools might be benificial to alleviate this concern.

Another concern is the cost. Current MAs are expensive as calling GPT4 frequently. We reported the average steps in Fig.4. Similarly, many prompting methods chose to enhance LLMs by generating more times, e.g., PHP. The actual expense in our experiments is the memory (similar to problems encountered in RAG). Concurrent works like [3] have proposed effective method to manage memories, which might be a possible solution. Besides, we found outputs of LLMs could be repetitive and redundant and hallucination led to repeated execution of actions. We suggest these issues as future research.

[1] Chen, J. et al. Skills-in-context prompting: Unlocking compositionality in large language models (2023).
[2] Chen, Wenhu et al. Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks.” ArXiv (2022).
[3] Chen, Howard et al. “Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading.” ArXiv abs/2310.05029 (2023): n. pag.