W1: Clarity of Framework and Implementation

Thank you for your helpful suggestions regarding the presentation of the paper. In response to your comment, we will revise the text in the Introduction and explicitly emphasize in Figure 1 that "executing computations in the CAEF framework requires multiple LLM queries." Additionally, we will restructure the paper to include concrete examples earlier on, which should make our approach easier to understand. We will also revise Figure 3 in Section 3 to clearly illustrate the inputs and outputs.

W2: Comparison with Baselines

Thank you for your thoughtful question; your understanding is correct and valid. We have conducted extended experiments to evaluate the baseline model’s performance under a CoT setup, as well as in a Turing machine-style computation similar to the CAEF framework. Details and results can be found in the global rebuttal section. As we can see, the results indicate that employing the single-query approach leads to a significant decline in model performance as the operand length increases. Additionally, due to the limitation of the context window, this method is incapable of handling scenarios involving operands with more than 50 digits. In contrast, adopting the multi-query approach without fine-tuning the model essentially fails to function effectively.

W3.1: Need for LLMs in Exact Arithmetic Computations

Thank you for your question. At this stage, we view our work as a methodological exploration rather than an application to specific real-world scenarios. A more detailed discussion of this issue is provided in the global rebuttal section.

W3.2: Pre-selection of Adapters

Thank you for your question. For a given operator, we are currently using a pre-selection method. However, there are existing methods that combine MoE architecture with LoRA, allowing automatically switching LoRA adapters. These methods could potentially address this issue. A more detailed discussion of this approach can be found in the global rebuttal section.

Questions

Thank you for pointing out the writing issues in the paper. We will correct these in the revised version.

Dear Reviewer,

As the rebuttal period is drawing to a close, I would appreciate your response to the authors' rebuttal at your earliest convenience.

Best Regards,

Area Chair

W1: Use of Turing Machine

Thank you for your question. One of the key assumptions of our work is that the model does the reasoning by itself, without using any external tools. At this stage, we view our work as a methodological exploration rather than an application to specific real-world scenarios. A more detailed discussion of this point can be found in the global rebuttal section.

W2: Basic Arithmetic Operations as Application

Thank you for your valuable suggestion. We share a similar perspective on this matter. While arithmetic tasks may appear straightforward, they are highly suitable for demonstrating the core principles of our approach. Our experimental results highlight that current LLMs struggle to achieve high accuracy in tasks involving large numbers. The proposed CAEF framework represents a foundational step towards "System 2" reasoning (as discussed in the global rebuttal section), specifically focusing on training LLMs to perform executing tasks. Moving forward, one of our ongoing work is to extend CAEF by incorporating search capabilities into LLMs to tackle more complex problems, particularly in the domain of automated theorem proving.

In the domain of automated theorem proving, existing methods, which rely on proof-step generation (each proof step is generated sequentially), often depend on external tools (e.g., Lean 4) to validate the correctness of individual steps and to estimate the optimal next step using search algorithms. We propose extending the CAEF approach to this domain, where models generating individual proof steps serve as the basic executors, and models responsible for devising search strategies act as the executor composers. By refining proof steps and providing richer contextual information throughout the proof process, we aim to enhance both proof generation quality and search efficiency. This approach aligns closely with the "System 2" reasoning paradigm outlined in the global rebuttal section.

W3: Risk of Cascading Errors

Thank you for your question. As shown by the experimental results in the paper and the extended results provided in the global rebuttal section, our approach does introduce cascading errors due to inaccuracies in individual steps. However, we mitigate this by achieving sufficiently high accuracy at each step through task simplification. For example, in the case of 100-digit addition, the executor's step accuracy needs to exceed 0.9999 to ensure an overall accuracy of 0.99. Despite the cascading errors, our results consistently outperform end-to-end generation methods.

W4: Misuse of Terminology: "Understanding"

Thank you for pointing out the inappropriate use of the term “understanding.” We acknowledge this issue and will revise the expression to a more suitable one in the revised version of the paper.

W5: Suggestion for Error Analysis Section

Thank you for your suggestion. We have added an analysis of this issue in the global rebuttal section. Please refer to it for further details. The analysis will also be updated in the revised version of the paper

Q1

Thank you. This issue is the same as the one raised in W1. Please refer to our response to W1 for further clarification.

Q2

Thank you. We conducted an additional experiment to investigate merging the aligner and executor into a single module using one LoRA adapter for the addition operator. The evaluation was performed on the same test set as in the paper, and the results are as follows:

The left side of slash shows the results after merging the executor and aligner, while the right side presents the original results of CAEF. From the results, we observe the following:

• The performance of the executor and aligner (O) shows a slight degradation compared to the original modular approach.

• The aligner (I), however, experiences a significant performance drop.

Under this setup, the bottleneck of the computation remains the reversal of operands rather than the computation itself, which aligns with the conclusions presented in our paper. Based on these findings, we believe separating the executor and aligner remains the preferable approach.

In the "Application of CAEF to Math Word Problems" section of the global rebuttal, we further discuss potential solutions for integrating these modular LoRA adapters. This could address the issues raised and provide a pathway to improve performance.

Suggestions

Thank you for pointing out the writing issues in the paper. We will correct these in the revised version.

W1: Clarification of Execution Process

Thank you for your comment regarding the expression in the paper. Your understanding is correct. We will revise the expression in the Introduction to clarify this point and explicitly highlight in Figure 1 that "executing computations in the CAEF framework requires multiple LLM queries."

W2: Computational Complexity

Thank you for your suggestion. For the seven operators implemented using the CAEF framework, we assume the longer operand has a length of $d$. Based on the computation mechanism of self-attention, the computational complexity of a single model inference is $O(d^2)$.

For the addition, greater_than, less_than, and equal operators, the computation is performed digit by digit, requiring at most $d$ model queries for a complete computation. Additionally, the aligner performs two representation conversions, resulting in a total of $d + 2$ model queries. Therefore, the overall computational complexity is $O(d^3)$.

For subtraction, the computational complexity of the auxiliary operators Reflection and Left_mask is also $O(d^3)$. Since subtraction involves one call each to Reflection and Left_mask, along with two calls to addition, the overall complexity remains $O(d^3)$.

For multiplication, the situation is slightly more complex. For a calculation of the form $a \times b$, we assume $\text{len}(a) = d_1$, $\text{len}(b) = d_2$, and $\text{len}(a \times b) = d_3$. The number of iterations in the loop is $b + 1$. During each iteration, less_than and two addition operations are performed, with the complexities as follows:

• For less_than, the longer operand has a length of $d_2$, resulting in a total complexity of $O((b+1)d_2^2)$ across all iterations.
• For the first addition, the longer operand has a length of $d_3$, giving a total complexity of $O((b+1)d_3^2)$ across all iterations.
• For the second addition, the longer operand again has a length of $d_2$, resulting in a total complexity of $O((b+1)d_2^2)$ across all iterations.

Thus, the overall complexity is the sum of these three parts, plus the two aligner conversions. The final computational complexity for multiplication is $O(bd_2^2 + bd_3^2)$.

For division, the situation is similar to multiplication. Assuming $\text{len}(a) = d_1$, $\text{len}(b) = d_2$, $a \div b = c$, and $\text{len}(c) = d_3$, the number of iterations in the loop is $c + 1$. The final computational complexity for division is $O(cd_1^2 + cd_3^2)$.

We will include this analysis in the revised version of the paper.

W3: Comparison in Section 4

Thank you for your question. The baseline setup involves the LLM generating results in a single-query mode. We completely understand your concern, and in response, we have conducted extended experiments comparing the model's performance under a CoT setup and a Turing machine-style computation, similar to the CAEF framework. Details and results can be found in the global rebuttal section. As we can see, the results indicate that employing the single-query approach leads to a significant decline in model performance as the operand length increases. Additionally, due to the limitation of the context window, this method is incapable of handling scenarios involving operands with more than 50 digits. In contrast, adopting the multi-query approach without fine-tuning the model essentially fails to function effectively.

W4: Modular Approach and Generic Solution

Thank you for your insightful suggestion. We completely agree with your concern. Initially, our idea was to design the LLM to imitate a universal Turing machine. However, we decided to pursue the current approach for two key reasons:

1. Choice of LoRA for Fine-Tuning: We selected LoRA as the fine-tuning method due to its lightweight nature. However, this approach has limited capacity for injecting knowledge into the model. Using a single LoRA adapter to encode all operators’ knowledge proved challenging. While full model fine-tuning might be an alternative, it comes with its own set of trade-offs.

2. Why LoRA Over Full Model Fine-Tuning: As noted in the paper, our goal is to enable the model to build upon previously learned operators while incorporating new computation logic. Full fine-tuning requires incremental training for new operators, which, if the logic is orthogonal to existing operators, could lead to catastrophic forgetting. Distributing knowledge across separate lightweight LoRA adapters provides a more modular and efficient approach.

Furthermore, there are emerging methods that combine MoE (Mixture of Experts) architecture with LoRA, which may offer potential solutions to this issue. A more detailed discussion about this can be found in the global rebuttal section.

W5: Performance in the Presence of Text

Thank you for your question. At present, our method does not directly address text-based tasks such as math word problems. However, in our vision, with the CAEF plugin, an LLM could seamlessly switch to CAEF for handling arithmetic calculations while generating reasoning steps for such problems. A more detailed discussion on this can be found in the global rebuttal section. This approach aligns with the response provided to W4.

Dear Reviewer,

As the rebuttal period is drawing to a close, I would appreciate your response to the authors' rebuttal at your earliest convenience.

Best Regards,

Area Chair

W1: Efficiency issue

Thank you for your thoughtful question. Your concerns are completely valid. At this stage, we consider our work to be a methodological exploration rather than a direct application to specific real-world scenarios. Addressing the efficiency issue is one of our ongoing efforts. For a more detailed discussion on this topic, please refer to the global rebuttal section.

W2: CAEF may fail in some unique digit patterns

Thank you for your insightful comment. We fully agree with your concern, as it aligned with our initial hypothesis when we first encountered this issue. However, after conducting a more detailed investigation, we arrived at the conclusion presented in the paper. For a more comprehensive discussion, please refer to the global rebuttal section.

While I appreciate your approach to conducting cutting-edge research, the issues present in the current proposal (also noted by other reviewers) cannot be overlooked.

Specifically, there are concerns about how to handle queries containing text, the cost associated with making multiple queries to the LLM, and whether modifying the network architecture might impact the original foundational capabilities of the LLM.

Therefore, my concerns about this paper remain unresolved, and I have decided to lower my score from 6 to 5.

Application of CAEF to Math Word Problems

First, we would like to clarify that, at this time, our method requires the manual selection of the active LoRA adapter, rather than enabling the model to autonomously select it. This limitation currently prevents the direct application of our method to solving math word problems.

However, our approach can be viewed as a plug-in component. There are several studies (refer to [1]) that focus on integrating Mixture of Experts (MoE) and LoRA techniques to enable automatic selection and switching of active LoRA adapters based on the input. These studies are orthogonal to our method, and we believe that combining these techniques with our approach would allow it to be effectively applied to math word problems. For instance, using the CAEF plug-in, an LLM could switch to CAEF for handling arithmetic calculations as part of the reasoning steps in solving a math problem.

Discussion on Model Errors and Error Patterns

We would like to provide further clarification regarding the “Prone to errors with repeated digit patterns” part. Our sampling methods for both the training and testing sets are consistent, ensuring that similar patterns are present in the training set. We have checked and verified this. Beyond this specific pattern, we have not identified any significant commonalities in the remaining error-inducing samples.

We attribute this issue to the internal mechanisms of generative models. A common failure pattern during inference involves the model repeatedly generating the same segment of text. This behavior suggests that transformer-based models may have difficulty tracking how much of a sequence has already been repeated, leading to the continuation of the same string. Consequently, we hypothesize that inputs with repeated patterns exacerbate the model's fragility, increasing the likelihood of inconsistencies between the input and the generated outputs.

Extended Evaluation

To address the issue of fairness in comparing CAEF with the baselines, we conducted two additional experiments on the addition operator:

1. CoT Group: We used the "detailed scratchpad" CoT prompt in [2] to evaluate three models: a LoRA-fine-tuned LLaMA 3.1 base model, LLaMA 3.1 Instruct model, and GPT-4o (consistent with the baseline in the paper). For this setup, the calculation results were obtained using a single-query approach.

2. CAEF-style Group: In this experiment, we instructed the LLaMA 3.1 Instruct model and GPT-4o to follow a CAEF-style multi-query approach. A 4-shot prompt was used to provide detailed Turing machine-style descriptions and examples.

The evaluations were conducted on the same test set as in the paper, and the results are summarized below:

1. CoT Group Results:

As the results shown in the Table, it is evident that the single-query approach, even when enhanced with a detailed scratchpad prompt containing explicit computational reasoning, struggles to handle scenarios with operands of 10 or more digits effectively. Additionally, because the detailed scratchpad method requires embedding all computational steps into a single query, context lengths can exceed 10k tokens for operands of 50 digits. This not only makes the method infeasible in large-number scenarios but also challenges the model’s ability to process such long contexts, even when supported by larger context windows.

2. CAEF-style Group Results:

In the CAEF-style group, neither the LLaMA 3.1 Instruct model nor GPT-4o achieved satisfactory results in the few-shot setting. This highlights that fine-tuning, as implemented in CAEF, is crucial for enabling the effectiveness of multi-query approaches.

[1] Cai, Weilin, et al. "A Survey on Mixture of Experts." arXiv preprint arXiv:2407.06204 (2024).

[2] Lee, Nayoung, et al. "Teaching arithmetic to small transformers." arXiv preprint arXiv:2307.03381 (2023).