We want to thank the reviewer for taking the time to analyze our work and provide meaningful insights and comments. Here below, we address the questions raised by the reviewer.

• Regarding dataset contamination: We can rule out that AIME2024 was part of the training set of the LLama and Qwen models that we analyzed, since their knowledge cutoff dates to the end of 2023, while AIME2024 was released at the end of 2024. However, for the other datasets, that is something that we cannot know for sure, since information on the data mix used for training is not made public, even for open weights models like Llama and Qwen. If there were data leakage, though, we’d expect that to also be reflected in the baselines. We therefore argue that measuring relative improvement over an appropriate baseline addresses most concerns regarding data contamination and its role in boosting specifically our method.

• Code execution tool ablation: Indeed, the use code tool is important as it enables the models to reliably solve operations like arithmetic operations in a more reliable way than autoregressive generation from the language model weights. However, we verified that the code generation capabilities are not by themselves enough to explain the final performance of our cognitive tools. To show that, we ran additional experiments where the baseline was endowed with the use of code and tested this enhanced baseline on AIME24, MATH500 and AMC. We notice that cognitive tools still consistently outperform also this enhanced baseline, across the board, except for Llama3.1-8B on MATH500 where the enhanced baseline becomes remarkably competitive to the cognitive tools approach.

• Accuracy of GPT-4.1 as a judge: Using GPT-4.1 as an automated judge is standard practice for evaluating complex, open-ended benchmarks like MATH500, where traditional scoring based on string matching is infeasible due to the diversity and ambiguity of valid answers. While automated grading is not flawless, prior work shows that GPT-4.1 achieves high agreement with expert human graders when properly instructed. In our experiments, all MATH500 outputs are uniformly evaluated using the official prompts employed for MATH500 benchmarking, ensuring that our assessment protocol is consistent with community-established standards, and are fairly scoring our method compared to the baselines.

• For Table 1: We provided in the supplementary material standard error related to these results. We will add them on the main paper in the camera-ready version of the paper, together with tests assessing statistical significance of the difference of importance in support of our work. We will also fix the grammatical errors and make sure to add the necessary details about the Smolbenchmarks dataset (the dataset in Table 2).

• On transparency and explainability: Thank you for the question, as we’re glad to elaborate on how our method enhances transparency and explainability. Our method enables each intermediate reasoning step to be explicitly mapped to a specific, well-defined cognitive tool, producing structured traces that expose the decomposition and function of each step. This contrasts with typical end-to-end or chain-of-thought prompting, where intermediate steps are generated as free-form text and can be difficult to interpret or attribute. Structuring traces according to tool calls also has the advantage of exposing the synergies that emerge in terms of well-defined operation. For instance, a preliminary analysis of the pattern of tool invocation revealed that calls to the tool recall related is highly correlated with the use of backtracking, which reflects a “recovery workflow” where the language models first recall multiple relevant pieces of information, try to follow them through one at a time, and backtracking when encountering failure to then pursue the next one. In the revision of the paper, we plan to provide more detailed such analyses of tool calls patterns that are enabled by the explainability aspect of modular cognitive tools, if given the chance.

• On reducing interference: Our approach is partly motivated by the observation in the previous literature that modularization is a mechanism that reduces interference by limiting information leakage across modules. In our specific case, compartmentalizing reasoning into separate cognitive tools, each with its own structured prompt and context, effectively achieves a separation of concerns where the language model is given a cleared context windows, with only the few relevant information provided by the main LLM, and the unique task of executing the tool being called. This is in contrast to a flat monolithic chain-of-thought prompting where all reasoning happens in one undifferentiated context and each generation step is potentially influenced by all the previous information present in the context window. In other words, we know that our cognitive tools are less subject to interference (since by construction the modular structure of the cognitive tools is implemented through a less cluttered context window with only the relevant information pertaining to the cognitive operation at hand), the question is rather whether there's any empirical evidence that a flat monolithic prompting approach suffers from the interference to which it is naturally exposed. For now, the main supporting empirical evidence is the higher performance that we report for cognitive tools over the non-modular baselines. Beyond improved accuracy, we’re currently considering a series of controlled experiments where we can track back errors to interference of multiple pieces of information in the context window (purposely inserted as distractors). We thank the reviewer for channeling our thoughts in this interesting direction, and hope that we will have something to show in the next revisions.

• On recall related tool: We found that the tool call tends to return 2-3 examples related with the initial question. These examples usually include reasoning steps and a final answer. Interestingly, the final answer proposed by the tool is however not always correct. Irrespective of that, the performance of the model does not seem to be affected by the correctness of the final answer of these examples, but mostly by the actual included reasoning steps. As an example, here we report a trace of a call to the recall related tool (LLama3.3-70B model on a sample from AIME24):

Query from the agent:

To continue solving the problem, I will use the recall_related tool to retrieve or generate analogous questions or problems for which a solution and the process to get to the solution is known.

recall_related({'question': 'Jen enters a lottery by picking $4$ distinct numbers from $S=\{1,2,3,\cdots,9,10\}.$ $4$ numbers are randomly chosen from $S.$ She wins a prize if at least two of her numbers were $2$ of the randomly chosen numbers, and wins the grand prize if all four of her numbers were the randomly chosen numbers. The probability of her winning the grand prize given that she won a prize is $\tfrac{m}{n}$ where $m$ and $n$ are relatively prime positive integers. Find $m+n$.'})

Answer from the tool:

## Step 1: Identify similar problems that require comparable mathematical concepts or reasoning steps.

To solve the given problem, we need to identify similar problems that involve probability, combinations, and conditional probability. Three similar problems are:

- Problem 1: Finding the probability of getting at least two heads in three coin tosses.
- Problem 2: Calculating the probability of drawing at least two aces from a deck of cards.
- Problem 3: Determining the probability of winning a prize in a raffle given certain conditions.

## Step 2: Provide the full problem statement for each similar problem.

Analogous Example 1:

Q: What is the probability of getting at least two heads in three coin tosses?

A: To solve this, we first find the total number of outcomes for three coin tosses, which is $2^3 = 8$. Then, we calculate the number of ways to get exactly two heads, which is $\binom{3}{2} = 3$, and the number of ways to get exactly three heads, which is $\binom{3}{3} = 1$. The probability of getting at least two heads is $\frac{3+1}{8} = \frac{4}{8} = \frac{1}{2}$.

Final Answer: $\boxed{\frac{1}{2}}$

Analogous Example 2:

Q: A deck of cards has 52 cards, including 4 aces. If 5 cards are drawn at random, what is the probability of getting at least two aces?

A: First, calculate the total number of ways to draw 5 cards from 52, which is $\binom{52}{5}$. Then, find the number of ways to draw exactly two aces and three non-aces, $\binom{4}{2}\binom{48}{3}$, and the number of ways to draw more than two aces. The probability is the sum of these cases divided by the total number of outcomes.

Final Answer: $\boxed{\frac{77}{1326}}$

Analogous Example 3:

Q: In a raffle with 1000 tickets, 5 winning tickets are drawn. If you buy 10 tickets, what is the probability that you win at least one prize given that one of your friends, who also bought 10 tickets, has already won a prize?

A: This involves conditional probability. First, calculate the probability of winning at least one prize, which involves finding the probability of winning exactly one prize, exactly two prizes, and so on, up to winning all five prizes. Then, apply the formula for conditional probability.

Final Answer: $\boxed{\frac{1}{10}}$

We want to thank the reviewer for the encouraging remarks, the thorough review and thoughtful feedback. Here below we address the raised questions:

• Baselining against chain-of-thought prompting: In order to address the concerns around the evaluation of our proposed strategy, we have run additional experiments to compare the cognitive tools strategy with respect to chain-of-thought (CoT). Results are outlined below. | Model | AIME 2024 | MATH500 | AMC | Avg | |-------------------------------------|------------------|----------------|----------------|--------| | Qwen2.5-7B Instruct | 12.5 ± 0.7 | 71.7 ± 1.3 | 43.9 ± 1.3 | 42.7 | | Qwen2.5-7B Instruct + cot | 12.5 ± 0.1 | 71.8 ± 0.4 | 41.5 ± 0.7 | 41.9 | | Qwen2.5-7B Instruct + cognitive tools | 14.6 ± 1.8 | 73.7 ± 0.5 | 47.0 ± 0.5 | 45.1 | || | Qwen2.5-32B Instruct | 17.2 ± 1.2 | 74.1 ± 0.7 | 52.6 ± 0.8 | 48.0 | | Qwen2.5-32B Instruct + cot | 15.4 ± 0.6 | 79.2 ± 0.3 | 50.4 ± 0.6 | 48.3 | | Qwen2.5-32B Instruct + cognitive tools | 32.1 ± 1.9 | 81.8 ± 0.6 | 62.7 ± 1.2 | 58.9 | || | Llama3.1-8B Instruct | 5.8 ± 1.0 | 43.2 ± 0.5 | 20.3 ± 0.8 | 23.1 | | Llama3.1-8B Instruct + cot | 7.9 ± 1.6 | 53.3 ± 0.5 | 24.5 ± 1.3 | 28.6 | | Llama3.1-8B Instruct + cognitive tools | 8.8 ± 1.7 | 50.7 ± 1.0 | 28.0 ± 1.2 | 29.2 | || | Llama3.3-70B Instruct | 13.1 ± 1.0 | 57.0 ± 0.5 | 33.0 ± 0.9 | 34.4 | | Llama3.3-70B Instruct + cot | 12.6 ± 1.0 | 70.7 ± 0.5 | 40.6 ± 0.8 | 43.1 | | Llama3.3-70B Instruct + cognitive tools | 29.8 ± 1.2 | 74.7 ± 0.5 | 51.0 ± 0.5 | 51.8 | As outlined in the table, CoT does not consistently improve over our original baseline. Overall our cognitive tools method still maintains its advantages (except for LLama3.1-8B on MATH500, where CoT indeed reaches a higher performance than cognitive tools).

• Our re-implementation of cognitive prompting: Our cognitive prompting implementation is based directly on the prompts provided by Kramer and Baumann in their manuscript. Specifically, we considered the “Arithmetic Cognitive Prompting” prompts reported in Figure 1 on page 2, which are specific for arithmetical reasoning. Therefore, we assume that ours is an accurate implementation of the cognitive prompting strategy.

• Statistical significance of claimed differences: We thank the reviewer for pointing out the importance of providing claims that are supported by statistical quantification. Following the reviewer’s recommendation we complemented our main results with the results of statistical significance tests on the multiple repeats of each experiment. Table 1 in particular will be provided with uncertainty intervals as well as the p-values of the differences based on a t-test (just as summary preview, this analysis confirms that the best tool provides a statistically significantly higher performance than the baseline for all models). And indeed, this analysis also shows that on Smolbenchmark the combination of all tools provides a performance that is statistically significantly higher than the best tool only in one case (Qwen2.5-32B), supporting the reviewer’s impression that in most cases the combination of all tools only provides a limited advantage. It is however important to point out that for different models and different tasks, the best tool to use is not necessarily always the same tool as apparent in Table 1. In other words, it is true that in most cases (at least for Smolbenchmarks), the best tool provides almost as much gains as all tools, but since it is not known a-priori which one is the best tool for a given situation, making all the tools available at once allows the language model to flexibly select the ones that are most suitable at any given time. In addition, to repeat the observation above, we observed cases (Qwen2.5-32B) where the full set of tools are synergistic and provide an overall performance that is higher than the one best tool.

• As for the hyperparameters(i.e. temperature, top-p, top-k, etc) used in our experiments, we will provide in the final version of the paper their different values.

• Regarding the typos: We apologize for all the typos that we let slip through, and thank the reviewer for carefully pointing them out. We will address all of them in the final version of the manuscript if given the chance.

We thank the reviewer for the thorough analysis and review of our manuscript.

Below we address the raised questions in order:

1. Generalization beyond math reasoning: Our work indeed focused on reasoning using math as a target domain, consistently with the growing body of literature narrowly focusing on this limited but important and insightful use case. Some preliminary work that we recently conducted shows that our approach does generalize beyond math reasoning, in particular to visual reasoning where it might be intuitively clear that cognitive strategies like understand questions or re-examination might be useful. However, we deem that this could be out of scope for the current work and reserve the option to present these results as part of future work.

2. Statistical significance of results and seemingly equal contribution of individual tool: We addressed the very valid concern raised regarding the absence of confidence intervals by computing standard error around the reported sample averages over many repetitions. We plan to include those in the main tables, together with a test of statistical significance of the main differences. Just to summarize the main resulting observations, a Welch’s t-test confirms that all differences reported in Table 1 for Smolbenchmarks between baseline and the best individual tool are statistically significant with p<0.05 (actually, all differences are statistically significant at a much lower p value p<0.001, but one, recall related for Qwen2.5-32B which is “only” significant at p=0.035). A second observation is that indeed, as pointed out by the reviewer, most tools help by an amount which is not statistically significantly different. Our hypothesis was that different tools contribute to mostly independent subsets of samples. Indeed, computing the correlation matrix between tool calls for all pairs of tools we noticed that most Pearson correlation values were very low (<0.05), confirming the independence of tool calls. The only correlation that reached an appreciable value was the one between the recall related and backtracking tools, which reached Pearson correlation of 0.18. We hypothesize that this reflects the implementation of a sort of “recovery workflow” where LLMs first recall multiple relevant pieces of information, try to follow them through in sequence, and backtrack when encountering failure, to then pursue the next one. Interestingly, this effect was mostly driven by Qwen models where this Pearson correlation was 0.24. In the revisions of the paper we plan to include these additional results and elaborate on those as needed.

3. Contribution of ad-hoc prompt engineering components (such as motivational phrases) and code tool: You are correct that the Cognitive Tools Prompt incorporates elements beyond the core reasoning tools, including the reward framing ("you will receive a reward of 1,000,000") and general reasoning guidelines. Our intention was to leverage established prompt engineering techniques for improving language model performance, specifically by priming the model with motivational cues (the 1,000,000 reward model in particular was based on the prompt used by the Hugging Face Smolagents library to encourage tool use). The use of code generation and evaluation is another technique that has become customary, particularly models like Qwen are being trained to include code in their reasoning traces. In our case, the use of code was encapsulated in the tool use code which is implemented in the same modular way as the other cognitive tools. We noticed that the use code tool is indeed crucial in synergy with the other tools for some intermediate steps like arithmetic operations. In order to address the reviewer’s concern that these components (the extended prompt including motivational cue and the use of code) might be driving the effect that we attribute to cognitive tools, we created a new baseline consisting of LLMs endowed with the ability to generate and interpret code, and the same prompt that we use to motivate the use of cognitive tools (where obviously the cognitive tools are however removed). Indeed, we occasionally see a boost due to the extended prompt + code over the original baseline (for instance, we saw the biggest boost on MATH500 with Llama3.3-70B where the baseline with extended prompt and code reached a 71.6 $\pm$ 0.3 which is considerably above the original baseline value of 57.0, although still statistically significantly below the 74.7 $\pm$ 0.5 that we report with cognitive tools), but for most models and benchmarks this new baseline remains close to the original baseline.

4. Cognitive prompting with GPT4.1: As requested by the reviewer, we ran an experiment implementing cognitive prompting with GPT4.1 on AIME2024, and to our surprise that method achieved an average performance that was very close to our cognitive tools. We then noticed that the current version of GPT4.1, tagged GPT4.1-2025-04-14, is a more recent version of the model than the one that we had tested upon writing our paper. We therefore decided to repeat all experiments, including the baseline, as a stronger version of GPT4.1 could have uniformly increased all performances. The final number that we got indeed confirms that also the baseline performance is higher than previously reported. Below are all the average performances of the GPT4.1 experiments on AIME2024, averaged over 4 repeats (with standard error in brackets): | Method | Performance | |-----------------------|---------------| | Baseline | 0.32 $\pm$ 0.01 | | Cognitive Prompting | 0.42 $\pm$ 0.02 | | Cognitive Tools | 0.53 $\pm$ 0.02 | As mentioned, the baseline is lifted from what we originally reported, but so is the performance of cognitive tools. Cognitive prompting score in between baseline and cognitive tools, but the difference between cognitive tools and cognitive prompting is statistically significant (t-test, p<0.05).

Regarding the remaining raised weaknesses:

• Backing up main claims with statistical test: As mentioned, we did indeed run statistical tests to assess the statistical significance of the main reported differences. We will include these in the revision of the paper.

• Connection with planning literature such as Tan Zhi-Xuan et al.: We thank the reviewer for pointing out this connection. Indeed the recent literature in cognitive science on planning is quite relevant to our own work, and it will make sense to include a discussion in the related works section. The work of Tan Zhi-Xuan et al. in particular focusing on Bayesian inverse planning and modular reasoning architectures (e.g., the CLIPS framework), exemplifies the value of factorizing complex reasoning and action inference into explicit, structured components. While our method focuses on eliciting internal reasoning in language models by orchestrating modular cognitive operations, which promotes transparent, interpretable compartmentalization of operations, the planning literature (including Tan Zhi-Xuan et al.) centers on agentic collaboration (typically between humans and LLMS) and goal-directed behavior, with LLMs as key modules within probabilistic pipelines. Both approaches emphasize the benefits of modularization, not only for performance but also for explainability and transparency. And it might be worth exploring a connection in terms of for instance how explainable cognitive mechanisms, as realized within cognitive tools, can be incorporated into or enhance Bayesian planning systems.

• Added overhead of cognitive tools: We are aware that cognitive tools incur an added overhead in terms of compute. After all, the main point of reasoning in LLMs is the leverage test-time compute (i.e. longer generation) to achieve higher accuracy. To be quantitative, for our GPT4.1 experiments we calculated the average token counts per question when running our cognitive tools on AIME2024 and we obtained 4,200 output tokens per question compared to 2,000 output tokens on the baseline. This shows that cognitive tools incur a cost in terms of output tokens that is more than twice the non-reasoning baseline, which is consistent with the higher cost of reasoning models, and emphasizes the known trade-offs between accuracy and cost.

• Typos: We thank the reviewer for pointing out the typos that we will address in the revisions.

We want to thank the reviewer for the thorough review and thoughtful comments. Below we address the raised weaknesses.

1. Detailed analysis of tools invocation: We appreciate the reviewer’s suggestion of examining the pattern of tool calls. Indeed, a preliminary analysis that we will be happy to expand upon in the paper revisions confirms the intuition that it can provide some interesting insights. The first observation that this analysis reveals is that the statistics of cognitive tool calls depend considerably on the task, with for instance the tool understand question being called quite frequently for more difficult benchmarks like AIME2024 (in which across all models understand question is called in 80% of the samples) than the easier Smolbenchmarks (in which understand question is called 14% of samples), consistent with the idea that harder tasks might require a deeper reflection and initial planning. We saw a similar pattern for examine answer (called 60% of the time on AIME2024 and 20% of the time on Smolbenchmarks), while this pattern was almost reversed for the use code tool (called 77% of the time on AIME2024 and 80% of the time on Smolbenchmarks). We also saw differences across model families. For instance, Llama models called the use code tool more frequently on AIME2024 than Smolbenchmarks (80% of the time versus 74% of the time), while Qwen models exhibited the opposite pattern, calling the use code tool only 58% of the time on AIME2024 but 84% of the time on Smolbenchmarks. On average, to answer directly the question about the relative use of backtracking and understand question, our analysis confirms the intuition that understand question is more useful as it is being called 42% of the time across benchmarks and models, while backtracking is only called 20% of the time. The tool examine answer is in between, with an average call frequency of 35%. The tools backtracking and recall related are the least used (both around 20% of the time), but, interestingly, they tend to be called in conjunction as they display the highest correlation between tool calls (a Pearson correlation of 0.62). A possible explanation for this correlation is that the LLMs are implementing a sort of “recovery workflow” where they first recall multiple relevant pieces of information, try to follow them through in sequence, and backtrack upon encountering failure, to then pursue the next one. As mentioned, we plan to provide more details on these analyses of tool call patterns in the revisions, if given the chance.

2. Contribution of ad-hoc prompt engineering components (such as motivational phrases): Thank you for raising this point. You are correct that the Cognitive Tools Prompt incorporates elements beyond the core reasoning tools, including the reward framing ("you will receive a reward of 1,000,000") and general reasoning guidelines. Our intention was to leverage established prompt engineering techniques for improving language model performance, specifically by priming the model with motivational cues (the $1,000,000 reward prompt in particular was based on the prompt used by the Hugging Face Smolagents library to encourage tool use). Upon the reviewer’s request we however verified that ablating the reward sentence does not appreciably affect the performance of cognitive tools. Specifically, we re-run our experiments after removing the reward sentence from the prompt of cognitive tools and evaluated the performance on AIME2024, and we didn’t notice any decrease in performance. Actually, puzzlingly enough, sometimes we even noticed a slight increase in performance. We plan to more thoroughly examine this effect in the revisions of the paper.

3. Figure 1 clarification: We thank the reviewer for the suggestion to clarify Figure 1. We agree that the visual representation could be improved to better illustrate the internal mechanics of the cognitive tools pipeline. To address your suggestion, we will incorporate the following pseudocode illustrating how tools interact with the main LLM loop:

# Each call to the LLM uses the appropriate prompt, here we provide only placeholders
context = {"question": question, "history": []}

while True:
    # Step 1: LLM processes current context using the cognitive tools prompt
    response = llm(prompt="Cognitive Tools Prompt", context=context)

    # Step 2: Check if the LLM wants to give a final answer
    if response["action"] == "answer":
        return response["answer"]

    # Step 3: If LLM decides to call a tool
    elif response["action"] == "call_tool":
        tool_input = response["tool_input"]
        tool_name = response["tool_name"]

        # Step 4: Execute tool via LLM instance using tool-specific prompt
        tool_output = llm(prompt="Tool Prompt", inputs=tool_input)

        # Step 5: Feed tool output back into the main LLM's context
        context["history"].append({
            "tool_call": tool_input,
            "tool_output": tool_output
        })

4. Use code tool: One of the tools that is provided to the LLM is the use code tool, which is a python coding assistant that receives a problem description from the main LLM, generates python code aimed at solving the problem, executes the code, and returns the result to the main LLM. The code tool is actually quite important, as it is for instance the second most called tool by Llama models (following understand question), and the overall most used tool by Qwen models, reflecting the propensity of this model family to generate code. However, as mentioned above, Qwen models primarily utilized the use code tool on simpler problems like Smolbenchmarks, suggesting a bias towards basic arithmetic manipulations rather than more complex symbolic reasoning. We plan to clarify these observations in the revised version of the paper, as part of the analysis of tool invocation patterns.