Buffer of Thoughts: Thought-Augmented Reasoning with Large Language Models

We thank Reviewer cwbi for the positive review and valuable feedback. We are glad that the reviewer found that our proposed BoT is more flexible and more efficient, and our method shows clear advantages over existing approaches. Please see below for our responses to your comments.

Q1: What embedding model is used?

A1: In our implementation, we used text-embedding-3-large as the embedding model. We also conduct in-depth experiments about the model size impact for our method. Please refer to global response for detail.

Q2: Information for compute resources?

A2: As mentioned in Line 214, we use NVIDIA A100-PCIE-40GB GPUs for local model like Llama3-8B and Llama3-70B.

Q3: How well the approach scale with different thought templates. If query does not match any of thought templates in the buffer, how well this approach works.

A3:

1. As depicted in Fig.9 and Fig.10, with the number of different thought template increase, the accuracy and reasoning efficiency of BoT steadily improves, it confirms that our method possesses excellent scalability. This is because the buffer manager continually expands the meta-buffer while also utilizing the thought templates obtained from previously solved problems to address subsequent similar problems. The possibility of retrieving suitable thought templates also increases, thus avoiding the need to construct reasoning structures from scratch and thereby enhancing inference efficiency.

2. As discussed in L160-163, for entirely new or unique problems, our method provides three general coarse-grained thought-templates, which could be assigned considering the distilled task. These coarse-grained thought templates ensure that new tasks follow appropriate reasoning processes. This broad guidance offers LLMs greater flexibility during the generative inference process, enabling them to efficiently address new and unique problems.

3. Due to character limits, please refer to Q1 in response to reviewer yiEf for more examples in demonstrating how well our thought templates work.

Q4: Language, typos and references

A4: We sincerely appreciate your attention to the details of our paper. We will carefully address the issues you pointed out regarding language, formatting, and citations, to enhance the professionalism and standardization of our paper.

Q5: Concern about error bars

A5:

1.It is indeed possible to compute error bars for our results. However, we chose not to include them in our initial version to maintain consistency with many prior works, such as ToT, Meta Prompting, and PAL, which also do not provide error bars. This approach ensures a unified basis for comparison across different methods.

2.In response to the reviewer's concern, we have now computed the error bars for our results to enhance the strictness and clarity of our results. The updated results, including error bars, are presented in the table below. We will incorporate these error bars into the final version of our paper to provide a more comprehensive and statistically robust presentation of our results.

As shown in the table, our method achieves stable performance, which further proofs that our method is effective, stable and robust.

Q6: What is logarithmic time?

A6: We perform a logarithmic operation on the obtained time, i.e., Logarithm time = $\ln(t)$. The reason why we do this is that the data range of the inference time between different methods is board, in such cases, the readability and effectiveness of the histogram in conveying information are compromised. By applying logarithmic transformation, the differences in bar heights within the histogram are reduced, rendering the chart more aesthetically pleasing and the information more clearly presented.

Q7: Why is the inference time of BoT not at least twice that of the Expert baseline?

A7: As mentioned in Q6, the formula for calculating the multiple of inference time should be $e^{\ln(t_1) - \ln(t_2)}$, rather than direct comparison. Furthermore, despite our method involving collaboration across multiple components, the inference time required by BoT for certain problems does not significantly exceed that of the baseline. Because our method transforms some reasoning tasks (such as Game of 24 and checkmate in one) by simplifying the multi-step reasoning or heuristic search process into generating a segment of code capable of solving the problem. Specifically, in the expert baseline, solving a problem from Game of 24 need to experiment with multiple combinations and conduct calculations, which is time consuming. On the contraty, our method transform this process into generating few lines of code and execute the code to get the answer, which requires significantly less time compared to the expert baseline, demonstrating the superiority of our approach in addressing certain complex problems.

Thank you for your reply, and we sincerely appreciate your suggestions! Following your suggestion, we will using a logarithmic scale of the y axis instead of logarithmic time in Fig.3 for the final version. Actually, in Fig.3, we have compared our method with PAL (Program-aided Language Models) which utilizes a python script to solve problem. And from the results in Fig.3, we can find that our BoT is faster than PAL because BoT retrieves proper thought templates for accelerating our reasoning process without reasoning from scratch.

If you have any further questions, please feel free to ask. Thank you once again for your invaluable contribution to our research.

Warm Regards,

The Authors

Dear Reviewer cwbi,

We greatly appreciate the time and effort you have invested in reviewing our paper. Your thoughtful questions and insightful feedback have been invaluable. In response to your queries, we provide more explanations about the effectiveness of our proposed approach, and additional analysis experiments.

If you have any further questions, please feel free to ask. Thank you once again for your invaluable contribution to our research.

Warm regards, The Authors

We thank Reviewer upCD for the positive review and valuable feedback. We are glad that the reviewer found that our BoT makes good contributions to the improvements of LLMs, and our method significantly improves LM's reasoning ability in various tasks. Please see below for our responses to your comments.

Q1: Generalizability and robustness of the proposed approach.

A1:

1. Breaking down a complex problem into step-by-step solutions has proven to be an effective way to enhance accuracy and robustness, as demonstrated by ToT, GoT, and Meta-prompting. Moreover, Our BoT additionally introduces a problem distiller to extract the core problem for input task and simplifies the problem-solving procedure by retrieving proper thought templates. Such high-level thought templates discard unnecessary specific details of problem-solving processes and can be instantiated with different problems adaptively, which effectively reduce possible noises caused by previous hierarchical reasoning methods (ToT, GoT).

2. The empirical results have also proved the generalizability and robustness of our BoT. The experiments of Fig.4 demonstrate that our BoT can maintain a robust reasoning process and reduce more noises compared to previous methods when faced with different tasks. And the experiments of Fig.9 reveal that our BoT can generalize to more novel tasks and continually improve the performance.

Q2: Comparison between automatically generated task templates with manually prepared templates.

A2:

1. We appreciate the reviewer's insightful comment regarding the quality of the automatically generated templates. To address this, we conducted comprehensive experiments to compare the automatically-generated templates with manually-designed templates in MATH dataset. Please refer to the PDF in global response for more detailed results.

2. The additional experiments showcase that the automatically-generated templates ensure consistency, efficiency and scalability for more accurate and effective problem-solving, also demonstrated by the results in Fig.9 and 10.

Q3: How template can be revised and improved?

A3:

1. Since there is no suitable metric to evaluate the quality of thought templates and BoT is a training-free reasoning fraemwork, we do not optimize the previously accumulated thought templates but instead add new thought templates. To be specific, we calculate the similarity between new thought templates and those in the meta-buffer to avoid redundancy and repetition. Using powerful LLMs like GPT-4, the quality of the extracted thought templates is relatively high, as demonstrated by our qualitative analysis in Fig.9 and 10.

2. As for the reason why the accuracy increase with more examples received in Fig.9, we have discussed this in Line 465-477. That is because when in the first round, the meta-buffer is empty and the buffer manager is still accumulating thought templates. With the accumulation of thought templates, BoT gradually enhances its ability by utilizing the thought-templates obtained from previously solved problems to help addressing subsequent similar problems, thus contributing to the increase in accuracy.

3. We sincerely thank you for your insightful suggestion. In future work, we plan to design a metric to evaluate the quality of thought templates, allowing for dynamic optimization of the accumulated thought templates. This enhancement may further improve the accuracy and robustness of our method.

Q4: What are the embedding models used for retrieval ? And analysis on the impact of retrieval quality.

A4: We used text-embedding-3-large as the embedding model. For better understanding, we here provide experimental analysis on the impact of retrieval quality on downstream task performance using three different-sized embedding models. Due to the page limits, please refer to the table and analysis in global response.

Q5: In Eq.5, whether we should replace the template based on similarity.

A5: It is noted that our update process only includes adding new thought templates, instead of replacing or optimizing existing ones . As mentioned in above A3, there is no suitable metric to evaluate the quality of thought template and BoT is a training-free reasoning framework. For future work, we will try to effectively evaluate the quality of thought templates, thereby enabling the dynamic optimization of meta buffer.

Q6: How quickly can this approach adapt to an unseen task? Number of examples to achieve reasonable accuracy? Analysis of adaptability and continual learning ability of BoT.

A6:

1. The adaptation is related with task diversity. For tasks with low diversity, such as "Game of 24", our approach can adapt very quickly. For tasks with higher diversity, such as those in MATH dataset, which includes a wide range of problem types (e.g., algebra, geometry...), more examples are required to accumulate a comprehensive set of thought templates. This is necessary to cover the various subdomains of the problems.

2. For tasks with low diversity, one or two examples are often sufficient to achieve reasonable accuracy. For tasks with higher diversity, a larger set of examples is necessary as mentioned above. This ensures that our method remains robust and generalizable across a wide range of problem types.

3. As shown in in Fig.9 and Fig.10 of Appendix, we conduct ablation study on buffer-manager, which demonstrate the extradionary adaptability and continual learning ability of BoT. With more examples received and high-level thought templates accumulated in each round, there is significant improvement in both the overall performance and reasoning efficiency of the model. We have discussed the underlying reasons for these improvements in A3. This continual learning capability is crucial for ensuring that BoT can effectively adapt to new tasks over time, maintaining the efficiency and robustness of our method.

Dear Reviewer upCD,

We greatly appreciate the time and effort you have invested in reviewing our paper. Your thoughtful questions and insightful feedback have been invaluable. In response to your queries, we provide further explanations about the generalizability and robustness of our proposed approach, and more details about our thought-retrieval procedure.

If you have any further questions, please feel free to ask. Thank you once again for your invaluable contribution to our research.

Warm regards, The Authors

We thank Reviewer yiEf for the positive review and valuable feedback. We are glad that the reviewer found that the proposed thought template and meta-buffer is novel, and our method achieve notable improvement across various tasks. Please see below for our responses to your comments.

Q1: How to solve entirely new or unique problems effectively?

A1:

1. As discussed in Line 160-163, for entirely new or unique problems, our method provides three pre-designed coarse-grained thought templates (in Appendix A.3) that can be instantiated based on the distilled task information. These templates offer a good starting point for reasoning when the relavant thought template does not exist.

2. Additionally, these coarse-grained thought templates ensure that new tasks follow appropriate inference processes. This general guidance offers LLMs greater flexibility during the reasoning process, enabling them to efficiently address new and unique problems.

3. During the cold-start reasoning process mentioned above, the buffer manager distills and summarizes new thought templates that can be used to solve similar future problems effectively. This adaptive process ensures that our method remains effective and robust, and could continually adapt to new problems.

4. To give a comprehensive illustration of this reasoning process for new or unique problems, we use the Game of 24 and solving a quadratic equation as examples below:

Game of 24

Input problem:

Let's play a game called 24. You'll be given four integers, and your objective is to use each number only once, combined with any of the four arithmetic operations (addition, subtraction, multiplication, and division) and parentheses, to achieve a total of 24. For example, if the input is 4, 7, 8, and 8, the output could be 7 * 8 - 4 * 8 = 24. You only need to find one feasible solution!

Problem Statement:

Given four integers 4, 7, 8, and 8, use each number exactly once and the four arithmetic operations (addition, subtraction, multiplication, division) to form an expression that equals 24. Parentheses can be used to define the order of operations.

Analysis:

Step 1: Problem Understanding: We need to find a combination of the four given integers and arithmetic operations that results in 24.

Step 2: Input Extraction: The four integers are 4, 7, 8, and 8.

Step 3: Output Requirements: The output should be a valid mathematical expression that equals 24.

Step 4: Constraints:

• Each number must be used exactly once.
• Only the four arithmetic operations can be used.
• Parentheses can be used to define the order of operations.

Step 5: Approach:

• Generate all permutations of the numbers.
• Generate all possible combinations of arithmetic operations.
• Evaluate all possible expressions formed by combining the numbers and operations with different parenthetical groupings.
• Check if any of these expressions evaluate to 24.

Transformed Python Code:

... Due to character limit, we are unable not show the code here, you can refer to A.1.5 in our paper ...

Solving a quadratic equation

Input Problem:

Solving a quadratic equation：

$2x^2-4x-6 = 0$

Solution:

Step 1: Identify the coefficients $a$, $b$, and $c$ from the quadratic equation $(2x^2 - 4x - 6 = 0)$:

• $a = 2$
• $b = -4$
• $c = -6$

Step 2: Calculate the discriminant $D$:

$D = b^2 - 4ac = (-4)^2 - 4 \cdot 2 \cdot (-6) = 16 + 48 = 64$

Step 3: Determine the nature of the roots:

• Since $D = 64 > 0$, the equation has two distinct real roots.

Step 4: Compute the roots using the quadratic formula:

$x = \frac{-b \pm \sqrt{D}}{2a} = \frac{-(-4) \pm \sqrt{64}}{2 \cdot 2} = \frac{4 \pm 8}{4}$

• Calculate the two roots:

  $x_1 = \frac{4 + 8}{4} = \frac{12}{4} = 3$

  $x_2 = \frac{4 - 8}{4} = \frac{-4}{4} = -1$

Answer: The solutions for the quadratic equation $(2x^2 - 4x - 6 = 0)$ are:

$x_1 = 3 \quad \text{and} \quad x_2 = -1$

In summary, as we mentioned in Line 160-163 and the example in Appendix A.3, we can effectively solve entirely new or unique problems. The example above further showcase the robustness of our method when encountering new tasks.

Q2: What happens in the case of unsolvable queries? Does the method allows for uncertainty based abstention?

A2:

1. For possible unsolvable queries, our method attempts to resolve the issue during the instantiation process. Subsequently, an additional inspector reviews both the problem and the reasoning process. If any issues are detected, the unreasonable parts are identified and handed over to the reasoner for revision. If it is concluded that the problem is unsolvable, the reasoning process is terminated immediately to prevent any pollution of the meta-buffer.

2. Due to the page limits, we did not discuss this scenario in our paper, and there were very few unsolvable queries in our experiments. However, we have implemented the inspector mechanism. We will add this part in the final version to provide a comprehensive overview of our approach.

We thank Reviewer hZUQ for the positive review and valuable feedback. We are glad that the reviewer found that the proposed framework is novel, and greatly enhances the accuracy, efficiency, and robustness of LLMs across multiple reasoning tasks. Please see below for our responses to your comments.

Q1: Dependence on Quality of Distilled Thoughts.

A1:

1. We address this by utilizing pre-designed prompt and leveraging state-of-the-art LLMs such as GPT-4, which are capable of generating high-quality, high-level thought templates that strike a balance between abstraction and specificity.

2. To demonstrate the adaptability and robustness of our method, we provide examples from two distinct datasets: GSM8K (grade school math word problems) and MATH (challenging high school math competition problems).

Example 1: GSM8K

Question: A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take?

High-level Thought Template:

Step 1: Identify the amount of each type of material required.

Step 2: Calculate any dependent quantities based on the given relationships.

Step 3: Sum all the quantities to find the total amount of material needed.

Example 2: MATH

Question: At a school, all 60 students play on at least one of three teams: Basketball, Soccer, and Mathletics. 8 students play all three sports, half the students play basketball, and the ratio of the size of the math team to the size of the basketball team to the size of the soccer team is $4:3:2$. How many students at the school play on exactly two teams?

High-level Thought Template:

Given:

Total number of students: T

Number of students in each team: A, B, C

Number of students playing all three sports: y

Step 1: Identify the total number of students T and the number of students in each team A, B, and C.

Step 2: Recognize the overlapping memberships and set up the equation:

$$A + B + C = T + x + 2y$$

Step 3: Substitute the known values for A, B, C, T, and y into the equation.

Step 4: Solve for x, the number of students playing exactly two sports:

$$x = (A + B + C) - T - 2y$$

These examples illustrate that our method can adaptively extract high-quality thought templates for problems of different complexity. For simpler problems from GSM8K, the high-level thought template is straightforward, while for more complex problems from MATH, the high-level thought template is correspondingly complicated. This adaptability and generalizability showcase the robustness of our approach.

Q2: Risk of Overfitting to Distilled Thoughts.

A2:

1. Our thought templates are designed to be high-level and abstract, which do not contain specific reasoning details from previous problems. This abstraction ensures the thought templates provide general guidance that would not cause overfitting problems of LLMs.

2. In scenarios where the meta-buffer contains thought templates derived from a limited set of problem-solving instances that differ significantly from the current problems, we reinstantiate the reasoning process using our manually pre-designed coarse-grained thought templates, as mentioned in Line 450. This approach avoids applying an incorrect type of thought template from the meta-buffer, thereby further mitigating the risk of overfitting.

3. As we mentioned above, our approach ensures that the model retains the flexibility needed to handle a wide range of tasks effectively while maintain adaptability and robustness across various problems.

Q3: Size impact on retrieval time and overall performance. Concern about the Scalability and Maintenance.

A3:

1. Many different problems share similar solutions that could be instantiated from the same high-level thought template. Consequently, even as the meta-buffer grows with more distilled thoughts, the number of unique thought templates remains relatively small as demonstrated by the template distribution of Fig.5. This ensures that our meta-buffer is a lightweight library, and its size has little impact on retrieval time. Moreover, a bigger size of meta buffer would improve the overall performance and the reasoning efficiency of the model as in Fig.9 and Fig.10, with empirical analysis in Line 465-477.

2. Many problems share the same reasoning patterns, we only need to save the high-level thought templates distilled from the solutions of various problems. This approach keeps the size of the meta-buffer relatively small, ensuring its scalability and making it easy to maintain.

Q4: How does BoT prioritize among conflicting reasoning strategies?

A4:

As illustrated in Line 134-144, for each thought template, we have a description $D_{T_i}$ that specifies the types of problems could be applied. BoT uses the embedding similarity between the distilled problem $x_d$ and $D_{T_i}$ to find the most suitable thought template, as mentioned in Eq.2. We only choose the thought template with the highest similarity to the current problem. This ensures that the chosen template aligns closely with the current problem, thereby prioritizing the most relevant reasoning strategy.

Q5: How to handle situations when thoughts are based on incorrect or outdated information?

A5:

1. The thought templates derived from the problem-solving process are high-level and abstract, which means they do not contain specific reasoning details or factual information. This abstraction helps to ensure that the templates provide robust and generalizable guidance without being affected by the incorrect examples or outdated data.

2. Compared to conventional methods that rely on in-context examples or specific databases, our approach leverages high-level thought templates to eliminate detailed reasoning steps. This abstraction helps to avoid potential errors that might be presented in more detailed templates, thereby enhancing the reliability and robustness of the reasoning process.