Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought

We express our sincere appreciation for your comprehensive feedback. We value the opportunity to address the concerns identified. Our responses to the enumerated points are as follows:

---

Q1: Can you provide more detailed examples of tasks that fall into each of the three RG categories? How do these categories affect the model’s optimization process?

R1: Thank you for your enlightening advice. Examples of our different inference granularities are as follows:

• CFRG:

A ship traverses the ocean waves. Below are entries from the ship's logbook:
- The ship sailed 31 kilometers north.
- Heading northward, the ship ventured 6 times further than it did yesterday.

How much distance has it covered from the beginning?

• PFRG:

A boat traverses the ocean waves. Below are entries from the boat's logbook:
- The boat navigated 42 kilometers southward.
- Navigating north, the boat traveled 570 times the distance it had managed yesterday.
- The boat journeyed 165 kilometers in a southerly direction.
- The boat navigated 339 kilometers northward.

What is the distance from its origin?

• CIRG:

Upon the vast expanse of the sea sails a vessel. Herein are the chronicles from the vessel's diary:
- The vessel traveled 564 kilometers towards the west.
- The vessel navigated 856 kilometers eastward.
- The vessel traveled 439 kilometers towards the west.
- The vessel sailed 990 kilometers west.
- The vessel journeyed 291 kilometers in a easterly direction.
- Navigating east, the vessel traveled 490 times the distance it had managed yesterday.
- The vessel navigated 161 kilometers westward.
- The vessel sailed 914 kilometers west.
- The vessel sailed 649 kilometers west.
- The vessel traveled 6 kilometers towards the west.

What is the extent of its travel from the starting location?

As shown in the example, the calculation amount and calculation steps of different cases increase, but the core logic of the problem does not change in any way. In fact, as shown in Figure 4 in the paper, different granularities receive different benefits from self-consistency, and PFRG significantly affects the performance of the Self-consistency optimization strategy. In addition, as shown in Figures 5 and 8 in the paper, different models and different strategies actually improve model performance from different angles by optimizing PFRG and CIRG.

---

Q2: How robust is the combination law of RG across different types of reasoning tasks? Are there specific scenarios where this law does not hold or requires adjustments?

R2: Since the combination law conforms to the weighted harmonic mean, it has excellent and robust properties for diverse scenarios. You only need to ensure relatively independent segmentation into several reasoning granularities, which can effectively utilize our framework. Specifically, for any CoT vertical domain problem, two reasoning granularities can be divided into task-planning and vertical domain solution, which satisfy that:

$$G=\frac{1}{\frac{1}{G_p}+\frac{1}{G_v}+k_1}$$

• If you ignore a certain reasoning granularity, it will only cause $k$ to increase.
• If your reasoning granularity is divided reasonably, it will make $k=0$.
• If you want to further divide $G_v$ into $G_{v1}$ and $G_{v2}$, it is also very convenient. There is no need to consider additional new formulas, because the following formula is satisfied:

$$G_v=\frac{1}{\frac{1}{G_{v1}}+\frac{1}{G_{v2}}+k_2}$$ $$G=\frac{1}{\frac{1}{G_p}+\frac{1}{G_{v1}}+\frac{1}{G_{v2}}+k_2+k_1}$$

---

Q3: This may raise concerns about the generality of the solution on other types of reasonings, for example in StrategyQA, or planning benchmarks.

R3: Thank you for your recognition of our work. In fact, we have conducted non-mathematical experiments. As shown in Figure 3 (c) and Figure 10 in original paper, we have conducted an analysis of multi-hop QA and even multilingual scenarios. In addition, in order to dispel your doubts, we have decomposed the Medical Knowledge Probing problem in detail according to the steps and related medical entities. As shown in Figure 2 in the supplementary material, the combination law is also satisfied in this benchmark.

In addition, as shown in Table 1 below, the MARP we proposed can also work on this data set and can achieve SOTA results on Medical Knowledge Probing, StrategyQA, and HotpotQA. Specifically, our observations are:

• Planning RG Optimization: The performance on Medical Knowledge Probing and StrategyQA has improved, and brought great token savings. It shows that our method effectively reduces the original planning RG and optimized the overall performance according to the combination law.
• Entity RG Optimization: For HotpotQA, MARP did not change token usage significantly but increased accuracy. This suggests that shorter demonstrations in HotpotQA benefit from a smaller planning RG, but instroduce larger the local entity RG. Therefore, according to the combination law, by appropriately keeping the planning RG, entity RG can be optimized based on MARP, which makes the problem difficulty less than the combined RG, thereby improving performance.

We will add more discussion in next version.

Table 1: Effectiveness of MARP strategies on different tasks.

[1] Yang et al. HOTPOTQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. EMNLP2018.

[2] Cheng et al. Adapting Large Language Models via Reading Comprehension. ICLR 2024.

[3] Geva et al. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. TACL 2021.

We extend our gratitude for your insightful feedback. We appreciate the opportunity to address the concerns presented. Below, we provide our detailed responses to each of the points raised:

---

Q1: Lack of Theoretical Analysis: While the paper provides an empirical framework and experimental validation, it may not delve deeply enough into the theoretical understanding of the concept of Reasoning Granularity (RG). A more rigorous theoretical foundation, although difficult, could strengthen the arguments and enhance the contribution of the paper.

R1: Thank you for your suggestion. In fact, our paper has a theoretical analysis in Appendix A.1. Specifically, for the two core concepts of this article, our theoretical analysis is as follows:

1. Reasoning Granularity: In fact, the existence of a universal RG upper limit has actually been done, so we don't need additional elaboration [1]. Based on the proof, it is also obvious that there are different upper bounds for different task conditions.
2. Combination Law: In fact, we provide a theoretical analysis of this formula in Appendix A.1 for combination law, and we prove that it is theoretically consistent with Combination Law under the condition that the RG is relatively independent. We will add more discussion in the next version.

[1] Towards revealing the mystery behind chain of thought: a theoretical perspective. NeurIPS 2024.

---

Q2: The RG framework, although validated across 25 models and 4 tasks, may not be fully generalizable to all types of large language models or reasoning tasks.

R2: Thank you for your insightful comment. In fact, our method can be generalized to other tasks.

Specifically, when encountering a new scenario, we will discuss how to utilize the framework and solve the problems effectively:

• Framework Utilization: Since the combination law conforms to the weighted harmonic mean, it has excellent properties. You only need to be able to ensure relatively independent segmentation into several reasoning granularities, which can effectively utilize our framework.

  Specifically, for any CoT vertical domain problem, two reasoning granularities can be divided into task-planning and vertical domain solution, which satisfy that:

  $$G=\frac{1}{\frac{1}{G_p} + \frac{1}{G_v} +k_1}$$
  • If you ignore a certain reasoning granularity, it will only cause $k$ to increase.
  • If your reasoning granularity is divided reasonably, it will make $k=0$.
  • If you want to further divide $G_v$ into $G_{v1}$ and $G_{v2}$, it is also very convenient. There is no need to consider additional new formulas, because the following formula is satisfied:
  $$G_v=\frac{1}{\frac{1}{G_{v1}} + \frac{1}{G_{v2}} +k_2}$$ $$G=\frac{1}{\frac{1}{G_p} + \frac{1}{G_{v1}} + \frac{1}{G_{v2}} +k_2 +k_1}$$

• Framework Generalization: As shown in Figure 3 (c) in the original article and Figure 2 in the supplementary material, we can also verify the existence of Combination Law on tasks such as HotpotQA and Medical Knowledge Probing. In addition, our proposed MARP strategy significant improves performance (2.23%-20.51%) and reduces token cost (-1.63%-188%) on these tasks and StrategyQA. Specifically, our observations are:

  • Planning RG Optimization: The performance on Medical Knowledge Probing and StrategyQA has improved, and brought great token savings. It shows that our method effectively reduces the original planning RG and optimized the overall performance according to the combination law.
  • Entity RG Optimization: For HotpotQA, MARP did not change token usage significantly but increased accuracy. This suggests that shorter demonstrations in HotpotQA benefit from a smaller planning RG, but instroduce larger the local entity RG. Therefore, according to the combination law, by appropriately keeping the planning RG, entity RG can be optimized based on MARP, which makes the problem difficulty less than the combined RG, thereby improving performance.

We will add more discussion in next version.

Table 1: Effectiveness of MARP strategies on different tasks for GPT3.5.

[1] Yang et al. HOTPOTQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. EMNLP2018.

[2] Cheng et al. Adapting Large Language Models via Reading Comprehension. ICLR 2024.

[3] Geva et al. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. TACL 2021.

Thank you for your valuable feedback. We appreciate the opportunity to address the concerns you have raised. Our responses to the specific points mentioned are as follows:

---

Q1: Although the paper has demonstrated the effectiveness of the RG framework across several models and tasks, it could further strengthen its claims by discussing how these findings might generalize to other types of reasoning tasks or different domains beyond the ones tested.

R1: Thank you for your insightful feedback. We totally agree with your comment.

In fact, since the combination law conforms to the weighted harmonic mean, it has excellent properties. You only need to be able to ensure relatively independent segmentation into several reasoning granularities, which can effectively utilize our framework. Specifically, for any CoT vertical domain problem, two reasoning granularities can be divided into task-planning and vertical domain solution, which satisfy that:

$$G=\frac{1}{\frac{1}{G_p} + \frac{1}{G_v} +k_1}$$

• If you ignore a certain reasoning granularity, it will only cause $k$ to increase.
• If your reasoning granularity is divided reasonably, it will make $k=0$.
• If you want to further divide $G_v$ into $G_{v1}$ and $G_{v2}$, it is also very convenient. There is no need to consider additional new formulas, because the following formula is satisfied:

$$G_v=\frac{1}{\frac{1}{G_{v1}} + \frac{1}{G_{v2}} +k_2}$$ $$G=\frac{1}{\frac{1}{G_p} + \frac{1}{G_{v1}} + \frac{1}{G_{v2}} +k_2 +k_1}$$

Based on this, the granularity of different sizes can be easily divided, making it more practical.

---

Q2: Can you provide some comparative results based on GPT4 or other models with stronger reasoning capabilities as baselines to demonstrate the performance improvement of RG in path planning?

R2: Thank you for your suggestion. As shown in Figure 1 in the supplementary material, GPT4o also conforms to the combination law. Moreover, compared with GPT3.5, both CFRG and IFRG have significantly improved. However, due to the current powerful capabilities of GPT4o, it is difficult to measure the IFRG of the model in other benchmarks such as HotpotQA. Therefore, we did not include GPT4o into the scope of verification in the main experiment.

In addition, as shown in Table 1 below, we found that on the GPT4o model, MARP strategy also achieved the SOTA effect.

We will add more discussion in the next version.

Table 1: Effectiveness of MARP strategies on BigGSM on GPT4o.

Thank you very much for your careful review and affirmation of our paper.

Q1: Check the writing and grammar. Some occasional typos or misused commas.

R1: Thank you for your constructive suggestions. We will correct these issues one by one in the next version.

Thank you for your insightful feedback. We appreciate the opportunity to address the concerns raised. Below are our responses to the points mentioned:

---

Q1: When evaluating RG on multi-hop question answering, the difficulty of sub-questions in each hop is measured by the number of entities, which may not necessarily be the case. Can you justify this choice?

R1: Thanks for your constructive comment. In fact, the reasoning path of the answer to this question is completely affected by several core entities and entity relations for multi-hop data construction, which are also mentioned in HotpotQA original paper[1]. Inspired by this, we measure the number of entities as the difficulty of sub-questions in each hop .

---

Q2: Why does the categorization use 10% and 90% as the cut-off points? Is there statistical support for this categorization? Since most tasks fall in the difficult range of 10% to 90%, what insights does this framework offer for optimization within this range?

R2: Thanks for your insightful feedback. Our intuition for using this classification is that a model with 90% accuracy actually means almost complete mastery, while 10% accuracy means absolutely no ability to do it.

In addition, we have conducted preliminary experiments on multiple models and found that no matter how the prompt changes, the accuracy difference will not exceed 2%. The specific information is shown in Table 1, and we will provide more discussion in subsequent versions.

Table 1: The performance of different prompts at different reasoning granularities.

---

Q3: How would the RG framework perform different types of reasoning tasks beyond those covered in the experiments?

R3: Thank you for your insightful comment. From an application perspective, our mechanism framework is universal and can quickly adapt to a variety of new scenarios.

For example, when encountering a new scenario, we will discuss how to utilize the framework and solve the problems effectively:

• Framework Utilization: Since the combination law conforms to the weighted harmonic mean, it has excellent properties. You only need to be able to ensure relatively independent segmentation into several reasoning granularities, which can effectively utilize our framework.

  Specifically, for any CoT vertical domain problem, two reasoning granularities can be divided into task-planning and vertical domain solution, which satisfy that:

  $$G=\frac{1}{\frac{1}{G_p} + \frac{1}{G_v} +k_1}$$
  • If you ignore a certain reasoning granularity, it will only cause $k$ to increase.
  • If your reasoning granularity is divided reasonably, it will make $k=0$.
  • If you want to further divide $G_v$ into $G_{v1}$ and $G_{v2}$, it is also very convenient. There is no need to consider additional new formulas, because the following formula is satisfied:
  $$G_v=\frac{1}{\frac{1}{G_{v1}} + \frac{1}{G_{v2}} +k_2}$$ $$G=\frac{1}{\frac{1}{G_p} + \frac{1}{G_{v1}} + \frac{1}{G_{v2}} +k_2 +k_1}$$

• Framework Generalization: As shown in Figure 3 (c) in the original article and Figure 2 in the supplementary material, we can also verify the existence of Combination Law on tasks such as HotpotQA and Medical Knowledge Probing. In addition, our proposed MARP strategy significant improves performance (2.23%-20.51%) and reduces token cost (-1.63%-188%) on these tasks and StrategyQA. Specifically, our observations are:

  • Planning RG Optimization: The performance on Medical Knowledge Probing and StrategyQA has improved, and brought great token savings. It shows that our method effectively reduces the original planning RG and optimized the overall performance according to the combination law.
  • Entity RG Optimization: For HotpotQA, MARP did not change token usage significantly but increased accuracy. This suggests that shorter demonstrations in HotpotQA benefit from a smaller planning RG, but instroduce the larger local entity RG. Therefore, according to the combination law, by appropriately keeping the planning RG, entity RG can be optimized based on MARP, which makes the problem difficulty less than the combined RG, thereby improving performance.

We will add more discussion in next version.

Table 1: Effectiveness of MARP strategies on different tasks.

[1] Yang et al. HOTPOTQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. EMNLP2018.

[2] Cheng et al. Adapting Large Language Models via Reading Comprehension. ICLR 2024.

[3] Geva et al. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. TACL 2021.