Decompose, Analyze and Rethink: Solving Intricate Problems with Human-like Reasoning Cycle

We appreciate your affirmation of the contribution of our paper. For your concerns:

Q1: The process for obtaining decomposition demonstrations in the logic heuristics lacks a detailed explanation.

A1: Thank you for your comments. We have provided a detailed explanation for the obtaining of question decomposition demonstrations and examples of prompts for the Decomposition Stage in Appendix B.1. We use a BERT encoder to transform target question Q and human-annotated question decomposition demonstrations into vector representations. Then we use cosine similarity to select the top K (K=3 in our setting) similar demonstrations as logic heuristics.

Q2: Lacking further discussion or evaluations on question decomposition.

A2: Thank you for your insightful comments. Other methods that use question decomposition to solve complex problems typically decompose the original question into sub-questions through a simple prompting method and then solve them step by step, such as the least-to-most approach. In contrast, the DeAR framework we propose not only uses logic heuristics during the Decompose stage to enhance the logic of the question decomposition but also provides more refined planning and updating of the problem-solving process during the Analyze and Rethink stages. This further ensures the reliability of the problem-solving process, preventing the spread of errors. Additionally, the reasoning tree generated by DeAR makes the reasoning process more interpretable.

Q3: The effectiveness of the “self-check” mechanism is not well evaluated.

A3: Thank you for your insightful comments. Here, we verify the effectiveness of the self-check method by comparing the predict accuracy of DeAR and DeAR(w/o self-check) on the ScienceQA dataset. DeAR(w/o self-check) refers to the version where the self-check part is removed during the Analyze stage, while the rest of the implementation process is identical. The experimental results are shown in the following table, from which it can be seen that under different backbones, DeAR has higher predict accuracy, demonstrating the necessity of the self-check method for correcting errors.

Q4: The paper could benefit from improvements in presentation.

A4: Thank you very much for your suggestions. We will carefully review and correct any writing errors.

Thank you for your insightful feedback once again. We hope that our response addresses your concerns and questions. As our author-reviewer discussion nears its end, we'd appreciate knowing if your concerns are resolved. We are open for any further discussion if needed.

We appreciate your affirmation of the motivation of our idea, and the implementation of our approach DeAR. As for your concerns:

Q1: Question decomposition examples.

A1: Thanks for your insightful comments. In our approach, the logic heuristics provided in the problem decomposition prompt vary dynamically depending on the question. The logic heuristics presented in Table 5 are specific to a particular case of a question. For a given question, we select problem decomposition examples from the demonstration pool based on the calculated cosine similarity. Different examples may be selected as part of the prompt for different questions (see Appendix B.1 for a detailed description). This method can effectively adjust the prompt according to the differences in questions, allowing for better decomposition tailored to the characteristics of each problem compared to a fixed prompt.

Q2: Using LLMs for answer scoring.

A2: Thanks for your valuable comments. In section 4.2, line 217, we mentioned that scoring for the answer could also be achieved using other voting or classification methods. In the ToT method, a comparison was made between "value" and "vote" scoring approaches, demonstrating that both are effective for verifying the accuracy of answers[1]. In our paper, for simplicity, we explored the method of directly generating scores using the backbone LLMs , and we will supplement the results obtained using voting in the future.

[1] Tree of thoughts: Deliberate problem solving with large language models.

Q3: An ablation study about Decompose Stage.

A3: Thanks for your insightful comments. Given that each stage in our method is essential for constructing the reasoning tree, it is challenging to perform an ablation study by simply removing one stage. If we were to conduct an ablation study that eliminates the decompose stage, then the subsequent analyze and rethink stages would be hindered, because these stages rely on analyzing and updating the sub-questions that result from the decomposition process. A possible way to validate the effectiveness of the decompose stage is to replace its prompt with prompts from other methods, such as those used for problem decomposition in the Least-to-Most approach. In the table below, we have included supplementary comparative experiments on ScienceQA that demonstrate the superiority of our designed decompose stage. As shown in the table, the performance will decline after we replace our prompt with Least-to-Most decomposition prompt, indicating the effectiveness of our method. We will consider designing additional experiments to further validate the effectiveness of different stages.

Q4: Typos and grammatical problems.

A4: Thanks for your suggestions, we will correct these errors in the revised version.

Q5: Human cognitive reasoning discussion.

A5: Thank you for your insightful comments and for raising a philosophical consideration regarding the comparison between ANNs and human cognition. Regarding the human cognitive reasoning discussion in our paper, we included it with the intention of drawing parallels to human problem-solving strategies, like the decompose, analyze and rethink stages, which can provide intuitive understanding and potentially guide the development of more natural and effective AI systems.

Q6: The first question example in B1.

A6: Thank you for your comments. For each question, we employ cosine similarity to select the most semantically similar questions from the demonstration pool to construct decomposition examples. For instance, regarding the original question "Does the actress who played Elizabeth II speak fluent Arabic?", the questions chosen from the demonstration pool are "Will Queen Elizabeth be buried in the Pantheon?", "Was Elizabeth II the Queen during the Persian Gulf War?", and "Does Elizabeth II reign over the Balearic Islands?". These selected questions and their decomposition examples might contain additional information. However, compared to direct prompting methods (such as using the least-to-most decomposition prompt for prompting), this method is more effective, as we have also demonstrated in our response table for Q3.

Thank you for your valuable feedback. We hope that our response has effectively addressed your concerns and questions. As our author-reviewer discussion comes to a close, we would appreciate if your concerns have been resolved. We are always open to further discussion if necessary.

We appreciate your acknowledgement of on our study motivation, model design, experimental results, and presentation. Your suggestions are insightful for us.

Q1: The effectiveness of the “self-check” method in “Analyze Stage” may need further validation.

A1: Thank you for your insightful comments. Here, we add an ablation study focusing on the self-check mechanism, utilizing the ScienceQA dataset for our analysis, as illustrated in the table below. The results demonstrate that across three LLM backbones, the DeAR model outperforms its counterpart without the self-check method, thereby validating the self-check method's efficacy. Although other methods that employ LLMs for self-correction may not be sufficiently effective, our experiments have demonstrated that incorporating a self-check method during the Analyze stage is very necessary. We intend to incorporate these findings into the updated version of our work.

Q2: More detailed explanation of case studies.

A2: Yes. We'll break down how DeAR enhances the reasoning process with a real example from Figure 9 in Appendix C.4. Imagine we need to solve a comparison question, "#2": "Who is younger between these two directors?" In the Decompose stage, DeAR breaks "#2" into simpler sub-questions, "#3" and "#4", asking for the ages of the directors of "Zakhm" and "Telefono Rosso," which are more manageable for Large Language Models (LLMs) to figure out. Once we have the answers to these sub-questions, DeAR moves on to the Analyze stage. Here, it not only gets the specifics but also spots and fixes a mistake: it corrects the age of Mahesh Bhatt from 70, born in 1954, to the accurate 76 years old, born in 1948. With the correct information in hand, in Rethink stage, DeAR then revisits the original question and makes the necessary update, correcting the initial guess of "Mahesh Bhatt" to the right answer, "Nanni Moretti." This step-by-step approach allows DeAR to catch and correct any faulty reasoning along the way, stopping errors from spreading. In the next version of our work, we'll add more such detailed examples to paint a clearer picture.

Q3: Is the method effective for more complex tasks?

A3: Thank you for your comments. We conduct further experiments based on GPT-4, particularly on the more challenging MATH dataset, to address your inquiry. The results are presented in the table below. For more complicated questions in MATH, DeAR also performs better.

[1] Self-consistency improves chain of thought reasoning in language models.

[2] Large language model guided tree-of-thought.

Q4: Could the authors provide a more detailed efficiency analysis based on GPT-3.5?

A4: Yes. On the ScienceQA dataset, using GPT3.5 as backbone, to ensure a fair comparison, we compare DeAR (with parameters b=1.58 and d=3.62) with ToT, which has the closest values for branch b and depth d (b=3, d=4). We've looked at the average number of API calls for each question, and ACCs on test set, as shown in the table below. It's clear that our method makes fewer API calls on average, which means less time under the same conditions, and achieves higher ACCs at the same time. We'll add more detailed experiments about the average input tokens in the updated version.

Thank you for your constructive feedback. We sincerely hope our response has answered your concerns and questions. As we near the end of this discussion, we would appreciate it if you could let us know whether all your concerns have been addressed. We are open to further discussion if needed.

We appreciate your positive comments on the novelty and efficiency of our DeAR and the affirmation of its superior performance over SOTA methods.

Q1: Lack of ablation studies to analyze the contribution of each individual step.

A1: Thanks for your insightful comments. The construction process of our proposed reasoning tree is such that the three stages—decompose, analyze, and rethink—are indispensable. If we conduct a ablation study that omits one of these stages, for example, removing the decompose stage, then both the analyze stage and the rethink stage would be unable to proceed, as the latter two stages must analyze and update the sub-problems generated from the decomposition. Similarly, eliminating the analyze stage would result in the inability to obtain the rationales for each node, thereby preventing the rethink stage from taking place. Removing the rethink stage would also render the first two stages pointless; the entire framework would then devolve into using a zero-shot approach to directly solve the problem at the root node and obtain the result.

Here, the only part where an ablation study can be reasonably conducted is the self-check method within the analyze stage, as removing self-check will not structurally affect the other two stages. Therefore, we have added an ablation study on the self-check method using the ScienceQA dataset, as shown in the table below. It can be observed that, based on different LLM backbones, DeAR consistently performs better than DeAR without self-check, which also proves the effectiveness of the self-check method. We will include this experiment in the updated version.

Q2: Human evaluations.

A2: Thank you for your comments. We have adopted some approaches to help minimize the bias of annotators towards rationales of varying lengths. For example, we provided each annotator with detailed annotation instructions, allowing them to select the most logical response from the answers given by different models, as shown in Figure 6, Appendix C.2. At the same time, we performed multiple random samplings from the dataset, each time with a different set of annotators, to further prevent the unreliability of results due to the subjective factors of individual annotators. We have five annotators, which is a similar number to that used in other studies employing human evaluation methods, such as in [1]. We will include more details about the sampling and annotation process in the updated version.

[1] Guiding Mathematical Reasoning via Mastering Commonsense Formula Knowledge

Q3: The values for threshold hyperparameters.

A3: Thank you for your insightful comments. We set the thresholds by conducting the threshold combination experiment in Section 5.6. We selected the threshold combination that yields the highest reasoning accuracy for our configuration.

Q4: Typo about table’s name: Table 3 should be Tabel 4.

A4: Thank you for the reminder, we will correct it in the updated version.

Q5: How large is the demonstration pool?

A5: For the ScienceQA dataset, we randomly selected some questions from each topic in the training set and annotated 500 examples as a demonstration pool. For GSM8K and StrategyQA, since their training sets already have annotations for problem decomposition, we directly chose 500 items from them as the demonstration pool. We will include this in the updated version.

Q6: I notice that the authors employ a cosine similarity-based strategy to pick appropriate demonstrations when constructing prompts at the Decompose stage, is the same strategy used to test the performance of the baseline prompting methods?

A6: Thank you for your question. Selecting demonstrations for problem decomposition prompts will only be effective for methods that include a problem decomposition step. Among the baselines in this experiment, only the least-to-most method includes a problem decomposition step. Therefore, for least-to-most, we use the same demonstration pool and cosine similarity selection method as DeAR. As for other baselines, such as CoT, ToT, GoT, since they do not include a problem decomposition step, naturally we do not select decomposition demonstrations for prompting.

Q7: How to assign the proper values if there is no validation set available?

A7: Thank you for your question. A portion of the data from the training set can be selected as a validation set to verify the effects of different threshold combinations, and the optimal combination can be chosen for testing on the test set.

We greatly appreciate your feedback and hope that our responses have addressed your concerns. As we approach the end of our author-reviewer discussion, we would be grateful if your concerns have been resolved. We remain available for any further discussion if needed.

We appreciate your affirmation of the motivation of our paper, the significance of our experimental results and the novelty of DeAR.

Q1: The iterative nature of the cycle may lead to increased computational demands.

A1: Thank you for your comments. In section 5.7, we compare the efficiency of our framework with different variants of ToT/GoT with ChatGLM3-6B backbone. As shown in Figure 5, compared to state-of-the-art ToT/GoT methods, the point corresponding to our method achieve better ACC with less time.

Q2: Include comparisons with more strong baselines (one variant of ToT**[1]** and CoT+SC**[2]**).

A2: Thank you for your comments. We conducted further comparisons with one variant of ToT**[1]** and CoT+SC**[2]** based on GPT-4 backbone, on a more challenging dataset MATH, to address your inquiry. The results are presented in the table below.

Q3: More case studies.

A3: Here, we use the case in Figure 9, Appendix C.4, to further explain how the reasoning process benefit from DeAR’s decomposition, analyze and rethink. First, to answer the comparison question “#2”: Which of these two directors has a smaller age?”, our framework decomposes it into sub-questions “#3”: What is the age of Zakhm’s director?” and “#4”: What is the age of Telefono Rosso’s director?”, which are easier for LLMs to solve. Second, in the Analyze stage, DeAR obtains the answers of sub-question #3 and #4, and also corrects the wrong answer “Mahesh Bhatt was born in 1954, he is 70 years old now” to the right one “Mahesh Bhatt was born in 1948, he is 76 years old now”. After that, the corrected answer to #3 is used to update the answer of #2, and corrects #2’s answer “Mahesh Bhatt” to “Nanni Moretti”. Through the above process, DeAR is able to help correct wrong reasoning steps and avoid error propagation, which is crucial in enhancing model’s reasoning ability. We will include more detailed cases in the revised version.

Q4: Does the method work with stronger LLMs?

A4: Thank you for your insightful comments and valuable feedback. In response to your interest, we have conducted further experiments using the GPT-4 backbone to robustly illustrate the effectiveness of our DeAR framework. As indicated in the response to "W2," we present these results to demonstrate the superiority of our approach. On the MATH dataset, a comprehensive benchmark that challenges models with a variety of mathematical reasoning tasks, DeAR has demonstrated superior performance compared to different SOTA methods, including CoT, CoT-SC, ToT, a variant of ToT, and GoT.

[1] Self-consistency improves chain of thought reasoning in language models.

[2] Large language model guided tree-of-thought.

Thank you for your valuable feedback. We hope that our response adequately addresses your concerns and questions. As our discussion draws to a close, we would appreciate knowing if all your concerns have been resolved. We are open to further discussion if needed.