Exchange of Perspective Prompting Enhances Reasoning in Large Language Models

Thank you very much for your insightful comments and feedback on our manuscript. Below, we provide a detailed response to the points you have raised.

Regarding Weaknesses

W1: ...usage of PEC and QR may have limitations. While these techniques might appear effective for simpler tasks...

We fully agree with your insight regarding the potential for EoP-induced errors due to question augmentation. In fact, the performance gain of EoP is not from the rephrasing question. Our experiments show that the rephrased questions perform worse than the original questions, as seen in below table:

However, when combining the outputs from both branches using the EoP framework, the overall performance improves. We analyze the enhancement results come from two main factors: (1) Error Correction, where insights from one branch can rectify misinterpretations from another, thereby improving problem analysis accuracy, and (2) Complementary Information, where merging branches provides more extensive and holistic insights to problem-solving. We have added a detailed discussion of these findings to the manuscript (line 275 ~ 285).

W2: ...the time and cost should be reported as the efficiency and the effectiveness of the methods should be a trade-off.

We have indeed carefully designed the EoP framework to mitigate this issue. To ensure efficiency, the iteration process in EoP will terminate upon meeting one of the following conditions: (1) Consensus Across Branches, (2) Stability Within Branch Consequently, the required number of interactions remains low. We provide a detailed comparison between EoP and PHP as shown in the table below.

So EoP does not lead to a significant increase in inference cost. We have added these details in the manuscript (line 260 ~ 269, line 365 ~ 367).

W3: ...It would also be interesting to see how EoP performs on more challenging benchmarks, such as OlympiadBench

To address concerns about the generalizability on challenging benchmarks, we have conducted OlympiadBench test. The results are as follows, EoP outperformed PHP by 2.6% for Qwen-2.5-7b and by 3.5% for Qwen-2.5-72b for OlympiadBench dataset .

The consistent and reliable results highlight the EoP’s robustness and its potential for enhancing reasoning capabilities. We have added detailed experimental results into the manuscript (line 260 ~ 269).

W4: ...Testing on other models, such as Llama 3.1, would strengthen their claims.

See feedback of w3, we have conducted experiments using the open-source LLMs Qwen-2.5-7b and Qwen-2.5-72b. The results are positive.

W5: ...it would be interesting to compare EoP's performance with massive sampling approaches...

We compared EoP with SC, for fair compare, we keep the reasoning interaction at the same level. The results are as below, EoP can outperforms SC. We have added a detailed results to the manuscript (line 257 ~ 269).

W6: ...The prompt and some hyperparameters are missed like the temperature used.

Thank you for raising this important point. The prompt we used are inclueded in the appendix A, we have added the LLM setting in the experiment description. (line 215)

We appreciate your feedback and are committed to addressing the concerns raised in your comments.

Q1: Why do rephrased questions consistently exhibit a lower pass rate compared to the original ones? Are LLMs capable of accurately rephrasing complex questions, especially those with intricate logical flows, such as Olympiad-level problems?

(1) We analyze that the lower pass rate for rephrased questions is due to the need for the LLM to have an accurate understanding of the problem. If the model's understanding is biased, the rephrased question will also be biased, leading to incorrect answers. Therefore, using the "rephrase question then answer" paradigm might reinforce these errors.

(2) Accurately rephrasing complex questions is a very challenging task, requiring the model to have a comprehensive and deep understanding of the problem.

If we examine the performance of LLMs in mathematical reasoning tasks, we can find that the majority of errors can be attributed to two main points: <1> Question Misunderstanding, and <2> Value Calculation Error. Among these, Question Misunderstanding is the more significant factor.

Different ways of presenting a problem can expose different aspects of the problem. We can leverage this by exchanging the different pieces of information presented, which can achieve a complementary effect. This allows the LLMs to gain a more comprehensive understanding of the problem, and this is the core idea behind our proposing EoP. Our experiments show that for more difficult problems, EoP can achieve better effectiveness. For example, as shown in Figure 4 (line 396 ~ 413) in our manuscript, performance improvements increase with the level of difficulty.

Q2: Why does EoP underperform in InterAlgebra and Precalculus tasks on the MATH dataset, especially when compared to previous state-of-the-art reasoning methods?

ToRA achieves the best performance on these tasks. However, ToRA relies on program-based methods, which require generating and executing specific code. This introduces additional complexity and dependencies. In contrast, EoP directly uses CoT reasoning, which does not depend on any code interpretation. This makes EoP simpler and more flexible to implement.

When comparing EoP with PHP (both using the same CoT paradigm), EoP demonstrates significant performance improvements:

• On the InterAlgebra dataset, EoP achieves an 8.9% improvement over PHP.
• On the Precalculus dataset, EoP achieves a 7.0% improvement over PHP.

These results suggest that while EoP may not match the performance of program-based methods like ToRA, it still represents a significant advancement within the CoT framework.

Q3: From my perspective, a single iteration in EoP requires double the inference time compared to PHP. Therefore, when considering the overall cost and time, EoP demands approximately four times the resources of PHP.

When examining the reasoning phase, take the interaction numbers using the Math dataset with Qwen-2.5-72B as example. The table below shows the distribution of interaction numbers:

(Note: Each rephrasing consumes one LLM query, so the actual total number of LLM queries = interaction number + 1. Here, we focus on the interaction number.)

The average interaction number can be calculated as follows: $2 \times 80.2 \\%+4 \times 3.94 \\%+...+18 \times 0.02 \\%=2.8456$

From the table, we can find that the interaction number is 2 in 80.2% of the cases, indicating that the original branch and augmented branch often reach a consensus in the first round without additional information exchange. True information exchange between the two branches occurs in only 19.8% of the cases, and the performance improvement of EoP is primarily attributed to this portion.

Please feel free to contact us if you have any further questions.

Thank you very much for your insightful comments and feedback on our manuscript. Below, we provide a detailed response to the points you have raised.

Regarding Weaknesses

W1: Redundant Expressions .... but instead add to the burden of reading. I think it is better to simplify this section.

Thank you for your suggestion to simplify Section 2.1. When organizing the paper, we indeed considered the readability and ease of understanding for our readers. To achieve this, we included many figures to provide visual aids that can help readers grasp the key concepts of the EoP framework more easily. However, while figures are useful for providing an intuitive understanding, they still lose some important details and nuances that are essential for a comprehensive understanding of the method. Therefore, we aimed to balance visual simplicity with technical depth by including detailed explanations and formulas in Section 2.1. This section is designed to be a comprehensive and accurate reference for readers who wish to delve deeper into the EoP framework.

W2: Limited Novelty... Some existing work have proposed such algorithm for prompt engineering

We acknowledge that rephrasing and rewriting prompts to improve model accuracy is a well-explored area, and several existing works have proposed various techniques for prompt engineering. However, our findings reveal a nuanced aspect that sets our work apart from previous research.

In our study, we observed that rephrasing the questions did not consistently lead to improved performance. As shown in Table 3, the performance of the augmented (rephrased) branch was actually lower than the original branch for both prompt settings:

This suggests that rephrasing questions can sometimes introduce inaccuracies or nuances that negatively impact model performance. Despite this, we found that combining the outputs from both the original and augmented branches using our EoP framework resulted in a significant improvement in overall performance, demonstrating the effectiveness of leveraging complementary information from different branches.

The novelty of our approach lies in the integration and ensemble of multiple branches, rather than the individual rephrasing techniques themselves. By utilizing the complementary information from different branches, the EoP framework can mitigate the negative effects of rephrasing and enhance the robustness and accuracy of the model's responses.

Thank you very much for your insightful comments and feedback on our manuscript. Below, we provide a detailed response to the points you have raised.

Regarding Weaknesses

W1: ... increase in computational cost ... The paper does not appear to discuss computational cost in detail.

We have indeed carefully designed the EoP framework to mitigate this issue. To ensure efficiency, the iteration process in EoP will terminate upon meeting one of the following conditions: (1) Consensus Across Branches, (2) Stability Within Branch Consequently, the required number of interactions remains low. For clarity, we provide a detailed comparison between EoP and PHP. The average number of interactions required in the reasoning phase is shown in the table below.

So the EoP framework is efficient, despite employing two branches, EoP does not lead to a significant increase in inference cost. We have added these details in the manuscript (line 260 ~ 269, line 365 ~ 367).

W2: ...Solely using GPT may not be sufficient to showcase how EoP outperforms other existing baselines...

Thank you for raising this important point. To address concerns about the generalizability of our method beyond the GPT family, we have conducted additional experiments using the open-source LLMs Qwen-2.5-7b and Qwen-2.5-72b. Furthermore, to verify performance on more challenging datasets, we have added the Olympiad dataset. The results are as follows:

The consistent and reliable results across various datasets and LLMs highlight the EoP’s robustness and its potential for enhancing reasoning capabilities. We have added detailed experimental results into the manuscript (line 260 ~ 269).

W3: ... its applicability to a broader range of reasoning tasks, such as GPQA and other challenging datasets, remains unexplored...

Based on the feedback from W2, we added the Olympiad dataset for testing. EoP outperformed PHP by 2.6% for Qwen-2.5-7b and by 3.5% for Qwen-2.5-72b.

W4: ... the paper does not investigate the potential for EoP-induced errors that might arise from question augmentation...

We fully agree with your insight regarding the potential for EoP-induced errors due to question augmentation. Your concern is well-founded, and it highlights an important aspect of our framework that we should address more thoroughly.

To clarify, the performance gain of EoP is not from rephrasing the question. In fact, our experiments show that the rephrased questions often perform worse than the original questions. As demonstrated in Table 3 (line 287~300), we provide the performance metrics for each branch:

These results indicate that the augmented branch consistently performs slightly worse than the original branch. However, when combining the outputs from both branches using the EoP framework, the overall performance improves. Specifically, the EoP framework achieves a higher accuracy than either branch alone.

We analyze the enhancement results come from two main factors: (1) Error Correction, where insights from one branch can rectify misinterpretations from another, thereby improving problem analysis accuracy, and (2) Complementary Information, where merging branches provides more extensive and holistic insights to problem-solving. So EoP can effectively mitigating the potential inaccuracies introduced by rephrased question.

We have added a detailed discussion of these findings to the manuscript (line 275~285).

Regarding your questions

Q1: ... Olympiad-style questions.. how many reasoning iterations would be required in the EoP process?

See feedback of W3.

Q2: what happens if smaller models, such as llama-3.1/3.2, Qwen2-72B, or perhaps other closed-source models, are used?

See feedback of W2.

Q3: ... rephrase questions inaccurately, ... prevent error propagation within the iterative feedback loop?

See feedback of W4.

Thank you very much for your insightful comments and feedback on our manuscript. Below, we provide a detailed response to the points you have raised.

Regarding Weaknesses

W1: This work is a bit incremental compared to PHP. The all use previous answers to refine ...

Indeed, PHP has been a significant source of inspiration for our research. However, our motivation and objective are different. While PHP focuses on refining its own reasoning paths using the previous generated outputs, our approach, EoP aims to incorporate external perspectives to overcome the intrinsic capacity constraints of LLMs. Besides, the flexibility of our framework allows for the integration of various types of external insights, not limited to just answers from another branch.

W2: Due to the iterative and two-branch nature of this method, this will significantly increase the cost.

We have indeed carefully designed the EoP framework to mitigate this cost issue. To ensure efficiency, the iteration process in EoP will terminate upon meeting one of the following conditions: (1) Consensus Across Branches, (2) Stability Within Branch. Consequently, the required number of interactions remains low. We provide a detailed comparison between EoP and PHP.

So despite employing two branches, EoP does not lead to a significant increase in inference cost. We have added these details in the manuscript (line 260 ~ 269, line 365 ~ 367).

W3: ... Table 1 and Table 2 look marginal compared with the previous state-of-the-art methods.

We understand your concern regarding the marginal improvements observed in the arithmetic reasoning tasks. It is important to note that the arithmetic datasets, such as the Aqua dataset, often consist of short and straightforward questions. For example, a typical question from the Aqua dataset might be:

"A trader sold an article at a profit of 20% for Rs.360. What is the cost price of the article?"

These questions are relatively easy for LLMs to understand, and the additional perspective from the other branch provides only a finite improvement However, the true strength of our EoP framework is dealing with more complex and nuanced questions. Consider a more challenging problem like:

"A worker receives an annual wage of 20, which he always deposits into a savings account at the end of the year. By the end of the third year (when he makes the third deposit), he wants to have at least 66,200 in the account to finance the purchase of a house. What is the minimal compound interest rate that the savings account must provide? Express your answer as a percentage, but do not include the percent sign".

In such complex scenarios, LLMs can easily get mixed up and make mistakes because they lack a deep understanding of the problem. EoP is specifically designed to address these challenges by allowing the two branches to exchange perspectives and collaboratively refine their solutions.

W4: The choice of datasets is narrow because it focuses mostly on math and arithmetic reasoning.

While our current focus is on mathematical reasoning, we recognize the importance of extending our method to other types of reasoning tasks. We view this as a promising direction for future work. The principles underlying our EoP framework, such as the collaborative refinement of solutions through multiple perspectives, are general and could potentially be applied to a wide range of reasoning tasks.

W5: Only GPT family models (GPT-3.5-turbo and GPT-4) have been tested.

To address concerns about the generalizability of our method beyond the GPT family, we have conducted additional experiments using the open-source LLMs Qwen-2.5-7b and Qwen-2.5-72b. The results are as follows:

The consistent and reliable results across various LLMs highlight the EoP’s robustness and its potential for enhancing reasoning capabilities. We have added detailed experimental results into the manuscript (line 260 ~ 269).

Regarding your questions

Q1: Does your method work for other types of reasoning tasks, such as logical reasoning, or BBH dataset?

see feedback of W4.

Q2: Does your method work for other models other than GPT?

see feedback of W5.

Dear authors,

Thanks a lot for your rebuttal. It is great to see that your approach is working well on another LLM Qwen for a more challenging dataset Olympiad. I have read other reviews and your rebuttals as well. I am still a bit concerned that the novelty is limited and the improvement is not significantly large compared to the additional cost. So I am keeping my score as 6 (I am still in favor of acceptance!).