Breaking Mental Set to Improve Reasoning through Diverse Multi-Agent Debate

• Q5: Explanation about degraded performance and mistakes made with additional rounds when using DMAD. Why does LLaMA model's performance drop after the third round with DMAD?

This is mainly because of the typical mistakes discussed in the answer to Weakness 2. For example, in the 2nd round, Agent 1 and Agent 2 get the right answer while Agent 3 gets a wrong one. Then the judge with Self-Consistency gets the right answer. However, in the 3rd round, Agent 1 and Agent 2 mistakenly obtain the wrong conclusion, while Agent 3 correctly modifies its wrong answer. Then the judge with Self-Consistency gets the wrong answer. (Situation 1)

What we hope is, Agent 3 correctly revises its answer, and Agent 1 and Agent 2 retain their correct answers. Another ideal situation is, if two agents get wrong answers while the other one is right, all agents or at least two agents reach correct solutions in the next round. (Situation 2)

DMAD will more frequently experience that agents get different answers due to its diversity compared to MAD. If there are more Situation 2 than Situation 1, the performance will upgrade, otherwise, it degrades, which is why llama's performance drops after the 3rd round.

When to stop the debate?

So it is important to determine when the debate ends. Our method DMAD can be over in just 2 rounds and gets good performance, even surpassing MAD in 5 rounds. Besides, we can also design different criteria to determine when the debate ends, such as:

1. Consistency-2: If there exist 2 agents getting the same answer, the debate ends. Otherwise, the debate continues unless reaching its maximum number of rounds.

2. Consistency-3: If all the 3 agents get the same answer, the debate ends. Otherwise, the debate continues unless reaching its maximum number of rounds.

3. Self-Determine: Set another model as a judge to determine whether the debate should be over. The judge can receive all agents' solutions in each round and save them in its history.

4. Hybrid: In each round, if all agents get the same answer, the debate ends. Otherwise, use Self-Determine to judge whether the debate should end or continue.

We set the maximum number of the debate round to 5 with Gemini on ScienceQA. With the stop criteria, MAD and DMAD can get a relatively high accuracy with a low average round. Results show that the criteria of Consistency-3 is the best. However, DMAD with the stop criteria of Consistency-3 needs more overhead with nearly 5 debate rounds.

• Minor Comments

Thanks for your careful review of our paper. We have revised the minor errors you mentioned in the first two items. We have also added an explanation that the agents actually receive raw markdown text.

Thanks again for your valuable feedback. We hope our responses can address your concerns. Looking forward to further communication with you.

• Q3: Examining setups with more than 3 agents, with automatically generated diverse methods and manually selected ones.

According to our answer to Q2, it may be difficult for the model to simply propose sufficiently high-quality and diverse strategies. So it is hard to guarantee the performance of DMAD with automatically generated strategies. Besides, manually selecting more high-quality strategies does not necessarily mean much higher diversity. Finding more sufficiently diverse reasoning methods is challenging. We calculate $diversity$ for $k=4$ strategies. The improvement is limited compared to $k=3$. For LLaMA, $diversity$ of $k=4$ is even equal to $k=3$ with $\\{CoT, SBP, PoT\\}$.

What's more, assuming we do have sufficiently diverse high-quality strategies, we think the performance may first increase, then stabilize, or even decrease at last when increasing agents. This is because, when there are too many agents, the strategies are so diverse that it leads to confusion. It may be more difficult for different agents to reach consensus, and more easily to make typical mistakes discussed in the answer to Weakness 2.

Nonetheless, we cannot rule out the possibility that this automatic setup is feasible and performs better, as we have not conducted systematic and comprehensive experiments to verify it. We greatly appreciate your insightful suggestion. It is worth leaving for future work exploration.

• Q4: Confusion about Eq. 3 and Eq. 4.

The messages stored in history $h$ are in order. The message of the agent itself is appended into $h$ at first, and others afterwards. This is consistent with the original prompt of MAD for a fair comparison, which places the agent's own solution in front while others at the back. We will emphasize this in the revised version to help readers have a better understanding.

• Q1: "In Table 1, for the Alg. column and the LLaMA model, the value 72 achieved by DMAD is not the highest (c.f. MAD (All CoT) and CoT - SC); is this a typo or a correct results (in the latter case, please correct the text bolding)?"

Thanks for your feedback! We have corrected the text bolding.

• Q2: "Do you believe there would be merit in attempting to set up another LLM to propose reasoning strategies, instead of having them fixed?"

This is an interesting idea. However, based on our experiments and experience, we're concerned that this approach may not lead to a better performance, especially on smaller and weaker models. The only advantage may be its automation. The reasons come from two aspects:

1. Quality: The quality of the automatically generated prompt is not guaranteed. Instead, we select published prompting strategies that have been widely used and their effectiveness has been widely proven.

2. Diversity: The diversity of the automatically generated prompt is not guaranteed. Instead, we can subjectively choose diverse strategies or use a designed metric to select.

Assuming we have $K$ candidate basic reasoning strategies $\\{R_i\\}^K_{i=1}$ and want to select $k$ diverse ones, run each strategy $N$ times, and record the problems which $R_{i}$ correctly solve at least once as $P_{i}$ (Note that the definition of $P_{i}$ here is different from $P_{i}$ when introducing mental set). Note all problems on the measured dataset as $P_{all}$. We can define the diversity of the selected strategies $\lbrace R_{s_i}\rbrace ^{k}_{i=1}$ as

$diversity =\frac{\|\cup \\{{P_{s_i}\\}}^{k}_{i=1}\|}{\|P\_{all}\|}\in [0, 1], s_i \in \\{1, 2, ... \,K\\}, s_i \neq s_j \, for \\, \\, i \neq j.$

This represents the proportion of total questions that the selected $k$ methods can answer correctly. The more diverse these methods are, the larger this proportion should be.

We run $N=3$ times for each reasoning strategy in $\\{{R_i} \\}^{4}_{i=1}=\\{CoT, L2M, SBP, PoT\\}$ and select $k=3$ strategies to calculate $diversity$, and test DMAD with different strategy groups on MATH with GPT-4o-mini. We can see $diversity$ of {CoT, SBP, PoT} is the highest. Experiment results show that using the strategy group with larger $diversity$ can get better results.

• Weakness 3: Broader applicability.

1. Statistical significance.

We run experiments with GPT-4o-mini and LLaMA-3-70B-Instruct on MATH 3 times to calculate the average accuracy and standard deviation. Statistical experiments demonstrate that DMAD outperforms other MAD settings. DMAD on LLaMA-3-70B-Instruct also gets better average accuracy than MAD (All PoT).

2. On smaller models.

We test {MAD (All CoT), MAD (All SBP), MAD (PoT), DMAD} on LLaMA-3-8B-Instruct on MATH. Results demonstrate DMAD is also effective on smaller models.

3. On more challenging reasoning dataset of MMLU [1].

We supplement experiments on a more challenging benchmark, MMLU [1]. We test GPT-4o-mini on the subset "abstract_algebra" of MMLU and run 3 times. As this dataset consists of multi-choice questions and some options are not numbers, we replace PoT with Least-to-Most (L2M) [2]. DMAD also outperforms other MAD settings on this challenging multi-hop reasoning task.

[1] Measuring Massive Multitask Language Understanding. ICLR 2021.

[2] Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. ICLR 2023.

What is mental set?

In our paper, we introduce a new concept of mental set according to the psychological theory. Here we provide a specific definition for it. Denote MAD (All CoT), MAD (All SBP), and MAD (All PoT) as $M_{1}, M_{2}$ and $M_{3}$ respectively. When using a kind of MAD method $M_{i}$ to solve a problem, if all agents consistently get wrong answers in all debate rounds, we assume that $M_{i}$ is unable to correctly solve the problem. Record all such problems for $M_{i}$ as the set $P_{i}$, and get $P=P_{1} \cap P_{2} \cap P_{3}$. For a problem $p \in P_{i}$, if it satisfies $p \notin P$, we define that the problem $p$ causes mental set of $M_{i}$, and define $p$ as the mental set problem of $M_{i}$. It means although $M_{i}$ constantly gets wrong solutions, the model can correctly solve the problem by changing to another method.

We record the mental set problems of each method on MATH with GPT-4o-mini, and the problems among them which at least one agent of other methods correctly solves. All methods have 3 agents and debate for 2 rounds. The results show that DMAD can more effectively solve other methods' mental set problems.

• Weakness 1: Limited novelty.

We would like to emphasize our novelty and differences compared with other methods leveraging diverse solutions.

1. We introduce a new concept of mental set and provide a specific definition and systematic analysis as above, which is inspired by the psychological theory of mental set.
2. Self-Contrast just contrasts diverse reasoning solutions to get better performance than simple IO solutions. However, they don't compare their method with each distinct reasoning strategy. So it's not sure if using diverse solving perspectives all outperforms completely using one perspective among them. Meta-Reasoning Prompting dynamically chooses the best reasoning method to solve problems, while it only achieves a balanced performance between various reasoning methods and is often inferior to a specific one in most cases. Different from them, we conduct a more comprehensive analysis and prove that leveraging diverse reasoning methods can outperform all single methods with the same setting.
3. Our experiment shows that DMAD has the potential to break the mental set of single reasoning methods. For example, for the same problem, all agents in MAD (All IO) constantly get wrong answers in all 5 rounds. While in DMAD, an agent thinking with IO correctly solves the problem after receiving messages from agents thinking from other perspectives in the 2nd round. This also holds for MAD (All CoT), MAD (All SBP), MAD (All PoT), MAD (All CCoT) and MAD (All DDCoT). We state this in Section 4.3.4 in the initial submission.

• Weakness 2: Investigation of typical mistakes made during debating with DMAD.

Apart from the failed case we discuss in Appendix, another typical mistake is that sometimes the agents may be affected by other agents' wrong solutions, changing its original correct answer to a wrong one. This is not the special case of DMAD, but a common issue of MAD, attracting many scholars to try to study how to alleviate this problem. However, this is not the core of our paper. We just introduce DMAD to help solve problems that MAD struggles with, improving the upper bound of its reasoning capability.

Sincerely appreciate for your meticulous review of our paper! We thank for your valuable insights, which could be much helpful for improving and clarifying our paper. We would like to address your each concern in detail.

We summarize our responses as follows:

1. We provide a specific and systematic definition for mental set, connected with the corresponding psychological phenomenon. (In Response 1/5)

2. We emphasize our novelty and differences compared with other methods leveraging diverse solutions. (In Response 1/5)

3. We discuss about typical mistakes of DMAD and MAD. (In Response 1/5)

4. We conduct abundant experiments to prove the broader applicability of our method. (In Response 2/5)

   (1) Statistical significance. We re-conduct our experiments on MATH with GPT-4o-mini and LLaMA-3-70B-Instruct 3 times to get the average accuracy and standard deviation. Our method DMAD consistently outperforms other MAD methods.

   (2) We test on a smaller model, LLaMA-3-8B-Instruct. DMAD also achieves the best performance.

   (3) We conduct experiments on the subset of a more challenging benchmark, MMLU [1], where DMAD also outperforms other methods.

5. We elaborate our opinions on automatically generating prompting strategies (In Response 3/5) and setting more agents (In Response 4/5).

6. We answer the questions about equations. (In Response 4/5)

7. We explain the degraded performance of LLaMA in the 3rd round. (In Response 5/5)

8. We design 4 different stop criteria to determine when the debate ends. (In Response 5/5)

We also respond to other minor questions, and have added these supplementary experiments in the revised paper. We hope our responses can address your concerns. Looking forward to further communication with you.

[1] Measuring Massive Multitask Language Understanding. ICLR 2021.

I thank the authors for their comprehensive response. After consideration, I decided to keep my initial assessment — the work, while incremental, would in my opinion represent an interesting and valuable addition to the community.

• Adding experiments on more vision-centric tasks/datasets.

In our submitted paper, we conduct experiments on both LLMs and MLLMs to further verify the effectiveness of DMAD on different modalities. We test 3 MLLMs on two vision reasoning benchmarks including multiple aspects such as object recognition, optical character recognition, knowledge answering, math vision-question answering, language generation, and spatial awareness, which requires MLLMs to solve vision-centric problems according to the given image.

We believe our evaluated datasets and tasks provide evidence to support the conclusions drawn. However, we greatly value the reviewer’s insights and would be happy to explore additional tasks or datasets that require different types of reasoning or demonstrate other properties, if the reviewer has specific recommendations.

• Why self-reflection methods struggle on multimodal tasks compared to language-only tasks?

We discuss it in Appendix C.3. This may be due to the underconfidence of MLLMs, which tend to believe that their initial answer is incorrect and modify it, even if most of their answers are right. Here we compared Self-Refine results of MLLMs on ScienceQA and LLMs on MATH. We can find MLLMs are more likely to change their right answers to wrong ones.

(Maintain: The answer remains unchanged. Right -> Wrong: A right answer is changed to wrong. Wrong -> Right: A wrong answer is changed to right. Wrong -> Wrong: A wrong answer is changed but remains incorrect.)

However, this may be affected by specific prompts. The evaluation prompt in Self-Refine, "Review your previous answer and find problems with your answer.", and the refinement prompt, "Based on the problems you found, improve your answer.", may lead models to nitpick on correct answers and find problems of right solutions. Therefore, we change the evaluation prompt to "Review your previous answer and determine whether your previous answer is right or wrong." to get feedback, and use "Based on your judgment, improve your answer. If your previous answer is judged as wrong, modify it to be correct. Otherwise, keep your previous answer." to refine. On LLaVA and Gemini, this prompt setting performs better than Self-Refine, while still getting worse results than before modification.

• Were the number of trials/random seeds adjusted for experiments on MLLMs?

No. On the the same benchmark and model, we strictly test the performance of different methods using completely identical settings, only the prompts are different. Although MLLMs are computationally intensive, we test LLaVA, Gemini, and GPT-4o on the whole MM-Vet benchmark. We also test LLaVA and Gemini on all ScienceQA samples, and test GPT-4o with 100 random samples with the sample seed 0 for reproduction. We clarify these in Section 4.2.2.

• Visualizing more of the ablations in the format of Figure 1.

Thanks for your advice. We have added such a figure in Appendix.

Thanks again for your valuable feedback. We hope our responses can address your concerns. Looking forward to further communication with you.

• Computational overhead including tokens and inference time of DMAD and MAD during the debate rounds.

Thanks for your advice. This is important for real-world applications. Based on our record of experiment results on Gemini on ScienceQA, we use a word as a token to calculate the tokens overhead of traditional MAD, i.e., MAD (All IO), and DMAD during the debate rounds.

We can see that (DMAD tokens / MAD tokens) becomes larger as the debate round increases. This is because the selected diverse strategies generate more tokens. The diverse solutions are added to each agent's debate history in each round. The accumulated history of diverse solutions will cause more and more overhead. Nonetheless, DMAD in 2 rounds gets better performance and needs lower overhead and calls than MAD when MAD achieves its best performance in 5 rounds.

When to stop the debate?

Besides, it is important to determine when the debate ends to reduce additional overhead. Although we run all experiments in the fixed round for a fair comparison, which is consistent with the settings of the paper of MAD, we can also design different criteria to determine when the debate ends, such as:

1. Consistency-2: If there exists 2 agents getting the same answer, the debate ends. Otherwise, the debate continues unless reaching its maximum number of rounds.

2. Consistency-3: If all the 3 agents get the same answer, the debate ends. Otherwise, the debate continues unless reaching its maximum number of rounds.

3. Self-Determine: Set another model as a judge to determine whether the debate should be over. The judge can receive all agents' solutions in each round and save them in its history.

4. Hybrid: In each round, if all agents get the same answer, the debate ends. Otherwise, use Self-Determine to judge the debate should end or continue.

We set the maximum number of the debate round to 5. With the stop criteria, MAD and DMAD can get a relatively high accuracy with low average round. Results show that the criteria of Consistency-3 is the best. However, DMAD with the stop criteria of Consistency-3 needs more overhead with nearly 5 debate rounds.

The inference time of DMAD and MAD has the same trend as tokens. We re-conduct experiment on MATH with gpt-4o-mini and record the tokens overhead and inference time of DMAD and MAD. We run 3 times to calculate the average accuracy and standard deviation. Our DMAD approach effectively strikes a balance among different approaches that require varying amount of inference computation while achieving optimal performance. We hope the reported results can be helpful.

• What is mental set?

In our paper, we introduce a new concept of mental set according to the psychological theory. Here we supplement a specific definition for it. Denote MAD (All CoT), MAD (All SBP), and MAD (All PoT) as $M_{1}, M_{2}$ and $M_{3}$ respectively. When using a kind of MAD method $M_{i}$ to solve a problem, if all agents consistently get wrong answers in all debate rounds, we assume that $M_{i}$ is unable to solve the problem correctly. Record all such problems for $M_{i}$ as the set $P_{i}$, and get $P=P_{1} \cap P_{2} \cap P_{3}$. For a problem $p \in P_{i}$, if it satisfies $p \notin P$, we define that the problem $p$ causes mental set of $M_{i}$, and define $p$ as the mental set problem of $M_{i}$. It means although $M_{i}$ constantly gets wrong solutions to $p$, the model can correctly solve the problem by changing to another method.

We record the mental set problems of each method on MATH with GPT-4o-mini, and the problems among them which at least one agent of other methods correctly solves. All methods have 3 agents and debate for 2 rounds. The results show that DMAD can more effectively solve other methods' mental set problems.

• What kind of diversity is most effective?

We initially chose divergent prompting strategies by intuition. CoT solves the problem step by step, but may struggle with problems involving complex theorems. SBP first extracts relevant principles and then solves the problem according to them, but may weaken the ability to solve problems step by step. PoT uses Python programs, but this is mostly suitable for problems involving numbers. Therefore, these strategies are diverse and can complement each other.

To offer a more quantitative analysis, we design a more objectively quantitative metric. Assuming we have $K$ candidate basic reasoning strategies $\\{R_i\\}^K_{i=1}$ and want to select $k$ diverse ones, run each strategy $N$ times, and record the problems which $R_{i}$ correctly solve at least once as $P_{i}$ (Note that the definition of $P_{i}$ here is different from $P_{i}$ when introducing mental set). Note all problems on the measured dataset as $P_{all}$. We can define the diversity of the selected strategies $\lbrace R_{s_i}\rbrace ^{k}_{i=1}$ as

$diversity =\frac{\|\cup \\{{P_{s_i}\\}}^{k}_{i=1}\|}{\|P\_{all}\|}\in [0, 1], s_i \in \\{1, 2, ... \,K\\}, s_i \neq s_j \, for \\, \\, i \neq j.$

This represents the proportion of total questions that the selected $k$ methods can answer correctly. The more diverse these methods are, the larger this proportion should be.

We run $N=3$ times for each reasoning strategy in $\\{{R_i} \\}^{4}_{i=1}=\\{CoT, L2M, SBP, PoT\\}$ and select $k=3$ strategies to calculate $diversity$, and test DMAD with different strategy groups on MATH with GPT-4o-mini. We can see that using the method group with larger $diversity$ can get better results.

We thank the reviewer for considering our idea of mental set is interesting and commending our sufficiency of experiments. We also appreciate the valuable and insightful feedback provided, which could be much helpful to better prove the effectiveness of our method. We would like to address each of your concerns.

We summarize our responses as follows:

1. We conduct experiments on the subset of MMLU, where our method DMAD also outperforms other methods. (In Summary Response)
2. We provide a specific and systematic definition for mental set, connected with the corresponding psychological phenomenon. (In Respone 1/3)
3. We design an objectively quantitative metric to measure the diversity of selected basic prompting strategies, and conduct several experiments to prove the effectiveness of this metric. (In Response 1/3)
4. We report the computational overhead including tokens and inference time of DMAD and MAD during the debate rounds. DMAD achieves optimal performance with balanced overhead compared with other MAD methods. (In Response 2/3)
5. We design 4 different stop criteria to determine when the debate ends to reduce excessive tokens. (In Response 2/3)
6. We test different MLLMs and LLMs using 2 different self-reflection prompts, proving and analyzing the poor performance of self-reflection on MLLMs. (In Response 3/3)

We also respond to other minor questions and have added these supplements in the revised paper. We hope our responses can address your concerns. Looking forward to further communication with you.

• Experiments on more challenging multi-hop reasoning tasks such as MMLU.

Thanks for your suggestion. We use GPT-4o-mini to conduct experiments on the subset "abstract_algebra" of MMLU and run 3 times to get the average accuracy and standard deviation for each method. As this dataset consist of multi-choice questions and some options are not numbers, we replace PoT with Least-to-Most (L2M) [1]. DMAD also outperforms other MAD settings on this challenging multi-hop reasoning task.

[1] Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. ICLR 2023.

• Question3: Why focusing on mathematical reasoning and visual understanding?

We conduct abundant experiments in multiple aspects, not limited to mathematical reasoning and vision understanding. In our paper, we conduct experiments on mathematics, biology, chemistry, and physics reasoning. We also test on Scientific Vision-Question Answering (VQA) including natural, social, and language science, and MM-Vet involving various visual capabilities such as object recognition, optical character recognition, knowledge answering, math VQA, language generation, and spatial awareness. These benchmarks have been widely used by researchers.

What's more, we supplement experiments on a more challenging benchmark, MMLU [1]. We use GPT-4o-mini to conduct experiments on the subset "abstract_algebra" of MMLU and run 3 times to get the average accuracy and standard deviation for each method. As this dataset consists of multi-choice questions and some options are not numbers, we replace PoT with L2M. DMAD also outperforms other MAD settings on this challenging multi-hop reasoning task.

• Question 4: What criteria determine the ideal number of debate rounds? Is there a smarter stopping criteria?

To fairly compare with MAD, we adopt all the same settings in their paper, which executes MAD in the fixed round. Without considering this, we can design different criteria to determine when the debate ends, such as:

1. Consistency-2: If there exist 2 agents getting the same answer, the debate ends. Otherwise, the debate continues unless reaching its maximum number of rounds.

2. Consistency-3: If all the 3 agents get the same answer, the debate ends. Otherwise, the debate continues unless reaching its maximum number of rounds.

3. Self-Determine: Set another model as a judge to determine whether the debate should be over. The judge can receive all agents' solutions in each round and save them in its history.

4. Hybrid: In each round, if all agents get the same answer, the debate ends. Otherwise, use Self-Determine to judge whether the debate should end or continue.

We set the maximum number of the debate round to 5 with Gemini on ScienceQA. With the stop criteria, MAD and DMAD can get a relatively high accuracy with a low average round. Results show that the criteria of Consistency-3 is the best. However, DMAD with the stop criteria of Consistency-3 needs more overhead with nearly 5 debate rounds.

Thanks again for your valuable feedback. We hope our responses can address your concerns. Looking forward to further communication with you.

• Weakness 2-1: Weak theoretical foundation and empirical evidence.

Our idea is inspired by the psychological theory of mental set, which refers to the cognitive tendency to approach problems in a particular way based on past experiences, learned behaviors, or established habits. This may hinder diverse thinking and make it difficult to solve a problem in the mental set way. However, if thinking in a different way, it may surprisingly find it is easy to address the problem. We observe that LLMs have an analogous phenomenon. MAD with a fixed prompting strategy may always get wrong answers to a problem, while changing to another prompting strategy can correctly solve the problem.

Here we provide a specific definition for it. Denote MAD (All CoT), MAD (All SBP), and MAD (All PoT) as $M_{1}, M_{2}$ and $M_{3}$ respectively. When using a kind of MAD method $M_{i}$ to solve a problem, if all agents consistently get wrong answers in all debate rounds, we assume that $M_{i}$ is unable to correctly solve the problem. Record all such problems for $M_{i}$ as the set $P_{i}$, and get $P=P_{1} \cap P_{2} \cap P_{3}$. For a problem $p \in P_{i}$, if it satisfies $p \notin P$, we define that the problem $p$ causes mental set of $M_{i}$, and define $p$ as the mental set problem of $M_{i}$. It means although $M_{i}$ constantly gets wrong solutions to $p$, the model can correctly solve the problem by changing to another strategy.

We record the mental set problems of each method on MATH with GPT-4o-mini, and the problems among them which at least one agent of other methods correctly solves. All methods have 3 agents and debate for 2 rounds. The results prove the connection between LLM reasoning and the mental set phenomenon in psychology, and DMAD can more effectively solve other methods' mental set problems. In the initial submission, we state that DMAD can solve mental set problems of other MAD methods in Section 4.3.4 and list many examples in Appendix D.

• Weakness 2-2: Statistical significance.

We run experiments with GPT-4o-mini and LLaMA-3-70B-Instruct on MATH 3 times to calculate the average accuracy and standard deviation. Statistical experiments demonstrate that DMAD outperforms other MAD settings. DMAD on LLaMA-3-70B-Instruct also gets better average accuracy than MAD (All PoT). This implies the gains come from method diversity rather than the debate process itself to some extent.

We appreciate your comprehensive review and valuable feedback of our paper. We would like to address each of your questions.

We summarize our responses as follows:

1. We elaborate the connection between our idea and the psychological theory of mental set, and provide a specific and systematic definition for it, proven by our experiment results. (In Response 1/4)
2. We re-conduct our experiments on MATH with GPT-4o-mini and LLaMA-3-70B-Instruct 3 times to get the average accuracy and standard deviation. Our method DMAD consistently outperforms other MAD methods. (In Response 1/4)
3. We report the tokens overhead and cost of all methods on MATH. DMAD achieves optimal performance with balanced overhead compared with other MAD methods. (In Response 2/4)
4. We explain why choosing CoT, SBP, and PoT, and also test other groups including Least-to-Most prompting. We design an objectively quantitative metric to measure the diversity of selected basic prompting strategies, and conduct several experiments to prove the effectiveness of this metric and the gains do come from diversity. (In Response 2/4)
5. We test on a smaller model, LLaMA-3-8B-Instruct. DMAD also achieves the best performance. (In Response 3/4)
6. We emphasize our novelty and differences compared with other methods leveraging diverse solutions. (In Response 3/4)
7. We conduct experiments on the subset of a more challenging benchmark, MMLU [1], where DMAD also outperforms other methods. (In Response 4/4)
8. We design 4 different stop criteria to determine when the debate ends to reduce excessive tokens. (In Response 4/4)

We also respond to other minor questions, and have added these supplements in the revised paper. We hope our responses can address your concerns. Looking forward to further communication with you.

[1] Measuring Massive Multitask Language Understanding. ICLR 2021.

• Weakness 3: Model capability of scaling.

We test MAD (All CoT), MAD (All SBP), MAD (All PoT), and DMAD with LLaMA-3-8B-Instruct on MATH. Results demonstrate DMAD is also effective on smaller models.

However, LLaMA-3-8B-Instruct using DMAD still gets lower accuracy (26.9%) than LLaMA-3-70B-Instruct using CoT (41.43%). We analyze that the upper bound of LLaMA-3-8B-Instruct using DMAD is still lower than 41.43%. We calculate $diversity$ for {CoT, SBP, PoT} and get 0.3829. This means {CoT, SBP, PoT} can only correctly solve 38.29% of the problems, which is lower than 41.43%.

• Weakness 5: Incremental rather than novel contribution.

We would like to emphasize our novelty and differences compared with other methods leveraging diverse solutions.

1. We introduce a new concept of mental set and provide a specific definition and systematic analysis as above, which is inspired by the psychological theory of mental set.
2. Self-Contrast just contrasts diverse reasoning solutions to get better performance than simple IO solutions. However, they don't compare their method with each distinct reasoning strategy. So it's not sure if using diverse solving perspectives all outperforms completely using one perspective among them. Meta-Reasoning Prompting dynamically chooses the best reasoning method to solve problems, while it only achieves a balanced performance between various reasoning methods and is often inferior to a specific one in most cases. Different from them, we conduct a more comprehensive analysis and prove that leveraging diverse reasoning methods can outperform all single methods with the same setting.
3. Our experiment shows that DMAD has the potential to break the mental set of single reasoning methods. For example, for the same problem, all agents in MAD (All IO) constantly get wrong answers in all 5 rounds. While in DMAD, an agent thinking with IO correctly solves the problem after receiving messages from agents thinking from other perspectives in the 2nd round. This also holds for MAD (All CoT), MAD (All SBP), MAD (All PoT), MAD (All CCoT) and MAD (All DDCoT). We state this in Section 4.3.4 in the initial submission.

• Question 2: Could the system dynamically adjust its reasoning methods during the debate process?

We assign each agent to think with a fixed reasoning method in DMAD. It's interesting to teach each agent to dynamically adjust its reasoning method. We mention that we leave it for future work in Appendix A "LIMITATIONS AND FUTURE WORK", "We can also dynamically guide agents to choose the method they deem appropriate independently. We leave this to future work and believe this may further improve the models’ reasoning performance." However, from another perspective, our experiments have proven that an agent thinking with a fixed method can break its mental set by getting insights from other diverse solutions. So it may not be necessary to design a dynamic adjustment strategy.

• Weakness 1: Resource efficiency analysis. Thanks for your advice. We report the token overhead and cost for each method in the experiment on MATH with GPT-4o-mini. DMAD balances the overhead of MAD (All CoT), MAD (All SBP), and MAD (All PoT), and can achieve the best performance than other methods.

• Weakness 2-3: Whether the gains are truly from method diversity or simply from the debate process itself?

• Weakness 4: Limited implementation guidelines.

• Question 1: Why choosing CoT, SBP, PoT? How about Least-to-Most prompting?

We initially chose divergent prompting strategies by intuition. CoT solves the problem step by step, but may struggle with problems involving complex theorems. SBP first extracts relevant principles and then solves the problem according to them, but may weaken the ability to solve problems step by step. PoT uses Python programs, but this is mostly suitable for problems involving numbers. Therefore, these methods are diverse and can complement each other. Instead, Least-to-Most (L2M) breaks down the problem into progressive sub-questions and answers them to get the final answer. It looks somewhat similar to CoT which gradually solves the problem in steps.

To offer a more quantitative analysis, we design a more objectively quantitative metric. Assuming we have $K$ candidate basic reasoning strategies $\\{R_i\\}^K_{i=1}$ and want to select $k$ diverse ones, run each strategy $N$ times, and record the problems which $R_{i}$ correctly solve at least once as $P_{i}$ (Note that the definition of $P_{i}$ here is different from $P_{i}$ when introducing mental set). Note all problems on the measured dataset as $P_{all}$. We can define the diversity of the selected strategies $\lbrace R_{s_i}\rbrace ^{k}_{i=1}$ as

$diversity =\frac{\|\cup \\{{P_{s_i}\\}}^{k}_{i=1}\|}{\|P\_{all}\|}\in [0, 1], s_i \in \\{1, 2, ... \,K\\}, s_i \neq s_j \, for \\, \\, i \neq j.$

This represents the proportion of total questions that the selected $k$ methods can answer correctly. The more diverse these methods are, the larger this proportion should be.

We run $N=3$ times for each reasoning strategy in $\\{{R_i} \\}^{4}_{i=1}=\\{CoT, L2M, SBP, PoT\\}$ and select $k=3$ strategies to calculate $diversity$, and test DMAD with different strategy groups on MATH with GPT-4o-mini. We can see $diversity$ of {CoT, SBP, PoT} is the highest. Experiment results show that using the method group with larger $diversity$ can get better results. This further proves gains are from method diversity, not the debate process.