MAgICoRe: Multi-Agent, Iterative, Coarse-to-Fine Refinement for Reasoning

We appreciate the reviewer's thorough feedback. To answer the question regarding research focus, whether the proposed framework addresses all three issues, and what are the connections between our proposed modules, we would like to highlight that refinement is a multi-faceted process with several challenges. For example, applying uniform refinement across all samples risks over-correction, which can degrade overall performance. At the same time, refining “the right samples” only once may be insufficient. As a result, a framework that aims to address excessive refinement may end up suffering from under-refinement. Besides this tradeoff and beyond identifying which samples require refinement, another critical question is how to refine them effectively. MAgICoRE jointly tackles these challenges with its multi-agent design and the incorporation of reward models.

To further illustrate this, we conduct an additional ablation to illustrate the importance of tackling these issues jointly. In the ablation, we only address one issue at a time. The following table refers to the settings:

And the following are the results.

The results show that

• “Only Address Issue 1” is less effective due to the absence of PRM-based targeted feedback and iterative refinement.
• “Only Address Issue 2” incorporates PRM scores for all samples, but this sacrifices efficiency and still underperforms the full setup used in MAgICoRE.
• “Only Address Issue 3” performs the worst, as it uniformly refines all samples without leveraging PRM-based targeted feedback. Thus, all three issues must be addressed jointly, as MAGiCoRE does. These results are further described in lines 455-472 in the revised pdf.

More in-depth discussions should be placed in the paper

We appreciate the reviewer’s valuable feedback on the writing, and we made several improvements to enhance clarity: (1) moved related work to the appendix to relax the space limit. (2) added more method details and motivations to Section 3.2. (3) stressed that the three refinement issues are tied and that we are jointly addressing the three issues instead of independently. These modifications are highlighted in blue in the revised version.

Why do we need to perform a complex iteration multi-agent process?

The main idea of the iterative multi-agent framework is to break the refinement process into two stages: feedback generation and refinement, so that the feedback can incorporate PRM step-wise scores to enable better error localization. Empirically, our results in Table 1 and Figure 3 in the original version demonstrate that the proposed multi-agent refinement approach performs better and continues to improve with additional iterations. In contrast, Figure 3 shows that baseline methods like Self-Consistency and Best-of-k fail to benefit from the iterative process. While any test-time scaling method can, in theory, be applied to this subset of hard samples based on our classification, our findings indicate that simply increasing the number of responses per question (as seen in the 40-way and 120-way Self-Consistency experiments), yields limited benefits. On the other hand, the proposed multi-agent framework is more effective.

Thank you once again for your valuable feedback and suggestions. Since the PDF update window closes in 24 hours, we’d greatly appreciate your confirmation on whether our additional positive results—MAgICoRE's improvement over 4 new baselines and the detailed token cost comparison that shows its cost-effectiveness—along with our responses, address your comments. If they do, we kindly ask you to consider revisiting your score.

Dear Reviewer u3Va,

As the deadline for the discussion period approaches, we wanted to follow up and see if our additional experiments and responses have addressed your comments. If they have, we would greatly appreciate it if you could revisit your score accordingly.

Best,

The Authors

The three technical designs to address the three major challenges seem isolated

While our framework does consist of multiple connected components to address the three interrelated issues, we would like to clarify that we are not solving each issue one by one, but instead, solving the three issues jointly through a communicative multi-agent system, as each module in our system re-uses components and relies on the other modules. That is: To address “Excessive refinement” issue, we adopt reward models (ORM and PRM) to categorize each instance as easy or hard, and only refine hard instances. To address “Inability to localize errors”, the PRM is re-used to obtain targeted feedback via the “reviewer” agent. This helps the reviewer LLM localize the error and generate helpful feedback. To address “Insufficient refinement”, we propose a multi-agent interactive refining framework that communicates between the reviewer and refiner and refines based on information from the same PRM and ORM to iteratively refine the output. This only applies to hard instances, and we re-use the ORM to evaluate whether the refinement enhances quality.

Thus, the approach to the challenges is tied together by the use of the RMs and the communication between agents (the Solver, the Reviewer, and the Refiner).

As per your suggestion, in addition to our existing ablations in Table 2 - Table 5 in the original pdf, we have added an additional ablation study for each issue individually to clearly demonstrate the significance of addressing all issues jointly. The setup and results are presented below:

The results show that

• “Only Address Issue 1” is less effective due to the absence of PRM-based targeted feedback and iterative refinement.
• “Only Address Issue 2” incorporates PRM scores for all samples, but this sacrifices efficiency and still underperforms the full setup used in MAgICoRE.
• “Only Address Issue 3” performs the worst, as it uniformly refines all samples without leveraging PRM-based targeted feedback.

Thus, all three issues must be addressed jointly, as MAGiCoRE does. We have added these results and a discussion of them to lines 455-472 in the revised pdf.

Thank you once again for your valuable feedback and suggestions. Since the PDF update window closes in 24 hours, we’d greatly appreciate your confirmation on whether our additional positive results—MAgICoRE's generalization to two other domains and its improvement of GPT4o-mini—along with our responses, address your comments. If they do, we kindly ask you to consider revisiting your score.

Dear Reviewer eYto,

As the deadline for the discussion period approaches, we wanted to follow up and see if our additional experiments and responses have addressed your comments. If they have, we would greatly appreciate it if you could revisit your score accordingly.

Best,

The Authors

We appreciate the reviewer’s positive feedback regarding the novelty of our idea (“very interesting”) and the promise of our results (“very promising”). Please find the answers to the questions below.

The writing of Section 2 could be clearer.

We have addressed this by moving the related work to the appendix, and adding further details on our method (lines 235-277 in the revised pdf). We also added more motivations to Section 2.3. The changes are highlighted in blue.

The classification of this method is not very clear (intrinsic or extrinsic).

According to [1], MAgICoRe is categorized as an extrinsic method. However, the only external feedback we utilize is step-wise scores, without any natural language feedback from a more powerful model. The rationale for using RM scores is that LLMs are not effective at identifying their own errors or self-correcting [2, 3]. Additionally, we do not rely on a large dataset of examples, whereas RAG would require a substantial corpus of solved problems to retrieve from. These problems would need to be similar to the test problem at hand.

Moreover, it is not certain whether, even given this corpus and a strong retriever, models could in fact improve and refine their output. Doing so would involve understanding what commonalities the different retrieved solutions have and how these differ from the proposed solution, which is a challenging problem involving latent error localization. In contrast, PRM makes these problems explicit by directly identifying and localizing errors. If the reviewer is aware of any relevant works in this area, we would greatly appreciate any references.

[1] https://arxiv.org/abs/2310.01798

[2] https://arxiv.org/abs/2311.08516

[3] https://arxiv.org/pdf/2406.01297

The framework is a combination of multi-agent and RM for selective refinement.

The scope of our work is indeed to perform multi-agent refinement. Multi-agent systems have been widely applied to enhance faithfulness [4], reasoning [5], and translation [6], but there has been limited success in demonstrating their effectiveness for refinement tasks. Thus, while we acknowledge that MAgICoRE combines multi-agent systems with reward models for selective refinement, one of its key contributions lies in showcasing the effectiveness of multi-agent systems for LLM refinement.

[4] https://arxiv.org/abs/2305.14325

[5] https://arxiv.org/abs/2309.13007

[6] https://arxiv.org/abs/2305.19118

SVAMP is a dataset with 700 training and 300 testing samples.

We thank the reviewer for their careful attention. In SVAMP’s original paper (https://aclanthology.org/2021.naacl-main.168/) and code base (https://github.com/arkilpatel/SVAMP), we didn’t find that they have an official split for training and testing. Therefore, we follow previous work [7,8,9] to use all 1000 samples for evaluation, since we think a larger test set can reduce variations.

[7] https://arxiv.org/abs/2205.11916

[8] https://arxiv.org/abs/2309.17452

[9] https://arxiv.org/abs/2309.05653

Please specify the subtasks in MMLU as used.

We thank the reviewer for their careful attention. We are using the MMLU-Math subset curated by MAmmoTH [10]. We have added this in line 323.

[10] https://arxiv.org/abs/2309.05653

Please add GPT4 for a subset of the test datasets.

Following prior work [11,12,13] and considering the high cost of running GPT-4 across all datasets, we focus on Llama3 and GPT-3.5 as base models. However, we conducted an additional experiment using GPT-4o-mini on the MATH 500 dataset, primarily comparing 40-way SC with the weighted SC variation that incorporates PRM scores for vote weighting. Note that in Figure 4 in our submission, we found that 120-way SC actually decreases the performance on MATH. The results demonstrate that MAgICoRE can also enhance GPT-4o-mini’s performance, albeit with a smaller margin of improvement compared to Llama3-8B and GPT-3.5. We have added these results in lines 489-500.

[11] https://arxiv.org/pdf/2305.14934

[12] https://arxiv.org/pdf/2311.07961

[13] https://arxiv.org/pdf/2305.14325

What is the cost of MAgICoRE for comparison?

We include a token count comparison in Figure 5 of the revised version. We find that scaling 40-way Self-Consistency (SC) to 120-way SC largely increases token usage but yields a limited performance improvement (and even a drop on MATH). In contrast, MAgICoRE effectively improves the performance with a larger token budget. Notably, MAgICoRE uses fewer tokens on datasets such as SVAMP, GSM8K, and SAT compared to 120-way SC, while achieving superior performance. These results are further described in lines 997-1020 in the revised pdf.

Thank you once again for your valuable feedback and suggestions. Since the PDF update window closes in 24 hours, we’d greatly appreciate your confirmation on whether our additional positive results—MAgICoRE's improvement of GPT4o-mini and the detailed token cost comparison that shows its cost-effectiveness—along with our responses, address your comments. If they do, we kindly ask you to consider revisiting your score.

Dear Reviewer cViA,

As the deadline for the discussion period approaches, we wanted to follow up and see if our additional experiments and responses have addressed your comments. If they have, we would greatly appreciate it if you could revisit your score accordingly.

Best,

The Authors

We thank the reviewer for recognizing the effectiveness and efficiency of our method. Please find the answers to the questions below:

MAgICoRE relies heavily on external reward models

We acknowledge that external reward models play an important role in our framework. While MAgICoRE does utilize external reward models, our framework is modular and can readily incorporate new reward models as they emerge. This gives MAgICoRE an advantage, as the community is actively improving reward models (as evidenced by the rapid rate of improvement on reward model benchmarks) [1]. Our approach is thus complementary to and enhanced by progress in reward modeling, rather than constrained by it.

Furthermore, existing work has found that LLMs often struggle to recognize their own errors but can effectively correct them when the mistakes are explicitly identified [2]. External feedback on error localization is therefore critical for refinement to success. While it is possible to train a custom domain-specific error-identification model, this approach is often data-dependent and prone to obsolescence. In contrast, MAgICoRE's modular design overcomes this limitation by enabling the integration of new state-of-the-art models as they become available. Moreover, our results on generalizing to commonsense and logical reasoning show that even when RMs are unavailable, we can use sufficiently strong LLMs like GPT4o-mini in place of trained RMs. We have added this discussion to lines 514-524 in our revised pdf.

[1] https://arxiv.org/abs/2403.13787

[2] https://arxiv.org/abs/2311.08516

More evaluations should be conducted across a broader range of tasks

We focus primarily on math reasoning due to the current development of the Process Reward Models (PRM), which has been more math-focused. We believe ongoing advancements in reward models will help overcome this bottleneck [5]. Moreover, PRMs are domain-specific models, so creating a new PRM for a domain is a major challenge and likely a topic for one (or several) full papers.

Nevertheless, based on your suggestion, we add the following experiments on a commonsense reasoning task: ARC-Challenge [3], and a logical reasoning task: Date Understanding [4]. We randomly sample 200 instances and use GPT4o-mini as a PRM for our experiments. Specifically, we prompt GPT4o-mini to provide step-wise correctness scores without any textual explanations or reasoning, acting the same as a PRM (indeed, we manually verified that no explanations were generated by manually evaluating 100 examples). This approach ensures that our Solver (Llama3-8B) does not have access to explanations from a stronger model (GPT4o-mini). Instead, the Reviewer (Llama3-8B) relies solely on the step-wise numerical scores from GPT4o-mini to generate feedback, and the Refiner (Llama3-8B) uses this self-generated feedback for improvement.

Results below show that MAgICoRE generalizes well on commonsense and logical reasoning tasks, outperforming 120-way SC by 2.5% on commonsense reasoning and 8.0 % on logical reasoning. Full results and discussion can be found in lines 473-489 of our revised pdf.

[3] https://arxiv.org/abs/1803.05457

[4] https://arxiv.org/abs/2206.04615

[5] https://arxiv.org/abs/2403.13787

The optimal stopping point is unclear

We would like to point here to the results of 5 iterations shown in Figure 3 and Table 12 in the appendix (lines 350-363 in the original, lines 365-377 in the revised pdf). Our empirical findings suggest that 2 to 3 iterations generally yield the best performance. We would also like to note that the stopping criteria can flexibly align with budget constraints, as performance continues to improve with each iteration. For instance, on the MATH dataset, even if the process is stopped arbitrarily at iteration 3, (although the optimal number is 5 ) it would still surpass the performance of earlier iterations, such as iteration 1.

Thank you once again for your valuable feedback. We hope our response has addressed all of your questions and will allow you to revisit your score, otherwise we would be happy to engage further and address any further questions you might have in the remaining few days of the discussion period.

We thank the reviewer for highlighting that “the writing is clear” and “the method is efficient and effective”. We provide detailed answers to each question below.

comparison with methods that also use a PRM.

Thanks for this suggestion – we have added new experiments using Self-Consistency with PRM scores. Specifically, based on the approach taken by [1], we perform weighted Self-Consistency where the weight for a reasoning chain is the product of the step scores. While this baseline underperforms MAgICoRE, we would like to highlight that it is also inefficient as it uniformly generates 120 samples per question with the PRM scores (MAgICoRE is adaptive, whereas this baseline is not). The results are as follows:

We find that using PRM scores as aggregation does improve the performance over 120-way SC alone, but underperforms our method by 2.8% on average, underscoring the importance of using PRM to generate targeted feedback, and using that feedback for fine-grained refinement. We have added this in lines 864-895 of our revised pdf.

The method assumes that LLMs can improve through fine-grained stepwise refinement.

We thank the reviewer for raising this important point. Our method indeed assumes that LLMs can improve with fine-grained, stepwise refinement, which we believe is backed by our empirical results. In Table 3, we compare the effectiveness of feedback generated from random versus PRM-predicted scores, showing that feedback with PRM scores leads to higher performance on MMLU (66.4 vs. 67.3) and MATH (43.8 vs. 46.0), suggesting that fine-grained PRM feedback enables effective LLM refinement. We also agree that further testing is valuable, so we conducted additional oracle experiments to compare the role of the PRM score for refinement.

Specifically, we sample 500 instances from the Math-Shepherd dataset [2], which includes gold label correctness for each step. We evaluated four conditions: (1) No feedback, where the LLM self-refines; (2) Random PRM score, where feedback is generated from random PRM scores; (3) PRM predicted score, where feedback is based on predicted PRM scores; and (4) Oracle PRM score, where feedback uses the gold correctness labels.

Aligned with Section 4.2 and Table 3, “PRM predicted score” outperforms “No feedback” and “Random PRM score”. Most importantly, using oracle PRM scores achieves the highest performance. This indicates that given reliable stepwise scores, LLMs can effectively refine their solutions and improve, and these findings are also supported by [3]. We further describe these results in lines 906-915 of our revised pdf.

[1] https://aclanthology.org/2023.acl-long.291/

[2] https://huggingface.co/datasets/peiyi9979/Math-Shepherd

[3] https://arxiv.org/pdf/2311.08516

Thank you once again for your valuable feedback and suggestions. Since the PDF update window closes in 24 hours, we’d greatly appreciate your confirmation on whether our additional positive result—MAgICoRE's improvement over a strong PRM-based baseline—along with our responses, address your comments. If they do, we kindly ask you to consider revisiting your score.

Dear Reviewer ywAv,

As the deadline for the discussion period approaches, we wanted to follow up and see if our additional experiments and responses have addressed your comments. If they have, we would greatly appreciate it if you could revisit your score accordingly.

Best,

The Authors