ReMA: Learning to Meta-Think for LLMs with Multi-agent Reinforcement Learning

Weakness 1: Not compared to meta-thinking baselines

We sincerely thank the reviewers for their thoughtful and constructive comments. We have carefully addressed the requests regarding additional baseline experiments. Due to time and computational constraints, we focused on adding two representative baselines: Multi-Agent Debate (MAD) and Self-refine, one as a strong multi-agent meta-reasoning baseline and the other as a strong single-agent meta-thinking baseline.

For both two experiments, we use the prompts and implementation in [1]. Note that MAD and self-refine are inference-time methods, we test them under a computation close to ReMA. For MAD, we use 2 agents and let them interact 2 rounds. While for Self-Refine, we let the LLM to first propose an answer and then prompt it to critique and refine the answer only once. We use a temperature of 0 for most generations except the 1st round of MAD to maintain diversity.

We summarize the additional baseline results below:

Multi-agent Debat (MAD) vs Self-refine vs ReMA (Ours)

Compared to these inference-time methods, our proposed method (ReMA) achieves overall higher performance. This clearly demonstrates the advantage of our multi-agent reinforcement learning-based meta-thinking training strategy.

While MAD represents a strong multi-agent reasoning baseline, and self-refine serves as a classic single-agent meta-thinking method, our method still outperforms both significantly. Specifically, ReMA leverages structured multi-agent interaction optimized by reinforcement learning, enabling it to systematically improve reasoning capabilities beyond inference-only methods.

Reference

[1] Du et al., "Improving factuality and reasoning in language models through multi-agent debate," ICML2023.

I would like to raise my score to "accept" -- based on the rebuttal (which I discussed elsewhere in a different comment).

Weakness 1: Concern about computational efficiency

We appreciate the reviewer’s insightful comment about computational efficiency and scalability. We fully acknowledge this limitation. This has always been one of the primary challenges in MARL, and yet mainstream training frameworks such as VeRL still offer limited native support for multi‑agent RL. However, there are already increasing efforts trying to solve this issue: e.g. (i) parameter sharing across agents (we tried in this work) (ii) PD disaggregation that reuses cached states, and (iii) an asynchronous actor–learner pipeline. These design choices allow our method to keep near‑linear training speed as the number of agents or task complexity grows. In fact, the community is likewise beginning to focus on scalable solutions, such as Multiverse[1] and SGLang[2]. And we are willing to further iterate on our codebase moving forward. We hope these additions resolve the reviewer’s concern and are grateful for the opportunity to clarify.

Weakness 2: Comparisons with existing multi-agent meta-reasoning approaches

We sincerely thank the reviewers for their thoughtful and constructive comments. We have carefully addressed the requests regarding additional baseline experiments. Due to time and computational constraints, we focused on adding two representative baselines: Multi-Agent Debate (MAD) and Self-refine, to cover key concerns raised by the reviewers.

For both two experiments, we use the prompts and implementation in [3]. Note that MAD and self-refine are inference-time methods, we test them under a computation close to ReMA. For MAD, we use 2 agents and let them interact 2 rounds. While for Self-Refine, we let the LLM to first propose an answer and then prompt it to critique and refine the answer only once. We use a temperature of 0 for most generations except the 1st round of MAD to maintain diversity.

We summarize the additional baseline results below:

Multi-agent Debat (MAD) vs Self-refine vs ReMA (Ours)

Compared to these inference-time methods, our proposed method (ReMA) achieves overall higher performance. This clearly demonstrates the advantage of our multi-agent reinforcement learning-based meta-thinking training strategy.

While MAD represents a strong multi-agent reasoning baseline, and self-refine serves as a classic single-agent meta-thinking method, our method still outperforms both significantly. Specifically, ReMA leverages structured multi-agent interaction optimized by reinforcement learning, enabling it to systematically improve reasoning capabilities beyond inference-only methods.

Question 1: More details on how exploration is structured in the multi-agent setting compared to single-agent models?

Thank you for your helpful and constructive suggestions! Below we outline the two main mechanisms through which ReMA structures exploration more efficiently. (i) Hierarchical decomposition shrinks the search space: single‑agent RL must discover high‑level planning, sub‑goal tracking, reflection and correction within one long autoregressive rollout. The agent therefore explores an undifferentiated action space that mixes meta decisions with low‑level reasoning tokens, leading to sparse rewards and early convergence to sub‑optimal patterns; ReMA decouples those roles. A high‑level meta‑thinking agent proposes a compact plan, while a low‑level reasoning agent executes it. Because each agent now solves a simpler sub‑problem, the effective search space factorises; this yields more diverse trajectories and faster policy improvement, as shown in Figure 3. (ii) Context‑engineered workflow accelerates computation & execution: In multi-agent systems, the agents can exchange succinct messages instead of the entire history. Consequently the context window processed per forward pass is shorter, which saves the computations. Moreover, multiple agents can solve sub-tasks in parallel, which accelerates the task execution process.

Question 2: Are there any plans to extend ReMA for multi-agent cooperation in collaborative tasks, such as cooperative problem-solving or resource-sharing scenarios?

Thanks for your spot-on insight. We are actively exploring broader applications of the ReMA framework, and a promising next step is an AI‑Scientist‑style expert ensemble, in which ReMA’s meta‑agent is promoted to a principal investigator who assigns sub‑problems to a team of domain‑expert agents with different tools working cooperatively.

Reference

[1] Yang et al., "Multiverse: Your Language Models Secretly Decide How to Parallelize and Merge Generation," arXiv:2506.09991v2, 2025.

[2] Zheng et al., "SGLang: Efficient Execution of Structured Language Model Programs," arXiv:2312.07104v2, 2023.

[3] Du et al., "Improving factuality and reasoning in language models through multi-agent debate," ICML2023.

Weakness 1: Missing important baselines

We sincerely thank the reviewers for their thoughtful and constructive comments. We have carefully addressed the requests regarding additional baseline experiments. Due to time and computational constraints, we focused on adding two representative baselines: Multi-Agent Debate (MAD) and Self-refine, to cover key concerns raised by the reviewers.

For both two experiments, we use the prompts and implementation in [1]. Note that MAD and self-refine are inference-time methods, we test them under a computation close to ReMA. For MAD, we use 2 agents and let them interact 2 rounds. While for Self-Refine, we let the LLM to first propose an answer and then prompt it to critique and refine the answer only once. We use a temperature of 0 for most generations except the 1st round of MAD to maintain diversity.

We summarize the additional baseline results below:

Multi-agent Debat (MAD) vs Self-refine vs ReMA (Ours)

Compared to these inference-time methods, our proposed method (ReMA) achieves overall higher performance. This clearly demonstrates the advantage of our multi-agent reinforcement learning-based meta-thinking training strategy.

While MAD represents a strong multi-agent reasoning baseline, and self-refine serves as a classic single-agent meta-thinking method, our method still outperforms both significantly. Specifically, ReMA leverages structured multi-agent interaction optimized by reinforcement learning, enabling it to systematically improve reasoning capabilities beyond inference-only methods.

Reference

[1] Du et al., "Improving factuality and reasoning in language models through multiagent debate," ICML2023.

Weakness 1: Credit assignment remains an issue & Question 1: Suggestion of a method of incorporating intermediate rewards/values

Thank you for your constructive review and highlighting the credit‐assignment challenge. Here we address both Weakness 1 credit assignment issue and Question 1 incorporating intermediate rewards/values.

As you point out, “poor reasoning steps would lead to negative rewards for both.” In our manuscript, we emphasize that we develop two agents collaboratively, where the meta‐thinking agent generates guidance for the reasoning agent. When the reward is negative, the agents search for a better collaborative strategy.

For the papers you recommand (https://arxiv.org/pdf/2410.08115 and https://arxiv.org/pdf/2412.01928), they estimate the value of each turn using a Monte Carlo method, which we agree is very promising and plan to explore in future work. For example, sampling multiple reasoning trajectories for each meta‑thinking decision and averaging their rewards to compute the advantage, yielding more accurate credit assignment. We will discuss these works in our manuscripts.

We also propose two additional methods for intermediate rewards/values.

1. Baseline reuse: We can leverage our existing “no‑meta‑thinking” samples as a baseline to reduce reward variance and support credit assignment.
2. Tabular Markov‑chain modeling: For tasks with a limited state space, we can treat the problem as a tabular Markov chain, akin to GiGPO[1] , and compute more accurate state values for advantage estimation.

Weakness 2: Improvement is not significant. RL baselines on Qwen models have often have higher results.

We thank the reviewer for their thoughtful comments. As shown in Section 4.2 Table 1, ReMA achieves substantial improvements on individual hard benchmarks, notably: AMC23 with Llama3-8B-Instruct: ReMA → +20.00% (2.50% → 22.50%), AIME24 with Qwen2.5-7B-Instruct: ReMA → +13.33% (6.67% → 20.00%). These results demonstrate large gains on challenging tasks. Our ablation study in Section 4.3.1 (Fig 6) further disentangles the contributions of turn‑level ratio clipping and the learned meta‑thinking policy, confirming that the observed improvement is directly attributable to the guidance provided by the meta‑thinking agent.

We acknowledge that Qwen models fine‑tuned with RL can achieve larger improvements. However, most recent work applies RL to base models, which start from a lower initial performance and are hard to sample MAMRP-style data for training. In contrast, our approach leverages the full potential of already capable models and further enhances their reasoning ability.

Weakness 3: The method seems quite fragile to different configurations and hyperparameters.

Thank you for highlighting these important points. We agree that multi‑turn ReMA can be sensitive to some hyper-parameters and training configurations. That’s why we introduced turn‑level ratio clipping combined with parameter sharing between agents, which noticeably reduces collapse and stabilizes learning (Section 4.3.1, Question 5 and Figure 6). We are actively developing more robust credit‑assignment mechanisms and training recipes to further enhance stability.

Question 1: Suggestion of a method of incorporating intermediate rewards/values

Thanks for your constructive comments. Please see above reply to Weakness 1 and Question 1.

Question 2: How does ReMA compare to simply prompting a single model with meta-thinking instructions and then performing RL on the subsequent model to best use these steps?

We appreciate the reviewer’s detailed critique. We view the question from two perspectives: (1) the high‑level meta‑thinking agent, specifically, the quality of the meta‑thinking instructions it generates; and (2) the low‑level reasoning agent, its ability to follow those instructions and the instructions’ suitability for that agent.

In our manuscript, we address these two aspects in separate sections respectively. In Section 4.2.1 and Figure 3, especially the comparison between the first and third settings, RL from base and RL under Meta-thinking. Figure 3 shows that this meta‑thinking guidance significantly boosts performance, demonstrating that the good instructions help with RL training of the reasoning agent. In Section 4.2.2 and Figure 4, we restrict the meta‑thinking instructions to “DECOMPOSE”, “REWRITE”, or “EMPTY”. Different models eventually converge to different meta-thinking instructions, the small model selects “EMPTY”, and the large model adapts to problem difficulties at the end, showing that we need different meta-thinking instructions for different models. We hope these experiments fully address your concerns.

To eliminate confusion, we will add at the end of Section 4.2.2: “Sections 4.2.1 and 4.2.2 together demonstrate that RL with high-quality meta-thinking instructions enhances performance, and that the model subsequently selects instructions appropriate to its own capability.”

Question 3: Replacing Meta-thinking Agent with another baseline agent

We sincerely thank the reviewers for their thoughtful and constructive comments. We have carefully addressed the requests regarding additional baseline experiments. Due to time and computational constraints, we focused on adding two representative baselines: Multi-Agent Debate (MAD) and Self-refine, to cover key concerns raised by the reviewers.

For both two experiments, we use the prompts and implementation in [2]. Note that MAD and self-refine are inference-time methods, we test them under a computation close to ReMA. For MAD, we use 2 agents and let them interact 2 rounds. While for Self-Refine, we let the LLM to first propose an answer and then prompt it to critique and refine the answer only once. We use a temperature of 0 for most generations except the 1st round of MAD to maintain diversity.

We summarize the additional baseline results below:

Multi-agent Debat (MAD) vs Self-refine vs ReMA (Ours)

Compared to these inference-time methods, our proposed method (ReMA) achieves overall higher performance. This clearly demonstrates the advantage of our multi-agent reinforcement learning-based meta-thinking training strategy.

While MAD represents a strong multi-agent reasoning baseline, and self-refine serves as a classic single-agent meta-thinking method, our method still outperforms both significantly. Specifically, ReMA leverages structured multi-agent interaction optimized by reinforcement learning, enabling it to systematically improve reasoning capabilities beyond inference-only methods.

Reference

[1] Feng et al., "Group-in-group policy optimization for llm agent training," arXiv:2505.10978, 2025.

[2] Du et al., "Improving factuality and reasoning in language models through multi-agent debate," ICML2023.

Furthermore, the default GRPO baselines on Qwen 2.5-7B-Instruct do not improve performance, and this is not consistent with baselines in other literature such as [1] and [2].

In fact, we use REINFORCE++ in our experiments in Table 1. Different algorithms may lead to different performance, and we successfully demonstrate the performance of our framework based on the same underlying optimization algorithm. We also attach a comparison with GRPO as follows for your reference, hope this resolves your concern:

GRPO performance comparison at the final step of 400

Due to time constraints, we compared VRP(single-agent GRPO) and ReMA-GRPO (parameter sharing) under a single-turn setting, using the same hyperparameters and trained Qwen2.5-7B-Instruct on the MATH dataset. To illustrate performance trends, we report the sliding average test accuracy at every 100 steps (see the table below for more details).

We observe that single-agent GRPO starts with a higher performance, showing the strength of the base instruct model. However, its performance fluctuates during training and shows limited overall improvement.

In contrast, ReMA-GRPO starts slightly lower but improves steadily and consistently, eventually surpassing GRPO on average. This demonstrates ReMA’s stronger stability and generalization over time, with stable growth and the potential for further improvement after training more steps.

Regarding the referenced literature, for [1], we only observe GRPO training results without verified rewards, which fall outside the scope of our work. Nevertheless, we will include a discussion of this work in our manuscript. As for [2], the results in Table 1 clearly show that training from an instruct model also yields marginal improvements (e.g., AceCoderRM: 46.7 → 47.9, CodeR1-LC2k: 46.7 → 48.0).

GRPO training performance on test datasets per 100 steps VRP-GRPO (Qwen2.5-7b-instruct)

ReMA-GRPO (Qwen2.5-7b-instruct)

Question 2: Baselines of other type of agents and setting of fixing meta-thinking agent training reasoning agent

Thank you for the clarification. First, as requested, we have added Multi-Agent Debate and Self-Refine as additional baselines, and ReMA outperforms both on the majority of tasks. We hope the baselines and performance satisfy your requirements. Second, the baseline that fixes a meta-thinking module and then RL-trains a second agent corresponds exactly to the third setting in Section 4.2.1. In that setting, we first SFT a base model with GPT-4o’s meta-thinking instructions, freeze it, and then RL-train a separate base model for low-level reasoning with the meta-instruction of the fixed meta-thinking agent. Comparing this to the first setting, where we directly RL-train a base model, shows that meta-thinking yields the performance gains illustrated in Figure 3. We apologize for the confusing description and will revise the manuscript accordingly.

Reference

[1] Prabhudesai, Mihir, et al. "Maximizing Confidence Alone Improves Reasoning." arXiv:2505.22660 (2025).

[2] Zhao, Andrew, et al. "Absolute zero: Reinforced self-play reasoning with zero data." arXiv:2505.03335 (2025).

We hope the above clarifications have addressed your concerns. If there are any remaining questions or points that require further explanation, we would be happy to provide more details. If you find that we have adequately addressed your comments, we kindly ask you to consider revisiting your evaluation. We sincerely appreciate your time and constructive feedback.

We thank the reviewer for the constructive feedback and address the additional concerns below.

Weakness 1: Lack of novelty because of no credit assignment

Thank you for raising the question about credit assignment. We would like to restate our contributions. We are the first to introduce a multi-agent meta-thinking reasoning process and extend it to a multi-turn interaction setting. To support this framework, we adjust the reinforcement learning algorithm and introduce a turn-level ratio to stabilize training. Our extensive experiments and ablation study demonstrate the effectiveness of both the framework and the training method.

Although we do not invoke classical RL credit‐assignment algorithms in our experiments, our additionally proposed bi‐level framework in Appendix C.5 achieves the same effect. This method is not designed to credit assignment, but we can extract the performance gain of meta-thinking and reasoning agents directly during training, like Monte Carlo state value estimation method in the papers you recommanded.

We will incorporate additional credit-assignment techniques, such as Monte Carlo estimation of state values, as previously proposed, in our future version.

Weakness 2: Improvement is not significant

We appreciate your detailed inspection on our experiments and insightful feedback. Here we present our detailed response w.r.t. your comments:

The variance in performance improvements is still quite high. ReMA has performance improvements as well as drop on different benchmarks.

We acknowledge that the performance improvement is not particularly significant. As noted in lines 208–210 of our manuscript, it is inherently challenging to achieve substantial gains when starting from a well-optimized instruct model. Given that both ReMA and the baselines are trained on a split of the MATH dataset and evaluated on separate test sets, it is reasonable that ReMA shows improvements on some benchmarks while experiencing drops on others. This is precisely why we introduced a diverse set of test benchmarks and reported the average performance improvement across them.