MALT: Improving Reasoning with Multi-Agent LLM Training

Dear Reviewer,

Thank you for your time and valuable feedback! We are glad you found our work novel, promising, and an important contribution. We appreciate the opportunity to further improve our paper and discuss promising future avenues.

We also provide a detailed set of additional experiments on another model (Qwen 2.5 1.5B Base) in our response to Reviewer rgPo. Once again, MALT demonstrates significant performance improvements on GSM8K, and we will expand our experiments and analysis with the Qwen model family to MATH and CSQA for the camera ready version of our paper.

 

Questions

1. We are very happy to discuss how MALT can be used towards training reasoning models. DeepSeek R1 [1] discusses how high quality cold start data, especially when containing reflection and verification, can be very useful for the reinforcement learning phase. These behaviors and their usefulness for RL, such as backtracking and verification, are also studied in [2]. MALT’s credit assigned reasoning chains, even with our current Generator, Verifier, and Refinement Model setting can be thought of as one reasoning chain for the purposes of generating this cold-start mid-training data. This chain will inherently include verification and backtracking steps due to our multi-agent setup. More specifically, as the reviewer envisions, high-value paths can be used to directly train models to learn the basic cognitive structures and steps required for these behaviors, which RL can then further amplify. Our search based credit assignment strategy yields various benefits to further improve the quality of such data. MALT can allow for the identification of paths, for e.g., where the Generator produces an answer that leads to a low assigned value but still has some correct downstream outputs in the tree. These reasoning chains that converted low value initial generations into high value final outcomes could be specifically used to teach important backtracking and verification behaviors that improve the reasoning performance of the model. Moreover, low and high value paths can be used for preference optimization to further improve the model’s knowledge of these behaviors before the online RL phase to post-train reasoning models. In general, as described in Section 3, MALT can incorporate any meta-strategy, which can then be used as synthetic training data to improve reasoning models. We will provide an even more detailed description of this in the Appendix of our camera ready version.

2. We performed qualitative experiments early on in our research, where lowering $n$ or $\theta$ reduced the quality of our synthetic training data. With more compute that remains beyond our current scope and resources, both $n$ and $\theta$ can be increased to collect even higher value paths as positive signals while balancing how much training data is generated. In the camera ready version of our paper, we commit to providing a small-scale experiment and analysis that varies the branching factor $n$ to 2 and 5 and increases $\theta$ to 0.7 to evaluate the change in SFT and DPO post-training performance of Generator model. Once again, we would like to sincerely thank the reviewer for highlighting the significance and novelty of our work, and engaging in an important discussion about a possible future direction that MALT can be very useful for.

[1] Guo et. al, “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning”

[2] Gandhi et. al, “Cognitive Behaviors that Enable Self-Improving Reasoners”

Dear Reviewer,

Thank you for your time and valuable feedback! We are glad you found our paper technically sound, novel, and well motivated. We appreciate the opportunity to further improve our work and answer your questions.

In order to address several of the points raised, we provide a rigorous set of experiments using Qwen-2.5-1.5B Base as the underlying model for MALT and evaluate on GSM8K. We provide our results in the table below along with a detailed discussion.

 

Here, Single Agent refers to the Generator and MV refers to Majority Voting@3. On the Base model, DPO improves the Generator's performance beyond STaR on the same data, and is done after SFT. MALT surpasses all baselines by a clear margin. Our anonymized inference logs are available at https://shorturl.at/EdtVc

 

Weaknesses

1. For Llama 3.1 8B Instruct, we provide a justification in Appendix A.10 for not using DPO. Our results with the Qwen Base model show that DPO on a non-overtrained base model for the same data distribution after SFT shows notable improvements (68.75% compared to 63.50% with SFT only).
2. Our results on the Qwen Base model show that STaR MA+MV surpasses MA+MV on the Base model. Overall, in the case of our Llama Instruct model, our results across all 6 benchmarks show that MALT can lead to notable performance improvements. Our experiments on a Base model here further substantiate our claims, where the relative improvement is even higher than Instruct models (due to a higher scope for improvement) and shows how MALT can offer substantial benefits in both settings.
3. We appreciate the reviewer’s feedback on experiments related to ablating the branching factor. While a detailed experiment remains beyond our current scope due to compute constraints, we commit to providing a small-scale experiment and analysis that varies the branching factor $n$ to 2 and 5 and increases $\theta$ to 0.7 to evaluate the change in SFT and DPO post-training performance of the Qwen Generator model.
4. To address the noise levels, we provide a detailed set of experiments with the Qwen model, where MALT on GSM8K shows clear improvements. Therefore, despite some noise (common for small models and because we use 4 seeds), we show improvements for two model families across a range of benchmarks.

 

Questions

1. Yes, absolutely! We have provided a detailed set of experiments using Qwen 2.5 1.5B Base on GSM8K for this discussion. We commit to expanding these experiments to MATH and CSQA along with a detailed analysis in our camera ready version.
2. For the Instruct model, using the Generator in SA mode is equivalent to our STaR SA baseline. For the Base model, it is equivalent to our DPO baseline.
3. We hypothesize that if the number of G, V, and R samples go up at training, this would allow for more synthetic multi-agent data to be generated. While this would require additional computational resources, we expect more synthetic data to be useful and will evaluate with a branching factor of 5 in our camera ready version, as described above.
4. In the case of Llama 3.1 8B Instruct, the Generator is trained with SFT only. However, it sees roughly the same amount of data (i.e. training questions), only without the incorrect preference example which was used for DPO. After the credit assigned synthetic data generation process, all models had roughly a similar number of training examples. MALT therefore remains an important source of producing multi-agent role-specific post-training data, regardless of whether only SFT or both SFT and DPO are used.
5. We completely agree. MALT can potentially be extended to soft reasoning domains, where multi-agent setups can improve performance with the guidance of an outcome based Generative (Generalist) Reward Model [1] or similar soft verification function. MALT provides an extensible method for multi-agent setups to learn via credit assignment and post-training on any domain.

We sincerely appreciate the reviewer’s feedback and have conducted important additional experiments to answer questions. We commit to providing a detailed analysis of our results with Qwen 2.5 1.5B Base on GSM8K, MATH, and CSQA, along with experiments related to modifying the branching factor. We would really appreciate it if the reviewer would consider raising their score further.

[1] Lie et. al, “Inference-Time Scaling for Generalist Reward Modeling”

Dear Reviewer,

Thank you for your review and for noting that MALT leads to performance improvements and that our value-iteration credit assignment removes the need for human labels or oracle models. We address the concerns raised in your review below:

1. Novelty: we would like to clarify that MALT is more than reused parts.

• Unlike AutoGen or Debate Agents, and multi-agent debate frameworks, which repurpose a single model with different prompts, we learn specialized agents with dedicated functions. More specifically, we do not use these frameworks for any part of our method other than drawing inspiration from multi-agent LLM setups.
• We show how our method does credit assignment without extra supervision, relying only on outcome rewards for training the agents. We also provide a theoretical justification for the same.
• The nontrivial combination of the above with SFT + DPO yields a new post-training paradigm for complex multi-agent reasoning and demonstrates important empirical improvements.

2. Majority voting fairness: all baselines, including every STaR variant, were evaluated with the same MV budget. In this fair setting, MALT (which includes MV) outperforms baselines on six benchmarks and even generalizes to unseen tasks (eg, AIME, MMLU-Pro, GSM-Symbolic). We also provide a detailed discussion of why we use MV and how it improves MALT significantly compared to other baselines in Section 5.2 (Self-consistency). Moreover, in our response to reviewer rgPo, we provide further evaluations on another model family, where MALT clearly surpasses all other baselines.

We would also like to highlight the novelty and importance of our paper as recognized by reviewers rgPo and s2iA, who state that it is “novel, an important contribution, technically sound, well-motivated, has many degrees of freedom built in, and provides significant gains on reasoning benchmarks”.

We hope this clarifies why we believe MALT is a novel training paradigm rather than an engineering tweak, and we appreciate any further feedback you may have. We would be grateful if, given the information and clarifications here and in the paper, the reviewer might consider reevaluating and increasing their score.