ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaboration

[Improvement over baselines] We thank the reviewer for their interest in our experimental results. However, we believe that there might be a misunderstanding regarding our empirical results. In Table 1 of our paper, we show that our method achieves significant improvements (well outside the confidence intervals) against other MAD methods (both trained and untrained).

In particular, just to point to a few, using the LLama 3 model on the BoolQ dataset, we are 8.1%, 8.2%, 11.2%, 11.9%, 9.6% and 7.8% better than SoM(2x), SoM(4), Persona, DebateTune, SFT and DebateGPT respectively. Similarly, for Mistral on the SCIQ dataset, we are 4.9%, 4.8%, 4.4%, 4.2%, 4.6%, 3.4% better than SoM(2x), SoM(4), Persona, DebateTune, SFT and DebateGPT respectively. To better illustrate the improvements, we have created the table below to illustrate the average improvements of our method compared to existing baselines.

These results illustrate an arguably significant improvement upon baselines and this trend holds across most datasets as well as models. However, we agree that for smaller models (Gemma-2-2b) this is not always the case; our method is equivalent to baselines (SCIQ, ARC), and only worse than baselines for MMLU. We will better highlight these cases in the final version of our paper.

Here positive numbers = our model performed x% better than the baseline and negative meaning our method performed x% worse than the baseline.

Our Method Improvement Over Baselines (Llama-3)

Our Method Improvement Over Baselines (Mistral)

Our Method Improvement Over Baselines (Gemma-2)

[Further training to improve results] We expect that further training iterations will improve the performance of our method. However, as pointed out above and seen in Table 1, our method only requires two rounds of training to outperform baseline methods in all but one case.

[Metrics other than accuracy] For each of the baselines that we examine, accuracy is the standard evaluation metric.

[Comparison with other critic/judge baselines] We propose a multi-agent debate approach, and as such, we compare our method to two other multi-agent debate methods (SoM and Persona). Additionally, our method uses debate data to train models and we thus compare it to other methods that also use debate data to train models (DebateTune and DebateGPT).

Following the reviewer's suggestion, we have run additional baseline experiments using the critic/judge-based approach in Int-Debate. Below we compare our results with SoM (i.e., vanilla debate), Int-Debate, and our two versions of ACC-Debate. We see that typically, Int-Debate does not outperform SoM, and more importantly performs significantly worse than our approach (ACC-Debate) in most cases.

Llama-3

---

Mistral

---

Gemma-2

Most works that use critic/judge models only observe superior performance in settings where an external LLM judge is used to evaluate the correctness of a given answer (e.g., QuALITY); two notable works being https://arxiv.org/pdf/2305.19118 and https://arxiv.org/pdf/2402.06782 . Generally, there is little evidence in the literature that critic/judge-based MAD methods are superior to those without critic/judge models, when using untrained models, in the absence of an external judge.

[Generalizability] We would likely to clarify that in our experiments, we train and test models on the same task (partitioning each task into a training and testing set). Thus, the generalizability of each actor-critic team to unseen domains is unknown. We have added this to our limitation section (in pruple text).

[Extension to an arbitrary number of agents in debate] The ACC-Debate pipeline is easily extendable to the case of larger numbers of agents. Note that both the reward for partial trajectories $r(z^{(t)}, x, y)$ (line 197) as well as the DPO loss (line 308) are independent of the number of agents in the debate process. As a result, Algorithm 1 can be applied to debate with any number of agents; the only modification required is to change the implementation of the function OneGuidedDebateRound(), which simply runs one round of debate with guided prompts. For simplicity, let's say that we want debate with $n$ actor models (i.e., traditional MAD), then OneGuidedDebateRound() is trivial to implement by simply prompting each of the $n$ actor models with the guided debate prompt provided on line 842. Moreover, since our training pipeline uses LoRA fine-tuning, the computational overhead from adding more models is minimal since only one base model needs to be held in memory.

Lastly, we would like to thank the reviewer again for their insightful comments and attention to detail to improve our paper. We hope that the above has addressed any outstanding questions and that the reviewer would consider raising their score if all the questions have been appropriately answered.

[Clarity of Figure 1] We apologize for the confusion regarding Figure 1. We went through quite a few iterations of this figure, attempting to make it more compact. In hindsight, doing so also removed some important details from the figure. We have included an updated version of this figure in our current draft and updated the PDF on OpenReview. Please let us know if this new version provides more clarity on our method.

[Clarification on the use of guided and natural trajectories in ACC-Debate] ACC-Debate uses both guided and natural trajectories. First, we generate natural debate trajectories, which are then used as the starting position for guided trajectories. This is outlined on line 223-227 of Algorithm 1. The natural debate response $z^{(t-1)}$ is used as the starting position for the natural debate trajectory (line 223) as well as both guided trajectories (lines 226 and 227).

[Typo in Equation 5] You are correct, this is a typo on our end. The percent improvement in equation (5) should be $\frac{\text{acc}_4 - \text{acc}_0}{\text{acc}_0}$. We have fixed this in the current PDF uploaded to OpenReview.

[Type of errors made by actor and critic after training with ACC-Debate] We observe two primary areas of failure in the actor-critic team after training with ACC-Debate. First, as you mention, there are instances where the actor is incorrect but is not persuaded by the critic. Second, we also notice instances in which the critic provides incorrect, but highly persuasive feedback. Balancing the persuasiveness of the critic and the persuadability of the actor is a delicate balance. Despite these occasional errors, we generally observe that ACC-Debate strikes an effective balance (as illustrated by the example in Figure 4).

[Ablation on accuracy and number of debate rounds] We agree that examining accuracy as a function of the number of debate rounds is an important evaluation. We would like to highlight that this ablation was provided in Figure 3 (for BoolQ) and Figures 6, 7, 8, 9 (for the other datasets). These figures illustrate that accuracy increases over debate round, up to a point before plateauing. Accuracy typically plateaus after 2 rounds. However, there are some cases in which accuracy continues to increase over all 5 rounds. We note that this relationship between accuracy and debate rounds is common for most debate methods.

We would like to thank the reviewer again for their comments to improve the clarity and quality of our paper. We hope the information provided has resolved any remaining questions; if all concerns have been adequately addressed, we kindly ask the reviewer to consider raising their score.

[Training on guided debate may cause models to reach the correct answer through flawed means] We do not believe that there is a notable risk of our method causing models to reach correct answers through flawed means. We would expect the guided debate trajectories in ACC-Debate to improve the model's ability to both reach the correct answer as well as to follow correct reasoning/discussion steps to get there.

To verify this hypothesis, we conducted the following experiment for each model and dataset combination:

• For both an untrained actor-critic team and a trained actor-critic team (trained via ACC-Debate) we randomly sample 500 debates which resulted in the correct answer after 5 rounds of debate.
• We then prompted GPT-4 to evaluate the correctness of the reasoning and discussion steps taken by the actor-critic teams in each of the 500 debates.
• We then compute the average accuracy of the reasoning and discussion steps as evaluated by GPT-4.

In the table below, we show the accuracy of our method (ACC-Debate Accuracy) and the accuracy of the untrained actor-critic team (Untrained Accuracy), as well as the difference. We observe that, in nearly all cases, our method attains competitive, and sometimes superior, accuracy compared to the untrained actor-critic team.

GPT-4 Evaluated accuracy of reasoning and discussion steps for correct answers

Llama-3

Mistral

Gemma-2

We will include this discussion and these experimental results in the supplement of our final version.

[Clarity of experiments] When esitmating the reward function for a given trajectory $z^{(t)}$, task $x$, and answer $y$, namely $r(z^{(t)}, x, y)$, we use one-step roll-out with Monte Carlo estimation. This is a common estimation procedure which, despite the name, is amazingly simple. Starting from response $z^{(t)}$, we simulate one additional round of debate multiple times. The average accuracy of these simulations is then the estimated value of $r(z^{(t)}, x, y)$. We hope to make our work as clear and well presented as possible. If there are any other areas where you believe clarity can be improved, please let us know.

[Clarity of Figure 1] You are correct in that natural debate trajectories can be either positive or negative. Our goal with Figure 1 was to show a minimum complexity version of our pipeline, but we agree that this figure can cause confusion. We have included an updated version of Figure 1 in the main body of our paper and updated the PDF on open review. Please let us know your thoughts on this new figure.

[Typo and equation reference] Thank you for pointing these out. We have updated these in the current PDF uploaded to OpenReview.

Lastly, we would like to thank the reviewer again for their comments to improve our paper. We hope the information provided has addressed any remaining questions and that, if all concerns have been adequately addressed, the reviewer would consider raising their score.

[Optimality and Equation 1] Equation 1 is an optimization of the actor's accuracy at the final round $T$, and optimizes the actor ($\theta_a$) and the critic ($\theta_c$) jointly. Thus, $\theta_a^*, \theta_c^*$ are definitionally global optima of Equation 1.

Attaining these optimal values is unlikely in practice, as is the case with nearly all deep learning optimization schemes. We do not set out to produce an algorithm that attains optimal values for Equation 1, but rather an algorithm that attains reasonably strong performance in practice. We observe that this is the case as our approach attains higher accuracy than all baselines on all combinations of dataset and model type (with the single exception Gemma-2 on MMLU).

For clarity, we also note that in Equation 1, max-max formulations (i.e., $\max_{\theta_a}\max_{\theta_c}$) are equivalent to joint-max formulation (i.e., $\max_{\theta_a, \theta_c}$).

[Evaluation of the reward for partial debate trajectories $r$] As mentioned on line 200, the partial trajectory reward $r(z^{(t)}, x, y)$ can either be learned, or can be approximated via heuristics. In our experiments, we use heuristics to approximate the reward, namely a one-step roll-out with Monte Carlo estimation. This means that we simulate one additional round of debate multiple times (starting from response $z^{(t)}$), the value of $r(z^{(t)}, x, y)$ is then set as the average accuracy of these simulated rounds.

In practice, one could also learn this reward function. However, doing so would introduce additional computation overhead compared with heuristics. We choose to use heuristics as they are more efficient and empirically result in a high-performing debate team; we leave learned reward functions for future work.

[Choice of benchmarks] We use 5 standard benchmarks in our experiments and select these benchmarks as they provide a diverse set of tasks. We note that three of our benchmarks (MMLU, BBH, and ARC) contain math and reasoning questions. We agree that GSM8K and MATH are common benchmarks in literature, but one can only include so many benchmarks in a single paper, so we elected not to include these benchmarks, relying instead on the fact that MMLU and BBH have math problems.

We hope that we were able to address all the reviewer's questions in the above and hope that the reviewer would consider increasing their score.