A Theoretical Issues:

[Theoretical Results] We aim to provide theoretical results that help explain why specific behavior would be observed in the debate process. For example, Theorem 5.2 is intended to help understand why one would observe tyranny of the majority LLM debate. While not surprising to observe tyranny of the majority in debate, we are first to establish this observation rigorusly. Our result complements the main findings of [1], which outlines a similar phenomenon in in-context-learning; when many of the examples share an underlying pattern, models are more likely to give responses that also have that shared underlying pattern (Figure 1 in [1] outlines the basic idea). We don't wish to overrepresent the impact of our results, but we do genuinely believe that the theorems we provide are informative.

[Parameter Representation and $\theta$] We agree that in isolation, the representation of $\theta$ in Assumption 4.1 is unclear. We discuss the role of $\theta\in \Theta$ in detail, starting on line 89. To address this, we propose to change the language of Assumption 4.1 to read:

Assumption 4.1 For a given latent concept sapce $\Theta$, the probability of generating response $\mathbf{z}_i^{(t+1)}$ is conditionally independent of both the responses $Z^{(t)}$ and the task $x$ when given any concept $\theta\in\Theta$ and model parameters $\phi_i$, i.e., $\mathbb{P}\big(\mathbf{z}_i^{(t+1)}|\theta, \mathbf{x}, Z^{(t)}, \phi_i\big) = \mathbb{P}\big(\mathbf{z}_i^{(t+1)}|\theta, \phi_i\big)$.

[Communication] Models communicate with each other only through tokens and do not share their latent space.

B Methodological Concerns

[Simplistic Multi-Agent Dialogue] We agree that debate as a paradigm fails to fully leverage the potential of a multiple LLMs. Our main goal with this paper is to provide a more rigorous formalization of debate. Given the growing number of papers focusing on multi-LLM debate (which all use a very similar setup to ours), we believe that our work is valuable.

We would also like to point out that debate is quite different from traditional ensemble methods. Stoping debate at round $t=0$ would be equivalent to a traditional ensemble. With that said, we would be very interested in seeing general multi-LLM interactions explored further, this is outside the scope of our work.

[Latent Concepts and a Reliance on Heuristics] We follow the latent concept model of [1, 2]. The true latent concept space will not be accessible; we use proxies such as sentence embeddings. We fully agree that our method's weakness is its reliance on heuristics. However, our theory informs the development of these heuristics. In our experiments, these heuristics are effective.

Implementation and Experimental Issues

[Theorey and Implementation] We use our theory to help guide and motivate the development of our heuristics. While we use proxies for statistical distance (namely, the distance between sentence embeddings), the choice of this proxy, as well as what we do with that proxy, are the direct result of our theoretical results. Beyond just informing the heuristics, our theoretical results also help provide some intuition as to why these heuristics work. In particular, it may not be obvious a priori why diversity pruning effectively reduces tyranny of the majority (as demonstrated in Figure 2). Both theorem 5.2 and the formulation diversity pruning (given on line 232) help explain this effectiveness more intuitively than just empirical observation alone.

[Model Choice] We use Llama-3, Llama-2, Mistral-v0.2, and GPT-3.5. Both Llama-3 and Mistral-v0.2 are state-of-the-art models. While we agree that Llama-2 (which has been replaced by Llama-3) and GPT-3.5 are not state-of-the-art, they are still quite capable models. Our experiments show that our method is effective on a diverse set of models, including quite capable models such as Llama-3.

[Improviemnt in the First Round] Our method achieves improvement over the baseline across all rounds (not just the first round). For both vanilla-debate and our method, the largest average gain is at round $t=1$. Both methods continue to improve over subsequent rounds (Figure 4). Ultimately, we observe significant improvement over vanilla-debate (in most cases) even after 10 rounds of debate (Table 1).

[Example about misconceptions] Our example of misconceptions indicates that misconceptions represent a particular type of error, specifically erroneous associations between more fundamental underlying ideas (concepts).

[Choice of KL Divergence] While we could choose other distance metrics, we choose KL divergence specifically to capture the surprise (in a technical sense of the word) when using one of the responses $\mathbf{z}_i$ as a predictor of $\theta$ compared to using the original task $\mathbf{x}$ as a predictor of $\theta$. Under this interpretation, the asymmetry of KL is desirable for quality pruning and misconception refutation. In the case of diversity pruning, because we sum over all pairs of $\mathbf{z}_i, \mathbf{z}_j$, a symmetry of KL is lost. We agree that the unbounded nature of KL can be undesirable.

[Ablation of Each Intervention] We provide an ablation of each intervention in Table 3 of the supplement and point to this table on line 314 of the main text.

[Related Work] There are quite a few relevant papers that have come out after the submission deadline that we plan to add. If you know of any other papers that we have missed, please let us know. We find this area deeply intriguing and would be very grateful for pointers towards any interesting work that we may have overlooked.

References

[1] Xie, Sang Michael, et al. "An explanation of in-context learning as implicit Bayesian inference." ICLR (2022).

[2] Hui Jiang. A latent space theory for emergent abilities in large language models. arXiv preprint 354 arXiv:2304.09960, 2023.

Dear Authors: Thank you for your response. I will adjust the rating accordingly.

Dear Area Chair,

Please determine whether the omission of the "limitations" section in the submitted checklist is a significant issue. Consistency in review policy is crucial for fairness, especially given that both of our NeurIPS submissions were desk-rejected due to checklist-related issues.

Thank you.

---

Dear Authors,

According to the NeurIPS policies, any violence of formatting instructions (as indicated in the Call for Papers: https://neurips.cc/Conferences/2024/CallForPapers -> Formatting instructions) results in a desk rejection. Your paper was rejected since the NeurIPS paper checklist is part of these formatting instructions (see https://neurips.cc/Conferences/2024/CallForPapers -> Formatting instructions -> Point 3). Unfortunately, we are unable to allow further submissions or amendments to submissions as the system is now closed.

We are aware that this is not a decision you expected, but we must follow the NeurIPS policies. The decision is final and cannot be reverted.

We are sorry to hear the unfortunate desk rejections from your end, and thank you for reminding us of this. In the future, we will pay closer attention to it.

At the time of submission, we had a different interpretation of the limitation requirement. While a separate section is encouraged, we thought the paper flows better when we discuss the limitations right in the main text, and labeling Yes simply states that we are aware of our limitations and have discussed them in the main text. (in the checklist, we added the justification that "Justification: We mention the limitation of our approach in the conclusion section and experiment section").

That said, we would like to be clear that we fully agree with the reviewer's comments regarding our discussion of limitations; our initial discussion of limitations was insufficient and should be provided in its own section (please see our general response for our proposal).

I pasted the statement of rejecting my papers. Of course I have no authority what-so-ever to make that decision.

Good luck!

[Applying Debate to Other Settings] We think that applying debate to settings beyond QA tasks would be very interesting. As of now the majority the works on debate focus on QA tasks (while some investigate translation). We hope that our framework inspires further generalization to other settings.

[Typos and Grammar] Thank you for pointing these out. We will address these in the final version of our paper.

[Diversity Pruning and Mutual Information] It would be more accurate to say that we are maximizing the amount of unique information present in the responses $Z'^{(t)}$, i.e., maximizing the information entropy present in $Z'^{(t)}$. Our use of the term "mutual information" arose during editing. Initially, we had said, "the amount of unique information mutually present" and then edited this phrase down to “mutual information” on lines 35 and 232, without realizing that we had inadvertently invoked a technical term conveying something other than our intention. This is an oversight on our part; we will modify the language around diversity pruning from "mutual information" to "information entropy".

[Theoretical Assumptions] While we agree that we make some stylized assumptions, we believe that these assumptions are reasonable in the context of debate and LLMs. For example, we formulate debate in a Markovian manner (i.e., each agent can only see one round of history). This Markovian property is not unique to our paper and is common in most papers on LLM debate (e.g., [Du, Yilun, et al.]). It would be interesting to consider cases where agents have access to longer histories, especially with the rapidly growing context windows of modern models (e.g., 128K token window of the newly released Llama 3.1).

As for Assumption 4.1, we would like to provide some clarification. At each round $t$, the question $\mathbf{x}$ and the previous responses $Z^{(t-1)}$ are formatted into a prompt, which is passed to the model. Assumption 4.1 says that the model's generation at time $t$ (namely $\mathbf{z}^{(t)}$) is independent of the question and the other models' responses at time $t$ given the latent embedding $theta$ and model parameters $\phi$. This assumption can be thought of as saying that once a prompt has been passed through the encoder of the model, the original prompt (i.e., $\mathbf{x}$ and $Z^{(t)}$ in our case) is no longer needed.

[Majority Answers and Figure 2] For the results in Figure 2, we force the models to provide specific answers using special target prompts (an example for BoolQ is provided below). We design these prompts to be as similar to the original prompts as possible (see line 576 of the supplement for the original BoolQ prompt). We use the targeted prompt 20 times for each target answer to obtain a diverse set of responses. Any response that does not provide the target answer is removed. We will add this explanation to the experiment section (after the first sentence of line 285) and the following example prompt to Section B.1 of the supplement (after line 617).

You will be given a yes-no question which is based on a passage.
You should use the passage to provide an answer of {_TARGET_ANSWER_}.
You should give a brief justification for that answer,
and you must provide a final answer {_TARGET_ANSWER_}.
\n Question: {_QUESTION_}
\n Passage: {_PASSAGE_}

The values shown in Figure 2 are averages over all questions in each dataset (see line 273 for these values). We show one plot for each dataset, denoted by each plot's title in Figure 2.

If anything else is unclear about the experimental design, please let us know; we are always eager to improve the readability of our paper.

[Running Time] This is a good point; debate as an inference procedure can be costly, and our method introduces additional overhead. We believe that the performance gain from our method justifies the increased inference time. Below is a table showing the average run time for a single question (in seconds) for both the vanilla debate method and each of our interventions. We use 6 models and report the average time to complete 1 question, averaged over 3000 questions from BoolQ. The comparisons are vanilla debate (VD), diversity pruning (DP), quality pruning (QP), misconception refutation (MR), and a combination of all three interventions outlined by Algorithm 1 (All 3).

When combining all three of our interventions, we see that the additional overhead compared to vanilla debate is not too severe. This is due primarily to the fact that diversity pruning and quality pruning remove responses from $Z^{(t)}$, which shortens the prompt lengths at round $t+1$, which in turn speeds up the inference time of each model. We will add this table to Section B of the supplement and reference/discuss this table on line 312 of our experiments section.

For Llama-2, Llama-3 and Mistral, we use a single Tesla-v100-32GB GPU, an Intel i9 CPU, and the VLLM acceleration library (https://docs.vllm.ai/en/latest/). For GPT-3.5 we use the openai api and use the AsyncOpenAI class to handel the prompts in batches of 500 (https://github.com/openai/openai-python).

References [1] Du, Yilun, et al. "Improving factuality and reasoning in language models through multiagent debate." arXiv preprint arXiv:2305.14325 (2023).

[Citations and Specificity] Thank you for pointing this out. We agree with both points regarding the related work. We will add named citations rather than numbered citations. The specific model versions are provided in Table 2 of the supplement, but we will add this information to line 280 of the main body.

[Figure 3] We will make this figure large and change the dashed green line to a dashed blue line to further distinguish these values.

[Difference Between Debate and Our Method] Our method (namely our three interventions) is an in-processing modification to the original debate procedure in which we leverage a latent concept space $\Theta$ to modify the responses $Z^{(t)}$ before these responses are used at the next round $t+1$. We provide an overview of the vanilla debate procedure (proposed in [Du, Yilun, et al.]) on line 73 of our paper. To help clear up this point, we propose to cite [Du, Yilun, et al.] at the beginning of our discussion on debate (line 73), and to state after line 88 that “The key distinction between our method and vanilla debate is that we will leverage a latent concept space (discussed next) to modify the responses $Z^{(t)}$ in-between round)”.

[Method and Task are Both Called Debate] We agree that overloading the term debate can cause confusion. We propose the following fix: we will refer to the debate baseline [Du, Yilun, et al.] used in our experiments as "vanilla-debate" (VD), and we will refer to our method as either "Ours" or "Debate via Latent Concepts" (DLC).

[Small Errors] Thank you for pointing these out. We will make these fixes in the final version of our manuscript.

[Role of $a$ in Formula (3)] In Formula (3), the variable $a$ is an "answer extraction" function explained on line 80 of our paper. If $\mathbf{z}$ is a string, then $a(\mathbf{z}))$ is the extracted answer from that string. For example, suppose the question is "What color is the sky?" and $\mathbf{z} =$ "During the day, the sky is blue.", then $a(\mathbf{z}) =$ "blue." In practice, $a$ is implemented either through regular expression matching (as in the case of BoolQ, MMLU, and MathQ) or an LLM judge (as is the case in TruthfulQA).

[Misconception Refutation] In practice, misconception refutation works as follows. A model $f_1$ provides a response $\mathbf{z}$. A second model $f_2$ is then asked to identify a list of misconceptions in $\mathbf{z}$. The first model $f_1$ is then reprompted for an updated response that addresses the misconceptions raised by $f_2$ ($f_1$ and $f_2$ can be the same model). The intuition for why this works is very similar to why methods such as self-refinement are effective [Madaan, Aman, et al.] (also called self-reflection); judging is often an easier task than generating. However, it is also important to note that there is a limit to how well LLMs can self-identify their own mistakes as pointed out by [Tyen, Gladys, et al.], LLMs can have a difficult time identifying their own errors in reasoning tasks, but (as the title suggests) can fix erroneous reasoning steps when being told that a given step is wrong (even without being told what the correct step should have been). This observation might indicate that misconception refutation scales in the effectiveness of $f_2$ (responsible for misconception identification) more than the effectiveness of $f_1$.

References

[1] Du, Yilun, et al. "Improving factuality and reasoning in language models through multiagent debate." arXiv preprint arXiv:2305.14325 (2023).

[2] Madaan, Aman, et al. "Self-refine: Iterative refinement with self-feedback." Advances in Neural Information Processing Systems 36 (2024).

[3] Tyen, Gladys, et al. "LLMs cannot find reasoning errors, but can correct them!." arXiv preprint arXiv:2311.08516 (2023).