Let Models Speak Ciphers: Multiagent Debate through Embeddings

The experimental results are only on 4 datasets

We would like to clarify a minor discrepancy in the number of datasets mentioned. Our experiments were conducted on five datasets, including GSM8K, Formal Logic, High School Math, Professional Psychology, and Arithmetic.

the improvements are relatively small, and no confidence intervals are reported.

Our approach demonstrates consistent improvements of 1-3.5% across five datasets and six different models without any further training or fine-tuning. Thus, we believe that the improvement is significant.

We ran 5 experiments for each method in Table 1 to obtain the confidence intervals for LLaMA2, reported as below:

Overall, Single Answer has the highest variance, ranging from 1 to 3% across five datasets. This increased variance is attributed to its token sampling process during token generation. Major@5 and NLD methods reduce variance by ensembling diverse responses. However, they still rely on a similar token sampling process, leading to relatively high variance. In contrast, CIPHER is not affected by this issue, thanks to its deterministic embedding generation. For any given input, CIPHER consistently produces the same embeddings, which are then translated into identical texts via a nearest neighbor search across the vocabulary set.

It's important to note that running these baseline methods are time-consuming, regarding that we need at least 5 runs for each to obtain a valid standard error estimation. While we are working on confidence intervals for additional experiments, as of now, we can only provide these intervals for LLaMA2 in Table 1 as part of our rebuttal. We plan to include all the results in the final camera-ready version of our work.

I didn't see any discussion on validation set for hyper param setting

As our approach focuses solely on inference without involving any training phase, the conventional use of a validation set for hyperparameter tuning was not applicable. Additionally, we swept over temperatures to guarantee we report the best of each method, hence no need to set a validation set.

according to the appendix it seems like different temperatures were used for the main baseline (NLD) and for CIPHER? Why is that? This raises some additional concerns about the experimental setup.

We employ Bayesian optimization to sweep over some temperature pairs for all the baselines and our method. For a fair comparison, we reported the best performance of each method in our paper. We also included an ablation study on the temperature, as reported in Figure 5, to give a deeper understanding of how the temperature affects debate performance.

As each method prefers different temperatures to obtain the best results, it is not fair to use the same temperature. For example, Majority Voting (major@5) usually requires higher temperatures compared to Single Answer, to have a more diverse response for the ensemble.

That said, we still conducted some results on the same pairs of temperatures (in brackets) for both CIPHER and NLD baseline as follows:

In general, CIPHER works better than NLD for these temperatures reported above, but again, this is not a good setting as each method favors different temperature settings. As discussed in Sec 5.2, CIPHER often prefers low and high-temperature agents to pair together. This explains its lower accuracy compared to NLD on two datasets, as shown in the table above, particularly in Arithmetic, where CIPHER prefers a large difference in temperatures.

We thank the reviewer for the useful comments on our work. We address the comments below.

The method requires access to the output embeddings of the model and the full output probability, so cannot be directly applied to LLMs served via API.

Our research is focused on open-source language models due to its transparency and reproducibility. While open-source models provide insight into their internal mechanisms, APIs often obscure the distinction between the AI model and the engineering components, e.g., incorporating specialized engineering for specific canned responses.

While there method is described in several sections, the definition of the method still didn't feel precise enough. For example, I assume that at each step of the autoregressive generation the input token embedding is the previously aggregated token embedding and not a single sampled token? Or do you still sample tokens for decoding and just convert the full sequence to aggregated embeddings after?

We are grateful to the reviewer for pointing out the need for greater clarity. In our approach, at each step $t$ of the autoregressive generation, we utilize an aggregation of all previous $t−1$ generated token embeddings as the input for the next token generation.

We have revised Section 3.2 of our manuscript to reflect this clarification. The updated notation, now denoted as $\overline{emb}^{(1:t-1)}$, has been introduced to clearly represent the aggregated input at each step of the generation process. Please refer to the revised version for a more detailed and precise explanation.

There is a lot of discussion on the effect of sampling temperature, including ablation experiments etc., that I am confused about. From my understanding of the method (e.g., Equation 2), there is no temperature parameter in the method. The only effect of temp is in the final generation of the answer by one of the models. Therefore, I don not understand why are two temperatures reported, or the ablation in Figure 5.

We apologize for any confusion caused by our initial presentation of Equation 2 and appreciate the opportunity to clarify. In Equation 2, our method generates a new embedding by calculating the weighted average of all token embeddings within the vocabulary set. Here, the “weight” for each token is determined by its respective probability, which is derived from the softmax of the logits. Similar to conventional natural language generation processes, the temperature parameter $T$ controls the smoothness of the probability distribution, thereby shaping the output embedding. To enhance clarity and facilitate better understanding, we have revised notations and Equation 2 in our manuscript. We decided to break it down into two separate equations as follows:

$\bar{emb}^{(t)} = \sum_{i = 1}^{V} p_{vocab_i}^{(t)} \cdot emb_{vocab_i}$

where

$[p_{vocab_i}^{(t)}, \dots, p_{vocab_V}^{(t)}] = softmax \\{ logit(emb_{prompt}, \bar{emb}^{(1:t-1)} ) /T \\}$

In addition to the equation, we have revised Section 3.2 of our manuscript to clarify on the temperature:

”Role of temperature. The temperature $T$ in Eq.1 and Eq.3 controls the smoothness of the probability $p^{(t)}_{vocab_i}$. When $T \rightarrow 0$, both CIPHER's embedding generation and natural language generation result in greedy generation. In contrast, a large $T$ leads to a uniform averaging and sampling over the whole vocabulary set for CIPHER and natural language generation, respectively. We find that choosing proper temperatures for the debaters plays a pivotal role in the performance of CIPHER debate and natural language debate. Thus, to ensure fairness of our empirical evaluation, we utilize Bayesian optimization [1] to select the best performing temperatures for each method in our experiments in Section 4. Moreover, we conduct sensitivity analysis on the temperatures in Section 5.2.”

[1] Bayesian Optimization: Open source constrained global optimization tool for Python

We believe these changes will effectively address the comments and improve the overall clarity of our method's description.

Thank you for your detailed reply. The updated paper now indeed better explains the role of the temperatures in the method and clarifies some of my initial confusion. I appreciate your work on revising the paper and answering my questions.

I would be very interested in examining the experimental performance of the proposed method. However, I still have two concerns with the current version and response:

1. I think you confused my question about confidence intervals with computing variance. CI help provide a statistically more reliable estimate of the performance. One option to obtain such is to use bootstrapping which doesn't require any new runs of the model and works for both deterministic or random outputs.
2. I disagree with your answer that a validation set is not required here. Sweeping hyper parameters over the test set and reporting best performance is not a valid evaluation setup. Especially given the sensitivity of the performance to the temperature value as you yourself state above and as shown in the temperature heat map figure.

I think you confused my question about confidence intervals with computing variance. CI help provide a statistically more reliable estimate of the performance. One option to obtain such is to use bootstrapping which doesn't require any new runs of the model and works for both deterministic or random outputs.

We appreciate your suggestion regarding the use of bootstrapping to estimate confidence intervals (CIs). However, while confidence intervals are valuable in many statistical analyses, it is not standard in the related literature. ​​ For example, related works such as [1, 2, 3, 4, 5, 6] presented their findings without reporting confidence intervals. In addition to the std reported for LLaMA2-70B, we’re working on adding std for the rest of the table, and will incorporate the results in our camera ready version. While we believe that including standard deviation adequately demonstrates the robustness of our method, especially in comparison with baseline models, we still show the bootstrapped 95% confidence intervals (in brackets) for our results as follows.

LLaMA2-70 in Table 1

LLaMA-65B, Table 1

Fig 3 for multiagent debates across different models

[1] Self-Consistency Improves Chain of Thought Reasoning in Language Models

[2] SELF-REFINE: Iterative Refinement with Self-Feedback

[3] Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback

[4] Teaching large language models to self-debug

[5] Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate

[6] Improving Factuality and Reasoning in Language Models through Multiagent Debate

The authors conducted experiments on five common reasoning datasets, can this method be tested on agent-related leaderboards

We kindly ask the reviewer to clarify which agent-related leaderboards might be relevant. If the leaderboard suits our framework, we would be happy to test and report it in our final version.

In formula 2, whether the response embedding will be adjusted, how are the results of different weights?

In Equation 2, our method generates a new embedding by calculating the weighted average of all token embeddings within the vocabulary set. Here, the “weight” for each token is determined by its respective probability, which is derived from the softmax of the logits. Similar to conventional natural language generation processes, the temperature parameter $T$ controls the smoothness of the probability distribution, thereby shaping the output embedding. We also added a discussion on the temperature in the revised version in Sec 3.2, as follows:

”Role of temperature. The temperature $T$ in Eq.1 and Eq.3 controls the smoothness of the probability $p^{(t)}_{vocab_i}$. When $T \rightarrow 0$, both CIPHER's embedding generation and natural language generation result in greedy generation. In contrast, a large $T$ leads to a uniform averaging and sampling over the whole vocabulary set for CIPHER and natural language generation, respectively. We find that choosing proper temperatures for the debaters plays a pivotal role in the performance of CIPHER debate and natural language debate. Thus, to ensure fairness of our empirical evaluation, we utilize Bayesian optimization [1] to select the best performing temperatures for each method in our experiments in Section 4. Moreover, we conduct sensitivity analysis on the temperatures in Section 5.2.”

To enhance clarity and facilitate better understanding, we have revised the notations and Equation 2 in our manuscript. We decided to break it down into two separate parts, now Eq.2 and Eq.3, as follows:

$\bar{emb}^{(t)} = \sum_{i = 1}^{V} p_{vocab_i}^{(t)} \cdot emb_{vocab_i}$

where

$[p_{vocab_i}^{(t)}, \dots, p_{vocab_V}^{(t)}] = softmax \\{ logit(emb_{prompt}, \bar{emb}^{(1:t-1)} ) /T \\}$

Why are the results in table1 and table2 completely different, how many rounds are used in table1？

The settings in Table 1 and Table 2 are different and are stated in the captions. Specifically, in Table 1, we facilitate debates between two identical LLaMA-family models, with temperature being the only difference between the two debaters. In this case, we report the accuracies of the final responses from the debater operating at the lower temperature. In Table 2, we facilitate debates between one LLaMA-65B (first version) and one LLaMA2-70B. In this case, we report the accuracies of the final round responses from the two debaters separately for a thorough analysis. In both Table 1 and Table 2, the total number of rounds is 3. We added this to the caption for Table 1 in our revised version.

Can this method be used for different models with the same tokenizer, for NLD, different models can communicate each other.

Yes. In Table 2, we applied CIPHER debate between LLaMA-65B and LLaMA2-70B.

Limited Generalizability. As the authors described in the limitations, this method is only applicable to LLMs that share a common vocabulary. For different types of LLMs, aligning embeddings is a difficult task.

As stated in the Limitation section, requiring a shared tokenizer is a limitation of our work and is out of the scope of this paper. Extending CIPHER debate to models with different tokenizations will require tokenization/embedding alignment, which would be an interesting direction for us to further explore. Such an alignment can potentially be tackled by utilizing methods in multimodality literature, such as [1, 2]. This literature deals with broader questions related to aligning various modalities.

We would like to note that, even though CIPHER debate can only be applied to models sharing the same tokenizer, we verify that it is effective in pushing the reasoning performance of the open-source models beyond the current state-of-the-art methods.

[1] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

[2] MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

From Figure 10, the language of CIPHER is still difficult to analyze.

As described in the Introduction Section: “First, we do not necessarily need to understand the intermediate debates amongst LLMs. Second, natural language generation uses only one token to represent the model’s belief over the entire vocabulary, which risks losing information embedded within the model output logits.” The main takeaway of our approach is to trade interpretability for information preservation during the debates. Yet, we can still interpret the communications made by the debater operating at low temperatures (e.g., blue debater, now Figure 11 in the revised version) while using another debater with a higher temperature to gather more information. In practice, keeping at least one debater at a low temperature is necessary to collect the final result of the debate.

Which experiment result can support the statement “our approach can generalize across a wide array of LLMs, enabling even smaller LLMs to unlock the benefits of debate and achieve better performance than majority voting"? Is there any experiment of smaller LLMs like LLaMA-2 13B or others?

Prior work in LLM debate and self-critique (e.g., [1, 2]) show that only some state-of-the-art LLMs, such as GPT-3.5, GPT4 are capable of taking advantage of debates and discussion. These models are over 100B parameters and closed-source. In our work, we propose CIPHER that unleashes the potential of smaller open-source models in debates. By “generalize across a wide array of LLMs”, we mean that one can apply CIPHER to smaller sized models such as LLaMA-family ones, ranging from 7B (reported in Appendix B.2, Fig. 8a) to 70B (table 1). We chose to mainly use LLaMA2-70B in our experiments because it’s one of the largest and most up-to-date models at that time. Additionally, we showed the results across different model architectures in Fig 3 to evaluate the robustness and generalizability of our method.

[1] Self-refine: Iterative refinement with self-feedback

[2] Improving language model negotiation with self-play and in-context learning from AI feedback.

Why is the performance of CIPHER worse than natural language debate when Round=1?

In Table 2, when round=1, it means the model only gives a direct response. By choosing responses of the larger model (“LLaMA2-70B” columns) as the final answers, our method is not worse, but rather slightly better than the baseline at round=1. That said, CIPHER is designed for communication in debate settings. Thus, we need debate rounds to fully take advantage of the method. We added the results by round for ablation study purposes, to see how the performance changes over rounds.

I suggest incorporating the meaning of Round=1 into the figure caption

Thank you for the suggestion. We will incorporate the meaning of round=1 to the figure caption in the camera-ready version.

in Figure 4(a), the performance of CIPHER at rounds = 1 seems weaker than NLD

In Figure 4(a), CIPHER's performance is lower than NLD in the first round. In general, the performances of NLD and CIPHER are quite similar in the first round, as shown in the performance of LLaMA2-70B and LLaMA-65B in Table 2. As discussed earlier, the first-round result is the direct response without any debate. CIPHER is designed for communication, thus we need a few rounds of debate to fully take advantage of it.

why does the performance of LLaMA2-70B (CIPHER) decrease on Arithmetic when Round=2?

It's a good observation! In Table 2, we paired LLaMA2-70B with a smaller model, LLaMA-65B.

In the first round, LLaMA-65B's performance was low, resulting in a large gap between the two models, especially on the Arithmetic dataset.

In the second round, LLaMA2-70B received responses from LLaMA-65B, which were much worse compared to its own performance. This confused LLaMA2-70B, leading to a decrease in its performance in this round. However, for LLaMA-65B, this round saw a significant boost in performance, as it obtained much better answers from LLaMA2-70B.

In round 3, we observed an increase in the performance of both models after the gap between them was reduced.

In general, after a decrease in the second round, LLaMA2-70B recovered and boosted its performance in the third round. Note that this drop in the second round was also the case for NLD, where LLaMA2-70B experienced a decrease in the second round.

Regarding the drop in performance, an interesting question arises: how can a bad debater, such as a dummy debater, affect the final debate performance? We investigate this in Appendix B.2, where we analyze the performance upper bound (debate with an expert debater) and the lower bound when using nonsensical feedback from other debaters. We observe that debate can be detrimental when the model has low capacity (Fig. 8a), but it does not pose much harm in the case of a more capable model (Fig. 8b)

We thank the reviewer for the valuable and detailed feedback. We appreciate the advice on the presentation and have revised the paper accordingly in our new version. We address the comments as below.

It is unclear how the Convert-and-Aggregate function works, in particular how the responses from multiple debaters are distilled into a single response.

We write the Convert-and-Aggregate function in the debate algorithm within a general framework, to accommodate the various distillation strategies in the current literature. In our revision, we made extra elaboration on what such a function can be in the context of LLM debates and also justify our choice. In general, we can apply random pick, majority voting [1], involving a judge as suggested in a concurrent work [2], and lower temperature (our method). We now clarify in Section 3.2 the following:

“To close the debate, at Convert-and-Aggregate step (Line 6), we convert the embedding responses back to natural language using nearest neighbor search over the vocabulary set, then aggregate them to obtain the final response. In most cases, LLM debaters typically reach a consensus answer by the final round, as observed in [3]. When divergence in final responses occurs, majority voting [1] or random tie-breaking are often used. However, majority voting may not be suitable for open-ended questions (e.g., summarization tasks) where multiple correct answers exist, as in the game of 24 [4], and scenarios where debates involving only two agents. Thus, in our experiments, we select the response from the debater operating at the lowest temperature as the final answer. This approach is computationally efficient as it only requires running inference on one model at the final round. Meanwhile, it also achieves comparable accuracy with the best performing debater, as evidenced in Fig.5”

[1] Self-Consistency Improves Chain of Thought Reasoning in Language Models

[2] Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate

[3] Improving Factuality and Reasoning in Language Models through Multiagent Debate

[4] Tree of thoughts: Deliberate problem solving with large language models

The "Result collection" and "Metrics" paragraphs in Section 4.1 are the first time in the paper that differing temperatures are mentioned. If an optimization procedure is being used for temperature selection as part of the method, then this should be described in detail along with the rest of the method in Section 3.

Thanks for pointing it out. We have incorporated the Result collection section along with the temperature selection procedure to Section 3.2, answered in the above question. We also dedicated a section for the role of temperature and how it can affect the debate resulting in the new revision, Section 3.2 as follows:

”Role of temperature. The temperature $T$ in Eq.1 and Eq.3 controls the smoothness of the probability $p^{(t)}_{vocab_i}$. When $T \rightarrow 0$, both CIPHER's embedding generation and natural language generation result in greedy generation. In contrast, a large $T$ leads to a uniform averaging and sampling over the whole vocabulary set for CIPHER and natural language generation, respectively. We find that choosing proper temperatures for the debaters plays a pivotal role in the performance of CIPHER debate and natural language debate. Thus, to ensure fairness of our empirical evaluation, we utilize Bayesian optimization [1] to select the best performing temperatures for each method in our experiments in Section 4. Moreover, we conduct sensitivity analysis on the temperatures in Section 5.2.”

[1] Bayesian Optimization: Open source constrained global optimization tool for Python

The temperatures used should all be clearly reported, and whatever process is used for temperature selection should either also be applied to the other baseline methods where relevant, or ablated away in a separate experiment to highlight potential sensitivities to this hyperparameter, or both.

We apologize for the confusion. The temperatures used for the reported accuracies are already included in Appendix D (tables 3, 4, 5). To further aid reproducibility, we added all other temperature settings in two additional tables (Table 6 and Table 7) in Appendix D.1. These new tables correspond to the temperatures depicted in Figure 4. We included a reference to this Appendix in the main text of our revised version, at the beginning of Section 4.1 “Experimental Setup”, for easier reference as follows:

“For all the methods, we use Bayesian optimization to select temperatures, which are reported in Appendix D for reproducibility of our results.”

For both CIPHER and the baselines, we swept over temperatures using Bayesian hyperparameter optimization to make sure that the accuracy reported is optimized for each method. Additionally, we also investigate the effect of temperature, as shown in Contour plots in Fig. 5 for an ablation study.