Scalabale AI Safety via Doubly-Efficient Debate

We are very happy to hear that you find that our approach offers a unique proposition that would be of relevance and appeal to practitioners in the field! Thank you for your encouraging words!

"Could the authors clarify the connection between their protocols and the framework introduced in [1]? [1] Du Y, Li S, Torralba A, et al. Improving Factuality and Reasoning in Language Models through Multiagent Debate[J]."

The biggest distinction is that the method of [1] is test-time only invervention. That is, several models provide answers to a question, then are prompted to re-evaluate their answers based on the output of the other models and so on until a final answer is output.

In contrast, our paper is a training-time intervention. That is, we train the models via reinforcement learning to answer questions by winning debates against opposing models. So in our case the debate happens at training time and the goal is to produce models that give honest, correct, and verifiable answers.

Additionally, our paper proves formal theoretical guarantees showing that our debate training method needs only a fixed constant amount of human feedback per question, even when the length of the human-verifiable argument for the correct answer is very long. This is the main high-level takeaway message of our theorems, and it has no counterpart in [1], which is an empirical, test-time method.

We are very encouraged by your comment that our proposed combination of adversarial provers and human oversight is a promising direction in an increasingly important topic. We hope that our proposal can accelerate much needed progress on this subject.

"Can you present a simple example of a false proof that survives the protocol, because of inconsistencies or errors in the (human) oracle?"

Please see our response below regarding the example of the oracle machine $M$ that computes majority. This example shows that if we are attempting to find the majority vote of $n$ questions where the correct probability for yes for each is $\frac{1}{2} - O(\frac{1}{\sqrt{n}})$, an error of just $O(\frac{1}{\sqrt{n}})$ in the human oracle is enough to flip the answer from 0 to 1. Thus, using debate to conduct polling on a very evenly contested issue can result in large error, even if each individual answer is only slightly biased.

An important point to make here is that after training with debate, one can easily verify at test time if an oracle machine $M$ is non-Lipschitz at $\mathcal{O}$ by simply running the protocol by asking $A$ to argue that $M^{\mathcal{O}}(x) = 1$ and then running it again while asking for $A$ to show that $M^{\mathcal{O}}(x) = 0$. If $A$ is able to win the debate in both cases, then it is clear that $M$ is not sufficiently Lipschitz and the results cannot be trusted.

"The probabilities in Definition 3.2 stood out: are these merely illustrative (so that any result could be replaced by arbitrary constants $a$ and $b$ ), or would even qualitative results derived be overturned by use of different fractions (e.g. are there critical values for these numbers)?"

The probabilities in Definition 3.2 are just illustrative constants, all that is needed is that they are bounded away from $1/2$. A key point here is that the probability is over the internal randomness used by $M$. Therefore by simply running the machine $M$ independently $k$ times on the same input $x$ and taking the majority vote of the answers, the probability of outputting the wrong answer goes to zero at a rate exponential in $k$.

"Definition 6.1: can you provide an example of an oracle that is not -Lipschitz?"

Let $\epsilon = \sqrt{\frac{10}{n}}$ and $\mathcal{O}$ be the oracle that outputs 1 with probability $\frac{1}{2}-\epsilon$ on every query. Let $M^{\mathcal{O}}$ be the machine that, on input $x$, takes the majority vote of the outputs of $\mathcal{O}$ on the $n$ strings $y^1,\dots,y^n\in\{0,1\}^l$ lexicographically following $x$. Then $M$ is not $K$-Lipschitz for any $K < O(\sqrt{n})$.

To see why consider the oracle $\mathcal{O}'$ which outputs 1 with probability $\frac{1}{2} + \epsilon$. By standard concentration bounds, the probability that the majority over $n$ samples from $\mathcal{O}$ is 1 is at most $\exp(-O(\epsilon^2 n))$. On the other hand, the probability that the majority of $n$ samples from $\mathcal{O}'$ is 0 is $\exp(-O(\epsilon^2 n))$. Thus by our choice of $n = 10\frac{1}{\epsilon^2}$ a change in the probabilities of the oracle by $2\epsilon$ can result in a change in the output of $M$ by nearly 1. Thus $M$ is not $K$-Lipschitz at $\mathcal{O}$ for $K < O(\frac{1}{\epsilon}) = O(\sqrt{n})$.

"FYI"

Thank you for this list of related debate/argument concepts. Your pointers contain some very interesting ideas that could be used extend or change our debate model, and to reason about debate more generally. We look forward to engaging with these ideas actively in future work. Thank you again!

We are very happy to hear that you found our paper well written and enjoyable to read with practical examples provided throughout the main paper that clearly motivate the protocols studied in this work, and are helpful for understanding technical details! This type of encouragement means a lot to us.

"I don't have specific clarification questions. However, it would be great if the authors could provide a discussion related to my comments about the weaknesses of their work. More specifically, some discussion on whether it would be possible to set up an experiment based on LLMs which showcases the utility of their framework."

At the scale of the examples given in our paper (e.g. writing long contracts or conducting meta-analyses), LLMs are not currently good enough to "get-off the ground" in an RL training setup where one model produces outputs and the other points out flaws. In particular, the first model will likely always have many flaws in its outputs.

However, we believe that it will soon be possible to set up smaller-scale but useful experiments with LLMs utilizing this framework. The simplest version of such an experiment would be to train two models $A$ and $B$ in an adversarial RL setup where the human judge determines who wins. For example, a natural setting could be math or logical reasoning problems that require many steps of chain-of-thought to solve. Here the model $A$ is supposed to provide long chain-of-thought answers to questions, and $B$ is supposed to point out an incorrect step in the chain-of-thought. Human raters are then shown the step pointed out by $B$. If the rater judges the step to be incorrect $B$ wins, otherwise $A$ wins. The base models are then trained by playing against each other to win at this task. Our method applied to this task would allow the number of human ratings required to scale with the number of problems solved, rather than the number of problems times the number of steps per problem.

The key properties that such an experiment testing our theory needs to have is that:

1. It is possible for the models to break down complex arguments into simpler, human-judgeable steps.
2. The arguments made are long enough that the efficiency gained by only requiring human feedback for a single step is significant.

1. "Could authors give an example to show the difference between the proposed debate method and existing works such as Irving et al. (2018)? Why the proposed method could get a better bound intuitively?"

The key difference between our approach, and that of Irving et al. (2018) is that we explicitly require that the debater model arguing for the truth can do so efficiently. In contrast, Irving et al. (2018) assume that both debaters have unlimited computational power. To see why this can make a difference, consider the example for Theorem 6.2, in which we want to use a debate between LLMs to perform a meta-analysis of scientific literature on whether caffeine improves test-taking performance. Suppose that in this example, the truth is that the literature shows that caffeine does not improve performance. The model $A$ could attempt to $p$-hack the analysis by selecting a subset of existing studies that give the desired result, and then inventing an after-the-fact reason to reject all the studies that were not contained in the selected subset.

The key difference between our approach and that of Irving et al. (2018) now arises with regard to the model $B$, which is tasked with arguing for the truth (i.e. that caffeine does not improve performance). In the approach of Irving et al. $B$ is modelled as having unlimited computational resources. So $B$ would be allowed to employ brute-force strategies in order to show that $A$ has $p$-hacked. For example, $B$ could re-run the analysis on every subset of studies the same size as the subset selected by $A$, and show that the opposite conclusion is reached a large majority of the time. Note that this strategy requires $B$ to run a potentially exponentially large number of meta-analyses in order to refute just one meta-analysis run by $A$.

Thus, using the approach of Irving et al. it is possible that the debater arguing for the truth may necessarily need to spend exponentially more compute in order to actually win the debate. Our approach explicitly requires the design of protocols where the debater arguing for the truth must run efficiently. In fact, the protocols we design allow the debater arguing for the truth to win by spending only $O(T\log T)$ compute time, where $T$ is the time that it would take to solve the problem without debate. To rephrase this in the context of the meta-analysis: if it is possible to correctly perform a meta-analysis via $T$ forward passes of an LLM, then our debate protocol requires only $O(T \log T)$ forward passes in total to guarantee that the debater arguing for the truth will win.

2. "How to implement the method in practice to help researchers to train an LLM?"

In practice, in the simplest version of our protocol, one would train two models $A$ and $B$ via reinforcement learning to play the following zero-sum game:

1. Sample a question $Q$ from the dataset.
2. Prompt $A$ to give a chain-of-thought answer to $Q$.
3. Prompt $B$ to point out a step in $A$'s answer which is incorrect.
4. Show the step pointed out by $B$ to a human rater. If it was incorrect, $B$ wins, otherwise $A$ wins.