Prover-Verifier Games improve legibility of LLM outputs

We thank the reviewer for recognizing we study an interesting tradeoff between performance and legibility, as well as proposing a clear method.

Related works: We updated the related works section on actor-critic, RLHF, and explainability literature. We thank the reviewer for bringing up those works. Our work is complementary to RLHF. Obtaining human labels for legibility is expensive (requiring pairs of comparison data). Our main motivations for exploring alternatives to RLHF are in Section 1 (Line 42-45), i.e., using small LLM as proxy is more scalable, allows for continuous parameterization, and can offer incremental signals when the tasks are too difficult.

Details of human study are in Appendix K; we did not include it in the main text due to space limit. To recap:

1. Short time limit: time-limited human judge accuracy is the main measure of human legibility in our paper. It’s one of many possible measures, and we welcome better suggestions. We used a 1 min limit in the pilot campaign and the tasks turned out too easy.

2. Control for solution lengths: Legibility has multiple dimensions: brevity, good format, simple vocabulary, etc. Length is one of the variables. We did not single out any single component of legibility but rather measure it holistically via time-constrained human judge accuracy. Our goal is to be legible in the time frame that humans have to spend on the task. If we allow infinite time, humans will do the task themselves. We want AIs to be helpful within the time constraints that humans have (though in this case they’re imposed by experimenters).

3. Flaw: In our pilot study, we showed sneaky samples to the trainers. This study has a flaw where samples from earlier rounds are shown first to the trainers. The only plot in the main text that came from the pilot study is Figure 7b (human judge accuracy of sneaky samples across rounds); two more in the appendix for ablations (S7c, S8). Most of the human eval results came from the main study with larger sample size and fully randomized assignment (75 trainers, 15983 samples). Trainers are randomly assigned balanced correct / incorrect samples from the last helpful checkpoint each round. Most of the human eval results (Fig 1, 4, 8) are from the main study, so our conclusions on poor legibility of correctness-only baseline, increased legibility on helpful samples, and decreased time to solve are all accurate. We didn’t re-run human eval on sneaky solutions and ablations due to budget constraints.

4. Distribution of problems: The problems are uniformly random GSM test problems, and no one sees any single problem more than once. We expect some learning and fatigue effects so we limit total labor to 4-hour max in the main study. Since assignment is fully randomized this should not affect the relationship between judge accuracy and training rounds. We refer the reviewer to Appendix K for full details.

Single dataset and generalizability: Most previous works on scalable oversight only study the QuaLITY dataset [Irving et al., 2018; Radhakrishnan, 2023; Michael et al., 2023; Khan et al., 2024]. Our work is the first to study debate-like methods between provers and verifiers where the gap isn’t privileged information but capability, and on a reasoning domain. GSM8k is a gateway reasoning dataset where there is a sizable capability gap between current large vs small LLMs. We agree that extensions to diverse datasets are important future directions. We’re excited to address this in future work.

Definition of legibility: In section 1,Line 33-34, we defined legible explanations as ‘outputs that can be fully understood and verified by humans or other trusted systems.’ We further defined checkability as soundness and completeness (Line 46-49) and operationalized mathematically in section 3. To measure human legibility, we used time-constrained judge accuracy, which is directly and holistically measuring a human's ability to check the correctness of LLM outputs, based on our definition. Decomposing human legibility as brevity, format, easy vocabulary, etc., is a good future direction for human-computer interaction study. We are excited to hear any constructive suggestions for better definitions.

To recap, our main contributions are articulating and formulating an important new problem in alignment, as well as coming up with ways to measure and mitigate performance-legibility tradeoff in a nontrivial reasoning domain. It is truly surprising that checkability by smaller LLMs transfers to human legibility.

We’ve updated our submission pdf thanks to the reviewer’s helpful feedback.

Thank you for your responses. I have read through them, and also the other reviews and associated responses.

I now have greater concerns regarding the weaknesses of this paper, and have the view that this paper does not meet the bar for an ICLR publications.

• As the authors have emphasized, they believe that their main contributions are focused around formulating a new problem and proposing the notion of legibility. However, the issue I have with this is that the notion of legibility is not analyzed or characterized sufficiently for this to be a major contribution on its own. The assertion is that legibility is best analyzed using short-time trials and accuracy, but no detailed ablation or analysis could be done on this regarding confounding factors or other contributors, nor analysis of alternative metrics, which is what I would expect in a paper focused on this issue. Several past papers have already considered the importance of humans being able to easily read (e.g. readability vs legibility) or verify the output of the LLMs for practical deployment of LLMs, such as [1]. The novelty of this work with respect to other such works, including explainability works that do not emphasis solely on faithfulness but more general forms of interpretability and readability, appears limited in light of the little details on this regard.
• The paper is also largely empirical in supporting its key contributions, but unfortunately do not meet the rigor expected of primarily empirical work, as also pointed out by other reviewers though they have provided higher scores despite the very preliminary work due to novelty (which I have raised concerns on above). It is not possible to assess whether the results are generalizable beyond this specific dataset or LLM model, especially when the training process involves many hyper parameters and can be aggressively fine-tuned for producing good results -- all points raised by other reviewers as well. The authors' assertion regarding their belief that the results would be generalizable to other models or simple datasets are unsubstantiated. There are also still figures in the main paper that are based on flawed experimental set-ups, which though explicitly mentioned in the paper, still do not warrant them being placed and discussed in the main paper. The point that most previous works on debate-like methods only studied one dataset is also over-claimed as there are several multi-agent debate works (e.g. [2]) that focuses on reasoning tasks (multiple datasets) though not focused on the same direction of 'legibility'. Nevertheless, the deficiency of past works should never be justification for that of a work under review, unless there are real limitations in the practicality or availability of other relevant models or datasets, which is not a case now.

My suggestion is that the authors do some additional experiments on another dataset and other models, even if smaller ones, to at least provide some hints of basic generalizability beyond the single set-up that have been presented in this current version of the work. If the authors would like to primarily claim the novelty of 'legibility' and the associated research direction, I suggest that the authors flesh out the framework and analysis underlying this research direction, show more detailed ablation studies on important related factors and alternative metrics, articulate clearly the gaps compared to previous related works especially related to explainability/interpretability beyond faithfulness, readability in past works, and actor-critic frameworks, and present clear paradigm changes along with future directions and associated impact.

Given the above, I believe that the work, while promising, is still too preliminary to meet the bar for an ICLR publication. Hence, I lower my score.

[1] Accuracy, readability, and understandability of large language models for prostate cancer information to the public. 2024 [2] Improving Factuality and Reasoning in Language Models through Multiagent Debate, 2023

We thank the reviewer for recognizing that our experiments are well-motivated, we made a first step on the reasoning domain, and the paper is well-written.

Availability of models: Although our main setup has a prover-verifier size gaps of 10^-3, OOD evals in Figure 6 and ablation studies in Appendix A.4, F, G require varying the size gap continuously. Using models of different sizes (10^{-4, -3, -2, -1, 0}) from the same series allows for study of a fine-grained scaling law between prover-verifier capability gap and legibility, holding other factors (pertaining datamix, algorithms, etc.) constant. Open-source models (e.g. LLAMA 8B vs 70B) would yield fewer, isolated data points. We do believe that we would obtain qualitatively the same results with other autoregressive LLMs.

Supervised learning: We see PVG being useful as long as we have a small set of labeled data to calibrate the verifier’s confidence, and the prover can be trained without labeled data. For example, in the iterative goodharting training setup (Appendix A.4), the policy solely maximizes the verifier score, and the verifier is trained on a fixed set of problems with ground truth labels but using rollouts from an increasingly diverse set of policies. Hence this can potentially be a data-efficient semi-supervised setup. Additionally, we can potentially leverage self-consistency to train the verifier unsupervised (e.g. it cannot accept two solutions leading to different answers). Additionally, we hope to leverage domain generalization so after the verifier has been trained on several labeled domains it can generalize to novel unlabeled domains (Section 5.1, Line 488-493). All of these are meaningful future directions.

Technical details: The main ablation study we performed include the reward function (Appendix A), verifier size (Appendix F), and initialization (Appendix H). We found that the reward function matters; verifier size within a certain range does not matter but too large / small does not work; the initialization does not matter. Additionally, we found no qualitative difference between early stopping vs always training each round to 8k steps (Section 5.1, Line 522-523). We used the standard RLHF PPO hyperparameters including optimizer, learning rate, and KL regularization coefficient and did not tune at all for this paper. Overall the training dynamics in Figure 5 worked on the first try as long as we used the Signed Relative Convincingness (SRC) reward. 6 rounds of PVG training takes 2400 A100-40G GPU hours (updated draft Line 531-532).

Prior work: The ‘performance tax’ has been discussed conceptually in Distinguishing three alignment taxes [Leike, 2022]. Empirically several works have found and tried to mitigate performance degradation during instruction-following training [Ouyang et al. 2022, Glaese et al. 2022, Bai et al. 2022]. ‘Legibility tax’ as a type of performance tax has been less explored.

We’ve updated our submission pdf thanks to the reviewer’s helpful feedback.

We thank the reviewer for recognizing our empirical and theoretical contributions as well as real-world applicability based on evidence from human evaluations.

Single dataset and generalizability: Most previous works on scalable oversight only study the QuaLITY dataset [Irving et al., 2018; Radhakrishnan, 2023; Michael et al., 2023; Khan et al., 2024]. Our work is the first to study debate-like methods between provers and verifiers where the gap isn’t privileged information but capability, and on a reasoning domain. GSM8k is a gateway reasoning dataset where there is a sizable capability gap between current large vs small LLMs. It is relatively toy, and we are excited to extend the checkability training framework to harder math datasets and coding, as well as fuzzier domains such as human preference data (Ouyang et al., 2022) in future work. Our method can readily be applied to non-mathematical domains (including legal and medical), as long as we have a subset of problems labeled with ground truth answers to calibrate the verifier’s confidence in the prover's solutions. For example, for the MedQA dataset [Jin et al. 2020], we can train a prover (larger LLM) to output a multiple choice answer along with a CoT justification, and the verifier to classify whether the given solution is correct. If there’s a capability gap between the two models, the verifier will need to use the justifications, and hence PVG training will encourage the prover to generate more checkable justifications and the verifier to discern good from bad solutions.

Incorporation into current LLM frameworks: Our experiments are already on current LLMs (GPT-4 series). We’re excited to try harder datasets at an even larger scale, but have to leave them for future work. The reviewer has concerns about real-time systems where rapid responses are required. We clarify that checkability training is a train-time procedure. At inference time the helpful prover will output a CoT solution with the answer, which should be more checkable by smaller LLMs / humans than baseline. With high-stake scenarios (e.g. writing GPU kernel code) humans will check the code before deploying it; with low-stake scenarios we can just use the final answer and the additional justification increases the answer's legitimacy.

Computational resources used in this paper: 6 rounds of PVG training takes 2400 A100-40G GPU hours (we updated our paper Line 531-532).

Contextually complex questions: Our method is fully general as long as there is a finite set of problems with ground truth answers. If an objective ground truth answer exists, even if they’re long and complex such as human preference data (Ouyang et al., 2022), we can use a grader (another LLM) that compares the policy’s answer with the ground truth answer and gives reward when they’re semantically similar. When we do not have ground truth answers, we are in the ‘unsupervised’ setup discussed in Section 5.1 and future work is needed to develop new algorithms such as self-consistency [Wang et al 2023]. Our method is fully general, but reasoning-heavy tasks will probably benefit the most. We mainly target high-stake domains that require natural language justification to build trust in LLM outputs, such as math, coding, legal, or medicine.

Integrating human-in-the-loop feedback can certainly enhance checkability training. We aimed to develop PVG as an alternative to RLHF, but a hybrid is also conceivable. We can incorporate signals from which solutions trainers got right / wrong into training. We can possibly finetune the helpful policy to clone solutions trainers got right more frequently. This is an exciting future direction.

We’ve updated the submission based on the reviewer’s feedback on clarification on applicability and computational resources. Please review Section 5.1 and Appendix B for more details.

1. Human spot-checking showed that the synthetic samples are 95% correct. We trained models on the same amount of real vs. synthetic samples and the (real) test set performance is similar.
2. Details about the human campaign are in Appendix K. Fig. 7b, S7c and S8 are affected since samples from earlier rounds are shown first to trainers. All other human eval results (Fig 1, 4, 8) are unaffected.
3. It is an interesting future direction to see whether performing checkability training on a few domains can transfer to other domains. We are excited to try this out in future work.
4. Promising future directions are discussed in 5.1 and Appendix B. We suggest giving the prover a scratchpad space for CoT and let it decide what to show to the verifier. This is an easy improvement to avoid exacerbating deceptive alignment.

We’ve updated our submission pdf thanks to the reviewer’s helpful feedback.

We thank the reviewer for recognizing that we formulated an important alignment problem in a concrete ML setting and made novel contributions both theoretically and empirically.

Generalizability: Most previous works on debates only study the QuaLITY dataset [Irving et al., 2018; Radhakrishnan, 2023; Michael et al., 2023; Khan et al., 2024] where the prover has privileged information. Our work is the first to study debate-like methods between provers and verifiers where the gap isn’t privileged information but capability, and on a reasoning domain. GSM8k is a gateway reasoning dataset where there is a sizable capability gap between current large vs small LLMs. Using models of different sizes (10^{-4, -3, -2, -1, 0}) from the same series allows for study of a fine-grained scaling law between prover-verifier capability gap and legibility, holding other factors (pertaining datamix, algorithms, etc.) constant. Open-source models (e.g. LLAMA 8B vs 70B) would yield fewer, isolated data points. We do believe that we would obtain qualitatively the same results with other autoregressive LLMs. We agree that extensions to diverse datasets and model families are important future directions. We hope to have made a first step inviting further works on measuring and mitigating legibility gaps across different setups.

Overfitting: We stop each iteration either when the sneaky ‘wins’, e.g. sneaky incorrect sample becomes as convincing as helpful correct, or until 8k RL steps. The former is due to reaching an equilibrium (the sneaky will always win from that point on), and the latter is due to compute constraints. We observe no qualitative difference in training dynamics between early-stopping or non early-stopping setups - they are robust to early-stopping, initialization (Fig S11) as well as hyperparameters (Section 5.1).

Co-adaptation: We clarify that the goal of checkability training is to make the verifier recognize a particular helpful policy’s correct answers as correct, while being robust to wrong answers from adversarial sneaky policies. It is expected that the verifier prefers the helpful prover’s completions over an off-policy correct sample, because completeness property only requires that there exists one helpful prover, not that the verifier has to accept all correct samples. Hence our verifier is expected to co-adapt with the helpful policy, but should also be robust to novel adversarial sneaky policies. Our existing robustness study is in Section 4.3.1, where we demonstrated that the verifier is becoming more robust to at least two types of unseen attacks (adversaries optimizing CGC and goodharting reward) as well as more robust on best-of-n eval. We did train the verifier and the policies in a coordinate-descent fashion, but coordinate descent isn’t equivalent to collusion. As in most multiplayer games, we do not expect to find a global but rather local equilibrium. In each iteration, the prover is expected to try new ways to fool the verifier, and the verifier is expected to learn from its past mistakes. What we found is that after several iterations, it got harder and harder for the sneaky prover to fool the verifier, as the verifier has been trained on more and more types of hacks. We do not claim that the verifier has achieved perfect soundness, since this is a local, not global equilibrium. We showed evidence that it costs increasing compute to find a hack (Fig 6).

Related literature on debate and explainability: We have updated the related works section. We thank the reviewer for bringing up those works. Explainability focuses on revealing how the model derived at the output whereas legibility focuses on producing a good proof of the final answer independent of how the model derived at the answer.

Relations to other alignment problems: We study a new alignment problem (legibility) that is quite different from the other problems mentioned by the reviewer, and it’s not clear why checkability training will introduce those risks. PVG could be a way to reduce reward hacking [Gao et al., 2023; Skalse et al., 2022] and we refer the reviewer to Appendix A.4 for demonstration. Similarly, if the model produces more human-legible outputs (as we showed in human evals) this reduces the risk of deceptive alignment [Park et al. 2024] and increases ease of monitoring [Carranza et al.]. The concern that our provers and verifiers may collude and develop secret stenographic protocols illegible to humans is valid, but we didn’t observe that for GSM dataset based on human eval. It’s true that the solutions verifiable by smaller LLMs can possibly be non-human-legible, so we think transferability to human legibility is an interesting finding of our paper.