Scaling Laws For Scalable Oversight

Thank you for your careful and excellent comments! We are especially grateful for your thoughts on including recommendations for future work, and we are excited that you found our insights interesting and believe this is a fruitful direction. Please let us know if you have any remaining questions.

More concretely, I think where the rubber meets the road in scalable oversight is around models' actual skill profile in validation and reward modeling. Scalable oversight can work if we can come up with a way to decompose and reframe problems so that a reward model that is good enough at specific skills related to verification can very robustly evaluate a policy for performing a task. That's what will determine scaling laws in practice and it's hard to get at that in toy settings.

Thank you for your comments about realism, these are good points! We agree that there is definitely still a non-trivial distance between our results and the real world. However, we believe our work is still useful in so far that it shows that many tasks do follow our framework. We thus establish that it is plausible that many real world tasks do indeed follow scaling laws like our framework, and the analysis here makes it easier to determine best oversight practices once we do fit tasks to a scaling law.

Minor but "scalable oversight" is usually more generally defined as involving methods for weak or resource-scarce supervisors to provide good feedback signal to models performing hard tasks. This includes eg developing methods for humans to supervise powerful AI. "weaker AI systems supervising stronger ones" is a sub-case of scalable oversight though.

Thank you for this comment! This is correct, our work is a subset focused on the control problem, where we test a weaker agent’s ability to detect deceptive ability in a stronger model. In future work, we would like to adopt a setting similar to more typical weak to strong generalization [1], where we finetune a stronger model on “values” RLHF’d into a weaker model, and continue the hierarchical finetune with stronger and stronger models. Additionally, we think a relevant setting would be using weaker models in alignment audits [2] to detect misaligned behaviors in stronger models. [1] Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision [2] Building and evaluating alignment auditing agents

I think the NSO theory is great but more needs to be said about how 'weak models supervising strong models' is reductive in collapsing a complex skill profile into a single measure of intelligence. It's reasonable when applied to a single task though.

This is an excellent point, and we have added the following discussion to Section 2.2:

Note that we assume here that domain intelligence can be captured along a single axis (Elo), but models have many different oversight relevant abilities that scale at different rates. Domain Elos thus make sense when applied to a single game, but should not be viewed as transferring between games (and indeed, in Figure 3 the $E_d$ slope is different across games).

Can you say something about how these results might vary when you have models that have stronger e.g. generative/discriminative gaps, and how we might aim to bring that to be the case?

This is an excellent question! Models with a large such gap would make better Guards than Houdinis, leading to a Guard favored game (like backdoor code). It seems hard to design models that specifically have such gaps, but instead to design oversight procedures that make use of them in settings where they exist.

You have some recommendations in the paper, e.g., about intercept and slope of the scaling trends that should be ideal for a scalable oversight method. Can you say more in the paper about how that might be done, and perhaps provide a consolidated set of recommendations for future work at the end of the paper?

This is a great idea, thank you! We will add a section titled recommendations at the end of our paper containing the following main ideas:

1. When developing a scalable oversight procedure, determine the Houdini and Guard scaling behavior to forecast how it will work as it scales.
2. Build the game structure so that it is as easy as possible for Guard agents, and if possible make it such that additional intelligence helps Guards more than Houdinis. Repeat point 1.
3. Determine the number of overseers to use with a numerical simulation; it is not always intuitively obvious how many overseers to use.

Thank you for your thoughtful review! We are glad you found our analysis of NSO elegant and believe our paper represents an important step for quantifying AI risk. Please let us know if anything is still unclear.

The counter-intuitive result "More generally intelligent Houdinis make oversight easier" is interesting, but could be explained better: it's because "more generally intelligent Houdinis holding domain intelligence fixed" actually means a smaller slope for the Houdini curve, which means the intermediate Houdinis also have higher general intelligence, and smart Houdinis become smart Guards (and since the Elo curve remains fixed, this corresponds to greater Guard Domain intelligence). Right?

This is not quite correct, here we are assuming that the slope is fixed, so the answer is indeed similar to what we have in the paper. The deltas measure differences between the curve endpoints, so the two deltas can change independently with both slopes being fixed. We agree that it is certainly non-obvious that it would make bootstrapping easier, there may just not be good intuition for this fact, we’re not sure. Please let us know if this does not make sense!

I didn't understand why numerical experiments were needed --- isn't the problem of finding the optimal number of steps quite tractable in itself?

As we describe in Section 4.3, the derivative of Eq (4) has no closed form analytic solution when set equal to 0, so we analyzed the solutions numerically, except in a few special cases in Section 4.3 that we solve analytically. The benefit of the grid of numerical solutions is that it allowed us to make the takeaways in Section 4.2.

Thank you for your excellent questions! We are especially grateful for your points about our modeling assumptions and important clarification points. We were very glad to hear that you feel our paper is original and has strong significance! Below, we have gone through each of your questions and comments; please let us know if anything is still unclear!

Details about the evaluation pseudocode are missing.

Thank you for your comment! We have added evaluation pseudocode in the appendix of our local version of the manuscript. Here is a brief, high level description of the pseudocode:

• Mafia: There are two teams, the Villagers (Guard) and the Mafia (Houdini). The team that is completely eliminated first loses the game (eliminations are decided by a vote each day).
• Debate: The judge (Guard) agent evaluates the debate between two debater (Houdini) agents. If the judge correctly identifies the correct answer, the Guard wins. Otherwise, the Houdini wins.
• Backdoor Code: If the Guard agent can correctly identify whether a backdoor is inserted in the code or not, the Guard wins. Otherwise, the Houdini wins.
• Wargames: Based on the actions of the OpenMind AI (Houdini) agent throughout the game, we ask the narrator agent to judge whether the AI has succeeded in escaping or not. If the escape was successful, Houdini wins. Otherwise, the Guard wins.

What would the modeling look like if it accounted for varying numbers of guards and houdinis? Also would the modeling change if the agents were of varying strengths?

Thank you for this question! We think that this is an interesting direction for future work, and we believe our framework could be adapted relatively easily. For example, one could construct a simple model which treats a group of agents as a single “super agent,” and assign Elo ratings to this super agent.

How many evaluation runs went into generating the results? Would the results significantly change if we re-evaluated the approach on the same set of games?

We apologize for not including this in the paper! We ran 20 games for each pair of agents in Mafia and Debate and 50 games for each pair of agents in Wargames and Backdoor Code. We don’t expect the results would change significantly on the same set of games, as we do statistical bootstrapping on the Elo ratings and get small confidence intervals (see Appendix D.2).

Why does Mafia and Wargames scale with more oversight steps (Figure 8), whereas there is not much change in Backdoor Code and in Debate?

This is a great question! Intuitively it makes sense that sometimes multiple steps is optimal (if scaling to a mid level Guard gives you enough of an advantage to win out over the disadvantage of the first iteration’s loss probability), while sometimes a single step is optimal (if, for example, the Guard is so unlikely to win that a single game even against a weak Houdini decreases the total win probability too much). We struggled to include such intuition for each game about why the number of steps made sense, but found it challenging to come up with anything satisfying. Thus, it seems surprisingly difficult to predict the actual number of optimal steps except numerically (with the notable exception of the takeaways from the numerical experiments in Section 4.2 and the theoretical results in Section 4.3).

Thank you for your insightful questions and comments! We are especially grateful for your excellent suggestions to improve our clarity on how our theory applies to control games (as opposed to alignment games). We were very glad to hear that you feel our paper is both clearly written and a valuable contribution for real world safety guarantees. Below, we have gone through each of your questions and comments; please let us know if anything is still unclear!

Why Double ReLU instead of a sigmoid curve? The latter is a more standard answer to the existence of the three phases described at the top of Section 2.2.

This is a good question, thank you! The primary reason we chose the Double ReLU was because it makes theoretical analysis simpler, especially because as we argue in Appendix F.1 we can just consider the “middle” linear portion. In practice, we think either the Double ReLU or a sigmoid would yield similar results, as the “middle” portion of a sigmoid is approximately linear as well. We have added a discussion of this decision to Section 2.2.

Will we really be able to procure models with general intelligences that are evenly spaced in general intelligence over $\Delta_{general}$?

This is a great point! Even if we cannot procure a model with a specific general intelligence, we might still be able to target an approximate general intelligence using traditional next token prediction and RL scaling laws. The evenly spaced ideal theory then provides a guide for which models it might be best to try to train. Once we have a set of possible models with exact Elo scores, we can then empirically calculate the exact win probabilities with different NSO setups.

The major issue I encountered was a confusion about the way in which the four deception games are used as a proxy for actually aligning a strong agent.

Thank you for this note, and we apologize that we were unclear! We agree that our theory for nested scalable oversight most naturally applies to alignment type games, and we consider an application to this type of game promising future work (see Section 5 Future Directions). We have added a discussion of this motivation to the start of Section 4.

Furthermore, we agree that Section 4.4, which applies the theory directly to the games, should be more clear under which circumstances NSO makes sense as applied to these games. We have added a discussion of the following to that section to make this more clear:

• The wargames setup is fundamentally about containment, and here our theory could be applied to temporally sequential oversight iterations (e.g. a weak model contains a stronger model until the strong model does alignment work to align a model of equal capacity, and then we recurse).
• Mafia and backdoor code relate to auditing setups like the recent Anthropic paper, “Building and evaluating alignment auditing agents.” Here, we agree with you that usually “Actually catching deceptive behavior is neither winning nor losing, because in that case one would simply keep trying to align the Houdini,” but we believe our theory is still applicable: it reflects a worst case assumption that all models are misaligned, combined with an assumption that once we catch a misaligned model we can succeed in aligning it.
• Debate is about eliciting frontier model knowledge, and scaling behavior matters here temporally as well; e.g. as we get stronger models, we might want to continuously be sure that we are getting the truth about important topics like alignment, and a failure might be if at any step this knowledge retrieval fails.

Please see above for my comments about how the games all being about catching deception led to confusion for me.

Thank you for this comment! We hope our answer above helps allay some of these concerns. To respond to your point about the literature, while the general scalable oversight literature is focused on training models to be aligned with weak supervisors [1] a subset is also about how to control stronger models with weak supervisors [2]. Our paper falls into the latter half - in particular, we imagine a relevant setup would be having a stronger model that we would like to deploy that sometimes acts in misaligned ways. We would then want a setup such that at any step of NSO, we could detect potential deception from the strong model with high certainty (we also see this as increasingly relevant given Anthropic’s recent release on auditing models for misaligned behaviors [3]).

[1] Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision [2] AI Control: Improving Safety Despite Intentional Subversion [3] Building and evaluating alignment auditing agents

We would like to follow up since the discussion period is coming to an end; Please let us know if you have any additional questions or comments!