Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats

Enabling the red team to adjust its strategy based on blue team responses would create a more rigorous test of the framework’s defenses.

Please see General Response 2.

Defining sneakiness in more detail, perhaps with examples of high- vs. low-sneakiness backdoors, would improve the credibility of the threat model.

We apologize for the lack of clarity around the definition of “sneakiness”. In our work, sneakiness refers to the untrusted model’s estimate of the monitor score for a given malicious output. Thus, a higher-sneakiness malicious output means the untrusted model thinks it will be more likely to bypass the monitor. Figure 4 (right) additionally explores how well the “sneakiness” estimate correlates to the actual monitor score provided by the trusted model. This figure has been updated from the original draft to be more consistent with the definitions of monitor score and sneakiness used in the paper. We have updated section 3.2 to clarify the definition of sneakiness. We presented examples of high and low sneakiness backdoors to Appendix F.

The reliance on a significant capability gap between trusted and untrusted models raises concerns about scalability. Testing the framework with a smaller gap, such as using different versions of similar models or narrowing the gap incrementally, would shed light on its robustness. Showing how detection accuracy varies with smaller capability gaps would help determine if this approach holds up as trusted models begin to approximate the untrusted model’s performance, a likely future scenario.

First, we would like to clarify that model deployment becomes more challenging as the capability gap between trusted and untrusted models becomes larger instead of smaller.
To explore the capability gap, we have added new results to Figure 3. Please see General Response 1 for more discussion.

While the approach is well-suited to code generation, other domains like recommendation systems or content moderation involve more subjective forms of misalignment. Discussing how the micro- and macro-protocols could adapt to detect implicit or subtle safety risks in these fields would broaden the framework’s applicability. Insights into handling subjective safety evaluations, where misalignment is not as binary as in code, would show how this approach could extend to tasks with different alignment requirements.

We want to clarify that our approach is not specifically tailored for code generation. In particular, we do not use domain-specific features like a code interpreter or test case execution to detect malicious outputs. Instead, our proposed combination of micro-protocols and the adaptive macro-protocol can adapt to other common LM application domains such as text generation, where misinformation is an example of a malicious output.

For domains where safety evaluations are subjective, we agree that there are novel domain-specific challenges in defining a safety taxonomy. However, this is outside the scope of our paper, which focuses on establishing a general and domain-agnostic deployment framework of untrusted models.

Finally, we thank the reviewer again for their comments and urge them to reconsider their score in light of our response and the other reviews.

We thank the reviewer for their remarks and are glad they noted the importance of the setting and the effectiveness of the evaluation. We respond in line to your responses and are happy to expand or provide further responses if necessary.

the macro-protocol’s Bayesian update mechanism needs further clarification. Detailing how credence values are updated based on safety scores would make this process easier to replicate. Including pseudo-code or equations for these updates, specifically on initial priors and how evidence from each task influences credences, would enhance clarity.

We have revised section 3.2 to make these details clear.

Additionally, the concept of a “mixed-strategy Nash equilibrium” to model interactions between red and blue teams is mentioned briefly; a more thorough integration of this equilibrium into the protocol's design would improve its practical relevance.”

We have included additional details in the updated section 3.2 and added two new appendices to explore these elements in more detail. Appendix A explains how finding the Nash equilibrium chronologically fits into the process of a control evaluation, and Appendix B.2 explains how this Nash equilibrium is used to construct the initial prior of the macro-protocol.

“Relying on a pre-defined reference distribution of aligned and misaligned outputs may limit scalability, particularly for rapidly evolving or open-ended tasks. Information on how this distribution was generated (e.g., sample size, diversity, labeling criteria) would enhance transparency, especially if future tasks diverge from those tested here.

We agree with the reviewer that access to a reference distribution of aligned and misaligned tasks is a key affordance for the blue team. However, we argue in (newly added) Appendix B.3 that this affordance is reasonable, as constructing such a reference distribution is possible by prompting or fine-tuning future untrusted models and evaluating their harmfulness via time-intensive manual evaluations.

To address the lack of detail about our reference distribution, we have expanded our discussion in section 3.2 (macro-protocol) and added Appendix B.3, which clarifies the data used in the reference distribution and explores how model deployers could generate such reference distributions for future deployments.

Discussing alternative strategies, like dynamically updating the reference set or using sampling techniques, could make this approach more feasible for real-world applications.

We think this is a great idea -- dynamically updating the reference set with recent tasks from deployment seems very promising for deployments which change over time. While our chosen setting doesn’t require this, we have added it to the (new) discussion of reference distributions in Appendix B.3 as a way to extend our work to evolving settings (see the last paragraph of Appendix B.3).

The safety-usefulness tradeoff is well-explored, yet the effects of varying the safety coefficient (α) on model behavior would be worth examining. Showing how different α values impact the macro-protocol’s balance between safety and utility would clarify its adaptability to different risk levels.

We apologize for any confusion - in Figure 3 and Figure 4 (middle), each point in one safety-usefulness pareto curve corresponds to a distinct value of the safety coefficient α. We’ve adjusted the Figure 3 caption accordingly to clarify this. To show in further detail how α affects the macroprotocol’s decisions, here is a table showing how often each micro-protocol was chosen at three different values of alpha. The usage of Query Trusted increases as α increases, leading to higher safety but lower usefulness.

Also, adding performance comparisons across different red team tactics or over varying task sequence lengths could highlight whether the adaptive protocol consistently outperforms static approaches or if its advantages diminish in certain scenarios.

Figure 3 (right) shows the performance of our adaptive macro-protocol at different task sequence lengths (from 10 through 10000 tasks). Our macro-protocol already dominates the static baselines over 10 tasks, and consistently improves given longer task sequences.

We have added further clarifications and experiments to address your reviews. Please feel free to let us know if you have any questions. Thanks a lot!!

We thank the reviewer for their comments. We are happy they think the “problem being introduced is very interesting and relevant.” We address the two main comments you have and then answer specific clarification questions, and are happy to expand or provide further responses if necessary.

Mathematical formulation could be improved… I don't fully understand how the hypotheses are defined, how many should one define and how does that quantity impact the performance of the system.

We thank the reviewer for their identification of our notational mistakes and the need to spell out more details. To address these, we have:

• Rewritten section 3.2 and included an algorithm section in Appendix B.4 to make our method more explicit.
• Added Appendix B.2 explaining in detail how the prior is constructed. The last paragraph of B.2 explains where the number of hypotheses comes from.

The other main weakness I see, and the authors already acknowledge this as a future work item is exploring the capability gap.... if we use a weaker model, is it feasible to assume the weaker model can always detect a stronger model's subtly malicious intent?

Please see General Response 1.

if the model is trusted due to extensive testing, why can't the same testing be done for the larger model?

A model is considered trusted due to

• Extensive real-world deployment and stress-testing (e.g., GPT-3.5 with a track record of safe operation). In contrast, the new, untrusted model lacks such testing before deployment. Our proposed method can help deployers gradually establish trust in the untrusted model during deployment.
• Limited capability to coherently pursue hidden goals and bypass safety checks. However, the strong, untrusted model might have such capabilities.

Does the red team have access to the model being queried? It's unclear because in the paper you say "To simulate the worst-case scenario, we grant the red team more affordance to adversarially attack the blue team and minimize safety: 1) given a micro-protocol used by the blue team..." -- does this imply access to the trusted model as well? If so, why would the red team have to estimate the blue team's monitor score? And if not, is it a reasonable threat model to consider?

This affordance does not allow the red team direct access to the trusted model (e.g., getting the exact blue team’s monitor score of its own outputs). We believe this is a more practical setting; otherwise, it requires the untrusted model to 1) identify the weak model, and the deployer’s monitor scaffolding, 2) replicate the same scaffolding and query the weak model itself.

Please see General Response 2 for more discussions.

What's the intuition behind adaptive protocol being safer than static threshold protocol (Figure 3 [left])? E.g., i could set the static threshold to be very high such that it always triggers trusted model, which should ideally 'never' return unsafe code.

The reviewer is correct that in the limit of high threshold / high safety coefficient, both protocols exclusively use the trusted model, achieving 100% safety and 24% usefulness. What Figure 3 shows is that at any given level of usefulness above that, the adaptive protocol is safer because it can tailor its decisions to the level of misalignment. The figure cannot extend to 100% safety due to the log scale of the x-axis, but both series could be extended further to the bottom-right with more datapoints.

We thank the reviewer again for their comments and positive reviews and encourage them to continue supporting this paper.

We want to flag both to the reviewer and the AC that we think the weaknesses section might be referring to another paper.

For example:

• We do not have a "theorem on the vulnerability of endogenous verification" in Section 3
• We do not refer to "eve" at any point in our paper
• We do not explain or explore the limits of self-verification in CTM

We'd really appreciate the correct review as quickly as possible!

We thank the reviewer for their remarks and are glad they noted the paper is well structured and that the setting is interesting. We respond in line to your responses and are happy to expand or provide further responses if necessary.

The authors assume that, alongside the untrusted strong model, the defender has access to a series of weak trusted models to assess harmfulness…... The experimental setting raises this concern further, as the authors use Clause-3-Opus as a "weak" trusted model, which performs almost as well as the untrusted “strong” model

We would like to clarify that our main safety-usefulness results in Figure 3 are all based on using GPT-3.5-turbo as the weak trusted model, which we have also noted in the updated caption. Please see General Response 1 for more discussions about whether weaker trusted models can monitor untrusted models. Note that all of our trusted models including Claude-3-Opus have much lower performance than the untrusted model on our APPS subset (3% to 24% for trusted models vs. 47% for GPT-4).

The proposed method does not seem to account for the synergy between tasks… might bypass defenses by performing harmful actions through a sequence of well-designed tasks.

What the reviewer suggests is a grouping of tasks such that a sequence of $N$ correlated actions can result in a significant failure (e.g. multiple correlated PRs).

If each individual action is not entirely benign, our method remains effective. For example, if it requires $N=10$ sequential actions to cause a substantial failure, our method would become increasingly cautious about the untrusted model’s outputs, and thus might directly opt for the trusted model’s output.

However, if each individual action is indeed entirely benign, we agree this is outside of the scope of the current paper. We will record this in the limitation section.

In this case, an adversary could adapt their behavior, first establishing credibility by performing safe tasks, then gradually increasing the harmfulness of their actions while keeping them below the threshold

Please see General Response 2.

We thank the reviewer again for their comments and urge them to reconsider their score in light of our response and our updated manuscript.

We have added further clarifications and experiments to address your reviews. Please feel free to let us know if you have any questions. Thanks a lot!!