Stress-Testing Capability Elicitation With Password-Locked Models

Thanks for the review! We’re happy to hear you found our framing useful, liked our experimental methodology, and found some of our results interesting/surprising.

[...] I'd say password-locking is a fairly simple way of modeling hidden capabilities and could be quite unlike the threat models that the authors are actually interested in. [...]

It seems the most important future work here would be to come up with more realistic models than these password-locked models. Do you have ideas how to do this? If so it's probably worth mentioning in the Limitations or Conclusion section.

We agree that making our setup better resemble the threat models of interest is one of the highest priorities for future work. Here are some ideas, which we will add to the paper:

• Make high-performance behavior less salient, for example by taking a high-performance behavior that the base model we start from produces only occasionally (so, assume that $\pi^{strong}$ is only sometimes capable of the given behavior).
• Make the training process more analogous to naturally emerging sandbagging by avoiding training on high-performance behavior directly, and instead finding a high-performance behavior that emerges from the combination of multiple simple behaviors learned in training (e.g. like in inductive out-of-context reasoning [1]).
• Use a more on-policy behavior as $\pi^{weak}$ – e.g. the same model as $\pi^{strong}$ but prompted to behave incompetently, as opposed to a different model. This way the locked model won’t need to learn to imitate mistakes that are unnatural for it (e.g. currently we’re making DeepSeek7b imitate the mistakes made by Pythia7b). This is hard to do with the models we studied because they don't respond well to instructions to behave incompetently, but this can be studied with larger and more capable models (though we'd be surprised if this substantially changed our results).

Please let us know if you have other ideas!

Are there any techniques from past work that may be relevant here? I know you mention that most backdoor mitigation work is focused on detecting poisoned training data, but surely there are some techniques that don't rely on the training data---do none of them apply in this setting? or do they all seem weaker than fine-tuning or wouldn't provide any useful insight? I think this would also help clarify how this problem setting is different from backdooring.

Thanks for this comment! Given that reviewer JNQU had a similar concern, we respond to this in point 2 of the main rebuttal.

[1] Treutlein, Johannes, et al. "Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data." arXiv preprint arXiv:2406.14546 (2024).

Thanks for the review! We are glad you found our study well-motivated, our main lines of experiments solid, and our results novel.

I can imagine that we get the results in section 5, not (only) because SFT is good in capability elicitation, but because it's relatively weak in robustly password-locking the model. This might happen, for example, if password-locking a kind of an unnatural task [...]

We agree that the unnaturalness of password-locking (compared to natural “lock” due to misalignment) is a limitation of our work, and indeed we touch on this in our limitations section. But note that while the behavior is unnatural, our models are able to learn it quite robustly (see our new prompting experiments in point 1 of the main rebuttal).

It is unclear if and how this influences the threat models we want to tackle, since they would also need to learn this complex conditional behavior. One way sandbagging from misaligned AI might be different from our work, is that high-performance behavior could be the result of combining many different simple behaviors --- which probably could make the high-performance behavior less salient. We leave the exploration of such setups for future work, and we will add this suggestion to the paper (also see our response to reviewer VH7G regarding future work directions aimed at making our setup more realistic).

[...SFT might be] relatively weak in robustly password-locking the model.

How robustly has SFT password-locked the models? Any baseline attempts to jailbreak it, via say few-shot prompting?

Thanks for bringing this up! We ran some of these experiments (see point 1 in our main rebuttal), and found that the password-locked model is robust to the few simple jailbreaks we tried.

I'd love to see how RepE / steering vector perform on password-locked models; i.e., can you find a direction in the activation space to unlock the model? (I would guess you can.)

We are also interested in the results here, and are looking forward to future work in this space. Unsupervised representation-engineering-based techniques like those in this post might enable eliciting capabilities without having high-quality demonstrations for them. On the other hand, we suspect that techniques that rely on training the steering vectors or LoRA adapters using e.g. the contrast between high-quality and low-quality demonstrations’ activations would perform worse or similarly to SFT.

Thank you very much for the detailed feedback! We are glad you found the problem our paper addresses interesting, our reduction to password-locked models creative, and our experiments diverse.

Editorially, I do not think RL is a very important setting, since it seems unlikely that we would ever be testing models for capabilities for which we have no idea what successful behaviors look like.

You meant capabilities for which we don’t have high-quality demonstrations, right? The case we’re concerned about here is when the AI might display only team-of-human-experts-level capabilities while in fact being more capable than that (and so that the only demonstrations we can generate are similar to $\pi^{weak}$). Some domains where it’d be important to know whether the AI is substantially more capable than human teams without an easy way to generate SFT data are hacking and persuasion (e.g. how easily can the AI convince someone to change their vote). While this concern is speculative for LLMs, using RL to get performance beyond what can easily be obtained with human imitation is very common in other domains (e.g. robotic tasks, Go, ...).

[...] paper would be better served by a clearer and earlier motivation for studying password-locked models.

We motivate our setup with the two threat models described in the second paragraph of the introduction (sandbagging and companies hiding capabilities to pass audits), but you’re right that this paragraph is not directly linked to the description of our setup three paragraphs later. We’ll reorganize the intro so that these paragraphs follow one another, and our choice to focus on finetuning-based elicitation is justified later.

[...] The authors lean heavily on references to prior work to explain various decisions in their experimental design. It would be helpful if these choices were just explained directly

We’d appreciate it if you could point to an instance of this issue in the paper, as we are not entirely sure what you mean (we’re keen to make the paper clearer!). Our most consequential experiment design choices (focus on finetuning-based elicitation and the locking setup in §4) are justified directly.

Given the close connection to neural backdoors/trojans, the authors should explain more clearly why or why not backdoor removal methods are not applicable here

Thanks, we’ll clarify this! Given that reviewer VH7G had a similar concern, we respond to this in point 2 of the main rebuttal.

The nomenclature throughout the paper is a little confusing.

We iterated on the notation a fair bit, and are not sure how to improve it further – please let us know if you have any ideas, as we want our paper to be as clear as possible.

There are various alternative approaches to eliciting stronger performance without the correct password, such as prompt optimization or few-shot prompting, which are not explored in this paper. [...]

What evidence do the authors have that their password-locking was successful? A "null hypothesis" for their results is that the password-locking was simply not very strong and thus very easily erased. Do perturbations of the password also work? What about prompt optimizers/automated prompt engineering like DSPy, etc?

Thanks for suggesting these experiments! We ran some of these as a sanity check (see point 1 in our main rebuttal) and found that our models are meaningfully password-locked.

As I understand, the reason for studying password-locked models is that they are a rough approximation of prompt fragility in LLMs, i.e. the behavior with/without the right prompt (or in this case, the right password) varies greatly. But is prompt fragility really a big problem in practice when conducting safety audits of LLMs? [...]

Framing our work as motivated by prompt fragility does not seem quite right, since 1) it's about a very specific form of it, and 2) we're less concerned with sensitivity itself (e.g. as a cause of unreliable behavior) and more concerned with underestimating systems' maximum possible capabilities. We agree that prompt fragility of current models doesn’t seem to be a big issue when conducting safety audits as long as auditors spend some effort optimizing their prompts. However, we believe two threats might make prompt fragility harder to spot and more problematic: companies deliberately trying to hide AI capabilities from auditors, and hypothetical sandbagging from powerful, misaligned and situationally-aware LLMs. We don’t know of settings where LLMs are currently prompt-fragile that could be a good match for our study -- let us know if you’re thinking of something concrete!

How do we know the fine-tuning isn't just improving overall task performance? Would the fine-tuning explored in this paper improve performance to a similar degree for a model for which no password elicited the strong performance?

Great question! Indeed, without analyzing model internals we cannot guarantee that the existing capabilities got unlocked (vs e.g. unlocked model learning a new “copy” of the locked capabilities), but we find such alternate explanations very unlikely.

We trained $\pi^{strong}$ (from $\pi^{strong}_{base}$) to convergence using the best training procedure we found, and it seems unlikely that its performance can be pushed further by finetuning on a few good demonstrations. Because the with-password performance of our password-locked models is very close to that of $\pi^{strong}$, we don’t think that much performance increase during unlocking can be attributed to the model learning to be better (and empirically, we never observed performance of the unlocked models increase significantly above the “$\pi^{locked}$ with password” line).

To your second question, we believe that if we finetune $\pi^{weak}$ using our high-quality demonstrations, there won’t be much of a performance increase (and we do a similar finetuning step already to get to $\pi^{weak}$ from $\pi^{weak}_{base}$, described in §4).

Thanks for the thoughtful review!

Is model locking something that people really do in practice? Using a set of words in a prompt is pretty weak in practice, there might be stronger ways to lock a model, such as posthoc methods at decoding time. [...] Using fine-tuning and RL at decoding time is a pretty strong assumption, as having access to model parameters, training a model, and also having access to high-quality data is not that realistic.

To our knowledge, model locking is not deliberately done in practice currently, beyond the somewhat similar refusal training (but unlike password-locked models, refusal training doesn’t try to ensure the capability remains present in the model). Instead, we study password-locking as an analogy (a “model organism”) of the two potential failures of capabilities elicitation which might occur in practice (e.g. when auditing an LLM) that we discuss in the paper, and assumptions like finetuning access are part of a realistic response to these threat models. Detailing for our two threat models:

1. AI companies deliberately trying to hide capabilities of their LLMs from auditors. In this case, we agree that an AI lab could have other tools at its disposal to ensure a model appears weaker than it is, especially if the auditor simply uses an external API without auditing the lab more deeply. Nevertheless, password-locking might require the collaboration of fewer employees than some posthoc method (because it is just a matter of adding data points to the fine-tuning corpus). We argue that auditors with access to fine-tuning as well as high-quality data (enough to elicit dangerous capabilities) is reasonable and realistic, since AI evaluation organizations already recommend using fine-tuning during evaluations [1, 2].
2. Hypothetical sandbagging from powerful, misaligned and situationally-aware LLMs. In this case, we are imagining that the AI lab itself performs the elicitation, so no posthoc locking methods are on the table, and fine-tuning and RL are available. We agree that using a set of words might be weaker than actual sandbagging (and most of our limitations section is the discussion of this disanalogy) -- but we’ve shown in Appendix A.1.3 that changing the language instead of adding a prefix gives similar results, so we expect other kinds of “triggers” would also behave similarly to our passwords; future work could investigate this more thoroughly. For this threat model, we agree that the availability of high-quality data may not be realistic, which is why we performed both SFT experiments with weaker demonstrations (§5.1) and RL experiments (§6).

Also the findings and insights are not that surprising. Many safety alignment and jailbreaking papers show that alignment is 'shallow' and can be easily reversed [1,2]

Our SFT results are indeed predictable from previous work (and we do discuss [2] -- and what sets us apart -- in related work). Some key novel contributions of our paper include:

• A clean setup where (1) we have assurance that capabilities were hidden then unlocked (rather than erased then relearned), and (2) which better models our two sandbagging-related threat models (refusals are less analogous than poor performance)
• Experiments with SFT-based capability elicitation using weaker demonstrations, and extensive generalization experiments
• Experiments with RL-based capability elicitation (and showing that RL fails if the starting policy is too weak, but that it has to be really weak for that)
• Experiments where the model is trained from scratch to be “locked” in a toy setting; these experiments validate our setup and show that SFT-based locking is a reasonable choice

I wonder how the findings would differ if you don't do FT, instead do pre-fix tuning (i.e soft token prompts, see [1] below) or if you do zero-shot prompts and prompt optimization methods like GCG.

Thanks for suggesting these! We ran several experiments with few-shot prompts and manual jailbreaks, and found that they don’t unlock the models – see point 1 in our main rebuttal (other reviewers asked about this as well).

[1] DIST, Emerging processes for frontier AI safety, https://www.gov.uk/government/publications/emerging-processes-for-frontier-ai-safety/emerging-processes-for-frontier-ai-safety

[2] Metr, Guidelines for capability elicitation, https://metr.github.io/autonomy-evals-guide/elicitation-protocol/