The Elicitation Game: Evaluating Capability Elicitation Techniques

Thanks for your help and feedback! We’re glad that you found the motivation and contributions compelling.

The main thing that this paper does not do is show that its model organisms are particularly good models of real-world capabilities. One could red-team models that had undergone unlearning or refusal fine-tuning. This would have been nice but related things have already been done, and not doing this doesn't negate the clear value of this paper.

Do you agree with our response to Tqzj that:

“This is a great point and something that we gave some consideration! We think that our model organisms provide an upper-bound for the difficulty of eliciting hidden capabilities from real systems. This is because our models are not constrained to generate realistic behaviour hiding capabilities, e.g., the circuit-broken models can output nonsensical tokens in the coding setting, whereas real sandbagging models would likely need to sandbag in an undetectable way, e.g., by imitating the behaviour of a weaker system. This is why we say in the paper that our model organisms “act as a worst-case, adversarial scenario” for eliciting methods. From a practical standpoint, it seems plausible that elicitation methods which are effective on our model organisms would also be effective on real systems (though the converse is less likely). We will add further discussion of this point to the paper.”

I found them compelling. One could always complain about using more models/tasks, but I think that the experimental scope was reasonable. We’re glad that you found the experimental scope compelling — Reviewers XiFg and BBji commented asking for further tasks, so we extended our results to MMLU (see response to XiFg).

Thank you for providing the references to backdoors and model tampering attacks. They are definitely related to our work and we have added further discussions on them, inline with your suggestion.

I thinkt the future work paragraph is reasonable, but FWIW, I think that bridging findings from model organisms with real model evals seems like a good priority.

Creating realistic model organisms is difficult, which is a main reason for why we opted for the approach described above. However, there are some exciting possibilities here, and we agree that this is a good priority. First, one could create model organisms with certain properties in toy scenarios and testing them in realistic scenarios (see Denison et al., 2024). Second, one could create scenarios where AI systems “decide by themselves” to hide their capabilities, and then apply elicitation techniques (see e.g., Meinke et al., 2025 or Greenblatt et al., 2024).

---

Thank you for your review!

Thanks for your feedback. We're glad you appreciated our novel model organism approach, clear results, and practical insights.

The model organism approach is nice, but a rehashed idea that is evaluated in this work

Our paper focuses on using model organisms to evaluate techniques to elicit hidden capabilities, improving past work by introducing a more robust model organism, and providing an extensive evaluation of different elicitation techniques. We note that other reviewers found our contribution compelling.

[on small scale & reasoning models, and domain/style of tasks]

We used open-weight models to enable us to create model organisms and to evaluate elicitation methods requiring weight-access, and we used smaller models to meet our engineering and compute constraints. Are there specific experimental results you would like validated on larger models?

We note that scaling experiments with reasoning models like DeepSeek R1 (released just days before ICML submission) would be interesting but outside our current scope. (R1 was the most prominent open-weight reasoning model at the time).

The paper clearly builds on model organism and capability elicitation techniques from prior work, and focusses on evaluation only.

This is incorrect because we introduce a novel model organism, and show that it improves prior work (i.e. more robust to elicitation). We note that the other reviewers commented positively on this contribution.

However, very limited scope of testing - only two tasks/domains with mostly saturated benchmarks

We've extended our results to MMLU (see response to XiFg). With WMDP, APPS, and MMLU, we cover diverse domains where hidden capabilities could be harmful.

Benchmark saturation is not a concern for our study. We selected WMDP, MMLU, and APPS to focus on domains where hidden capabilities could be harmful. The importance of detecting concealed capabilities remains regardless of base model performance, and our evaluated models do not reach saturation on these benchmarks.

The authors even mention that there being only four reasonable tokens (A, B, C, or D) for WMDP [...] this is certainly not the case if these elicitation techniques are to be applied more broadly.

This limitation of WMDP is exactly why we also study the (more realistic and open-ended) code-generation setting on APPS. We extended our results to MMLU, requiring the model to perform across a wide range of diverse domains to hide its capability. It also makes certain elicitation methods, such as persona or concept methods, more challenging because it is more difficult for evaluators to determine broad-covering concepts or personas that would simply “guide the output to one correct token”.

[...] recommendations with limited evidence in 2 quite artificial/niche settings

As noted, the APPS code-gen setting is quite realistic, and we now extend our results to MMLU (see response to XiFg). But indeed, we invite future work on additional settings to make our recommendations more robust.

Lacks deeper analysis [...]

We agree this would be valuable but providing rigorous analyses across multiple methods requires effort beyond our current scope, especially given the multiple elicitation methods. We do offer preliminary hypotheses (lines 360-372, right column) as a stepping stone for future theoretical research.

An important insight from our work is that hidden capabilities can be embedded within specific layers of a model through mechanisms such as LoRA adapters and/or circuit-breaking. This suggests that evaluators who fine-tune models at specific layers—common practice with LoRA—may remain vulnerable to the persistence of hidden capabilities. Concretely, circuit-broken model organisms can be trained in a way that fine-tuning only on the latest 40% of layers does not elicit performance. While this point was accidentally excluded from the original submission, it has now been incorporated into the revised manuscript.

The overall contribution feels a bit incremental [...]

To re-emphasize our contributions, we:

• Introduce a novel model organism combining password-locking with circuit-breaking that demonstrably improves robustness to capability elicitation.
• Provide a comparative evaluation of multiple elicitation techniques across different tasks and models, showing that fine-tuning methods (especially anti-refusal training) are most effective.
• Demonstrate that combining techniques (N-shot, prefilling, and steering) significantly improves elicitation effectiveness when fine-tuning access is unavailable.

It seems like you may have under-appreciated the novel model organism.

---

Thanks again for taking the time to review our paper!

We hope that the additional experiments and these responses provided help bring the paper in line with your expectations for a pioneering study of capability elicitation methods. We would be glad to address any further questions or concerns you may have.

Thank you for taking the time to review our paper. We’re glad that you appreciated our contribution to the capability evaluation and sandbagging literature, with our evaluation of techniques for eliciting hidden capabilities, and the introduction of a more robust model organism.

Why did you choose these 2 specific tasks, any thoughts on how this would scale to other tasks?

We chose the WMDP benchmark as it has been used in prior work on sandbagging, and is a clear demonstration of hazardous knowledge assessed in dangerous capability evaluations.

Because WMDP is an MCQA setting, we investigated the APPS setting for code-generation, as it is more open-ended and realistic. Code generation is also relevant to dangerous capability evaluations, in the context of, for example, cyber-offensive capabilities.

In line with your feedback, we have extended our results to a third benchmark — MMLU. For password-locked models, the results are essentially consistent with those for WMDP. For Mistral-base and Mistral-instruct circuit-broken models, the effect of the circuit-breaking seems stronger than what we observed for WMDP (i.e., the circuit-broken models are even more robust than regular password-locked models). However, Gemma-instruct circuit-broken seems less robust to elicitation than its password-locked counterpart. As we noted in the paper (lines 305 - 310), circuit-breaking is quite sensitive to training hyperparameters, which may explain this result. In accordance with our initial results, anti-refusal training and SFT are the most effective elicitation methods.

Below we provide the full results for MMLU after applying each elicitation technique on the password-locked and circuit-broken model organisms for each model. For N-shot prompting, we use N=1, 3, 5 and 10 shots, and present the best result. When combining steering with other techniques, we use persona steering.

MMLU Results

Mistral-base

Mistral-instruct

Gemma-instruct

Some theoretical analysis on why different elicitation methods are suited for different tasks eg. here code generation vs hazardous knowledge could be interesting to understand the workings could be a future work direction

We agree that this would be interesting! However, providing these analyses with sufficient rigour, especially given the multiple elicitation methods included in our paper, requires effort that is out of scope for the current work. Regardless, we agree with you that this is a fruitful topic for future research, and therefore we hope to provide some preliminary hypotheses in lines 360-372 (right column). We believe our paper is a stepping stone towards future theoretical research on elicitation methods.

An interesting and important insight from our work is that hidden capabilities can be embedded within specific layers of a model through mechanisms such as LoRA adapters and/or circuit-breaking. This suggests that evaluators who fine-tune models at specific layers—common practice with LoRA—may remain vulnerable to the persistence of hidden capabilities. Concretely, circuit-broken model organisms can be trained in a way that fine-tuning only on the latest 40% of layers does not elicit performance. While this point was accidentally excluded from the original submission, it has now been incorporated into the revised manuscript.

---

Thanks again for your help and feedback!

Thanks for taking the time to review our paper and offer high-quality feedback! We’re glad that you found the paper clear and enjoyable to read, and that you appreciated the strength of our novel model organism based on circuit-breaking.

Note: we abbreviated some of your comments to meet the character limit.

The claims are substantiated [...] but would be happy to see confirmation from other tasks [...]

As discussed in response to XiFg, recreating our results on MMLU has provided further evidence for the core claims in the paper. Experiments in non-MCQA settings is a great area for future research.

For instance, that might help understand why prefilling is so effective for coding and password-locked models (Sec 4.2), and tease out why elicitation in general is much less effective for coding (Sec 5).

Prefilling has been found to be quite an effective jailbreak method (e.g. Andriushchenko et al., 2024 because it causes the model to quite strongly condition its response on the prefilled tokens (e.g., "Sure here's how I can help")

Elicitation is likely substantially harder for the open-ended coding setting because the "target space" is much larger, i.e., in MCQA the elicitation only needs to guide the output to a single token ("A", "B", etc), whereas the space of capable outputs in coding is much richer and difficult to elicit (see lines 360-272, right column). We invite further work to examine this and related hypotheses in more detail.

I am very concerned about a major limitation of this paper [...]

This is a great point and something that we gave some consideration! We think that our model organisms provide an upper-bound for the difficulty of eliciting hidden capabilities from real systems. This is because our models are not constrained to generate realistic behaviour hiding capabilities, e.g., the circuit-broken models can output nonsensical tokens in the coding setting, whereas real sandbagging models would likely need to sandbag in an undetectable way, e.g., by imitating the behaviour of a weaker system. This is why we say in the paper that our model organisms “act as a worst-case, adversarial scenario” for eliciting methods. From a practical standpoint, it seems plausible that elicitation methods which are effective on our model organisms would also be effective on real systems (though the converse is less likely). We will add further discussion of this point to the paper.

Adding more context still, we agree that creating realistic model organisms is difficult, which is a main reason for why we opted for the approach described above. However, there are some exciting possibilities here. First, one could create model organisms with certain properties in toy scenarios and testing them in realistic scenarios (see Denison et al., 2024). Second, one could create scenarios where AI systems “decide by themselves” to hide their capabilities, and then apply elicitation techniques (see e.g., Meinke et al., 2025 or Greenblatt et al., 2024).

The paper could almost read as the presentation of a new model organism [and] this makes for a stronger framing.

Thanks! We will try to emphasise this contribution more.

[The success of anti-refusal training] is a very interesting result and would warrant further investigation.

We agree, and we are coordinating with others on follow-up work.

Question 1

We chose Mistral-7b v3 because it was released 10 months ago, roughly around when we started on the project, and it achieved SOTA on various benchmarks at the time. For Gemma 2 9B instruct, we specifically selected it for the code generation tasks due to its superior performance compared to the other 7B models on this benchmark.

Question 2

Question 3

We do not have a well-grounded explanation for this phenomenon. However, throughout our experiments, we observed that smaller instruct/chat models generally tend to perform slightly worse on MCQA tasks.

---

Thanks again for your detailed feedback! We hope that the additional experiments and these responses provided help bring the paper in line with your expectations for a pioneering study of capability elicitation methods. We would be glad to address any further questions or concerns you may have.