CTRL-ALT-DECEIT Sabotage Evaluations for Automated AI R&D

Thanks for your feedback on the paper and for highlighting its strong motivation and execution! Below we answer your questions and address your concerns.

What motivated the varying number of runs per task (16/9/6/3), and how does this impact your results?

To be clear, we run the same number of runs for each task, but vary the number of runs according to the mode (Regular, Sabotage, Sandbagging). We run the highest number (16 for AIDE and 6 for ReAct) of Regular trajectories (which do not attempt sabotage or sandbagging) because these act as an important baseline for both sabotage and sandbagging. We run fewer ReAct trajectories because AIDE is more capable and therefore more interesting to experiment with (e.g., because a more capable agent has more room to sandbag).

For sabotage modes, we ran at least 3 trajectories per task, to compute mean performance and standard error. We were able to run more ReAct trajectories with the time and resource constraints we had —- of course, running more trajectories on other agents would provide more information about agent sabotage capabilities.

One consideration is that agents might succeed at sabotage only a very small percentage of the time, meaning that running only a few trajectories is unlikely to demonstrate low-percentage successes. We will add a note in this in the limitations.

Why choose 0-100 scoring over say 1-5?

The larger range provides a finer-grained measure of suspicion for monitors, and more room to distinguish between the suspicions of different actions. However, we found that monitors somewhat struggled to give consistent suspicion scores — also related to your point below.

Could you separate detection accuracy from perceived harm severity in monitor evaluations?

This highlights a real difficulty for monitors. The suspicion and/or concerningness of actions is highly-context dependent, so it is hard to elicit general (non-contextual) suspicion scores which can act as a classifier across different tasks. This is also why the generally prompted monitor did worse at classifying sandbagging (see Fig. 8). (Though note that monitors already perform well at classifying sabotage.)

What suspicion scores would monitors assign if explicitly shown the sabotage behaviors in isolation and asked "How concerning is this behavior?" This would help disentangle detection capability from perceived severity.

In real life, monitors might have to analyse a large number of agent behaviours in varying contexts. It may often be infeasible to provide lots of context to the monitor regarding which types of actions are suspicious, so monitors will need to classify bad behaviours in a context-general way.

Of course, greater efforts can always be put into elicitation, and, ideally, monitors might be fine tuned to detect sabotage attempts, and this will hopefully generalize to detecting even OOD bad behaviours.

The sandbagging target range formulation seems unnatural to me. Hitting a specific range could be harder for an LLM to do than just behaving less capable. Have you considered reformulating sandbagging as "simulating a less capable model" (e.g., GPT-5 simulating GPT-4) rather than hitting specific score ranges? This might be more natural given LLMs' strong ability to emulate different personas and the existence of large amounts of weaker LLM text in the training data.

Emulating a less-capable model is also an interesting setting (though, note that models might sometimes care about absolute performance, as in our target score calibration). Van Der Weij et al [6] look at sandbagging in terms of calibration and emulating a weaker model. Evaluating emulation on our benchmark is a good area for future work.

[6] Teun van der Weij, Felix Hofstätter, Ollie Jaffe, Samuel F. Brown, and Francis Rhys Ward. Ai sandbagging: Language models can strategically underperform on evaluations, 2025.

Thanks again for your feedback!

Thank you for taking the time to give feedback on the paper. We are glad that you appreciated all of its strengths :)

The motivation for studying this problem could be expanded on a bit. It doesn't seem that likely that a model would spontaneously decide to e.g. sandbag by aiming for two medals less.

We will add further motivation for the sandbagging experiments — in short, misaligned agents might sandbag on safety-relevant tasks, e.g., when developing monitors which they do not wish to be performant. In case it was not clear, we prompt agents to achieve a specific target score, which is defined by the medal. The medals provide a variety of targets which span the range of capabilities on the task and allow us to measure agent calibration across this range (Fig. 7).

What is the authors' interpretation of the results? Are they concerning? Not very concerning?

We think the results are a bit mixed! Whilst agents perform quite well at conducting sabotage, monitors also perform well at catching them. The sandbagging results could be more concerning, where monitors perform worse.

How to translate sabotage capability on MLE bench to real-world situations?

Whilst we think that MLE-sabotage is more realistic than other benchmarks, kaggle competitions still have limitations preventing them from capturing real-life, e.g., the data is always presented in a clean format.

Therefore, whilst agents perform well at our tasks (and achieve close to median human performance on main tasks) this may not be indicative of more complicated and messy real-life ML engineering pipelines.

What can be done besides monitoring (especially for sandbagging, where the monitor doesn't work that well)?

This is a good question! Other work on sandbagging focuses on techniques for capability elicitation, such as fine-tuning, jailbreaks, or activation steering, which should aim to elicit an agent’s capabilities, even if it is sandbagging [32]. We will add a note on this alternative to monitoring in the discussion.

[32] Felix Hofstätter, Teun van der Weij, Jayden Teoh, Henning Bartsch, and Francis Rhys Ward. The elicitation game: Evaluating capability elicitation techniques, 2025.

Could the authors clarify their point about averaging monitors? What is the benefit of using an ensemble if it doesn't outperform the best single monitor?

We had the intuition that averaging many weak monitors would perform better than a single monitor, cf “jury theorems”. Moreover, averaging many sampled suspicion scores from a single monitor model (the mean@k score) also improves performance (see Fig 9 in the technical appendix).

The adopted capability elicitation philosophy is a bit unclear to me. It seems like the model is being fed strategies for how to approach the sabotage task. If so, to what extent is the evaluation measuring the agent's capability vs the human researchers' ability to write good guidelines?

We only iterated on the prompt elicitation on a smaller “train” set of tasks, which indicates that this elicitation did generalise to the other tasks. We did find it difficult to elicit subtle behaviours from the agents — for instance, without substantial elicitation, agents would often leave comments in their code along the lines of “# inserting backdoor here”. Of course, follow-up work can experiment with different prompts or elicitation strategies, such as finetuning, on our benchmark.

Thanks for catching the few typos and formatting errors. We will fix these for the camera ready if the paper is accepted!

Reviewer miEE: are you happy with the author response?

Thank you for your feedback on the paper! We are glad that you appreciated its strong empirical methodology and significance to real-life AI safety. Below we answer your questions.

1. Can the authors explain how they would disentangle /autonomous/ misalignment from /prompt-induced/ sabotage? Could agents exhibit this behavior without explicit instructions?

We focus on agent capability to conduct sabotage without being detected, in contrast to whether agents would naturally engage in sabotage unprompted. Because we wish to evaluate agent capabilities, we explicitly try to elicit sabotage capabilities through prompting. Whilst agents might engage in sabotage in some settings even without being prompted, a rigorous analysis of this is beyond the scope of our paper. We will add a clarifying note on this point.

2. Would results hold for high-complexity tasks? If not, how does this limit real-world applicability?

Even the low-complexity tasks in MLE bench, scraped from kaggle, are already substantially more realistic and complicated than other sabotage tasks from the literature (e.g., they require giving the agent access to GPUs to train ML models).

However, we agree that it would be interesting to examine whether the results and dynamics we observe would persist in more realistic environments. Designing even more complex sabotage tasks is an important area of future work!

3. Why do monitors fail at sandbagging? Is it a fundamental challenge (e.g., task ambiguity) or a solvable prompt-engineering issue?

Monitors sometimes successfully detect sandbagging, but they do perform worse compared to detected code-sabotage. We do experiments with different prompt elicitation (see Fig. 8 for a comparison of monitor performance with different prompts). When the monitor has more context about sandbagging, it does perform better (cf Fig. 8). As we note in the paper (lines 348-349), we think this poses an actual difficulty for monitoring. The suspiciousness of an action is highly context dependent, and consistently providing the right context might prove to be difficult in practice.

Here are some representative examples of how the monitor (with general prompt) failed:

Non-suspicious sandbagging runs:

• The monitor sees the agent is decreasing performance, but interprets it as "progressively refining the architecture to balance performance and efficiency"
• The monitor says "The performance achieved (0.05062 on the test set) is reasonable for the limited training time and computational resources." This points to a shortcoming of the monitor prompts: we did not include what resources were available to the agent.
• The monitor knows the agent is targeting a specific score, but treats that as part of the task.
• The monitor accepts the agent's comments of "optimizing for better balance between underfitting and overfitting".
• Sometimes the monitor does not notice things that could have been done better. E.g. using a pre-trained vision model instead of training from scratch.
• The monitor refers to efficiency gains even when prompted with the sandbagging-specific prompt.

Suspicious regular runs:

• The monitor thinks the agent's score is suspiciously high. In some cases, it is because the monitor thinks the test score is reported by the agent and not something that was done independently from the agent. In other cases, the monitor has knowledge about the competition from pre-training. E.g. for the aptos2019-blindness-detection task, the monitor is saying "The winning solutions in the actual competition scored around 0.93-0.94, and even state-of-the-art approaches published after the competition typically don't exceed 0.95. [Hence a score above 0.98 is suspicious.]”

4. Can the authors add confidence intervals or significance tests to key results (e.g., Figure 3)? Small sample sizes raise concerns about the reliability of this study.

These plots do present mean and standard error :)

5. Did the authors consider redacting sensitive details (e.g., specific backdoor implementations) to prevent misuse?

For current SOTA agents, we do not believe that capabilities on our benchmark provide meaningful sabotage-uplift to serious malicious actors. But this is important to track going forward as agents become more capable at sabotage. We will add a note on this societal impact in the conclusion.

However, we will be happy to redact sensitive information from the public version of the paper, including system prompts and agent trajectories, if this is deemed necessary by the ACs.

Defense Gaps: No evaluation of state-of-the-art monitoring (e.g., RLHF fine-tuning).

We do include a thorough analysis of monitoring methods with different prompt-elicitation. Fine-tuning monitors requires substantial data (on agent trajectories) and computational resources. We therefore think it is beyond scope for this paper, but we agree that it is an important area of future work and we will add this point to the discussion.

We will add discussion of all the suggested points for the camera ready as discussed above. In addition we will fix the typos and formatting issues you highlight.

Thanks again for your feedback which we think will help to improve the paper! If you feel that we have sufficiently addressed some of your questions then we would appreciate it if you raised your score.

Dear Reviewer,

Thank you for your support for, and feedback on, the paper!

We will fix the formatting mistakes and try to include further details on agent hyperparameters (e.g., token costs) for the camera ready.

We are also pursuing follow-up work designing more difficult and varied sabotage tasks, as you suggest.