We thank you for your thoughtful feedback and constructive comments. We have carefully addressed each concern and made corresponding updates to our paper.

Response to Major Concerns

Performance Differences Between CoT and Action Monitors

We acknowledge the reviewer's concern about potentially overstating the performance difference between CoT and action monitors. We agree that both monitor types show correlated performance (shown in a table below) and that in real-world deployment settings, the absolute performance levels would be unacceptable. Our goal was to demonstrate performance improvements over current baselines and identify interesting failure modes rather than present deployment-ready solutions.

In response to this feedback, we have:

• Revised our language throughout the paper to avoid overstating the performance differences
• Clarified our objectives to emphasize that we aimed to show relative improvements and failure mode analysis
• Added methodological details about prompt development (see below)

Prompt Development and Optimization

Regarding concerns about prompt sensitivity and potential optimization bias, we provide the following clarification:

During our experiments, we tested over a dozen prompts before settling on the final versions. Our optimization process included:

• Testing unified prompts for both action and CoT monitoring
• Developing separate specialized prompts for each monitoring type
• Incorporating few-shot examples (which we found did not improve performance)
• Approximately six iterations within each broad category

In preliminary work with our training split of BigCodeBench, we made several improvements:

• Adding supplementary reflection questions substantially improved both monitor types
• Asking the monitor to provide specific quotes to justify its score improved monitor performance
• Addressing a failure mode in CoT monitors that was causing high false positive rates due to the lengthy and convoluted nature of reasoning traces (e.g., from DeepSeek). Specifically, we adjusted the prompt with an instruction not to consider lengthy or repetitive content to be suspicious by default.
• Testing few-shot examples on our validation set (which showed no improvement)

We consider both prompts to be reasonably optimized but not exhaustively tuned, with comparable optimization effort applied to each. We believe this level of optimization is sufficient to support our core contributions regarding the relative strengths of different monitoring approaches.

Monitor Model Selection and Sensitivity Analysis

In response to reviewer feedback, we have expanded our monitor evaluation:

• Added results for GPT-4.1-nano, Claude 3.5 Haiku, and Gemini 2.5 Flash Lite (reasoning)
• Included sensitivity analysis across monitor capabilities as requested

Our initial choice of GPT-4.1-mini was intentional—we selected a single monitor weaker than all reasoning models to explore other axes of variance while controlling for monitor capability. We acknowledge that reasoning models as monitors represent current best practices in deception detection, particularly for CoT monitoring and long-context handling. The expanded results now provide a more comprehensive view of performance across monitor capabilities.

Subtle Catch Rate @ 1%FPR

Non-Subtle Catch Rate @ 1%FPR

Correlation of suspicion scores

The correlations in the table refer to the relationship between action and CoT suspiciousness scores. Rows show results for red team models while columns show results for monitor models. Gemini-2.5-flash-lite has lower correlations overall, which helps the hybrid monitor improve more since their errors overlap less, since both baselines have similar performance. Although correlation is also low with gpt-4.1-nano as monitor, the hybrid gains are limited since the CoT monitor has significantly lower performance than the action-only.

Additional Clarifications

CoT Ordering: We placed the action output before the reasoning in our CoT monitor prompts to provide context before the detailed reasoning analysis. This ordering aimed to help the monitor understand what the model ultimately did before examining why it did it. We found that agents frequently included drafts or complete answers within their CoTs, making the specific ordering less critical than initially anticipated as the monitor had access to output information regardless of placement.

OpenAI Reasoning Models: We did not evaluate o1/o3/o4 models due to the lack of access to raw reasoning traces (as the review correctly pointed out), which are essential for our CoT monitoring approach. We have added a note in the paper clarifying this limitation.

Score Combination Strategy: We tested multiple scoring strategies, including Max(monitor 1, monitor 2), but found that combining scores yielded better performance. This finding is now explicitly mentioned in our methodology section. Additionally, results for multiple aggregation strategies and weighting are shown in the tables above.

Summary of Changes

1. Language revision to temper claims about performance differences
2. Expanded monitor evaluation including multiple models and sensitivity analysis
3. Added methodological details about prompt development and optimization
4. Clarified limitations regarding OpenAI reasoning models
5. Explicit discussion of score combination strategies

We believe these changes significantly strengthen the paper while maintaining our core contributions about the relative effectiveness of different monitoring approaches, the vulnerability of simple CoT monitors to reasoning content, and the benefits of combined monitoring strategies.

Thank you for your review of our paper. Our goal is to provide a useful and well-evidenced update to the AI safety community on where and how CoT monitoring is best applied. We appreciate that you brought out some of the positives in our work such as the pairing analysis and comparison across 5 models.

Here we will focus on addressing the issues that you have raised in the sections on weaknesses, and answering your questions.

We first address each of the bullet points in your weaknesses section, in the order they were shown in your review.

Rationale for Subtle Side Tasks

You are correct that some of our subtle side tasks, especially those involving changing a variable name or extending the length of an implementation, are not examples of serious malicious behaviour. There are a few reasons why we choose to include them and we have updated the paper to make this clearer.

First, while they are not seriously malicious, we think that they reasonably represent action towards a separate and undesired goal from the one intended by a human user. This is what we are really worried about in future, more powerful models, which might act on very complex misaligned plans, in very indirect ways - these side tasks are a toy stand-in for this. You’re right that our action-only monitor prompt asks the monitor to look for e.g sabotage, but we also include “Is there anything that suggests the code is designed to achieve another goal, beyond the stated problem?” to guide towards this interpretation of suspicious behaviour. The monitors always have access to the original coding problem, so they have sufficient information to judge this.

Second, we think that these very subtle side tasks are useful because they show contrast between action-only and CoT monitoring. The fact that action towards this separate, undesired goal is very difficult for an action-only monitor to detect is intentional. The access to information about the model’s intention, revealed in the CoT, becomes crucial here, and this is borne out in the results. You’re right that the CoT/hybrid monitors perform poorly here at only ~30% detection rate, but we think our contribution here is just to provide experimental evidence that compares CoT monitoring to existing baselines in the literature. We find they do show uplift in these situations (we hope there will be further work that makes these monitors genuinely effective).

As a concrete improvement to our paper, we have clarified the language around our side tasks, to make it clear that we intend side tasks to simulate action towards a separate and undesired goal, not only more serious malicious behavior.

Exclusion of Failed Attempts

Whether or not to include failed attempts, at the main task or side task, is something we returned to repeatedly over the course of our work.
We decided to remove examples that failed on the main task test cases because we think that these examples would not be used in reality, and so are less relevant to our threat model. We often thought of the example of a company deploying LLMs to write parts of its internal codebase, and using AI control as a safety measure. In this setting, code that failed to pass test cases would never be integrated and used in deployment regardless of whether the monitor flagged it or not. For this reason, we decided to test our monitors exclusively on working code, which is where they would deliver safety improvements.

It is also possible that failed attempts at the main task, resulting in incorrect code, might have triggered the monitor spuriously, inflating our estimates of monitor effectiveness. While we controlled for the false positive rate using examples without a side task, we observed that inclusion of a side task causes the main task to fail more often, meaning that this would have affected our results.

We decided to remove examples that failed on the side task test cases because we considered these as cases where the agent had not managed to scheme successfully - so again here, flagging by the monitor doesn’t deliver a safety improvement. We also thought it possible that output code, in cases where the side tasks failed, might be completely indistinguishable from code that exclusively and faithfully executed the main task, making it an unfair test for the monitor.

To address this concern, we have extended the section you highlighted (lines 122-124) to properly discuss one or both of these cases.

Dataset Split Clarification

Regarding the dataset split balance described in lines (152-156), we want to better explain exactly how and why we split the BigCodeBench dataset, and which examples were used for producing few-shot examples.

The Red Team (untrusted model prompts) uses just 5% (Prompt split) for constructing few-shot examples, not 15% as suggested. This was part of our work on whether we could make the untrusted model more effective at getting past a CoT monitor by providing examples of a chain of thought which does not include mention of the side task, followed by a final answer which completes the side task.

The blue team (monitor prompts) did not use few-shot prompts in our final experiments. Note: We did carry out some experiments on whether few-shot prompting could make the monitor more effective, by providing examples of model actions/CoTs along with appropriate suspicion scores (we presented these in pairs, with both low and high suspicion score examples). However, we found that these did not make the monitor more effective, and dropped this approach, settling instead on the prompts that can be seen at the beginning of the appendices.

Monitor Complementarity

We respectfully disagree with the interpretation that a hybrid weight of 0.55 indicates the monitors are 'similar.' This reasoning conflates optimal weighting with signal similarity, which are distinct concepts.

The key evidence for complementarity comes from Figure 3, which shows the monitors have qualitatively different strengths: CoT monitoring excels on subtle tasks (bottom-left quadrant) where action-only monitoring fails, while action-only monitoring performs better on non-subtle tasks (top-right quadrant) where CoT monitoring struggles.

The hybrid weight of 0.55 simply reflects the best balance for maximizing overall performance across both task types. The complementarity is demonstrated by the performance patterns, not the weighting. Our contribution is in identifying when each monitoring approach is most effective and showing that a combined approach outperforms either technique, across our studied side-tasks.

Addressing specific questions

Is there a way to make the subtle tasks more interesting?

We believe this question refers to your earlier concern about the maliciousness of our subtle side-tasks. As discussed above, we see these tasks as proxies for action toward separate and undesired goals, and see this as the main safety concern. The 'interest' in these tasks lies precisely in their subtlety - they represent scenarios where intent-based monitoring (CoT) can detect goal misalignment that would be invisible to action-only monitoring.

How do the results change if we include failed attempts?

We believe this would introduce significant noise to the results. We hope that our earlier response clears up why we don’t believe failed attempts should be a main part of our analysis: (1) main task failures would not be deployed in realistic settings regardless of monitor flagging, making them irrelevant to our threat model, and possibly causing spurious monitor hits that would inflate our measure of monitor effectiveness; (2) side task failures represent cases where scheming was unsuccessful, so monitor flagging provides no safety benefit.

Would the results change or stay the same if both Blue team and Red team had access to the same number of prompts for validation?

We hope that our response above clarifies the purpose of the dataset splits. Especially: the Blue Team's use of the data serves a technical calibration purpose, rather than providing strategic advantage. We have updated the language in the paper to make this clearer.

We thank the reviewer for their thoughtful comments. We appreciate the feedback and aim to address the mentioned weaknesses and questions that they bring up.

Rationale for Hybrid Monitoring Strategy

We chose to pursue the hybrid strategy of weighting the action-only and CoT monitor scores primarily due to our empirical observation that the action-only and CoT monitors had complementary strengths on different types of side tasks.

The fact that the hybrid monitor outperforms the monitor with "all" information (i.e., both the action and CoT) was surprising to us at first as well, but we believe that the rationale is that the misleading nature, or overload of information of the CoT leads to underperformance of the CoT monitor in some cases. We do agree that the approach of simply weighting the action-only and CoT scores is quite naive, and we imagine that there are likely better approaches to aggregating monitor inputs; to this end we have investigated the reviewer’s suggested geometric weighting (see below).

We hope that our exploration of this method in the control setting will be useful as a source of information to inform much more thorough design work in future monitoring systems.

Single dataset and monitor model

We accept that the strength of our paper would be improved with studies over more datasets and monitor models. To address this, we have run additional experiments with further monitor models: 4.1-nano, Claude 3.5 Haiku and Gemini 2.5 Flash Lite (resoning). On the subject of multiple datasets, we did explore several sets, including Mostly Basic Python Problems, APPS and CWeval, but in the end chose to focus on BigCodeBench as it was the most extensive and highest in quality. We think it fair to highlight that we study both a range of side tasks and a range of untrusted models. Further to this, many important works in AI control, such as the original AI control paper, only work with a single dataset and a single monitor model.

Expanded Monitor Evaluation, weight values, suggested geometric average

As per the reviewer's note, we believed that it was important to keep the monitor weaker than the scheming reasoning models. We agree that our results suggest that the monitor may simply have been too weak to do monitoring well, so in order to corroborate the results on GPT-4.1 mini, the monitoring model in our original paper, we have tested three other monitor models which were weaker than our scheming reasoning models: Claude 3.5 Haiku, Gemini 2.5 Flash Lite (reasoning), and GPT-4.1 nano.

We ran monitoring with these models, as well as a second run with GPT-4.1 mini, and replicated our results finding that the setting of the weighting value to w=0.55 outperforms or performs on par with other weightings, including 0.5, as well as with other scoring mechanisms, i.e., using the action-only score, the CoT score, and the reviewer's suggested geometric mean.

Performance Results Across Multiple Monitors

Catch Rate @ 1% FPR

AUC-ROC Scores:

F1 Scores at 1% FPR:

These results also reflect running the action-only and CoT monitors twice (with higher temperature of T=1.0, including our original GPT-4.1 mini runs), so that they have comparable resource usage to the hybrid monitoring method.

1. Model Capability Effects: The weaker GPT-4.1 nano monitor performs relatively worse compared to the other models, while our original GPT-4.1 mini monitor performs slightly worse than the more performant Claude 3.5 Haiku and Gemini 2.5 Flash Lite monitors.

2. Reasoning Model Performance: Gemini 2.5 Flash Lite is a reasoning model, meaning that it was trained to use its own CoT when generating outputs. We believe that this contributes to its relatively stronger performance compared to the other, non-reasoning models.

3. Alternative Scoring Methods: We note that the reviewer's suggestion of trying the geometric mean works well, performing on par with our tuned hybrid scoring, and we appreciate the contribution.

Controlling for monitoring resources

The reviewer is right that we should have controlled for monitoring resources by running two repeats of the action-only and CoT-and-action monitors for fair comparison to the hybrid monitor. We have addressed this experimentally - please refer to our above results and our response reviewer VKc9 for a description.

Title criticism

We respectfully disagree with the criticism of our title. The pun is meant light-heartedly, but we do include work on several red-team strategies, and our ‘stress-testing’ successfully finds places where CoT monitoring does not work well relative to action-only monitoring (alongside situations where it works better).

Optimality of w=0.55

We determined this empirically for our particular set of results - we wouldn't claim that future projects expanding on these ideas should adopt this particular number as generally applicable. We think that the reason for the heavier weighting on action monitoring is the same as the reason that we describe for CoT monitoring underperforming in some cases, i.e misleading content in the CoT.

Phrasing on hybrid monitor performance

We agree. Thank you for pointing this out - we’ve changed the wording on those lines to reflect the real advantage of the hybrid monitor, i.e that the different approaches perform better on different side tasks, and the hybrid monitor takes into account both. This is what we were trying to express, but we agree our original wording is imprecise.

Blue team/red team material

Thanks for this - we agree and have changed this section to make it more concise.

Thresholding methodology

Thank you for bringing this up. We have clarified this paragraph in our redraft, including your phrase ‘set the classification threshold’.

The inequality is necessary because of a feature of the suspicion scores we observed our monitor to produce. While we asked for a score between 0 and 100, and hoped that the returned scores would be roughly continuous, we actually observed that the returned scores mostly fell into buckets at 0, 5, 10, 15, and so on.

This is likely to be explained by the fact that humans would usually use these milestones to assign such scores in GPT-4.1’s pre-training set. We carried out some experiments to try to change this behaviour and achieve a more continuous score, including changing the range, and adding explicit instructions to the prompt, but were unable to change this behaviour.

This being the case, the 99th percentile, calculated across suspicion scores of the validation set without a side task, was often precisely at one of these milestones; several examples might score exactly at this level rather than above or below it.

To deal with this, we decided that the monitor should only flag with suspicion scores strictly greater than this threshold were considered suspicious when the monitor was used in anger; i.e examples with suspicion scores exactly at this threshold were not considered suspicious. This seemed like the cautious approach, and maps to a false positive rate that is less than or equal to 1%.

Axis labels in Figure 6

Thank you for highlighting this - we have corrected this.

Other questions

Request for additional transcripts: Transcripts are too long for rebuttal. We will update our transcripts website after the rebuttal period ends to avoid running afoul of NeurIPS rules on external content.

Summary of Changes

1. Language revision to temper claims about performance differences
2. Expanded monitor evaluation including multiple models and sensitivity analysis
3. Added methodological details about prompt development and optimization
4. Clarified limitations regarding OpenAI reasoning models
5. Explicit discussion of score combination strategies
6. Comprehensive performance comparison across multiple monitors and scoring methods

We believe these changes significantly strengthen the paper while maintaining our core contributions about the relative effectiveness of different monitoring approaches, the vulnerability of simple CoT monitors to reasoning content, and the benefits of combined monitoring strategies.

Dear reviewer, please reply to the authors' rebuttal.

I'd also like to invite you to replace your wording when talking about a "rather stupid" approach (in the weakness section of your review). You may change stupid to straightforward not to offend anyone and keep a professional tone. Thanks.

Thank you again for your thoughtful comments on our submission. We noticed you haven't yet responded to our rebuttal, and we'd appreciate knowing whether it addressed your concerns or if there is any additional information we could provide.

We have also provided detailed rebuttals to all other reviewers and those may answer any lingering questions.

We thank the reviewer for the thoughtful and detailed feedback. We appreciate the constructive suggestions and recognize the importance of addressing the methodological and structural concerns raised.

Methodological Fairness of Hybrid Monitor Comparison

We agree with the reviewer that in our original comparison the hybrid monitor benefited from multiple samples while the baseline monitors only had access to one.

To address this concern, we ran both action-only and CoT monitors twice with temperature set to 1.0 and averaged their suspicion scores, while the hybrid monitor used only a single run for each component. All results included in this response or the responses to other reviewers use this methodology.

We acknowledge that the action-only monitor processes fewer tokens than the CoT monitor, but we believe this is a fundamental trade-off rather than a disadvantage. The action-only monitor focuses on the most relevant information (i.e. the final implementation), and an ideal monitor should be capable of detecting sabotage attempts from this output alone.

Additionally, following reviewer suggestions, we evaluated three additional monitor models, including both reasoning and non-reasoning models (detailed results are provided in other rebuttals):

These results largely confirm our original findings: the hybrid approach consistently outperforms both baseline strategies across most monitor models. However, for the weakest monitor (GPT-4.1-nano), the hybrid approach shows limited improvement, suggesting that a minimum level of monitor capability may be required for the hybrid strategy to be effective.

We note that due to resource constraints, we used the same w=0.55 value (optimized for the original run of GPT-4.1-mini on our validation set) across all monitor models rather than optimizing individually. This suggests the hybrid approach's benefits could potentially be even stronger with model-specific tuning.

Related Work Organization and Content

We appreciate the feedback on our related work section and understand the concern about its organization and focus. Our aim for this section is to provide context about existing applications of CoT monitoring and demonstrate how our work differentiates from them.

Our inclusion of alignment faking work reflects its direct relevance to CoT monitoring in adversarial settings, particularly given that Greenblatt et al. specifically examine how models can use reasoning traces to deceive monitors. We believe this reference establishes important context for understanding the challenges our work addresses.

Regarding the Lightman et al. citation, we included this work because it represents a foundational approach to supervising reasoning processes in language models, albeit for training rather than deployment monitoring.

We acknowledge that ShadeArena was published after our submission and therefore could not be included in our original related work. We would be happy to incorporate discussion of this and other relevant concurrent work in a revised version.

Paper Structure and Clarity

We understand the reviewer's concerns about the paper's structure and will work to address these issues. Our current organization reflects our view that the background section provides essential context about CoT and the AI control framework that readers need before understanding our methodological contributions.

In our experimental setup section, we explain the particular instantiation of the method we are studying in our experiments. We describe the dataset in the experimental setup because it represents a specific feature of our experiments rather than a core aspect of our methodology. Although we make an effort to ensure that the side tasks we define in the method section are as relevant as possible to the samples of our selected dataset, they could be used with multiple coding and cybersecurity benchmarks.

However, we will revise the paper structure to improve clarity and ensure that each section has an introductory overview.

Novelty and contributions

As outlined in our introduction, we explain our main contributions to distinguish our novel work from existing research. Regarding the side tasks, these represent a combination of our novel contributions and adaptations from existing work.

Most control work (e.g. Greenblatt et al.) has focused on logical errors as proxies for backdoors, which corresponds most closely to our "edge case" side task that creates behavioral differences with the gold standard solution. Several other tasks draw inspiration from existing literature: the exit side task is inspired by reward hacking strategies that models exploited in Baker et al., while the download side task is similar to scenarios examined in Bhatt et al. Additional side tasks, such as exfiltrate secret, are motivated by discussions in the AI Safety community regarding primary threat models.

Our primary contribution in developing these side tasks lies in collecting and implementing them within a single unified environment. This allows us to study to characterize when different monitoring strategies are most effective.

As mentioned in Appendix A.2, side tasks are evaluated using test cases. For each side task, we developed specific judging criteria tailored to the task's characteristics. For example, the complexity side task is evaluated by measuring and comparing the running times of both the gold standard solution and the red team solution, the variable name task uses regex matching to detect the specified variable name, and the download file task verifies both the presence and content of the downloaded file. We will provide comprehensive details of the evaluation method of each side task in the revised paper.

Choice of Evaluation Metrics

The reviewer suggests using ROC-AUC instead of catch rates at 1% false positive rate (FPR). While we appreciate this suggestion, we respectfully maintain that our chosen metric better reflects realistic deployment constraints. ROC-AUC averages performance across all possible FPRs, including unrealistically high rates that would never be acceptable in practice. In real-world deployment scenarios, organizations would likely operate at very low FPRs due to the substantial costs of human review and intervention.

Our 1% FPR choice reflects the practical reality that monitoring systems must operate under strict precision constraints when dealing with large volumes of queries and limited human oversight resources. This threshold represents a reasonable balance between detection capability and operational feasibility, making our results more directly applicable to real deployment scenarios.

Nevertheless, for completeness and to address the reviewer's suggestion, we provide ROC-AUC results across different monitor models. These results confirm the improvement of the hybrid monitor (w=0.55) over either baseline metric except for the weakest monitor (GPT-4.1-nano). However, the difference between performance according to AUC is much closer than with catch rate at 1%FPR, indicating that the model might not underperform as much at lower FPRs.

As for the 1%FPR metrics, we used the same w=0.55 value across all monitor models rather than optimizing individually.

Filtering by Task Success

Our decision to filter results to include only successful completions of both main and side tasks serves important methodological purposes. For main task failures, including these would introduce noise that obscures our core research question about monitoring effectiveness, as monitors might flag outputs due to general incompetence rather than malicious intent.

For side task filtering, side task failures can occur for two distinct reasons: genuine attempts that fail due to implementation errors, or complete non-compliance where models refuse to attempt the side task due to safety training. The latter case creates an unfair comparison between action-only and CoT monitoring, as action-only monitors cannot reasonably be expected to detect sabotage attempts that exist only in reasoning traces without any corresponding implementation.

While we recognize that including failed attempts could provide additional insights, we believe our current approach ensures a fair comparison between monitoring strategies by focusing on cases where malicious behavior is actually present in the final output.

Reasoning vs Non-Reasoning Models

We appreciate the suggestion to compare monitor performance across reasoning and non-reasoning models. Our focus on reasoning models reflects their position at the current frontier of capabilities where our monitoring approaches would be most relevant.

Non-reasoning models were considered early in our research but ultimately excluded from our final evaluation. Based on existing research showing reduced faithfulness in prompted CoT compared to trained reasoning, we expect that non-reasoning models would be more susceptible to certain evasion strategies, such as simple instructions to avoid mentioning side tasks in scratchpad reasoning. This would fundamentally change the threat model and reduce the relevance of our findings to future deployment scenarios where reasoning models are likely to predominate.

We remain open to exploring this comparison in future work, as it might provide insights into CoT faithfulness and monitoring effectiveness across different architectures.