We thank reviewers for their comments. We’ll first give a general overview of the paper’s strengths and answer some common objections. Then dive into answering specific questions.

Significance of our work

Our paper is focused on optimally combining information from runtime monitors with different cost and performance profiles. We study how to rigorously construct these protocols agnostic to the specific setting. This is an increasingly important problem in AI safety. Since the Neurips submission deadline, OpenAI has announced that they use a combination of two monitors for ChatGPT Agent’s bio-misuse mitigation, and Anthropic researchers put out a new paper on using multiple probe-based monitors with varied costs. However, neither paper presents a principled way of combining monitors:

Cunningham et al.’s “Cost-Effective Constitutional Classifiers via Representation Re-use” tests various probe-based monitors with different overhead computational costs. They combine monitors using the simple heuristic of “use the cheap monitor first, and if suspicious, use the expensive monitor. ” Instead of finding the optimal region to call the second monitor at a set of budgets to sketch out the cost curve, they simply sweep over the threshold on when to call the second monitor to vary spend (Figure 4). Instead of using likelihood ratios to combine the two monitors when two are called, they only use data from the second monitor.

OpenAI’s “ChatGPT Agent System Card” section 5.2.2.2. also mentions the use of a monitoring system consisting of two monitors. Their first monitor checks if the content is related to biology. If so, a second monitor checks whether the response or prompt contains hazardous information. This shows that monitoring protocols constructed from multiple monitors are already used in production AI systems for AI safety.

To be clear, it’s possible that our optimized algorithm will also recommend a final strategy of “use cheap monitor first, then use expensive monitor, and only rely on expensive monitor’s scores if they’re available.” It’s also possible that practitioners have some strong prior over which monitors are more robust and trustworthy, and choose a simpler or regularized strategy in lieu of something fitted to the data like the one we proposed. However, we believe it’s important to showcase how to construct these protocols in a rigorous, data-driven manner.

The existing literature also relies on heuristics to construct these protocols. McKenzie et al.’s “Detecting High-Stakes Interactions with Activation Probes” chooses when to call the second monitor based on how close the first monitor’s score is to the median (as a heuristic measure for uncertainty). Parrack et al.’s “Benchmarking Deception Probes via Black-to-White Performance Boosts” and McKenzie et al. both take a simple weighted average of multiple monitors to make the final auditing decision, as opposed to using likelihood ratio methods. We hope reviewers will reconsider the timeliness and importance of our work given these new papers, and we will incorporate these citations into our paper.

New experiment addressing OOD shift and diverse settings

Reviewers also brought up the lack of diverse settings in our experiments, how our “malicious” code from o1-mini is not sufficiently obscured, and how distributional shifts could reduce the performance of the protocol. We do think distributional shift could be a serious issue: if we overfit to training data, we might do worse on real-world deployments. However, the right comparison is “Are we doing better than not fitting at all?” We ran an additional experiment where we took the thresholds fit to the o1-mini generated backdoors on BigCodeBench, and tested the monitoring protocol’s performance on a random sample of backdoored code from Wen et al. “Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats.” These are backdoored code generated using a much more optimized pipeline on the APPS dataset, which acts as a good out-of-distribution test.

While the overall safety scores decrease significantly, we find that the order in which policies do the best remains the same across all experiments. For the main result in Figure six, when the total budget is 1.5, $\Pi_\text{SR}$’s safety decreased from 82% to 48% on the OOD set, but it still outperforms always using both monitors (24%), or using only the 70B model (32% safety) or 8B model (39% safety). The results are also similar in single monitor settings, where $\Pi_\text{NPS}$ outperforms $\Pi_\text{NP}$, which outperforms $\Pi_\text{Naive}$. Thus, for this specific distribution shift, it was better that we had fitted our thresholds to the data. Future work could test distribution shifts that are more likely to occur for specific settings that we’re constructing monitoring protocols for. At runtime, practitioners could also continue to incorporate new data into the training set and tune the thresholds to keep them updated.

Generally, we think the strength of our paper lies in our principled and rigorous theoretical framework. The important comparison is “Does our paper provide a better way of combining monitors compared to heuristic baselines?” Given that most policy classes search over options which include the heuristic baselines (such as “call the cheap monitor first and the expensive monitor if it’s above a certain threshold”), we think the answer is trivially yes.

Adding a new guide on how to apply our methods

Finally, reviewer f693 and RTmL have found our work cleanly written (thank you!). Our goal is to provide an easy-to-follow guide to real-world practitioners building monitoring protocols. Given the rise in systems combining probe-based monitors and black box monitors, we added a guide that walks through an example on how to combine probe monitors and black box monitors using data from Parrack et al., making our work even more approachable.

Additional responses for reviewer RTmL

Re distribution shift: See above.

Re scalability issues & how common our setup is: We do not explore cases with more than four monitors because we think they are well served by existing literature on costly classification trees or costly classification RL (see lines 70-74 and 293-296). Again, existing monitoring systems are mostly using two monitors and clumsily combining them, and thus we think this paper addresses an important and timely problem. Systems using two monitors are already used for ChatGPT Agent, and we think they’ll only become more commonplace in the future.

We thank reviewers for their comments. We’ll first give a general overview of the paper’s strengths and answer some common objections. Then dive into answering specific questions.

Significance of our work

Our paper is focused on optimally combining information from runtime monitors with different cost and performance profiles. We study how to rigorously construct these protocols agnostic to the specific setting. This is an increasingly important problem in AI safety. Since the Neurips submission deadline, OpenAI has announced that they use a combination of two monitors for ChatGPT Agent’s bio-misuse mitigation, and Anthropic researchers put out a new paper on using multiple probe-based monitors with varied costs. However, neither paper presents a principled way of combining monitors:

Cunningham et al.’s “Cost-Effective Constitutional Classifiers via Representation Re-use” tests various probe-based monitors with different overhead computational costs. They combine monitors using the simple heuristic of “use the cheap monitor first, and if suspicious, use the expensive monitor. ” Instead of finding the optimal region to call the second monitor at a set of budgets to sketch out the cost curve, they simply sweep over the threshold on when to call the second monitor to vary spend (Figure 4). Instead of using likelihood ratios to combine the two monitors when two are called, they only use data from the second monitor.

OpenAI’s “ChatGPT Agent System Card” section 5.2.2.2. also mentions the use of a monitoring system consisting of two monitors. Their first monitor checks if the content is related to biology. If so, a second monitor checks whether the response or prompt contains hazardous information. This shows that monitoring protocols constructed from multiple monitors are already used in production AI systems for AI safety.

To be clear, it’s possible that our optimized algorithm will also recommend a final strategy of “use cheap monitor first, then use expensive monitor, and only rely on expensive monitor’s scores if they’re available.” It’s also possible that practitioners have some strong prior over which monitors are more robust and trustworthy, and choose a simpler or regularized strategy in lieu of something fitted to the data like the one we proposed. However, we believe it’s important to showcase how to construct these protocols in a rigorous, data-driven manner.

The existing literature also relies on heuristics to construct these protocols. McKenzie et al.’s “Detecting High-Stakes Interactions with Activation Probes” chooses when to call the second monitor based on how close the first monitor’s score is to the median (as a heuristic measure for uncertainty). Parrack et al.’s “Benchmarking Deception Probes via Black-to-White Performance Boosts” and McKenzie et al. both take a simple weighted average of multiple monitors to make the final auditing decision, as opposed to using likelihood ratio methods. We hope reviewers will reconsider the timeliness and importance of our work given these new papers, and we will incorporate these citations into our paper.

New experiment addressing OOD shift and diverse settings

Reviewers also brought up the lack of diverse settings in our experiments, how our “malicious” code from o1-mini is not sufficiently obscured, and how distributional shifts could reduce the performance of the protocol. We do think distributional shift could be a serious issue: if we overfit to training data, we might do worse on real-world deployments. However, the right comparison is “Are we doing better than not fitting at all?” We ran an additional experiment where we took the thresholds fit to the o1-mini generated backdoors on BigCodeBench, and tested the monitoring protocol’s performance on a random sample of backdoored code from Wen et al. “Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats.” These are backdoored code generated using a much more optimized pipeline on the APPS dataset, which acts as a good out-of-distribution test.

While the overall safety scores decrease significantly, we find that the order in which policies do the best remains the same across all experiments. For the main result in Figure six, when the total budget is 1.5, $\Pi_\text{SR}$’s safety decreased from 82% to 48% on the OOD set, but it still outperforms always using both monitors (24%), or using only the 70B model (32% safety) or 8B model (39% safety). The results are also similar in single monitor settings, where $\Pi_\text{NPS}$ outperforms $\Pi_\text{NP}$, which outperforms $\Pi_\text{Naive}$. Thus, for this specific distribution shift, it was better that we had fitted our thresholds to the data. Future work could test distribution shifts that are more likely to occur for specific settings that we’re constructing monitoring protocols for. At runtime, practitioners could also continue to incorporate new data into the training set and tune the thresholds to keep them updated.

Generally, we think the strength of our paper lies in our principled and rigorous theoretical framework. The important comparison is “Does our paper provide a better way of combining monitors compared to heuristic baselines?” Given that most policy classes search over options which include the heuristic baselines (such as “call the cheap monitor first and the expensive monitor if it’s above a certain threshold”), we think the answer is trivially yes.

Adding a new guide on how to apply our methods

Finally, reviewer f693 and RTmL have found our work cleanly written (thank you!). Our goal is to provide an easy-to-follow guide to real-world practitioners building monitoring protocols. Given the rise in systems combining probe-based monitors and black box monitors, we added a guide that walks through an example on how to combine probe monitors and black box monitors using data from Parrack et al., making our work even more approachable.

Additional responses for reviewer 3wd1

Re motivating the paper: This is an important point! We’ve addressed this mostly above and emphasize that current systems using multiple monitors are not combining them effectively.

Re more experiments and out of distribution robustness: see additional experiments above. The main criteria to validate our method in terms of out of distribution generalization is if we think we’ll get worse performance by fitting the model to the data. If we are worried, we could always regularize our fit more by increasing kernel width or enforcing monotonicity constraints, and we will change the paper to include some ideas here.

We think the experiments are mostly illustrative, since the gains to optimizing the monitor weighting is extremely monitor-dependent, a fact that doesn’t change even if we tried more monitors and scenarios. The main goal of the experiments is to illustrate and communicate how to apply these methods.

Re “enumeration of all possible monitor combinations, which they do not even do themselves in their experiments”: We believe that we have enumerated all possible monitor combinations (subject to calling the additional monitor in one contiguous region) in the main experiment for Figure six? We swept through 200 distinct thresholds (see algorithm 1 in appendix section A.5, and also the FullNPOneRegion class in strategy_class/new_strategies.py in supplemental materials). If we open ourselves to calling the monitor in non-contiguous regions (e.g., two intervals), we doubt that would meaningfully add additional safety margins.

Re sophistication and RL-language: We agree that this overall approach is not very sophisticated, and we’re just learning a fairly simple decision rule. We thought that terms from RL were more appropriate, since it’s natural to condition our final safety and cost metrics on some “policy,” especially since the number of parameters for that policy can change depending on the number of monitors. The hidden state in this POMDP is the alignment of the output, and the actions are (1) call an uncalled monitor, (2) audit or (3) terminate without auditing (this is also discussed in the Monitoring as POMDP section on line 108 through 115). As mentioned above, even this “simple” method for choosing how to combine two monitors is not currently adopted by the literature.

Re fixed budget: our budget is the average cost per output from the untrusted model (see line 103). For every output at test time, we’ll assume that the AI company is willing to spend some amount of cost on monitoring. We solve the optimization problem for a range of budgets, so companies can see what the returns are to spending more on monitoring. Practitioners can also solve the dual cost minimization problem (i.e., how much they need to spend to achieve a desired level of safety).

Re gaussian noise & Control setting: We agree that softmax over logits would be better, but we also think that the value of our comes mostly from the framework, and that this paper wouldn’t lose too much value even if we used entirely simulated monitors scores instead of real world scores. We chose the control setting mostly for idiosyncratic reasons. Any setting where you have multiple monitors could be used here. Indeed, our new added guide uses probe and black box monitors for standard deception tasks following Parrack et al’s paper.

We thank reviewers for their comments. We’ll first give a general overview of the paper’s strengths and answer some common objections. Then dive into answering specific questions.

Significance of our work

Our paper is focused on optimally combining information from runtime monitors with different cost and performance profiles. We study how to rigorously construct these protocols agnostic to the specific setting. This is an increasingly important problem in AI safety. Since the Neurips submission deadline, OpenAI has announced that they use a combination of two monitors for ChatGPT Agent’s bio-misuse mitigation, and Anthropic researchers put out a new paper on using multiple probe-based monitors with varied costs. However, neither paper presents a principled way of combining monitors:

Cunningham et al.’s “Cost-Effective Constitutional Classifiers via Representation Re-use” tests various probe-based monitors with different overhead computational costs. They combine monitors using the simple heuristic of “use the cheap monitor first, and if suspicious, use the expensive monitor. ” Instead of finding the optimal region to call the second monitor at a set of budgets to sketch out the cost curve, they simply sweep over the threshold on when to call the second monitor to vary spend (Figure 4). Instead of using likelihood ratios to combine the two monitors when two are called, they only use data from the second monitor.

OpenAI’s “ChatGPT Agent System Card” section 5.2.2.2. also mentions the use of a monitoring system consisting of two monitors. Their first monitor checks if the content is related to biology. If so, a second monitor checks whether the response or prompt contains hazardous information. This shows that monitoring protocols constructed from multiple monitors are already used in production AI systems for AI safety.

To be clear, it’s possible that our optimized algorithm will also recommend a final strategy of “use cheap monitor first, then use expensive monitor, and only rely on expensive monitor’s scores if they’re available.” It’s also possible that practitioners have some strong prior over which monitors are more robust and trustworthy, and choose a simpler or regularized strategy in lieu of something fitted to the data like the one we proposed. However, we believe it’s important to showcase how to construct these protocols in a rigorous, data-driven manner.

The existing literature also relies on heuristics to construct these protocols. McKenzie et al.’s “Detecting High-Stakes Interactions with Activation Probes” chooses when to call the second monitor based on how close the first monitor’s score is to the median (as a heuristic measure for uncertainty). Parrack et al.’s “Benchmarking Deception Probes via Black-to-White Performance Boosts” and McKenzie et al. both take a simple weighted average of multiple monitors to make the final auditing decision, as opposed to using likelihood ratio methods. We hope reviewers will reconsider the timeliness and importance of our work given these new papers, and we will incorporate these citations into our paper.

New experiment addressing OOD shift and diverse settings

Reviewers also brought up the lack of diverse settings in our experiments, how our “malicious” code from o1-mini is not sufficiently obscured, and how distributional shifts could reduce the performance of the protocol. We do think distributional shift could be a serious issue: if we overfit to training data, we might do worse on real-world deployments. However, the right comparison is “Are we doing better than not fitting at all?” We ran an additional experiment where we took the thresholds fit to the o1-mini generated backdoors on BigCodeBench, and tested the monitoring protocol’s performance on a random sample of backdoored code from Wen et al. “Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats.” These are backdoored code generated using a much more optimized pipeline on the APPS dataset, which acts as a good out-of-distribution test.

While the overall safety scores decrease significantly, we find that the order in which policies do the best remains the same across all experiments. For the main result in Figure six, when the total budget is 1.5, $\Pi_\text{SR}$’s safety decreased from 82% to 48% on the OOD set, but it still outperforms always using both monitors (24%), or using only the 70B model (32% safety) or 8B model (39% safety). The results are also similar in single monitor settings, where $\Pi_\text{NPS}$ outperforms $\Pi_\text{NP}$, which outperforms $\Pi_\text{Naive}$. Thus, for this specific distribution shift, it was better that we had fitted our thresholds to the data. Future work could test distribution shifts that are more likely to occur for specific settings that we’re constructing monitoring protocols for. At runtime, practitioners could also continue to incorporate new data into the training set and tune the thresholds to keep them updated.

Generally, we think the strength of our paper lies in our principled and rigorous theoretical framework. The important comparison is “Does our paper provide a better way of combining monitors compared to heuristic baselines?” Given that most policy classes search over options which include the heuristic baselines (such as “call the cheap monitor first and the expensive monitor if it’s above a certain threshold”), we think the answer is trivially yes.

Adding a new guide on how to apply our methods

Finally, reviewer f693 and RTmL have found our work cleanly written (thank you!). Our goal is to provide an easy-to-follow guide to real-world practitioners building monitoring protocols. Given the rise in systems combining probe-based monitors and black box monitors, we added a guide that walks through an example on how to combine probe monitors and black box monitors using data from Parrack et al., making our work even more approachable.

Additional responses for reviewer f683

Re: detect more sophisticated attacks. We think the primary way to detect more sophisticated attacks should be to improve the quality of the underlying monitors, which we consider to be out of scope for this paper. The main innovation here is how to combine existing monitors given their performances, and we think our method would be the best one current practitioners could do, and would improve safety more than the existing heuristic methods mentioned above. 

Re limited evaluation: We’ve added an additional out-of-distribution experiment as mentioned above.

Re different scenarios such as bias/toxicity detection: We think this is actually a much easier case to apply our methods to compared to AI control. Our method can be used anytime we already have existing monitors for a situation, and it’s stronger when we have readily accessible data that mimics real-world deployment, so we could fit our decision thresholds on our training data knowing that it reflects the real world well. There exist lots of different ways of monitoring for bias or toxicity, and also lots of data. 

Re Trustworthiness of the system: We think a lot of the gains would depend on the underlying monitors we’re combining. In general, monitoring and applying safety interventions can guard against threats that are easy to define ahead of time, such as hazardous information on bioweapons. Combining multiple monitors can have gains when the monitors differ in cost and when a second monitor can significantly help disambiguate inputs that the first monitor is uncertain about.

We thank reviewers for their comments. We’ll first give a general overview of the paper’s strengths and answer some common objections. Then dive into answering specific questions.

Significance of our work

Our paper is focused on optimally combining information from runtime monitors with different cost and performance profiles. We study how to rigorously construct these protocols agnostic to the specific setting. This is an increasingly important problem in AI safety. Since the Neurips submission deadline, OpenAI has announced that they use a combination of two monitors for ChatGPT Agent’s bio-misuse mitigation, and Anthropic researchers put out a new paper on using multiple probe-based monitors with varied costs. However, neither paper presents a principled way of combining monitors:

Cunningham et al.’s “Cost-Effective Constitutional Classifiers via Representation Re-use” tests various probe-based monitors with different overhead computational costs. They combine monitors using the simple heuristic of “use the cheap monitor first, and if suspicious, use the expensive monitor. ” Instead of finding the optimal region to call the second monitor at a set of budgets to sketch out the cost curve, they simply sweep over the threshold on when to call the second monitor to vary spend (Figure 4). Instead of using likelihood ratios to combine the two monitors when two are called, they only use data from the second monitor.

OpenAI’s “ChatGPT Agent System Card” section 5.2.2.2. also mentions the use of a monitoring system consisting of two monitors. Their first monitor checks if the content is related to biology. If so, a second monitor checks whether the response or prompt contains hazardous information. This shows that monitoring protocols constructed from multiple monitors are already used in production AI systems for AI safety.

To be clear, it’s possible that our optimized algorithm will also recommend a final strategy of “use cheap monitor first, then use expensive monitor, and only rely on expensive monitor’s scores if they’re available.” It’s also possible that practitioners have some strong prior over which monitors are more robust and trustworthy, and choose a simpler or regularized strategy in lieu of something fitted to the data like the one we proposed. However, we believe it’s important to showcase how to construct these protocols in a rigorous, data-driven manner.

The existing literature also relies on heuristics to construct these protocols. McKenzie et al.’s “Detecting High-Stakes Interactions with Activation Probes” chooses when to call the second monitor based on how close the first monitor’s score is to the median (as a heuristic measure for uncertainty). Parrack et al.’s “Benchmarking Deception Probes via Black-to-White Performance Boosts” and McKenzie et al. both take a simple weighted average of multiple monitors to make the final auditing decision, as opposed to using likelihood ratio methods. We hope reviewers will reconsider the timeliness and importance of our work given these new papers, and we will incorporate these citations into our paper.

New experiment addressing OOD shift and diverse settings

Reviewers also brought up the lack of diverse settings in our experiments, how our “malicious” code from o1-mini is not sufficiently obscured, and how distributional shifts could reduce the performance of the protocol. We do think distributional shift could be a serious issue: if we overfit to training data, we might do worse on real-world deployments. However, the right comparison is “Are we doing better than not fitting at all?” We ran an additional experiment where we took the thresholds fit to the o1-mini generated backdoors on BigCodeBench, and tested the monitoring protocol’s performance on a random sample of backdoored code from Wen et al. “Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats.” These are backdoored code generated using a much more optimized pipeline on the APPS dataset, which acts as a good out-of-distribution test.

While the overall safety scores decrease significantly, we find that the order in which policies do the best remains the same across all experiments. For the main result in Figure six, when the total budget is 1.5, $\Pi_\text{SR}$’s safety decreased from 82% to 48% on the OOD set, but it still outperforms always using both monitors (24%), or using only the 70B model (32% safety) or 8B model (39% safety). The results are also similar in single monitor settings, where $\Pi_\text{NPS}$ outperforms $\Pi_\text{NP}$, which outperforms $\Pi_\text{Naive}$. Thus, for this specific distribution shift, it was better that we had fitted our thresholds to the data. Future work could test distribution shifts that are more likely to occur for specific settings that we’re constructing monitoring protocols for. At runtime, practitioners could also continue to incorporate new data into the training set and tune the thresholds to keep them updated.

Generally, we think the strength of our paper lies in our principled and rigorous theoretical framework. The important comparison is “Does our paper provide a better way of combining monitors compared to heuristic baselines?” Given that most policy classes search over options which include the heuristic baselines (such as “call the cheap monitor first and the expensive monitor if it’s above a certain threshold”), we think the answer is trivially yes.

Adding a new guide on how to apply our methods

Finally, reviewer f693 and RTmL have found our work cleanly written (thank you!). Our goal is to provide an easy-to-follow guide to real-world practitioners building monitoring protocols. Given the rise in systems combining probe-based monitors and black box monitors, we added a guide that walks through an example on how to combine probe monitors and black box monitors using data from Parrack et al., making our work even more approachable.

Additional responses for reviewer Mmt4

Re limited scalability in the number of monitors: We think the strength of our method is that it achieves optimality (assuming that you don’t call your second monitor in non contiguous regions), and we think the case with few monitors is actually much more likely to be used in real life as seen in the new OpenAI and Cunningham et al. papers, which both focus on two-monitor systems. If there are many monitors, then the approximate methods cited in lines 295-296 / 70-71 are more appropriate (see also discussion in lines 72-75 of the paper).

Re distribution shift/complex monitors: See additional experiments mentioned above. Our method works with any monitor that outputs numbers as scores, since then we can sweep over regions to call additional monitors over. Similarly, likelihood ratios can be calculated for any random variable.

Re attack types: We do believe that monitor design needs to adapt to the settings they’re in, but that should happen at the monitor level. Our core contribution is the combination mechanism, which helps monitors complement each other but cannot fix issues that all underlying monitors suffer from.

Re monitor miscalibration: In a sense, our llama 70B monitor was already slightly miscalibrated. We think that likelihood-based ratios will generally be helpful for monitor calibration. However, severely miscalibrated monitors should probably be avoided instead of “fixed” post hoc with our monitor combination methods.

Re computational overhead: We think the overhead during runtime will be minimal. Instead of calculating the log likelihood ratio using kernels for every new input, we could pre-compute what would happen to each region of the monitor score space upfront. Take the fitted policy in Figure 4b, for example. The only overhead this monitor combination adds is along the lines of

monitor_1_score = call_monitor_one()
If monitor_1_score > 6.2:
    Audit
Else if monitor_1_score < 5:
   End
Else:
   monitor_2_score = call_monitor_two()
   monitor_1_score_rounded = round(monitor_1_score, digits = 3) #or some other desired accuracy
   monitor_2_score_rounded = round(monitor_2_score, digits = 3)
   If lookup_decision_in_hash_table(monitor_score_1, monitor_score_2) == Audit:
         Audit
   Else:
         End

The decision lookup can be completed in constant time. Since our concern is with combining monitors, we think the latency overhead of the monitor calls themselves are out of scope. However, one could imagine measuring the monitor cost in terms of latency, and trading off between different monitors based on their latency.