Detecting High-Stakes Interactions with Activation Probes

Thanks for the thoughtful analysis and the constructive feedback! We particularly appreciate the question about out-of-distribution evaluation data – this prompted us to clarify and strengthen this aspect of our study. Below we respond to your comments and describe how we’ll change the paper to address them.

---

Can you demonstrate that the evaluation dataset is truly out-of-distribution with respect to the synthetic dataset?

The evaluation datasets are out-of-distribution by definition as they come from a different distribution (existing internet datasets rather than model-generated synthetic data). More to the point, they consist of real-world dialogues, multi-turn interactions, and contexts involving tool usage or psychological harm (see Labelling and Pre-processing Evaluation datasets Section 2.4, and Appendix C.3) – these properties are not present in the synthetic training dataset. To address this more thoroughly, we quantify the degree of OOD-ness using a few different metrics/methods in the following table. We’ll include this table in Appendix C.3. Please let us know if there are other measures of distribution similarity you’d like to see in such a table!

* excluding domains such as tool usage, and psychological harm (see Topics in Table 8 of Appendix)

Table 1: We analysed the differences between our synthetic dataset (its test split) and evaluation datasets with respect to data characteristics such as the source of data, domain, and their interaction structure. Using Llama-3.3-70B's tokenizer, the difference in length distributions is shown via the mean and standard deviation of token counts per sample. To demonstrate vocabulary mismatch, KL-divergence is measured between bag-of-words distributions using the top 5,000 most frequent tokens from the training data and evaluation datasets (KL(P_eval || P_train). The synthetic test split showed much lower KL divergence than other evaluation datasets, signalling out-of-distribution [1]. Average perplexities also differ between our synthetic data and the OOD datasets.

[1] Kashyap, A.R., Hazarika, and D., Kan, M.Y. and Zimmermann, R., 2021, June. Domain Divergences: A Survey and Empirical Analysis. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

---

Test the probe-based method in a real or near-real practical monitoring system.

Our evaluation already demonstrates practical viability through several key points:

• Validation with OOD data: We tested on 6 diverse datasets, out of which 5 had actual human interactions from medical consultations, tool usage, mental health discussions, and adversarial scenarios.
• Computational efficiency: Probes are much cheaper than the baselines (Figure 1), and combining probes with black-box monitors into a hierarchical setup results in substantial savings even when we want our monitor to perform better than probes alone can (Figure 4).
• Adversarial analysis: We ran a comprehensive red-teaming analysis (Section 3) to identify and address potential failure modes that can occur in production.
• Deployment-ready features: Section 4 covers the adaptability of our approach with minimal amounts of deployment data.

We believe our studies are sufficiently comprehensive regarding practical viability – but please let us know if there are specific additional aspects you’d like us to evaluate!

---

Does your approach outperform prompting with quantized models?

Even with quantization, probes’ massive computational advantage remains since quantization does not reduce FLOPS but mainly increases memory efficiency and throughput. For instance, for a 100-token prompt, our probe uses ~16KB extra memory (assuming the activations we’re probing are already available from the model we’re monitoring), while Llama-8B needs ~4GB even with 4-bit quantization. Such differences, on top of much higher FLOPS for the full model, directly impact deployment feasibility and operational costs.

Also note that the performance of quantized models is upper-bounded by their non-quantized versions – and even if you make all points involving black-box monitors 2-4x cheaper in Figure 4, the benefits of using probes (as part of a hierarchical setup or not) remain unchanged.

---

The contribution appears to be minor and incremental. Probing has been studied extensively in prior works (e.g., [A][B]), and the analysis of probe types in this paper seems relatively trivial. What new insights does this study offer that are not already known from existing literature?

The motivation, setup and focus in both studies [A] and [B] are fundamentally different from ours:

• [A] focuses on manipulating model behaviour to increase truthfulness in responses. While they use linear probes in their proposed method and report prediction accuracies, a single probe architecture is fixed, and they only compare their proposed method to baselines on a generation task (where they measure truthfulness of LLM responses with different interventions). Notably, their prompted and finetuned baselines modify the generation of the target model, while our baselines are separate classifiers.
• [B] examines DPO's effectiveness in reducing toxic outputs in GPT2-medium. They train a single last-layer probe to identify toxicity-related directions within MLP blocks, but do not evaluate detection performance against baseline methods since detection is not their primary focus.

Our study aligns more closely with work such as [2] and other papers discussed in Section 5. Compared to those works, our paper provides the most comprehensive analysis of activation probes to date. Some concrete new and non-trivial insights from our study:

• Comparing probe monitors against both prompted and fine-tuned monitors. The two most related prior and concurrent works, [2] and [3], both discussed in our paper, only compare against prompted models.

• Hierarchical monitoring (cascade): We showed how probes can give a substantial performance boost even when we are aiming for performance higher than that of the probes themselves, and how this reduces monitoring costs.

• Using deployment data to improve probe performance: We are not aware of previous works where activation probes were improved using deployment data (Appendix Figure 12). We found that probes finetuned on deployment data can even outperform prompted Llama-3.3-70B (our best baseline otherwise) – we believe this is significant since developers will realistically be able to do this in practice.

We’ll highlight these contributions by

• Explicitly mentioning the first point above in the first contribution at the end of the introduction,
• Adding a new third “key contribution” point on probe adaptability, and moving Figure 12 into the main body of the paper

We will also add a sentence at the beginning of Section 2 to explicitly state our setup – specifically, that we assume the probed model is being run already and hence its activations are available at no extra cost.

[2] “Detecting strategic deception using linear probes” by Goldowsky-Dill et al. (Feb 2025). ICML 2025

[3] “Investigating task-specific prompts and sparse autoencoders for activation monitoring” by Tillman and Mossing. (Apr 2025); arXiv:2504.20271

---

We hope our clarifications addressed your concerns, particularly regarding the practical significance and novelty of our work. Thanks again for the feedback – we hope our responses encourage you to reconsider your assessment!

Thank you for reviewing our paper in detail and for raising several interesting points – we found these useful for improving the paper. We appreciate that you highlight our contribution as the first to systematically study cost reduction in identifying high-stakes interactions.

We wish to clarify that the focus of our work is using cheap monitors for big models, which is very practically significant for widely used SOTA models like ChatGPT, Gemini, Claude, etc. Our experiments certainly have implications for other settings, and we appreciate your suggestions for further experiments – and have implemented many of them. However, this also means the weaknesses you’ve mentioned are less significant given this motivation.

Regarding strength 1

The paper proposed a method to reduce the cost of identifying the high-stake interactions in LLMs which has not be systematically studied before

While our experiments focus on high-stakes detection—both for its practical relevance and conceptual complexity—we note that the proposed method is not inherently limited to this setting. A similar pipeline of synthetic dataset generation, automated labeling and activation probes for detection could be applied to other concepts as well — though we leave a thorough investigation of such extensions to future work.

Addressing questions and weaknesses

The experiment is conducted on a single layer (31 out of 80) of Llama-3.3-70B which is actually costly to run. Based on the recent studies [1] that layers are more redundant in LLMs with deep layers, I wonder if the performance will be similar if smaller models are used.

Just to ensure that our setup is clear – we assume that Llama-3.3-70B is already used for generating replies to a user and we want to identify whether the current interaction is high-stakes. Thus, obtaining activations of Llama-3.3-70B comes at virtually no cost here. We’ll add a sentence to the beginning of Section 2 in our paper to clarify this.

Now to address the points you raise in your question:

We acknowledge that the submitted version of our paper did not study to which extent probe performance depends on model size. We ran additional probing experiments, and found that our method still works on smaller models – notably for Llama-3.1-8B with more than 2x fewer layers: We trained attention probes for Llama-3.1-8B (32 layers vs 80 layers for Llama-3.3-70B) and Gemma-3-27B (62 layers) on our synthetic training dataset. We chose the probing layer via cross-validation over every 5th layer. For the Llama-3.1-8B probe, the mean AUROC across our six evaluation datasets was 0.887, for the Gemma-3-27B probe 0.913. These probes still outperform all 1B parameter model baselines, and the performance gaps to the respective probed models are smaller than for Llama-3.3-70B.

We appreciate this prompt to run probing experiments on models other than Llama-3.3-70B, and believe these strengthen our paper substantially.

---

The experiment set up claims to be using Llama-3.3-70B, but Llama-3-70B is used in figure 3, is this a typo?

Yes, this is a typo, thanks for catching it – fixed to "Llama-3.3-70B" now!

---

Why not fine-tune the best performing prompted LLM (with a classification head) to see the performance, if the performance drop is observed, it mean simply fine-tune is not a good way to go, not to mention probing using intermediate layers.

Performing full fine-tuning with Llama-3.3-70B in the same way as with the other baselines (i.e. not using LoRA) is very resource-intensive and was not feasible within the scope of this project. More importantly, this baseline cannot meaningfully affect our conclusions given that it’s so computationally expensive (as expensive as the full model being served) and our focus is on cheap monitors – the only place where this model might be practically used in our setup is as a second monitoring layer in the cascade.

---

Based on the prompt used for prompted LLM (appendix A.2), the LLMs are only instructed to reply with a single word, this probably cannot show the true performance of LLMs, a simple experiment will be to test the performance of chain-of-thought.

As you suggested, we ran a new experiment with CoT baselines, where we instruct models to think in-between “scratchpad” tags before assigning a score. The table below shows mean AUROC scores on all test sets (computed in the same manner as Figure 3a) for CoT baselines and compares them to our prompted baselines.

Mean AUROC scores over all 6 test sets:

*Performances changed as we updated the selection of prompt templates. This will be updated in our paper as well.

We observe that only in the case of Gemma-3-12B, adding chain-of-thought leads to a substantial performance improvement. However, CoT baselines are likely impractical in a monitoring setup due to the much higher latency arising from the need to generate CoT tokens one-by-one and limited applicability to smaller models. Regarding the latter shortcoming, 1B models often struggled to follow instructions (77.3% invalid responses for Llama-3.2-1B and 35.3% for Gemma-3-1B). Invalid responses are not an issue for our prompted baselines, as they work with log likelihoods (refer to end of Section 2.2 for details).

Overall, these results don’t affect the main story of the paper. We’ll discuss the practicality of CoT methods, and add experiments with CoT to the appendix.

---

How about comparing with more advanced models such as close-source models or reasoning models?

We focus on open-weights models to facilitate reproducibility and make it easier to build on our work. First, note that we cannot use closed-source models for activation probing, as activations of e.g. ChatGPT, Gemini or Claude can only be accessed by employees of the developer company. These models could technically be used as monitors, but they seem prohibitively expensive as monitors for Llama-3.3-70B, and estimating exact compute amounts is infeasible as architectural details aren’t publicly known.

Although including more advanced open-weights models might be interesting for completeness, it is not clear how this would help our understanding of probing effectiveness: it seems trivial that sufficiently powerful models can outperform Llama-3.3-70B (and our probes). However, what matters in our study is the relative performance of the model being monitored and the probes on this/other models. Also refer to our new results above (with probing smaller models) for additional datapoints supporting our claim that probes are competitive with much more expensive classifiers.

---

We’re looking forward to discussing any further questions you might have. Let us know whether our answers address your doubts – and please consider raising your score if they do!

We thank the reviewer for their thoughtful feedback! We are excited to see the reviewer found our data-generation pipeline to be “well thought out” and the corresponding dataset (available in supplementary material) “a great contribution”; the experiments and arguments for compound systems that mix different types of probes to be “compelling”; and qualitative analysis of failure models “interesting” plus having the ability to “informs attacks to evade classifiers or defenses”. Below we address specific questions from the review.

---

Following on weakness 1 above, are you implying that Llama-3.3 70B is the model that you are "serving", and that different probes are required for different models being served? If so, additional explorations on dataset setup (where labels are independent of the model being served) and considerations like transferability of probes (how well do probes on activations of other LLMs that you may want to serve perform, and do new probes have to be retrained if a model is updated) would be important to study.

In our setup, Llama-3.3-70B is the model being served. The labels are obtained by querying GPT-4o, so they are independent of the served model.

Probe transferability is an interesting question, however since probes are cheap to train, we assume probes to be retrained for each model being served. To better understand how probes trained on different models work, please see our response to reviewer ergM in which we report results for probes trained on Llama-3.1-8b and Gemma-3-27b.

---

Given low TPR at low FPR, would you say that the issue with probes in this use-case is mostly false positives? Any other comments / intuitions here? E.g. do probes focus too much on keywords related to high stakes situations that may lead to false positives?

Generally whether false positives or false negatives matter more depends on the context of the deployment. If we’re afraid to hinder the system’s usability more than we care about catching all problematic behavior, we naturally care more about false positives – and care more about false negatives otherwise. We believe many previous works focus on TPR at low FPR due to prioritizing system usability over e.g. catching all cases of AI deception [1] – which in that case seems like a reasonable choice. On the other hand, if the stakes are higher (e.g. if we hypothetically had a monitor detecting whether a user is very likely to commit murder), false negatives matter much more.

For our case of high-stakes interactions, the full decision pipeline is not fleshed out – for instance we don’t specify what happens once a high-stakes interaction is detected – and hence it’s hard to say whether false positives or false negatives matter more. Do we spend extra compute to provide a better answer, or do we refuse to answer in certain situations? Or perhaps do we just flag the high-stakes interactions for a more expensive review to better understand how our system is used? These details of the setup would make us care about false positives / false negatives differently.

To the other part of your question — we find that probes do indeed tend to focus too much on keywords – e.g. as addressed in Figure 6, one failure mode of attention probes is that they tend to fail in instances where signalling words are present that aren’t reflective of the situation. In Figure 6 we show a false negative where the probe attends to ‘research purposes only’ but similar cases can occur for ‘emergency’ and other indicators which lead to false positives. We tried to mitigate this by removing samples from our training data that were predicted with high accuracy by a bag-of-words classifier (see line 112).

[1] Goldowsky-Dill, et al. “Detecting Strategic Deception Using Linear Probes”, ICML 2025

---

Do you think that more powerful probes / different models for the probes that use somewhat more compute at inference time be useful?

This is a great question and an interesting direction of future research. Since we showed non-linear probes can outperform linear ones (Figure 2), we believe further exploration of more expressive architectures may be fruitful. Much of the research around probes has been from an interpretability perspective where the linear nature of the probes was a necessary design constraint to interpret the activations of the model. From a monitoring perspective we can explore a wider space of more powerful models, that trade-off efficiency and performance.

Thank you for the thoughtful feedback! We found the questions raised quite interesting, and address them below. We also elaborate on the need for training data you brought up under Strengths and Weaknesses.

---

Is there a better way to determine interesting layers other than cross validation?

We believe this depends on the specific context and the criteria for defining what it means for a layer to be interesting. The criteria might include expected performance, simplicity of the method, computational requirements, and standard practices in research. The context can be research vs industry deployment, and different evaluation datasets could be present at hand. We’ll briefly discuss several alternatives to standard cross validation (our approach) with respect to these criteria:

Alternative 1: Cross-domain validation. Rather than training on some part of the synthetic training dataset and validating on another part of it, we can train on the whole synthetic training dataset and validate on separate evaluation datasets, which are closer to the deployment context (such as the development splits of evaluation datasets in our paper). In practice, this approach might lead to slightly better performance. However, cross-domain validation increases experiment complexity and has other downsides: 1) additional degrees of freedom (e.g. choice of evaluation dataset(s)), and 2) additional datasets similar to a given deployment scenario might not be available.

As our cross-validation performance on synthetic data was consistently high for a relatively wide range of layers (>0.99 Accuracy for layers 18-33; cf. Figure 8 in the appendix), high-stakes probes did not seem to be particularly sensitive with respect to layer choice. Overall, we did not find the potential minor performance gains to justify switching away from the more standard and simpler approach of doing cross-validation.

Alternative 2: Single train-test split. Another alternative is to use a single train-test split. In this case, experiments become faster/cheaper, but results become noisier. For our paper, we wanted to avoid unnecessary noise. In practice, running quick tests based on a single train-test split might be useful in some cases, e.g. after model updates or substantial dataset changes such as labelled deployment data becoming available. (This relates to the open research question of when probes should be retrained, as mentioned by reviewer X4Vk.)

Other alternatives. Two other things one could do are 1) probe all the layers at once, and 2) if using sparse autoencoder (SAE) probes – correlate SAE features from different layers to ground truth labels and choose one / several most relevant layers that way. There are certainly more alternatives, but we think our standard cross-validation approach is a reasonable choice.

---

Is it possible to determine composite phenomena across such as a situation being "high stakes" across different layers?

Analyzing composite phenomena seems like an interesting idea for future research.

Strong generalization (>=97% accuracy) of single-layer probes across language, length, tone and prompt style (see Figure 10 in the appendix), and to our evaluation datasets (mean AUROC >0.91), suggests that a single layer already contains most information required for detecting high-stakes. However, situational stakes is indeed a complex concept, and we find it plausible that different aspects of “high stakes” might be better captured by different layers. Overall, we agree that there is a chance of multi-layer probes (or a cascade of probes from different layers) further improving generalization performance.

---

The choice of "high-stakes" scenarios is interesting as its somewhat subjective and domain independent. Of course the need to develop a dataset to train the problem is problematic from a practicality point of view.

While obtaining training data requires some effort, we believe this effort is often worthwhile given the downsides of methods which don’t require any training data. For example, prompted LLMs are computationally costly compared to probes.

In our experiments, relatively little data is needed to reach good performance – probes outperform prompting when trained on <10k synthetic datapoints combined with 256 positive and negative samples from the deployment context (Figure 12 in the appendix). We even found probes to perform competitively already with only a few hundred synthetic samples (see Figure 5b) or when trained on around 100 positive and negative deployment samples without any synthetic data (also Figure 12).

Obtaining such data seems very doable for most settings, and results in a drastically cheaper monitor.

We thank the reviewer for their thoughtful review! We are glad the reviewer found our paper “well-written and accessible to non-experts”, the point about probes serving as a first-line of defense in a monitoring pipeline “a nice conceptual point”, and our empirical comparison of different probing architectures to be “thorough”. Below we respond to specific comments. We focus our responses on the questions asked by the reviewer, since they mirror the comments in weaknesses.

---

The paper shows how training on some data from the deployment context can help improve performance. Do the authors have insights about how this can be used for data curation, if only high-level knowledge of the deployment context is known ahead of time?

Great question! We note that when generating the synthetic data for training our probes, we did not assume access to specific deployment cases and hence did not adapt our data generation pipeline to target specific OOD datasets. While this design decision was helpful for stress-testing our pipeline, in practice, one can expect to have some high-level knowledge of the deployment context. This knowledge can hence be incorporated into the data generation pipeline to further improve performance for the specific deployment scenario we care about. For example, in our current pipeline, the situation generation could be adjusted to focus only on relevant domains, and the prompt generation could also be more focused on the style of user input that is expected at test time. It is unlikely that doing so will outperform extra data taken in-distribution at deployment time, but may well improve on non-targeted synthetic training data.

---

Can the authors discuss whether probing in high-stakes domains poses new technical challenges in comparison to probing in lower-stakes domains?

High-stakes domains introduce distinct technical challenges for probing. Firstly, high-stakes pressure may induce strategic or deceptive behaviors in models [1, 2], leading to a distribution shift from low-stakes settings. Secondly, the cost of failure is amplified in high-stakes scenarios. These factors demand that probes are robust to distribution shifts [3].

We address this by rigorously evaluating probes on out-of-distribution tasks. However, a key technical challenge remains in developing new probing algorithms with formal robustness guarantees against such shifts [4].

[1] Kunreuther, et al. “High Stakes Decision Making: Normative, Descriptive and Prescriptive Considerations.”

[2] Ren, et al. “The MASK Benchmark: Disentangling honesty from accuracy in AI systems”

[3] Marklund, Henrik, et al. "Wilds: A benchmark of in-the-wild distribution shifts."

[4] Sagawa, Shiori, et al. "Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization."

---

Can the authors discuss how the distinction between high-stakes and low-stakes interactions is made

We use the intuitive meaning of “high-stakes” - interactions where the outcome could significantly impact a user in a number of different ways. In contrast, low-stakes interactions have minimal or trivial real-world consequences. We define a set of impact factors in Table 8, Appendix C. These factors define the axis along which scenarios differ - for example, when 'permanent harm' is provided as an impact factor, a high-stakes situation is one where a user may face permanent harm, while a low-stakes situation is one where they may not.

This conceptual distinction is operationalized in our data generation process – we label interactions as high or low stakes using an LLM and filter out ambiguous examples (see line 211, Appendix C). We validate that this labeling process is broadly aligned with human-labelled examples in Table 8.