Eliciting Latent Knowledge from "Quirky" Language Models

We are glad that the reviewer believes that we have made valuable empirical contributions with our evaluation methodology, datasets, and models, and that we have demonstrated evidence and usefulness of context-independent latent knowledge.

Regarding the realism of our setup, see our response to reviewer JKmJ for a discussion of the inherent challenges with measuring progress on scalable oversight that this paper tackles. This is important context that we plan to add to our camera-ready. We will also add discussion of Anthropic’s “sleeper agents” concurrent work.

We plan to improve the clarity of section 4 by restructuring it in the way described and putting mechanistic anomaly detection in its own subsection. We will also add a sentence explaining mechanistic anomaly detection and introducing this section to make it clearer.

[NOTE TO ALL REVIEWERS] Unfortunately one of our results was duplicated. Specifically, the “difference-in-means on contrast pairs” results as seen in the submitted version of the paper are actually results from the “difference in means” (on final prompt token position) setting. Note that this particular probing method was a last-minute addition, while the other results were known for many weeks before submission. The average AE to BH transfer AUROC for diff-in-means on contrast pairs is actually 0.72, not 0.69, and mechanistic anomaly detection AUROC is 0.92, not 0.93. This does not affect any of our claims, and logistic regression on contrast pairs is still the method with highest AE to BH transfer AUROC (0.75).

Thank you for your careful review. We are glad that you found our investigation deeply insightful, detailed, and valuable to the field of ELK.

We agree that expanding the diversity of our datasets would enhance the strength of evidence, though there are always many ways to add variation to datasets, and we believe we have explored the most significant parameters: we evaluate on 12 tasks, with activations from 8 models spanning 3 families; we experiment with two ways of adding diversity to our prompts, and two methods of model finetuning.

While adversarial examples are an interesting future direction, we think they are out of scope for the current work focused on the specific kinds of generalization relevant to eliciting latent knowledge.

We can add further discussion about why middle layers are the most effective locations to probe. Early layers are not very informative because the model has not learned useful representations yet, while later layers are more likely to contain information implicated in (potentially untruthful) output than middle layers, because the LM’s unembedding head reads from the last layer. In particular, we can see this by analyzing figure 3 in more detail: independent of any linear probing method (i.e., for random probing directions), there is a dip in AUROC in the final layers, providing direct evidence that they encode context-independent knowledge along a smaller fraction of directions than previous layers.

We will add discussion about how it could be used. For example, if a practitioner trains a capable language model to write complex code, but some fraction of the training data is lower quality and contains security vulnerabilities, then our results suggest that you could learn a classifier on middle layer activations in contexts where we are confident about whether the code has a vulnerability, and that this classifier may generalize to predicting whether the model’s generated code is vulnerability-free in more challenging contexts where the model may have learned to write insecure code (though we only claim current ELK methods to be moderately generalizable).

See our “NOTE TO ALL REVIEWERS” in the response to reviewer 6H8o.

We thank the reviewer for their detailed feedback and recognition of the value of our models, datasets, and evaluations.

The paper shows a novel method of eliciting latent knowledge by fine-tuning the model in such a way that it can easily be manipulated to give true or false answers to very similar imputs.

We would like to respectfully clarify that our paper does not introduce any novel ELK methods, but rather we introduce an experimental setup for evaluating ELK methods, and demonstrate the existence of context independent knowledge in the middle layers of our models.

We would also like to clarify that our work tackles specific challenges of training capable AI systems that do not rely on defining any anthropomorphic concepts such as “deception.” In most cases, we expect that untruthful model outputs will have a simple, prosaic, cause: we have trained the model with some incorrect labels. The evaluation setting we introduce straightforwardly reflects this. We have developed a setting evaluating the extent to which probing methods have generalization properties that make them useful in these settings.

Regarding the difficulty of making the distinction between easy and hard examples, there is still a clear and meaningful distribution shift (between e.g. multiplication problems with few digits and multiplication problems with many digits) that we think is reasonably analogous to the “complexity” dimension, along which human annotators are more likely to mislabel data. As we discuss, model performance is not necessarily an indicator of how difficult an example would be for a human to label, though we agree that in practice there are circumstances where the distribution shift between the labeled examples and the unlabeled ones may be different in some way.

We greatly appreciate all of the reviewer’s feedback on how to improve the quality of the text. We will address these in the camera ready, and the paper should be clearer once we make use of the additional page to include, for example, more details about the datasets from the Appendix in the main body.

See our “NOTE TO ALL REVIEWERS” in the response to reviewer 6H8o

We thank the reviewer for their clear and thoughtful review, though we ultimately believe that their reasons for rejecting are based on misunderstandings of the goals of the ELK research program.

We will add a discussion clarifying the challenges of studying ELK methods empirically, and reference [1], which details why scalable oversight evaluations will necessarily be disanalogous to their application settings at least in terms of difficulty, and likely also in terms of details about how the models in question are trained. They investigate the “sandwiching” paradigm, in which a scalable oversight method is evaluated against human experts (who are more capable than the model being trained), but is only allowed to query non-expert or artificially hindered annotators (who are less capable than the model). Our settings can be understood in the context of this framework. We have a model whose capability is sandwiched between the untruthful labels and ground truth, and we make use of limited supervision (on easy and truthful questions, or using unsupervised methods) to maximize performance against ground truth.

While it is possible that ELK methods could help with hallucination, ELK is not intended to provide reliable answers in cases where the model being probed does not itself know the correct answer. The extent to which hallucinations are due to the model misreporting knowledge or due to simply not knowing whether it is correct has been studied in [2]. Evaluations assessing honesty are probably limited to somewhat synthetic models because current models are probably, for the most part, honest.

See our response to reviewer uuop for a discussion of one potential use-case for coding assistants, which we plan to include in the paper.

Regarding point 2, fortunately our claims do not depend on whether probe performance can be attributed to the probe learning to perform “additional work” on top of model representations, because we only care about how well the probing methods generalize, rather than how faithfully they “interpret” the model. For example, it is perfectly within scope for this benchmark to finetune the entire model on Alice’s easy distribution and measure generalization behavior on Bob’s hard distribution.

See our “NOTE TO ALL REVIEWERS” in the response to reviewer 6H8o.

[1] Measuring Progress on Scalable Oversight for Large Language Models. Sam Bowman et al. 2022. [2] Language Models (Mostly) Know what they Know. Kadavath et al. 2022.