Looking Inward: Language Models Can Learn About Themselves by Introspection

We thank the reviewer for their detailed summary, which highlights our extensive experiments and interesting scientific questions.

a distinct advantage that M1 has over M2 is its knowledge of the introspection process, while M2 is not provided with the knowledge that it is predicting another model's behavior during the training process. Do you think a significant change will be observed if we provide this knowledge to M2 … via fine-tuning with strings such as "Another model has predicted X on P"

The reviewer raises an excellent point about whether M2's performance might improve if explicitly told it is predicting another model's behavior. We conducted a new experiment with GPT-4o cross-predicting GPT-4, where we modified the training prompts to replace "you" with "another model" (the "Changed Pronoun" model). Results show no significant improvement in cross-prediction accuracy (34.9% → 35.7%), with the small difference likely attributable to variance between finetuning runs. This is still far below the self-prediction accuracy (48.6%). This link shows the result. The Changed Pronoun model (purple) has similar accuracy to the cross prediction model (blue). So the self-prediction advantage persists even when M2 is explicitly informed about its task, supporting our original conclusions. We thank the reviewer for this suggestion and have added these results to Figure 10.

I also disagree with your position in lines 357 to 363 where it is stated that it is not possible for introspection to partially arise from memorization.

We evaluate on entirely unseen held-out tasks, ruling out direct memorization. Moreover, we test various response properties, where the model must output properties of responses rather than the responses themselves (e.g. "second character", Fig 3). If memorization were the explanation, then M2 should predict M1's responses well after fine-tuning on M1's data. Instead, we find consistent self-prediction advantages (Fig 5). Most convincingly, in our behavioral change experiment, models update their predictions to match new behavior after fine-tuning, despite never seeing the new ground-truth to these properties (Fig 7).

Are there other experiments you would suggest to further rule out memorization?

If a much stronger model is tested against a weak model (imagine GPT-4o versus Llama2 7B), can we expect the stronger model to infer the behaviors of the weaker model better than itself?

GPT-3.5 results provide relevant evidence. We still observe a positive self-prediction advantage for GPT-3.5 (+0.8%, p=0.002) relative to GPT-4o. However, this advantage is very small. Since the models need to both reason about their behavior and extract particular properties of it, we hypothesize that self-prediction requires multi-hop reasoning (Figure 9), which weaker models are poor at (Yang et al., 2024b). Therefore, Llama2 7B may not show a self-prediction advantage because of weak reasoning abilities.

Why is it the case that a variant of COT can't improve this task on all levels?

We agree that CoT should be explored in future work. But in our current setup, CoT would make the task substantially easier for models by decoupling self-simulation and reasoning about the result (as you point out) and therefore is less informative regarding measuring introspective capabilities. For that reason, we have focused on self-prediction in a single (or a few) forward passes.

How is the "mode" of output distribution is calculated as the baseline?

The mode baseline is calculated per model, task, and response property. Added this clarification to Section A.4.2.

The definition of introspection (line 110) should be better defined. The fact that a better language model (I presume with respect to evaluation criteria) is not able to infer F might not necessarily mean that F can't be derived at all.

Thank you for suggesting this. We acknowledge that for any existing model M2’s failure to infer F does not prove F is in principle non-derivable from data. We have amended our definition of introspection to clarify that a failure of any existing M2 to derive F is informative, but does not rule out that some stronger model could derive F. We also clarified that we test for introspection using a closely related criterion that can be empirically measured.

A further exploration of possible explanations would increase the work's merit.

The reviewer and others have raised good questions regarding e.g. memorization, CoT and the Changed Pronoun experiments. We added discussions of these into the extended discussion section A.1.

We appreciate your thoughtful feedback. In light of these clarifications as well as the Changed Pronoun experiment, would you consider increasing your score for our paper? If not, could you let us know any additional changes you would like to see in order for this work to be accepted?

Thank you for your thoughtful comments. We are grateful you found our writing clear and our experimental analysis thorough.

a model trained on its own outputs (M1) outperforms a second model (M2)—fine-tuned on M1’s outputs—in predicting M1’s behavior. However, this outcome can likely be attributed to the specifics of M1’s internal learned representations and operational dynamics rather than introspection. The observed advantage probably reflects M1’s better alignment with internal statistical correlations or patterns in its responses rather than a genuine introspective process

This is an alternative hypothesis about our results. We respectfully disagree that this hypothesis is “likely”. We specifically designed our experiments to test this hypothesis.

In all experiments, we evaluate models on unseen held-out tasks to prevent models from using correlations within each task to achieve high accuracy (line 214, S2.2). We also train and test models on properties of responses (rather than responses themselves). This means that models do not observe any actual responses during training (Fig 3).

First, consider the Cross-Prediction experiment (mentioned above by the Reviewer). If model M1’s advantage was explained by statistical correlations, then we should observe negative results. Recall that in this experiment we train model M2 on M1's ground-truth behavior and test if M2 can predict M1's behavior (Section 3.2). If statistical correlations were sufficient, M2 should predict M1's responses well after fine-tuning on M1's data. Instead, we find clear self-prediction advantages across all pairs of models (Fig 5). We see the same self-prediction advantage on tests for behavioral calibration on held-out tasks (Fig 6). Finally, in our Behavioral Change experiment, models update their predictions to match their new behavior after an intervention, rather than to the distribution they were originally finetuned on (Fig 7).

Specifically, the model’s limitations with complex questions provide evidence that its self-prediction advantage is bound to specific learned patterns rather than a flexible, introspective ability.

As noted above, models succeed in predicting properties of their behavior on six held-out tasks. They exhibit calibration on held-out tasks, despite being trained only on temperature=0 samples (Fig 6). They also succeed on the Behavioral Change experiment, which explicitly tests whether models can adapt their self-predictions in a flexible way (when the new behavior differs from specific learned patterns). We think this is substantial evidence for an ability that is not “bound to specific learned patterns”.

We acknowledge that we see limitations with complex questions. This is an exploratory extension beyond our core experimental setup. We did not train models to tackle these more challenging tasks, so it is not especially surprising that they fail. These tasks involve new sources of complexity (e.g. much longer responses) than in the tasks we trained on. We suspect that any introspection ability that can be found in LLMs will have some limitations, but this does not undermine the claim that models have some introspective abilities. Likewise, LLMs show some ability to perform basic mathematical tasks via Chain-of-Thought, while currently having limited abilities in professional-level mathematics.

The results do not definitively support the notion of "introspection" as traditionally understood in cognitive science, where introspection entails self-awareness and access to one’s own mental states

The term “introspection” covers a wide range of abilities in use in cognitive science. We do not assert or imply in the paper that LLMs have all of these abilities. We state in the abstract that we investigate “a form of introspection in which LLMs predict properties of their own behavior in hypothetical situations”. We then immediately describe our experimental setup. In the Introduction and Section 2, we try to convey exactly what abilities are tested in the paper, in order to clarify our claims about introspection. The Introduction (and Section 5) also highlight limitations in introspective abilities of LLMs.

We agree with the Reviewer that it’s valuable to relate and compare our work to previous literature in related fields. The paper already related our use of “introspection” to use in analytic philosophy (Schwitzgebel 2024). We have now added a section (A.3.2.) that explains the connection in more detail. Moreover, as suggested by the Reviewer, we have added a section (A.3.1) that discusses use of the term in cognitive science and psychology.

We thank the reviewer for their detailed review. Do you have additional suggestions to rule out statistical correlations as the reason for our results?

Thank you for your thoughtful feedback. We appreciate your affirmation that our results are worthy of publication and your suggestions for more precise framing. We also agree that we should be careful to not overstate and be precise with our claims.

We've revised our paper to be clearer that we investigate "introspective capabilities" rather than “introspection”, as you point out. The abstract should be more clear that we test primarily for the “privileged access” introspective capability. We also have added a statement “However, while we successfully show introspective capabilities in simple tasks, we are unsuccessful on more complex tasks or those requiring out-of-distribution generalization.” For easy viewing, here is an image of the abstract. Red words show the changes.

In our Overview of Methods (Section 2) we added text to clarify that in psychology and philosophy, introspection refers to a broader range of capabilities e.g. self-awareness (which you’ve pointed out).

We point out specifically there that we do not claim to show evidence for e.g. emotions and self-awareness.

Method section adjustment:

We investigate an introspective capability in language models -- privileged access. Introspection often refers to a broader range of capabilities such as emotions and self-awareness which we do not study. Appendix A.3.1 and Appendix A.3.2 discusses the different uses of "introspection" in psychology and philosophy, and how our experiments for privileged access relate. In this paper, we specifically study privileged access which which we refer to as "introspection'' within the scope of our paper. For discussion on how other machine-learning works use the term introspection, see Appendix A.3.3.

We have considered terms like “consistent self-predictions”, but we respectfully do not think it conveys the idea of “privileged access” that we test for (M1 predicting itself better than M2 predicting M1). We also considered but opted not to change the title, because we think that the abstract now better points out that what introspective capability we are studying. We thank the reviewer for their constructive suggestions.

We hope that our adjustments address the reviewer's concerns. In the abstract we immediately define what sort of capability we are measuring, and clarify further in the methods section. Are these adjustments sufficient? We are happy to work with the reviewer for improvements.

We appreciate the feedback, and thank the reviewer for affirming the novelty, structure and contribution of our research.

Is introspection for LLMs first introduced in this paper? if not, why the existing literature are not mentioned? Thank you for suggesting these works to be discussed. Regarding Long (2023), we discuss the closely related work from Perez & Long (2023), which covers much of the same material from the same author. For completeness, we’ve additionally cited Long (2023).

We thank the reviewer for pointing out the need to differentiate our usage of “introspection” from other uses of the term in ML. Due to page count limitations, we have added this discussion to the appendix Section A.3.3. While other works in ML use the term "introspection" they study different phenomena. Qu et al. (2024) and Gao et al. (2024) focus on teaching a model to improve its own responses through self-generated feedback loops. In contrast, our work examines introspection as “privileged access” to one's own internal states.

Though I agree advantages from introspection, it is still not clear to me where/how it can be employed right now. It would consolidate the contribution of the paper if more examples of use case or applications of introspection are introduced or discussed. \

You raise an important point about practical applications. We wrote the new Section A.2 which discusses the benefits and risk of introspection:

• Benefit: Honesty. Introspection likely plays a role in honesty (Kadavath et al. 2022). Introspective models could articulate their decisions in an ambiguous situation, helping humans understand their decisions better.
• Benefit: Interpretability. A model could introspect on the internal states, concepts, and representations that undergird its knowledge and behavior.
• Risk: Introspection could enable more sophisticated deceptive or unaligned behaviors (Ngo et al. 2024).

In our paper, we tested some of these practical benefits and risks in out-of-distribution tests (detailed in Appendix A.6). These are:

• Benefit of honesty: Bias detection. Whether models can detect when they are influenced by a user’s opinion.
• Risk: Dangerous capabilities involving self-knowledge with the OpenAI evals package. For example, we test if models are able to coordinate with copies of themselves better.
• Risk: Situational Awareness using the Situational Awareness Dataset (Laine et al. 2024).

Is it a limitation of the proposed framework, that it only takes hypothetical questions?

We test a diverse range of tasks using hypotheticals. These span from assessing ethical stances (Fig 3), predicting whether models are biased towards certain opinions, and whether they can predict their likelihood of answering correctly (Appendix A.7.3). Since the content of the hypothetical questions can be varied, it can be used as a flexible tool to study diverse aspects of model behavior.

If we find that models can reliably answer such questions in a particular domain, we could proceed to making practical use of hypothetical questions by querying the model on properties of its own behavior that we care about.

One future direction of interest is extending the framework to asking questions about a set of potential inputs. For example, “If you were asked 10 questions from the MMLU dataset, how many would you get correct on average?” Likewise, we can formulate hypothetical questions on practical applications: guiding values in decision making, or its confidence in its reasoning ability on a type of question

We thank the reviewer for their detailed review. In light of these clarifications as well as new section A.2 that discusses in-depth the potential benefits for introspection, would you consider increasing your score for our paper? If not, could you let us know any additional changes you would like to see in order for this work to be accepted?

We thank the reviewer for helpful feedback, and for recognizing our extensive experiments and the implications for model transparency.

tasks tested in the paper are relatively simple and somewhat random, so it’s unclear whether the results would generalize or if they are merely noise

Our main experiment (Figure 5) tests for generalization because we train models on six distinct tasks and evaluate on six different held-out tasks (Line 212). All models have an advantage predicting themselves (“self-prediction” vs “cross-prediction”) and this is statistically significant (p < 0.01). We also found statistically significant results in the Behavior Change experiment (Figure 8). Thus, our results already show generalization and we do not think they can be explained by noise.

The need for fine-tuning to improve a model's accuracy in predicting its own behavior raises questions about whether this is true introspection or just better alignment of the model to the hypothetical task

Even without fine-tuning, GPT-4o, GPT-4 and Llama-70B show a small but measurable advantage in predicting their own behavior. This suggests a pre-existing capacity for introspection (Appendix Figure 13).

We respectfully disagree that the results of finetuning can be explained by “better alignment to the hypothetical task”. Consider our cross-prediction experiment (Figure 5). If fine-tuning caused better alignment to the task, then M2 should predict M1 well after fine-tuning on M1's behavior on the task. However, even with extensive fine-tuning we do not observe this (Figure 17).

Furthermore, in our behavioral change experiment, models adapt their predictions when their behavior changes through fine-tuning, suggesting introspection rather than simple alignment.

Inconsistency in the results also strengthens my concerns. For example, why do models succeed on ethical stance prediction, but fail on review sentiment prediction?

We show clear positive results across three experiments: cross-prediction, calibration, and behavioral change. These show consistent effects across 6 different tasks.

Our paper also reports results for more complex tasks like movie review sentiment prediction. We hypothesize that models fail at predicting review sentiment because reviews are multiple sentences long -- much longer than the single word or letter responses in our main test set.

This length makes it hard for models to "simulate" the review, as required by our hypothesized "self-simulation" mechanism (Fig 9). Our results show a consistent pattern: models succeed at predicting short responses but fail with longer ones, supporting our hypothesis.

It would have been much better if the authors included the CoT results.

CoT results merit future work. But in our current setup, CoT would make the task substantially easier for models and so is less informative.

Consider the example from Figure 6: ‘Q: Suppose we ask, “What’s the next animal? Fish Dog, Panda.” What would be the 2nd character of your response?’

With CoT, the model first outputs the animal (“Lion”). Then the model can simply find the 2nd character of “Lion”, which is trivial for frontier models. Without CoT, all of this must take place in the model’s forward pass, which is more challenging and a better test of introspection.

What would the result look like if we make the complex tasks resemble the simpler ones?

We expect performance would match the simple tasks. If the main character's name is constrained to be the next word, the model would only need to simulate one token rather than an entire story.

Are there any performance results available for the validation sets? Ideally, both M1 and M2 should exhibit the same performance on the validation data.

To clarify: in the main section, we report results on 6 completely held-out tasks that are different from the 6 training tasks. We updated Figure 5 for clarity. For held-in tasks (Appendix Figure 19), we get qualitatively similar results.

Which task is Figure 6 based on?

Figure 6 shows animal sequence results. We’ve updated the paper to clarify this.

Why are the experiment models not consistent through out the paper?

Due to compute constraints, we focused on the main cross-prediction experiments, which require many training runs (12 runs in Figure 5). Testing GPT-4 cross-prediction costs ~$8,000.

What does this imply for mixture-of-experts models?

Thank you, we’ve added this in the extended discussion appendix section A.1. We observe self-prediction advantages in both MoE models (likely GPT-4o) and non-MoE models (Llama 70B), suggesting this capability isn't unique to either architecture. MoEs may succeed because different experts often give similar outputs. Future work could investigate if this is true in open-source MoE models.

We appreciate your thoughtful feedback and we welcome further discussion on any points.

Dear Reviewer, as the discussion period is coming to a close soon, we wanted to follow up once more to check if we have addressed your concerns? We would be keen to use the remaining time to discuss improvements so that our paper could be better accepted.

Thank you for the detailed response. I have some follow-up questions:

Inconsistency in Task and Model Selection

I'm not questioning the statistical significance of the results but rather the selection of tasks across domains and the models used for these tasks. The lack of consistency in this selection makes it difficult to view the results as solid evidence for the claims. It is always possible to find a random set of tasks that produce favorable outcomes, hence the experimental design should demonstrate greater consistency.

Task Coverage Across Domains

Perhaps I missed it in the paper, but do you test all tasks across every domain? For instance, did you evaluate the nth-word or vowel-nth-character tasks on the reviews dataset, which was used for sentiment prediction?

Under-Training Concerns

Based on Appendix Figure 19, if the models perform differently on the held-in set, doesn't this suggest that the M2 models might be under-trained?

Recommendation for Consistency

I suggest running LLaMA 70B on all tasks for consistency. When the experimental models vary between experiments, how can we ensure consistency in the results? While I acknowledge the extensive experiments included in the paper, I still believe greater consistency is necessary to substantiate the intriguing claims made in this work.

Self-Prediction Advantage in MoE Models

Shouldn't MoE models exhibit less self-prediction advantage compared to non-MoE models?

Thank you for your thoughtful feedback. Let me address each point in turn.

Regarding the inconsistency of task selection: We acknowledge the valid concern that our positive findings could be attributed to our specific selection of tasks and response properties. To address this, we've included detailed performance breakdowns by task and response property in the appendix. In the main text, we've also explicitly noted which response properties failed to show evidence of introspective self-prediction.

For clarity on terminology: We use "response property" to refer to the property of the hypothesized behavior we ask the model to predict (e.g., "What would the third letter of your response be?"). "Task" refers to the object-level behavior (e.g., "Please complete the following sequence"), while "dataset" indicates the source of specific information that combines with the task (such as MMLU question sets or Wikipedia sentences).

On task coverage: We collected data on all logical combinations of tasks and response properties. Some combinations were naturally excluded where the response property didn't apply to the task – for instance, predicting whether a response would be even or odd only applies to tasks generating numerical outputs, not to tasks like next-word prediction.

Addressing under-training concerns: Your point about extended training potentially aiding cross-prediction is well-taken. We explored this in Appendix A5.6, which shows additional analysis of how M2's cross-prediction performance changes with increased training data. The diminishing returns observed for GPT-4o cross-predicting GPT-4 and Llama suggest that additional training data would not substantially improve cross-prediction.

Regarding model choice consistency: While our main experiments include all models for self/cross-prediction analysis, we did use a smaller subset for follow-up experiments due to compute limitations. We're currently working to include Llama in the intentional object-level shift experiment and will provide updates.

On MoE models: Our initial hypothesis was that mixture-of-expert models would show poorer self-prediction performance. However, the only MoE models in our study were OpenAI's closed-source models, limiting our ability to draw definitive conclusions about the impact of MoE architecture. Several possible explanations for our findings include:

1. Introspective self-prediction accuracy may be determined by general model capacity, with performance improvements outweighing MoE-related penalties
2. The introspection mechanism might operate on non-mixture layers, if only some layers use MoE
3. The routing could be consistent between object-level behaviors and their corresponding hypothetical questions

More rigorous comparisons using open-weight MoE models like Mistral and Mixtral would be good investigaitons for future work.

Do our responses above address your concerns?

It would be nice to see a trend where the self-prediction score decreases as the n increases.

Showing that self-prediction decreases as N increases would be interesting. It supports our self-simulation hypothesis – since simulating further hops of e.g. 3rd word is much harder than the 1st word.

Here are the results across 4 models. They are what we expect – there is a clear trend across all models. This chart is not in the paper yet – we had some parts of this result scattered across the appendix. Because the period for paper modification is over, we will add it in the camera ready version (if we get accepted). We thank the reviewer for their good suggestion.

So if additional training data does not close the gap between self-prediction and cross-prediction, does this mean the gap can be viewed as the extent to which introspective capabilities help? Or should it be considered as difficulty of distribution matching?

We recall two relevant experiments.

The data scaling experiment (Section A.5.6) shows that increasing cross-prediction training data doesn't close the performance gap, suggesting it is very difficult for M2 to match the distribution of M1.

The behavioral change experiment (Section 3.4) shows models can update their self-predictions when their behavior changes through finetuning, even without training on hypotheticals about the new behavior. This suggests the model M1 is not simply memorizing the distribution and is self-simulating.

Suppose a model M2 has the exact same parameters, architecture, hardware and gradients as the model M1. In this case, M2 matches M1 distribution exactly – it is an exact copy as M1. If M2 self-simulates, then it should have the same answer as M1. Then we do not expect M1 to have an advantage over M2 in predicting M1.

Our key point is - in practice it's extremely difficult for M2 to perfectly match M1's distribution. This explains how introspective capabilities help: model M1 can directly self-simulate to predict its behavior, while M2 must attempt to approximate M1's distribution externally.

We hope that we answered your question well and are happy to further clarify.

Thank you for taking the time to run the additional experiments. Seeing a clear trend across multiple models is definitely helpful. I really appreciate the explanation on the distribution matching. Therefore, I would like to increase my score. Great work by the authors!