Meta- (out-of-context) learning in neural networks

Answering questions

Validation data and “performance on in-distribution questions”

Both descriptions are correct – we always evaluate using validation subsets. We use “in-distribution” as a description of the question types: in-distribution validation data has the same questions types as the training data, whereas the entity attribution questions are never present in the training data. We’ll make this clearer in the caption of Figure 2.

How does Figure 3 show in-context learning? It seems like a comparison to the data from Figure 2 would be needed for that.

In-context learning is the result of the setup for Figure 3: in this experiment, definitions appear in the same prompt as the question which the model needs to answer, and so the model learns to use these definitions in the prompt to answer questions. Comparison with Figure 2a tells us that out-of-context learning is a weaker effect than in-context learning, since consistent definitions help less when the model was previously trained on them VS the definition being in the prompt. We’ll add a sentence about this to the paragraph on “comparison with in-context learning”.

Why is the performance in Figure 4a lower than in the main experiment?

See our response to question 2.1 from reviewer zUXo about the same point. We will change the question type to be consistent with Figure 2b (right now it is not) and re-run the experiment to check whether something else is affecting performance.

Would it be possible to further elaborate on why the authors believe pretraining is not necessary?

Neither hypothesis for explaining meta-OCL discussed in Section 4 relies on pretraining. Below we will attempt to formulate a version of the “selective retrieval” explanation for the set inclusion experiment.

During the first finetuning stage:

1. The first few epochs result in middle-layer activations of each variable token containing information on the definition that was present in the training data – specifically, information about both the define-tag token and the number present in the definition. This is easiest to imagine for the (variable, tag, entity) definition order, where the model is trained to predict the tag and the number from the variable.
2. In the remaining few epochs, the model learns to use this information when answering questions. When predicting the answer, the model outputs the variable’s definition number IFF the variable’s define-tag is D-dot. (Note how this kind of selective retrieval/output mechanism can achieve low training loss AND is representable by a neural net.)
3. Realistically, the above two steps are intertwined and affect one another, and are also intermixed with the model just memorizing the training data without connecting definitions to questions, so the conditional output rule is not learned perfectly.

During the second finetuning stage, middle-layer activations of new variable tokens end up being structured similarly: they also contain info on the define tag and the number that followed the variable in the definitions. The selective retrieval mechanism from the first finetuning stage is unchanged, and results in the meta-OCL effect.

Why is the effect size for the set inclusion experiment so small?

In this experiment, presence & strength of OCL & meta-OCL depends a) number of entity-variable pairs in the training data, and b) number of QA examples per entity-variable pair. Increasing a) helps the model learn the task but also makes the task more difficult since the model needs to learn more variable binding relations. Increasing b) helps the model learn the task as well, but makes definitions less useful as the model can infer more from the QA pairs.

Consistently with this, we found that when we increase the number of QA examples per entity-variable pair, the first stage performance becomes very good (it’s easy to push it to almost 100% accuracy), but any effect in the second stage disappears.

We believe it should be possible to tune these and other parameters (such as how many integers are present in each question) and attain a stronger effect; still, our experiments are sufficient to show that pretraining is not needed for meta-OCL.

Thank you for a thoughtful review and suggestions.

Addressing weaknesses

Is it not enough to say that neural networks generalize differently for true and false statements and that this holds for both unseen questions and unseen definitions? What is meta- about this?

The fact that neural networks generalize differently in our experiments means that the model updates differently when trained on consistent-seeming and inconsistent-seeming datapoints in the second finetuning stage. This difference in updates is why we call this phenomenon "meta-learning". Our results can be explained without mentioning meta-learning (e.g. the "selective retrieval" hypothesis in Section 4); still, we believe that meta-learning is an interesting and helpful frame for understanding what's happening.

In Figure 12, the authors show that they observe similar effects in a single-stage setting, indicating that appealing to meta-learning might not be needed.

Indeed it is harder to view single-stage results as meta-learning, since it's unclear if the effect there comes from just training on all the data jointly, or a meta-learning-like effect. We employ the two-stage setting to disentangle these two explanations.

In general, [many versions of the experiments] is a good thing. However, I found the presentation a bit strenuous. In many places, an explanation is given about the setting, but then the reader is referred to the SI/Appendix for the results.

Would it be fair to characterize your concern as "the references to the Appendix are too frequent"? We are wondering whether having less granular references to parts of the appendix would be helpful (e.g. instead of two references in “varying model size and experiments with other models”, have one reference to a larger appendix section relevant for both).

The related work section felt like quite a stretch. I found the discussed work not very relevant to the present paper.

We’d appreciate any suggestions for the works that would be helpful to cover, or comments on which discussed works you believe to be the least relevant. We could see how the gradient alignment part of related work could seem extraneous given that we do not study this hypothesis in depth – was this the part that led to your concern?

Thanks for your thoughtful review! Addressing your points:

A

We are not sure we understand your explanation of meta-OCL here; it sounds similar to our last paragraph of Section 4, “the model learns the semantics of the define tags correctly”. Is this right? We would appreciate it if you could clarify this, as well as your sentence starting with “Similar arguments” (if that’s a different point).

Restating why we believe “meta-learning” is a valid characterization of our results:

• The first finetuning stage modifies the way the model will update on new definitions in the future, such that these future updates are more advantageous for generalization. Thus it is meta-learning.
• Or in other words, in the second finetuning stage, the model updates on the two types of definitions differently and in a way that’s helpful for generalization. Thus it has learned how to learn.

B

We define internalization as "treating a statement as true". This means the model should behave as if it "believes" the statement "xyz is a pseudonym for Cleopatra" and respond accordingly. This operationalization isn't absolute or binary, but our experiments do show more internalization of consistent definitions in this sense.

We don't consider “predict the second word of the bigram (<variable> <entity>)” a reasonable measure of internalization as we mean to study it. It could serve as an operationalization, but we believe our operationalization better captures the kind of generalization we’re interested in, and in any case, any systematic difference in how the model treats $\mathtt{D}_5$/$\mathtt{D}_6$ examples that indicates it has internalized $\mathtt{D}_5$ more would be enough. This is still the case even if your prediction about bigram completion is correct, which it might be since these bigrams are contained in definitions, and definitions are memorized the same regardless of the define tag (see Figure 8). We can run these experiments, but only a result in the opposite direction (internalizing $\mathtt{D}_6$ more) would challenge our findings.

C: The effect of 'implicitly learning about variables from QA-pairs' is much larger than the OCL effect.

This is true, and we do draw attention to it in the last paragraph of section 2.3 (>It is notable that $\mathtt{EM(\hat{QA}_4)}$ is not that far off from $\mathtt{EM(QA_3)}$, so less performance is lost due to replacing entities with variable names than one could expect). The reason for this strong effect is high mutual information between QA pairs about the same variable: e.g. if a QA pair about someone’s birth century is in the training data, it is easy to answer the test question about their death century. We discuss this in the last two paragraphs of Appendix A1, and will mention it in the main text.

The reason we don’t focus on this is because the difference in the effect sizes of learning from QA pairs VS learning from definitions does not matter for our main point, which is about the difference between the effects of the two types of definitions.

(It's also worth noting that the model doesn't fully learn the meaning of the variables from the questions, as evidenced by the $\mathtt{QA_3}$ line staying at zero on the entity association test set in Figure 2b.)

D

The define tags are meant to correspond to the reliable/unreliable sources in the wikipedia/4chan motivating example. We’ll make this clearer in the paper by coming back to the motivating example throughout, thanks!

E

Briefly, we believe situational awareness could arise because the model is trained on content that is from trusted sources and includes facts about the model (e.g. about its training process). We’ll clarify this in the text.

F

Yes, we agree. This is a somewhat involved argument that may require familiarity with the related works that we discuss. We don't see a good way around this, but since other reviewers also didn't find this content clear and valuable as it stands, we could expand this discussion and move it to the appendix.

G

We agree; our claim is that differentially internalizing one definition type more than the other in the second finetuning stage is not useful for the training loss. We know that these relations can be memorized without being internalized, and this is what happens with inconsistent definitions (see the training losses for the two types of definitions in Figure 8 being ~identical).

I thank the authors for their extensive reply to my review.

Did you only reply to my review or did you limit the visibility of your replies to the other reviews? (I think ICLR reviews and rebuttals are meant to be publicly visible, and I would appreciate seeing all replies.)

The authors have responded to a subset of my concerns, and I thank them for the efforts made. I will increase my score to a 5 but not higher, given that I still think the paper is misleading/lacking clarity in a variety of places. The authors have promised to remedy some of my concerns, but have not (as far as I can see) made any updates to the paper. Most importantly, I would like to see the authors clarify differences between meta-OCL and standard meta-Learning, as well as clearly state the effect size of meta-OCL.

A & B

Is there any evidence in your paper that the model updates differently in the second finetuning stage? I.e. do you show anywhere that the gradients actually change? Or is there only evidence that it it answers questions, such as 'What is the name of {xyz}?' differently? You define this as internalization, but my concern B proposes a different definition of internalization, for which one (I would guess) does not observe meta-OCL. Even though I think your definition of internalization is very reasonable, think my proposed alternative in B demonstrates, that, in some other sense, the model 'learns' just as much about the inconsistent facts as it does about the consistent facts.

I think it's important to make sure that readers of this paper do not wrongly assume you show meta-learning in the classical sense of changing how parameters are updated, and that your results here depend entirely on your definition of internalization. I think the current draft is not clear enough on this.

C [...] will mention it in the main text

Thanks!

C [...] which is about the difference between the effects of the two types of definitions

I think it's important to point out how strong the meta-OCL effect is relative to the 'implicitly learning about variables from QA-pairs'-effect.

D

Thanks! I'm still not sure your experiment follow this motivation very closely. Where does the 'replacing entities with random strings' come from? If you follow the motivation, wouldn't you just have (define-tag, fact or falsehood) and (question, answer) pairs, and then facts for the 'trustworthy' define tag are the correct answers? I guess to some extent this is related to what you are doing. In any case, I think it would be helpful for you to discuss this more.

E

I would have liked to see this clarification during the rebuttal period. As for now, it seems rather speculative to me.

F

This seems like a good idea.

G

I think it would be important for you to clarify this in the main text.

K

I think the important part for in-context learning is that it happens 'in-context' and without any gradient updates. I disagree that your experiments here are in the spirit of Brown et al. (2020). I think calling your setup here in-context learning will be misleading to readers.

L

Thanks for your clarifications!

H, I, J

Thanks!

H: How do you go from 'gradients in a minibatch are aligned' to 'gradients between define=consistent statements and corresponding QA pairs are aligned'? If gradients in a minibatch are aligned, that minimatch also contains 'define=inconsistent' statements. Why are these not aligned?

We assume that those gradients are more “alignable” (and hence become more aligned), because the examples share a common explanatory factor (e.g. that xyz is a pseudonym for Cleopatra); ‘define=inconsistent’ statements will not have such a common explanatory factor with any QA pairs, by construction. This assumption draws on connections with meta-learning and invariant learning described in Related Work. For instance, in “Learning explanations that are hard to vary” (Parascandolo et al., ICLR 2021) the authors argue that ‘parameter values for which different examples have a similar local loss surface (i.e. similar gradients)’ correspond to a model providing a similar explanation for those different examples.

I

We added a small clarification (in bold) to the first sentence of the selective retrieval hypothesis paragraph where we use the word “rely”: >the model learns to rely on them [define-dot definitions] more (e.g. retrieve them more frequently).

J

Thanks a lot for these comments and suggestions! We are considering the following edits.

(J1) We tweaked the sentence introducing meta-OCL and pointed to Figure 1b there: >More surprisingly, we observe such a difference in internalization even for statements that are equally compatible with other training examples, i.e. statements about variables for which no questions appeared in the training set (Figure 1b). We will edit this further as we agree that clarity here is especially helpful for understanding the rest of the paper.

(J2) For meta-OCL, we believe Figure 2 results are referenced quite clearly in Section 2.4, and the subsequent sentences explain why this is meta-learning. We will add a similar sentence when referencing Figure 2 in Section 2.3, and briefly recap the description of OCL from the intro.

(J3) We will slightly improve the notation by replacing $\hat{\mathtt{QA}}_4$ with $\mathtt{QA}_4^{\text{not replaced}}$ or similar, and $\mathtt{QA}_7$ with $\mathtt{QA}_7^{\text{unseen vars}}$ or similar. We will also point out the connection between {$\mathtt{\dot{D}}_1\mathtt{QA}_1$, $\mathtt{\bar{D}}_2\mathtt{QA}_2$, $\mathtt{\dot{D}}_5$, $\mathtt{\bar{D}}_6$} with the columns of Figure 1.

Thanks for your review! We are glad you found the phenomenon we study interesting and our experiments sound. We address the weaknesses below.

1. Improving clarity

• Re abstract: we are considering replacing the second sentence of the abstract with

  Using carefully designed synthetic experiments with LLMs, we establish the existence of a phenomenon we call meta-out-of-context learning (meta-OCL): LLMs learning how to update on new datapoints in a way helpful for generalization but not explicitly encouraged by the training loss.

  Would such a change address your concern about the abstract’s readability?

• We believe that while our current operationalization of internalization (“model treats a statement as true when generating new text”) can be formalized somewhat, trying to write this down in math would not make our paper clearer. For example, we could say that statement A is epsilon-internalized when “generated statements implying NOT A are epsilon times as frequent as generated statements that do imply A”. However, this is still very abstract (and requires an oracle that can tell whether one statement implies another), and so we believe it's best to leave the current informal operationalization as is.

  We think there are other promising directions to study meta-OCL more formally, such as formalizing a datapoint's usefulness for modeling other data (see our response to reviewer zUXo's Q1.1).

• Our experiments with transformers cover both decoder-only and encoder-decoder architectures, as well as both finetuning these models and training them from scratch. We also present an experiment showcasing meta-OCL with a ConvNet. We would be quite surprised if it was not possible to reproduce meta-OCL with LSTMs or RNNs – do you expect otherwise? Given this, we believe it is fair to claim the quoted level of generality, as opposed to e.g. making that claim only about transformer-based language models.

• We agree that our setup is quite involved, and would like to improve our description of it. In response to your comment we will modify the caption of Table 1 to point out the connection between {$\mathtt{\dot{D}}_1\mathtt{QA}_1$, $\mathtt{\bar{D}}_2\mathtt{QA}_2$, $\mathtt{\dot{D}}_5$, $\mathtt{\bar{D}}_6$} data subsets and the columns of Figure 1. See also our response to reviewer zUXo about potential improvements to our notation – we’d appreciate any thoughts you might have on this.

2. We agree that our two hypotheses for explaining meta-OCL (selective retrieval and gradient alignment, both in Section 4) are far from being conclusive explanations; we note this in the Limitations part of Section 6. Since the phenomenon we study is quite different from in-context learning, our analysis is very different as well – are you saying it is similar? We discuss the relationship between out-of-context and in-context learning in the corresponding paragraph of the related work section.

3. The primary usecase of our work is to better understand what kinds of learning LLM are capable of. We do point out several implications of meta-OCL in Section 6, specifically that this capability could be part of a mechanism for how LLMs might acquire situational awareness, as well as learn skills such as using a sophisticated decision theory from just being trained on descriptions of such skills. As for more practical usecases, we believe a better understanding of meta-OCL is necessary before it would be possible to say that e.g. one can instill some desirable property into a LLM by leveraging this capability.

4. We are not familiar with setups that involve finetuning a LLM in a self-supervised fashion after it was already finetuned with RLHF, which is why we did not include this kind of experiment. Following your comment we did run this experiment with Llama-2-7b-chat-hf out of interest, and the results are very similar to what we report for regular Llama-2-7b in Figure 16.

Answering questions

1.1. Replacing the arguments about “truthfulness” with “predictability” preserves the meta-learning framing: in the second finetuning stage, the model internalizes datapoints that look like they might be more predictive of other data (but the model never saw such data of which these datapoints are predictive). We believe formalizing this mathematically is a good avenue for future studies of meta-OCL; a metric similar to predictability that we think is highly relevant is something like “how much does retrieving knowledge from a given datapoint helps reduce the training loss on other training data”, which is also similar to Data Shapley [1].

1.2. We do not make any points about “how many updates does it take for the congruent-Define to be learnt”. Rather, we claim that the fact that the model learns to bind the variable bgn to the entity Darwin from being trained on the string “congruent-Define bgn Darwin” can be seen as meta-learning, and would be surprising to a substantial fraction of the ML community. We believe the ability of LLMs to internalize semantic content of the text (as opposed to shallow token co-occurrence statistics) is currently under-appreciated, and our study is an attempt to highlight and better understand this ability.

1.3. We might be missing your argument here: indeed in $\mathcal{X}_1$ (first stage finetuning data) define-dot definitions are predictive of other training datapoints, while define-dash definitions are much less predictive. This is not the case in $\mathcal{X}_2$ – here both types of definitions are equally predictive. The purpose of comparing the two is to demonstrate that the model learns to update on datapoints resembling those that were more/less predictive in the past in a way advantageous for generalization.

2.1. Thanks for catching this – we’re not sure why the numbers are so different, indeed it is a bit surprising that the last 5% of $\alpha$ should make as much of a difference. We are re-running this experiment to see if we can catch the issue. We will also change the question type and point out the connection with Figure 2b explicitly, thanks!

2.2. Agreed!

2.3. A setting with a given $\alpha$ is equivalent to that with 1-$\alpha$, so we don’t expect $\alpha$<0.5 experiments to be different from the ones presented in Figure 4a. The only difference is $\mathtt{\dot{D}}$ and $\mathtt{\bar{D}}$ changing places in which one is the more consistent/predictive one; in our case the define tags are random strings that are re-sampled across multiple seeds, so it does not matter which one we call $\mathtt{\dot{D}}$ and which $\mathtt{\bar{D}}$.

2.4. and 2.5. We did not understand the sentence below and would appreciate clarifications:

And so in Figure 4a, the D9 number is fully explained by “you just have more data that shows that the incongruent-operator is a random map”, it goes down in the same way as it does in Figure 2.

We believe the $\mathtt{\bar{D}}^\text{consis}_9\mathtt{QA}_9$ result is interesting because for high $\alpha$, the define tag pushes the model to internalize definitions less (since, as you say, there is enough data to learn the define-incongruent meaning), while the actual definition consistency pushes the model to internalize definitions more (internalizing specifically these definitions helps predict the answers to the training questions). This setting is different from that in the first stage of Figure 2, where these two factors always push in the same direction. It is also different from the second stage of Figure 2, where only the define tag affects internalization (since there are no training QA pairs for the model to tell whether the definitions are in fact consistent).

3. See our comment regarding notation on Weakness 4.

4. Question type sharing does indeed provide base rates for the answers. E.g. for a test question “what did xyz do” the model’s answer would be informed not just by the definition and the QA pairs about xyz, but also by the answers to this type of question for other variables in the training data. This is not an issue for us since our results are all based on the relative differences between the various data subsets.

[1] Ghorbani, Amirata, and James Zou. "Data shapley: Equitable valuation of data for machine learning." International conference on machine learning. PMLR, 2019.

Thank you for a thoughtful and constructive review.

Addressing weaknesses

1. We will mention the variable binding explanation ("the model learns the semantics of the define tags correctly" from Section 4) in the last paragraph of the introduction, which will hopefully make our paper clearer for future readers.

2. Regarding experiments from section 3 (set inclusion and MNIST): any systematic difference, even if small, is a positive result here. We do agree that our MNIST setup is quite far from any real-life computer vision setting; it is however very similar to the set inclusion setup, and the presence of meta-OCL there indicates that this phenomenon is not limited to transformers. (You might also be interested in our response to question 4 from reviewer Rz8f about the factors affecting set inclusion experiment's performance.)

3. Your comment closely mirrors points E and F from reviewer Vaqj. Following these comments, we plan to expand the paragraph on functional decision theory and move it to the appendix, and add a sentence clarifying the connection of our work to situational awareness.

4. Regarding notation: we agree our notation is quite involved, and would like to make it clearer and easier to understand. One small improvement is to replace $\hat{\mathtt{QA}}_4$ with $\mathtt{QA}_4^{\text{not replaced}}$ or similar, and $\mathtt{QA}_7$ with $\mathtt{QA}_7^{\text{unseen vars}}$ or similar. We’d also love to improve on $\mathtt{\dot{D}}$/$\mathtt{\bar{D}}$, but we don’t have good ideas for how to do it given that we also want to keep both the subscripts (for data subset number) and the superscripts (for whether a definition is consistent). We’d appreciate any suggestions you might have here.