Do Pre-trained Transformers Really Learn In-context by Gradient Descent?

We thank the reviewer for their review and request them to see our general response highlighting the changes that were made to the initial draft.

The authors make several arguments that do not prove the main thesis. the authors should explicitly say what is wrong in the work or setup of [0].

Please refer to our general response that clarifies our main thesis. As we clarify in the general response, our intention is not to prove the existing line of work wrong. Rather, we are studying the curious problem of whether the existing results stretch beyond the boundaries of their assumptions. This is clarified in our updated draft and we request the reviewer to take a look at it.

... authors claims that the because [0] made their analysis with linear regression then it cannot be comparable to a model trained with causal language modeling ...

We would like to clarify the difference between ICL and CLM objectives which has been detailed in our updated section 3.1. “CLM objective” refers to training of transformers on task-agnostic data. This general-purpose data does not specifically train transformers to perform ICL on a narrow family of tasks. In contrast, “ICL objective” refers to training of transformers exclusively and explicitly on the family of tasks, on which ICL is evaluated later. To gain insights into how ICL is achieved by LMs, choosing a simpler family of tasks like linear regression for analysis is perfectly reasonable, as long as the learning setup remains analogous to real world setups so that the conclusions can be reasonably extrapolated. Therefore, our problem is not with linear regression (the choice of family of tasks), rather how the transformers are trained and the misrepresentation of ICL in that setting.

… the authors should prove that the model trained with linear regression leads to a different solution

We have highlighted many fundamental differences between emergent ICL and the studied $\widehat{\text{ICL}}$ in previous works. We believe that these differences are reasonable evidence that they’re likely not the same. If one claims that they are indeed the same, we believe they should prove it.

the style of their article … adversarial against an emerging line of work

Thanks for sharing this concern. We certainly did not mean to appear adversarial to the prior work or dismiss them – these are excellent and innovative works. At the same time, the community in our academic discourse would benefit from added clarity on the implications of these works, which we worry may be misunderstood.

In our revision, we have tried to revise all the statements so that they’re objective and not over-claiming. If you feel that there is a place that would benefit from further precise statements, please let us know. Thank you!

font of some of the figures, consistency

We have tried to make the font larger and more consistent. Please let us know if anything else catches your eye.

We thank the reviewer for their time to review our paper and request them to look at our general response, which highlights changes made to the initial draft of the paper to improve its structure.

The proposed order sensitivity is interesting and yet requires more in-depth analysis ...

Beyond the comparison of the order sensitivity of ICL vs GD, in the appendix, we had results on the order sensitivity of variants of GD (like SGD and Adam) which provide a broader context. As suggested by another reviewer, we have brought these to the main text.

We also did ablations on the SGD comparison with varying batch sizes (added in the appendix of the updated draft), although it is unclear how the model would decide which order to use in-context examples in the formation of Akyürek et al., our comparisons show that ICL is still substantially more sensitive to order than SGD or Adam.

What if we use specific types of positional encoding to make ICL agnostic to the order of demonstrations?

We want to clarify that our objective with this paper is not to make ICL order-stable (although that’s an interesting problem to focus on). ICL does not necessitate order-stability as it emerges in real-world models like LLaMa or GPT3 that are shown to be sensitive to input order.

The LLaMA model (which we use for our experiments) uses relative (specifically, rotary embedding (RoPE, Su et al. 2022) positional embedding, which tends to be more robust to reordering context content. However, as our results in section 4 show, they remain order-sensitive. Whether there will be any future way of encoding information that is robust to order is an interesting goal for future research.

We review the key thesis of our work in the general response. Briefly, our work is motivated by curiosity about the generalization of the prior results (on an implicit equivalence between ICL and GD) to more realistic scenarios. To present our case, we show various forms of evidence that may indicate that ICL and GD (on full model or sub-model) work differently. For instance, the high variance of ICL performance wrt order permutations rather highlights it as a functional difference between ICL and GD.

... we can use the average prediction of many random orders ...

We note that our results, shown in section 4, have all been averaged across multiple runs where demonstrations and their order are randomly sampled. This is to make sure that any order-sensitivity behavior is not a one-off incident but average behavior over multiple runs.

The writing …

We regret that the writing of the paper was not up to the mark in the initial draft. We have fixed the mistakes the reviewer pointed out and hope they find the revised version much improved.

... the authors need to refute the probing experiments ...

Thanks for raising this question. First, we are not claiming to refute the theoretical formalism of Akyürek et al. However, taking their conclusions beyond the scope of their expressivity results needs extreme care.

As you hinted, Akyürek et al. use their “probing” experiments (Section 5 of their work) to show that a transformer's internal representations can be used to predict parameters or quantities that are necessary for implicit (internal) iterative optimization.

This cannot be taken as proof, for several reasons:

• The existing “probing” techniques are known to be problematic as they conflate “causation” vs. “correlation”. As the literature suggests, ”...the probing framework may indicate correlations between representations fl(x) and linguistic property z, but **it does not tell us whether this property is involved** in predictions of f” (Belinkov 2022).
• Even if we take their result as causal evidence, the probing experiment is done on quantities ($X^TY$), which could be used to perform many other algorithms (GD or otherwise). The question should not be whether a transformer representation can predict a solution, but how it is computed (the functional nature of the algorithm) – which is not covered by the probing setup.
• Last but not least, the authors use a Transformer that is already trained with ICL objective: “We take a trained in-context learner, freeze its weights, then train an auxiliary probing model”. Using the ICL objective has significant inductive biases (about the structure of the sequence it sees) which is different from real LLM training.

Belinkov, Yonatan. "Probing classifiers: Promises, shortcomings, and advances." Computational Linguistics 48.1 (2022): 207-219.

... come up with experiments to suggest that the model changes its internal mechanism across training ...

We do not claim that ICL is equivalent to any mechanism (GD or otherwise). With our evolution of weights result, we just want to highlight that the ability of LMs to perform ICL remains stable even with evolving parameters. This just means that any claim of equivalence should be with a mechanism that elicits a family of weights, rather than fixed sparse constructions.

Are the contextual examples concatenated with the test query…

We concatenate each ICL demo, including the test query, with ‘\n,’ which is similar to Gao et al'21. (they apply [MASK] in ICL for BERT). Our choice to fine-tune the model with a cross-entropy loss over the entire vocabulary was deliberate. This decision aligns with standard fine-tuning practices. While it's valid that placing more weight on relevant logits during training and inference could potentially refine the model's predictions, our primary goal was to evaluate the models in a setting that closely mirrors how they are commonly fine-tuned and deployed in real-world applications (for example, most of the works in the “instruction tuning” literature).

About overlap rate and cosine similarity?

In the figure, the purple dashed line represents the gap between two instances of ICL that vary solely in the order of demonstrations.

When the demonstration number equals 1, the overlap rate and cosine similarity metrics equal 1. This is because there is inherently only one possible order for a single demonstration. On the other hand, multiple orders are possible when the number of demonstrations exceeds one, introducing variability in the model’s output. In Figure 5, we include this comparison to show the impact that varying orders of demonstrations have on the model's ICL performance, as quantified by our three metrics.

We thank the reviewer for their detailed review and request them to see our general response highlighting the changes that were made to the initial draft.

Hypothesis under study in section 4

the authors assume equivalence between ICL and GD on the same model.

We have revised our draft so that the connection of Hypothesis 2 with prior work is clearer and we have included experiments on sub-model GD. Section 4 contains experimental results that involve (implicit) sub-models that are more comparable with the setup of Akyurek et al., and Oswald et al., i.e. hypothesis 2. Because we can not search over all possible subsets of the parameters, our experimental setup specifically focused on “intuitive subsets”` that were inspired by the construction of the earlier works: Von Oswald et al. hypothesized implicit model lies in $W_V$ of Transformers, while the probing experiments of Akyürek et al. suggest that this optimization happens in top layers (suggesting that initial layers focus more on representation learning).

We conducted an experiment where we only fine-tuned $W_V$ of a single layer, for different choices of higher layers (with ablations) in the LLaMa model in hopes of fine-tuning only the implicit model. Based on our notation (Figure 1), call this model $\widehat{\text{GD}}$. We then compare $\widehat{\text{GD}}$ to ICL on our three metrics. We found that the differences between the two remain significant. We have added this result in the updated draft (Figure 2).

Arguments against sparsity ... previous works simply aim to give an expressivity result on transformers

As we clarify in the general response, we are curious about the generalizability of these expressivity results beyond the scope of their assumptions. The previous “expressivity” works’ definition of ICL ($\widehat{\text{ICL}}$: our notation) is different from the emergent ICL in pre-trained transformers. Moreover, these works contain experiments and statements that imply stronger conclusions.

• In Von Oswald et al., there are a variety of experiments comparing ICL vs. GD and emphasizing the transformer weights with the hand-constructed weights. They conclude that: "... when training multi-layer self-attention-only Transformers on simple regression tasks, we provide **strong evidence that the construction is actually found**".
• You can see their stretching of the claims in the title of the published work: “Transformers learn in-context by gradient descent,” though a more accurate statement is “Transformers have the capacity to learn in-context by gradient descent”
• A similar weight comparison (implicit linear weight difference; section 4.1) is done by Akyürek et al, that implies these weights do emerge by training transformers. Although we agree that real models might find some dense counterparts of the simple constructions, more efforts should be made to find those families of weights.

These stronger conclusions, stretching beyond the boundaries of “expressivity,” have motivated our work. We provide a variety of empirical and theoretical arguments that interpreting ICL as [implicit] GD is not a foregone conclusion, and the community should view these results with a grain of salt.

Just to be clear, our intention is not to dismiss prior work such as Akyürek et al., Von Oswald et al., Dai et al., etc. They are important results, but we feel that the community would benefit from additional clarity about their implications.

Thanks for going through our detailed rebuttal!

... concerns about the exact hypothesis that the paper aims to verify in the experiments

As the reviewer already understands, previous works' claims are about the equivalence of $\widehat{\text{ICL}}$ and $\text{GD}$/$\widehat{\text{GD}}$. What we wanted to test with our experiments was whether this implies any equivalence between $\text{ICL}$ and $\text{GD}$/$\widehat{\text{GD}}$. Therefore, we performed two sets of experiments (Section 4) for comparing $\text{ICL}$ with $\text{GD}$ and $\widehat{\text{GD}}$.

1. In practice, $\text{GD}$ is implemented by fine-tuning the whole model on task-specific data, which is what we did. We found that $\text{ICL}$ and $\text{GD}$ behaved differently on our three evaluation metrics aimed at looking at functional equivalence. In fact, Dai et. al. in their experiments look at the same setting (whole model tuning) but only base their equivalence on raw performance which does not paint the whole picture about functional equivalence.
2. For $\widehat{\text{GD}}$, because it is hard to find where exactly would this hypothetical implicit model lie in a big "transformer" like LLaMa, we relied on the suggestion of reviewer uBmS to look for "intuitive subsets" of the model to consider as implicit models. We based our intuition on previous works which hypothesize the existence of these implicit models. Particularly, we used von Oswald et al.'s construction, which says that the implicit model lies in $W_V$ of the transformer, and Akyürek et al.'s probing experiments to guide us about which layers we should be looking at (deeper layers). Using these intuitions, we performed $\widehat{\text{GD}}$ updates in three different studies:
   • Fine-tuning $W_V$ in one of the deep layers (randomly choosing one of the last 4 LLaMa layers).
   • Fine-tuning $W_V$ in one of the middle layers (randomly choosing one of the layers in layers 16-20 of the 32 in LLaMa).
   • Fine-tuning all weights in 8 layers randomly selected out of the 32 in LLaMa.

In our updated section 4, we have presented results from all these experiments for $\text{GD}$ and our intuitive understanding of $\widehat{\text{GD}}$ when compared with $\text{ICL}$ and found that they function differently in all cases.

We agree with the reviewer that these comparisons for $\widehat{\text{GD}}$ are not exhaustive and do not invalidate the claim of equivalence completely, but as mentioned in our future work section, identifying implicit models used for GD in LLMs in a computationally feasible manner could be an interesting avenue of future research.

We thank the reviewer for their comprehensive insights and request them to look at our general response, which highlights what changes were made to the initial draft of the paper.

1. The GD is an ambiguous algorithm ...

…the proper claim to refute should be …

We are not suggesting that hypothesis 1 is what should be or can be proven. There are two hypotheses in our paper, one which is a more general notion of equivalence between ICL and GD, while the other is what previous works look at. We are just investigating whether the second implies the first in any way. In our general response and the updated draft, we have made the distinction very clear and request the reviewer to kindly read it.

2. Definition-1 and its relation to the strong claim, about experiment with sub-models (intuitive subsets of the parameters)

…authors compare ICL on the same model to GD on the same model with cross-entropy, which is inherently impossible to be equal…

... do not present experiments where only some parts of an LM is finetuned …

As the reviewer pointed out, it is infeasible to search over all subsets of sub-models to find if one corresponds to the hypothesized implicit model. At their suggestion, we have added experiments involving `“intuitive subsets” in Section 4. (Thank you for the suggestion!)

Our experimental setup specifically focused on subsets that were inspired by the construction of the earlier works: Von Oswald et al. hypothesized implicit model lies in $W_V$ of Transformers, while the probing experiments of Akyürek et al. suggest that this optimization happens in top layers (suggesting that initial layers focus more on representation learning).

We conducted an experiment where we only fine-tuned $W_V$ of a single layer, for different choices of higher layers (with ablations) in the LLaMa model in hopes of fine-tuning only the implicit model. Based on our notation (Figure 1), call this model $\widehat{\text{GD}}$. We then compare $\widehat{\text{GD}}$ to ICL on our three metrics. We found that the differences between the two remain significant. We have added this result in the updated draft (Section 4).

3. The GD part of the claim is a bit strong to be meaningful

... order sensitivity experiments are not related to SGD ...

We looked at the order-sensitivity for vanilla GD because that is what is studied in Von Oswald et al. We agree with the reviewer’s suggestion about highlighting the sensitivity study with SGD and Adam to show that even if Akyürek et al. allow for order sensitivity, ICL exhibits a lot more sensitivity than SGD/Adam. We have updated the order sensitivity figure (Figure 2) in the main text.

4. About the claim of Akyürek et al.

Akyürek et al. does not make the strong claim ...

While Akyürek et al. are more precise in their claims, there are phrasings that may lead to confusing conclusions of hypothesis 1. Most importantly, their use of ICL (training Transformers with ICL objective) is critically different from the emergent ICL in Transformers (the CLM objective on natural text). These two objectives elicit different families of transformer models ($\text{ICL}$ vs $\widehat{\text{ICL}}$, in our notation in Fig.1), as pointed out in our updated Background (section 2).

The misleading nature of conclusions is in fact evident in your own phrasing. As you suggested, Akyürek et al. argue “that Transformers can discover learning algorithms to achieve ICL **if it’s trained for ICL**”. The key assumption “... if it is trained for ICL” is easy to miss. Furthermore, if a model “is trained for ICL” is it really capable of ICL?! There is no work that shows the equivalence of this to the emergent ICL in real-world models like LLaMa. We have tried to clarify this distinction in our revised description in “background”. This is one of the first works that adopted the use of ICL objective and its framework has since been used in the several follow-up works that blindly equate $\widehat{\text{ICL}}$ with $\text{ICL}$.

Moreover, even Akyürek et al. seem to over-extend their conclusions' scope. For example, their section 4, which “aim[s] to explain ICL at the computational level by identifying the kind of algorithms to regression problems that transformer-based ICL implement,” is premised by saying that “it is these iterative algorithms that capture the behavior of **real learners**”, for which we did not find any concrete evidence. Or the title their published work is: “What learning algorithm is in-context learning?” though a more accurate title is “What learning algorithm can be simulated in-context by Transformers?”

Just to be clear, our intention is not to dismiss prior works. These are excellent and innovative works. Our hope is to bring in more clarity between the actual claims and the extend to which they can be generalized beyond their assumptions.

Next we address minor suggestions/questions:

Does the sparsity ratio changes from layer to layer of GPT-J?

The sparsity ratio does not change significantly for $W_K$ and $W_Q$ but drops a little for $W_V$ with deeper layers. If we believe the weights that simulate GD to be sparse (as in Von Oswald et al. and Akyürek et al.), this contradicts the probing experiments in Akyürek et al. which suggest that the optimization happens in higher layers. We have changed our sparsity figure (Figure 4) to reflect this, and also updated the appendix with more details.

Why do you switch between models GPT-J vs LLama?

While LLaMa is open-sourced, to our knowledge its intermediate training checkpoints are not released (which are used in Fig.3; notice that the x-axis is the checkpoint step). Therefore, we used a slightly smaller model (GPT-J, with ~6B parameters) with available checkpoints to compare the evolution of weights and ICL ability with training steps.

About SGD experiments:

We did not change the batch size which was fixed to 2. The reason is that, like Von Oswald et al., the batch size was equal to the total samples available in context and for Akyürek et al., the batch size is 1 by construction. On the suggestion of the reviewer, we do show an ablation with batch size = 1 for SGD/Adam in Appendix A. We shuffled the samples in all experiments and also did multiple passes (assuming multiple layers perform serial GD in the model).

1) The GD is an ambiguous algorithm

The new version is much clear in terms of argumentation, thank you!. But now the main argument of the paper has been substantially changed or had been made it clear to me. So, I am re-reviewing.

The argument the paper is refuting now is "Hypothesis-2 => Hypothesis-1" as written in the rebuttal. I am copying them here:

• H2: For a given well-defined task, there exist Transformer weights such that ICL is algorithmically equivalent to GD (whole model or sub-model).

• H1: For any Transformer weights resulting from self supervised pretraining and for any well-defined task, ICL is algorithmically equivalent to GD (whole model or sub-model).

Let me come up with three more hypotheses for clarity

• H1O: For linear Transformer resulting from meta training on the task family in H2, ICL is algorithmically equivalent to GD.

• H1A: For large enough Transformer resulting from meta training on the task family in H2, ICL is algorithmically equivalent to Bayes optimal decision rule

• H1LM: For an LM resulting from self-supervised pretraining on a large corpous, ICL is algorithmically equivalent to GD (whole model or submodel)

a) I think "H2 => H1" argument neither made/suggested by Oswald et al. nor Akyurek et al.

My understanding is that both papers shows H2 by theoretical constructions, and Oswald shows H1O and Akyurek shows H1A by experimentation. I think what could be problematic in these papers, and I think this is what the experiments of this paper tries to refute (H2, H1O/A) => H1LM. This claim is mildy suggest in Oswald paper, but not in Akyurek (my point in (4)). So, I would strongly suggest going back to this (refuting (H2, H1O/A) => H1LM) positioning, which is the way I understood at the beginning, or please clarify further.

b) To reiterate my worry in the previous (1): H1LM is too strong as GD is an unspecified algorithm. What succefully refuteed by the experiments of this paper is ICL cannot be specific form of GD with cross entropy loss on intuitive subsets of LM --- not all forms of GD.

So, overall what I delineate here still remains a major argumentation issue to me, I am happy to change my scores when this is fixed.

2) Thank you! This looks great!

3) Looks great!

4) Please see (1)

I really liked the new $ICL$ and $\hat{ICL}$ distinction in the notation.

• Ahn, Kwangjun, et al. "Transformers learn to implement preconditioned gradient descent for in-context learning." arXiv preprint arXiv:2306.00297 (2023).
• Li, Shuai, et al. "The closeness of in-context learning and weight shifting for softmax regression." arXiv preprint arXiv:2304.13276 (2023).
• Ren, Ruifeng, and Yong Liu. "In-context Learning with Transformer Is Really Equivalent to a Contrastive Learning Pattern." arXiv preprint arXiv:2310.13220 (2023).
• Zhang, Ruiqi, Spencer Frei, and Peter L. Bartlett. "Trained Transformers Learn Linear Models In-Context." arXiv preprint arXiv:2306.09927 (2023).