Understanding Prompt Tuning and In-Context Learning via Meta-Learning

We thank the reviewer for their feedback and suggestions and are very happy to hear how much they enjoyed reading our paper. Since the aim was to explain and show educational experiments, we particularly appreciate that the reviewer “learned a tremendous amount of new material”. Perhaps the main weakness, that all reviewers identified, is that the relevance of our results at frontier model scale (LLMs/VLMs and data such as text, images, or video) is hard to forecast. We have added this to our limitation section and significantly expanded our discussion section to provide additional details.

Overview of general changes/additions (across all reviewers):

• We discuss the limitations and difficulties with extrapolating our findings to LLM-scale, and some ideas how to approach this (we add to the initial limitations section at the end of the intro, and to the discussion section at the end of the paper). This issue was raised by all reviewers in some form, we added it to our response to HkrS, and kindly ask the other reviewers to read our response there.
• HkrS asked whether our results are brittle w.r.t. model size. We conducted an additional set of the main experiments (Fig. 1 and Fig. 3) with larger models, and findings are qualitatively equivalent. See our response to HkrS for the main numbers (we cannot provide the full plots that we added to the appendix).
• PHRz asked whether the superiority of Soft Prompting has a theoretical explanation. As speculated in L298-300, it may simply be the much higher dimensionality of embeddings vs. inputs in our case. We ran a control experiment (for Fig. 1 and Fig. 3) where we reduced the embedding dimensionality from 128 to 4, which largely cancels the superiority of ‘SoftPT’. Full plots were added to the appendix, the most important numbers are in our response to PHRz.

---

Specific response:

Weaknesses:

• [How would conclusions of the paper transfer to real-world experiments?] We have added another explicit limitation (end of the intro), and some discussion of this question to our discussion section (end of the main paper). See our response to HkrS, who raised a similar issue, where we show the full text that we added.

Questions:

• 1.a) This is correct - for the theoretical Bayes predictor there is only an input alphabet (of hard tokens). Soft tokens (i.e., real-valued inputs outside the alphabet) are outside this theoretical formulation, which makes it hard to make very general theory-based statements..
• 1.b) Unfortunately this is not easily possible. Even for SimplexPT, the inputs are not part of the alphabet of the idealized Bayesian predictor. What happens when we feed such inputs into an implementation of such a predictor depends on the implementation is unclear. E.g., we could implement a Beta-Binomial Bayes predictor in Python - what happens when we feed anything other than the tokens for ‘heads’ and ‘tails’ into this predictor is up to the implementation and not the Bayesian theory.
• Superiority of soft prompts: in our case, soft embedding prefixes are in a higher dimensional space compared to the input, which may give more degrees of freedom to “move” in off-distribution activation space and affect the internal representation of the predictor’s memory state. We have briefly speculated that this may be the reason for the superiority of ‘SoftPT’ in our case (L298-300). We have added a control experiment which confirms our speculation. We have added the following to the appendix and have clarified in our limitations section (end of introduction) why ‘SoftPT’ is superior to other soft prompt tuning methods in our setting.

“Results shown in Fig. A11 reveal that the superiority of 'SoftPT' over the other (soft) prefix tuning methods is largely explained by the much higher dimensionality of the embedding vectors, compared to input vectors (which are two-dimensional). In frontier models, the input dimensionality is typically higher than the embedding dimensionality, which could in principle result in soft input prefix tuning methods like `RealPT' outperforming embedding tuning. We leave the question of which prefix tuning method works best at frontier model scale to the large and active research community.”

Instead of the full Fig. A11 (which looks similar to Fig. 1 and Fig. 3), we provide here the most important information, which is the question how ‘SoftPT’ performance is affected by the change (compare with ‘RealPT’ in Fig. 1 and Fig. 3)

• 2a) There are different forms of meta-learning. Besides memory-based (which dates back a few decades see e.g. Learning To Learn by Thrun & Pratt 1998 and earlier work by Schmidhuber). Another common approach is MAML, where meta-training learns a shared parameter vector (a set of neural network weights) across tasks with the goal of being able to rapidly adapt (via SGD on the weights) to any particular task.
• 2b) Other papers, including Xie et al., have identified that in-context learning can be well described as the adaptation of a Bayesian predictor but not where this Bayesian predictor comes from in the first place (the connection to meta-learning theory).
• 2c) Yes, in principle LLM training can be viewed as implicit meta-learning (the notion of task may be very subtle, see Ortega et al. 2019 who has a discussion of implicit meta-learning). Perhaps more important than calling LLM pretraining implicit meta-learning is what we get out of this viewpoint.
• 2d) Strictly speaking, the Bayesian predictor arises at the endpoint of (perfect) meta-training. If the training process does not fully converge to that point, we have an approximate Bayes predictor but the theory allows us to say relatively little beyond that (whereas we can characterize the endpoint very well). Since it is not clear how close LLMs would be to this endpoint, we think it is important to add this disclaimer.
• 3. Thanks for flagging this. We have added more explanation to the main text and figure caption based on your questions. First, please see Fig. A7 a,b, which shows that the pretrained networks essentially use the vertical dimension in the plot (the first principal component) to keep track of the timestep of a sequence, and the horizontal dimension corresponds to the heads-to-tails ratio observed on the current sequence.
• 3a) The random coins trajectories are the traces of internal states that one gets by feeding in trajectories from the pretraining distribution (i.e., random coins). The colored single coin trajectories are sequences of internal states that one gets from feeding in sequences of the single coin with fixed bias (and prefixed by prompts tuned via the various methods).
• 3b) Since the trajectories of, e.g., SoftPT (dark blue lines) behave very differently from what one would get on pretraining sequences (gray lines), we can visually conclude that they are far off distribution for the Transformer (note that this cannot be seen visually for the LSTM, but because this is a 2D projection of a much more high-dimensional activation vector, it cannot be concluded from these plots that we are ‘on distribution’ for the LSTM).
• 3c) The red dots indicate the internal state of the neural network after it has consumed the prefix of length 6.
• 3d) Some differences in internal structure for different architectures are expected. What is interesting (Fig. A7 a,b) is that for both architectures the same two signals (number of steps and heads-to-tails ratio) dominate the variance of the internal state (the first two principal components). These are exactly the task’s sufficient statistics, and in this abstract sense the structure is very similar between both network types.
• 4. See lines 251-253 in the paper. These three predictors are different known (analytical) Bayesian predictors. For pre-training on ‘Random Coins’, PreBayes is a Beta-Binomial distribution. For fine-tuning to a single coin ‘TargetBayes’ is a Binomial distribution with the correct coin bias. For the Two-Coin Mixture ‘TargetBayes’ is a mixture of two Binomial distributions. ‘PreBayesPT’ results from taking the ‘PreBayes’ (Beta-Binomial), and prompting it with an optimal sequence of 6 hard tokens for the target task (which we find via exhaustive search).

Minor comments:

Thanks for pointing out the typo. Re longer sequences: the paper and main claims hold when considering results at n=49, which is where we have strict theoretical guarantees. We nonetheless think the additional results are important and interesting to get a sense of how robust the performance is for various tuning methods. E.g., it is conceivable that ‘SoftPT’ would produce very good performance up to the length for which the soft prompts were optimized (i.e. n=49), and then drops sharply because the off-distribution nature of the resulting internal activations rapidly “spirals out of control”, but we find this not to be the case.

We thank the reviewer for their feedback and suggestions, and are happy to hear that they enjoyed reading our paper and rated it as good on 3 out of 4 dimensions. Perhaps the main weakness, that all reviewers identified, is that the relevance of our results at frontier model scale (LLMs/VLMs and data such as text, images, or video) is hard to forecast. We have added this to our limitation section and significantly expanded our discussion section to provide additional details.

Overview of general changes/additions (across all reviewers):

• We discuss the limitations and difficulties with extrapolating our findings to LLM-scale, and some ideas how to approach this (we add to the initial limitations section at the end of the intro, and to the discussion section at the end of the paper). This issue was raised by all reviewers in some form, we added it to our response to HkrS, and kindly ask the other reviewers to read our response there.
• HkrS asked whether our results are brittle w.r.t. model size. We conducted an additional set of the main experiments (Fig. 1 and Fig. 3) with larger models, and findings are qualitatively equivalent. See our response to HkrS for the main numbers (we cannot provide the full plots that we added to the appendix).
• PHRz asked whether the superiority of Soft Prompting has a theoretical explanation. As speculated in L298-300, it may simply be the much higher dimensionality of embeddings vs. inputs in our case. We ran a control experiment (for Fig. 1 and Fig. 3) where we reduced the embedding dimensionality from 128 to 4, which largely cancels the superiority of ‘SoftPT’. Full plots were added to the appendix, the most important numbers are in our response to PHRz.

---

Specific response:

Weaknesses:

• [Educational experiments might not extend to real-world tasks]. We have added this as an explicit limitation and have expanded our discussion. Please see our response to HkrS for details.
• [Experiments are only conducted on small networks]. We have included an additional experiment in the appendix with twice the embedding dimensionality, twice the layer width and double the number of layers (two instead of one). Qualitative results are equivalent and all our main claims hold. The experiments mainly address HkrS’s question about how robust our results are to model scale. Please find more details in our response to HkrS. We do acknowledge that compared to frontier model scale, these networks are still “small”.

Questions:

• [How do we predict our experiments would translate to frontier models?] We have added this as an explicit limitation and have expanded our discussion. Please see our response to HkrS.
• [What actionable guidance do we have for practitioners?] Our original discussion section was meant to address this question, but rather than speculating we have formulated a number of hypotheses to investigate in the discussion. In particular:
  • Soft prefix tuning (SoftPT, RealPT, and modern methods based on it) should be superior to hard token tuning and traditional prompt engineering.
  • This may be particularly fruitful for in-context imitation learning, where one typically has small datasets available on which soft prefixes could be tuned, but the standard practice is to simply add a number of demonstrations to the context.
  • One take-away for practitioners based on our experiments could be to always use weight tuning (full weights or LoRA), as that yields the same or better performance as prefix tuning and does not suffer from its theoretical limitations. However, as we noted in the discussion this may be a wrong conclusion, as the two methods come with different practical implications and other works (e.g., Lampinen et al. 2025, Chan et al. 2022b) have found diminished neural plasticity after LLM pretraining, making weight tuning potentially less effective compared to prompt tuning.
  • Generally, we advise practitioners to compare prefix tuning and weight tuning, whenever possible, and pick which works best.
• [How does the Byaesian view on in-context learning explain the brittleness and sensitivity to non-semantic changes to prompts?] In short: these issues are outside our theory and there is not much that can be said about them from this particular angle (without significantly extending the theory). This does not invalidate the theory and our findings, it simply says that for certain (important) questions, the theory is not enough / not appropriate. For instance, we observed in previous work that network predictions and internal states on a coin-flip task are not fully invariant to the order with which samples are shown (an ideal Bayesian predictor only cares about counts of heads and tails, and is invariant to the order). This is not completely surprising, as learning perfect invariances over a large space of possible strings would be very difficult. So in practice, meta-trained neural nets can deviate from the ideal Bayesian predictor, and in these cases the idealized Bayesian theory cannot be used to make precise predictions about some very particular questions. Nonetheless, and despite these issues, our experiments show that the Bayesian theory is sharp and relevant on an abstract level, and theory tells us that, in principle, as models become even more expressive and optimizers get better, the theory becomes more relevant and accurate.

We thank the reviewer for their feedback and suggestions, and are happy to hear that they found our work very original and clear. Perhaps the main weakness, that all reviewers identified, is that the relevance of our results at frontier model scale (LLMs/VLMs and data such as text, images, or video) is hard to forecast. We have added this to our limitation section and significantly expanded our discussion section to provide additional details.

Overview of general changes/additions (across all reviewers):

• We discuss the limitations and difficulties with extrapolating our findings to LLM-scale, and some ideas how to approach this (we add to the initial limitations section at the end of the intro, and to the discussion section at the end of the paper). This issue was raised by all reviewers in some form, we added it to our response to HkrS, and kindly ask the other reviewers to read our response there.
• HkrS asked whether our results are brittle w.r.t. model size. We conducted an additional set of the main experiments (Fig. 1 and Fig. 3) with larger models, and findings are qualitatively equivalent. See our response to HkrS for the main numbers (we cannot provide the full plots that we added to the appendix).
• PHRz asked whether the superiority of Soft Prompting has a theoretical explanation. As speculated in L298-300, it may simply be the much higher dimensionality of embeddings vs. inputs in our case. We ran a control experiment (for Fig. 1 and Fig. 3) where we reduced the embedding dimensionality from 128 to 4, which largely cancels the superiority of ‘SoftPT’. Full plots were added to the appendix, the most important numbers are in our response to PHRz.

---

Specific response:

Weaknesses:

• [Experiments only on synthetic datasets.] We have added this explicitly to the limitations section at the end of the introduction and to our discussion section at the end of the paper. Please see our response to HkrS for the precise text that we added.
• [The Bayesian lens does not tell anything new about how prompt-tuning works.] We respectfully disagree. Our theoretical analysis, centered on the Bayesian view, reveals novel limitations of when prefix-tuning can and cannot work regardless of the tuning method, which we confirm to hold in simple experiments, but the theory holds at any scale (incl. LLM scale and beyond). We are not aware of any previously published work predicting or providing an explanation of why prompt-tuning to any single task works but prompt-tuning to any mixture of two or more tasks is at best suboptimal. Additionally, the Bayesian view emphasizes that prompting works via manipulating the sufficient statistics of a Bayesian predictor (in practice in ways that go beyond strict statistical conditioning, e.g., when using real-valued prefixes), which we have not seen spelled out in the literature before, and which we illustrate by recording how the internal state of networks is manipulated by various kinds of prompts. We thus used the Bayesian lens to tell us quite a bit new about how prompt-tuning works on a very fundamental level, and to a smaller degree mechanistically.
• [There are many ways to implement HardPT.] This is correct, and in general hard token tuning is a challenging problem with many published methods. We avoid this problem in our study by performing “exhaustive search” (L138, L260 in the paper). This means that we evaluate model performance for every possible hard-token prefix (2^6=64 possible hard prefixes in our case). Accordingly, the prefix we find cannot be improved with any other method of hard-token tuning.

Questions:

• [Is 1000 steps of pretraining enough?] Yes. We investigate pretraining curves for all our experiments (and they can be easily produced with the released code), and the 1000 steps were chosen via pilot experiments to suffice for convergence in all experiments with a comfortable margin.
• [Rename ‘Embedding-dimensionality’.] We agree the notation is a bit awkward, but it has the advantage that it does not need any further explanation / symbol. Since we only use it once in the paper, we have left it as is for now, but will change it if the reviewer feels strongly about it.
• [How was HardPT implemented?] As stated above (and L138 in the paper) via exhaustive search over all 64 possible binary hard token sequences of length 6.
• [Input dimensionality vs. embedding dimensionality.] Yes, the input dimensionality is 2 (binary inputs, i.e. two-dimensional one-hot encodings of ‘0’ and ‘1’). We pass inputs through a first linear projection (with learnable parameters) to compute an “embedding”.
• [What is the main takeaway from Fig. 1?] The main takeaway from Fig. 1 is that the pretrained network can be successfully prompted to behave like the Bayes-optimal predictor on the target distribution (TargetBayes) from the very beginning of eval sequences onwards. The effect of optimal prompting thus becomes very similar to weight-tuning on the target task in this case. We have added this main takeaway to the figure caption.

We thank the reviewer for their feedback and suggestions, and are happy to hear that they enjoyed reading our paper and rated it as excellent on 3 out of 4 dimensions. Perhaps the main weakness, that all reviewers identified, is that the relevance of our results at frontier model scale (LLMs/VLMs and data such as text, images, or video) is hard to forecast. We have added this to our limitation section and significantly expanded our discussion section to provide additional details.

Overview of general changes/additions (across all reviewers):

• We discuss the limitations and difficulties with extrapolating our findings to LLM-scale, and some ideas how to approach this (we add to the initial limitations section at the end of the intro, and to the discussion section at the end of the paper). This issue was raised by all reviewers in some form, we added it to our response to HkrS, and kindly ask the other reviewers to read our response there.
• HkrS asked whether our results are brittle w.r.t. model size. We conducted an additional set of the main experiments (Fig. 1 and Fig. 3) with larger models, and findings are qualitatively equivalent. See our response to HkrS for the main numbers (we cannot provide the full plots that we added to the appendix).
• PHRz asked whether the superiority of Soft Prompting has a theoretical explanation. As speculated in L298-300, it may simply be the much higher dimensionality of embeddings vs. inputs in our case. We ran a control experiment (for Fig. 1 and Fig. 3) where we reduced the embedding dimensionality from 128 to 4, which largely cancels the superiority of ‘SoftPT’. Full plots were added to the appendix, the most important numbers are in our response to PHRz.

---

Specific response:

Weaknesses:

• [Synthetic data too simplistic; models too small; how to extend the study to real world tasks]. We have made the following additions to the paper.

Addition to limitations section (end of intro): “This paper discusses fundamental properties of prefix tuning, which we illustrate with educational experiments where the focus is on clarity and being able to compare against a tractable Bayesian predictor. Accordingly, our datasets do not capture the full complexities of, e.g, large-scale language, vision, or robotics tasks. Similarly, our neural networks are small by modern standards, which means that our findings must be very cautiously extrapolated towards modern frontier model scale. Further rigorous and well-designed scientific studies are necessary to bridge the gap between our current work and modern large-scale ML practice, and we are optimistic that our fundamental results will inspire the design of such studies.”

Addition to the beginning of the discussion section: “While we have laid important fundamental groundwork in our current study, extrapolating our findings to modern frontier model (and data) scale is not straightforward. While the theoretical findings we presented, including the limitations of prefix tuning, hold at any scale, it is likely that additional practical issues arise at scale that are not captured by our current small-scale experiments. It is thus hard to predict the relevance and impact of our fundamental results on today’s frontier-model practice. For instance, one of our main results is that optimal prompting to a single target task is possible, whereas it is not for a mixture of tasks. We are confident that this holds even at frontier model scale (based on the theory), but it is unclear and highly non-trivial what constitutes a task for a LLM, and accordingly, whether this is a severe limitation or not. In our experiments, a task is simply an unobserved variable in a two-level hierarchical statistical model. At LLM scale, the structure is vastly more complex, with many more hierarchical levels, and potentially other statistical structures at play. The next step would be to carefully design data generators that are closer to natural language data, but still fully understood and well controllable, akin to the data generators used, e.g., in the `Physics of LLMs’ paper [1], and run our experiments of tuning to single tasks vs. tuning to mixtures of tasks at scale. With these caveats in mind, cautiously extrapolating our findings to frontier model scale, raises some questions for investigation [the orig. discussion section then follows...]”

• [How could we get TargetBayes for real world tasks?]. For many real-world tasks, all kinds of data modeling techniques are used to approximate TargetBayes. As models and optimizers get better, these approximations will get better. Note though, that we do not necessarily need TargetBayes: different prompt- or weight-tuning methods can be meaningfully compared against each other w.r.t. their cumulative log loss on a held out evaluation set. It would just be unclear how far these methods are from the optimum.
• [Is the result pattern consistent with other model sizes?]. We have observed the qualitative pattern underlying our main results very robustly across pilot experiments with different model sizes. To be rigorous, we have conducted an experiment with larger models, and have added the following to the appendix:

“Fig. A12 shows results for increasing the network size. Compared to the main experiments we double the embedding dimensionality (128->256), the width of layers (128->256), and the number of layers (1->2) [called LSTM Maxi and Transformer Maxi in the table below]. Qualitatively, our main claims hold. Particularly, that prefix tuning can be used to optimally adapt the pretrained predictor to the Single Coin target distribution, but cannot be used for perfect adaptation to the Two-Coin Mixture task. Anecdotally, we have observed our main results to hold robustly, as long as the network size and number of training and tuning steps is large enough. For too small networks, or networks trained or tuned too little, results become more inconsistent. From a theoretical perspective, too small networks violate the realizability condition, and networks with too little training violate the convergence condition. The theory thus does not apply to predict outcomes in this regime.”

Instead of the full Fig. A12 (which looks similar to Fig. 1 and Fig. 3), we only provide the most important information in the rebuttal: whether ‘SoftPT’ can be used to reach optimality or not? The answer is true for ‘Random -> Single’ but not for ‘Random -> Mixture’.

• Formatting issues: thanks for pointing out these typos. We have fixed them in our updated manuscript.

[1] Allen-Zhu, Zeyuan, and Yuanzhi Li. "Physics of language models: Part 3.1, knowledge storage and extraction." arXiv preprint arXiv:2309.14316 (2023).