Why In-Context Learning Models are Good Few-Shot Learners?

We consider the following baselines:

• ICL w/o adpt: The standard meta-learning setting, where meta-training is performed on all tasks without distinguishing between datasets. During meta-testing, no domain adaptation is performed, meaning that the 8/16/32 tasks are not utilized.
• ICL w/ adpt: The meta-training process is identical to that of ICL w/o adpt. While during meta-testing, the model adapts using 8/16/32 domain-specific tasks by fine-tuning all parameters (step = 5, learning rate = 0.0001, batch size = 8/16/32).
• M $^2$-ICL: Meta-level meta-training of an ICL model using the method introduced in Section 5.1. The domain adaptation process (i.e., the inner loop of meta-level MAML) is configured with step = 5, learning rate = 0.0001, and 16 tasks-for-adaptation per domain. Meta-meta-training and evaluation were conducted for 8/16/32 tasks-for-adaptation, respectively. During testing, the same adaptation process was applied.

Our implementation utilized an 8-layer transformer (without positional encoding) as the ICL model, with input features extracted by a ResNet-18 (512 dimensions). Although this model is relatively simple, it is sufficient to verify the effectiveness of meta-level meta-learning in improving ICL for real-world applications. The pipeline is generalizable, and the architecture can be replaced with more advanced models or adapted for other applications.

The results, presented in the table below and inlcuded in Appendix H of the revised manuscript, demonstrate that M $^2$-ICL significantly outperforms both ICL w/o adpt and ICL w/ adpt across all task settings. Specifically, adapting the ICL model with only 8 tasks often results in overfitting, while using 16 tasks shows limited improvement, and 32 tasks leads to marginal gains. In contrast, adapting M $^2$-ICL with just 8 tasks surpasses the average performance of other methods, and increasing the number of tasks for adaptation brings further improvements. Notably, ICL w/o adpt, ICL w/ adpt (32 tasks), and M $^2$-ICL (8 tasks) show comparable performance. These results highlight the effectiveness of the proposed meta-level meta-learning approach in enhancing few-task domain adaptation.

Meta-level curriculum learning:

For meta-level curriculum learning, there are existing works that demonstrate practical approaches with typical meta-learners on real-world problems. There are various strategies to determine which tasks should be learned in a given episode during meta-training. A common approach is knowledge-based curriculum learning, where task difficulty is defined using expert knowledge specific to the problem. For instance, [2] defines task difficulty based on interactions between semantically similar relations observed in a task sequence for continual relation extraction problems. Another approach is model-based curriculum learning, where task difficulty is determined based on the evaluation performance of the current model, as explored in [3][4]. More generally, to schedule task-sampling strategies in meta-learning, additional factors such as task diversity within a batch or other task-specific information can be considered, as proposed in [5][6][7]. These strategies can be extended to the pretraining of ICL models for real-world problems by replacing the meta-learner with the ICL model. We believe adopting such strategies could help construct an effective curriculum for complex, real-world datasets.

[1] Meta-Dataset: A dataset of datasets for learning to learn from few examples, ICLR 2020

[2] Curriculum-meta learning for order-robust continual relation extraction, AAAI 2021

[3] Curriculum meta-learning for next POI recommendation, KDD 21

[4] Progressive Meta-Learning With Curriculum, 2022

[5] Meta-Curriculum Learning for Domain Adaptation in Neural Machine Translation, AAAI 2021

[6] Adaptive Task Sampling for Meta-Learning, ECCV 2020

[7] The Effect of Diversity in Meta-Learning, AAAI 2023

Thank you for your comments and suggestions. We appreciate the opportunity to clarify and address potential misunderstandings in this discussion.

1. Numerous typos and a lack of clarity; add citations for MatchNet, ProtoNet and CNP in line 244.

We sincerely apologize for the typos that may have hindered your ability to fully engage with the paper. We greatly appreciate your careful reading and constructive feedback. In the revised manuscript, we have corrected the typos and added the citations for MatchNet, ProtoNet, and CNP on line 244.

2. lines 82–85 incorrectly claim that transformers are inherently permutation-invariant

This appears to be a misunderstanding.

First, in lines 82–85, the term “permutation-invariant” specifically describes “each layer,” referring to the combination of a self-attention module and a feedforward layer, which together form the basic building block of a transformer. Both of these modules process input as a set of vectors and are inherently permutation-invariant.

Second, a transformer becomes permutation-variant only when positional embeddings are added to the input, which introduces order sensitivity. However, using positional embeddings is not feasible when investigating a learning algorithm, as it would enforce a sequential structure on the training set. In this paper, we focus on using transformers as learning algorithms, where the natural training set does not exhibit sequential structure. Therefore, as mentioned in Appendix B and discussed in Section 6, positional embeddings are omitted. This approach aligns with the common practice in existing works aiming to understand ICL, such as [1][2][3]. [1] Transformers Learn In-Context by Gradient Descent, ICML 2023

[2] Transformers learn to implement preconditioned gradient descent for in-context learning, NeurIPS 2023

[3] Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning?, ICML 2024

3.line 453 mistakenly implies that all in-context learners are pre-trained, whereas studies such as [1] and [2] demonstrate otherwise".

This appears to be a misunderstanding.

Both [1] and [2] follow Eqn (13) to pre-train their ICL models. The key difference is that [1] and [2] restrict the pre-training prompts to consist of only a few examples per task, whereas typical LLM pre-training does not strictly adhere to this structure. Nonetheless, all these methods involve pre-training under the global objective described in Eqn (13), which can also be interpreted as a form of meta-training.

In contrast, our work introduces an additional layer of meta-training at the domain level, termed meta-level meta-learning. The relationship between ICL and M^2 -ICL is analogous to the relationship between a model trained with standard supervised learning and a model meta-trained using MAML.

[1] Sewon Min, Mike Lewis, Luke Zettlemoyer, & Hannaneh Hajishirzi. (2022). MetaICL: Learning to Learn In Context.

[2] Kirsch, Louis, et al. "General-purpose in-context learning by meta-learning transformers." arXiv preprint arXiv:2212.04458 (2022).

4. The related work section needs significant improvement. Rather than simply listing various meta-learning studies, it should focus on situating this work within the broader literature, especially by incorporating more relevant references on in-context learning and model generalization.

In our submitted manuscript, we had addressed existing works on various perspectives of understanding ICL in the Introduction. We had discussed a broader range of related works in Appendix A, including topics such as ICL, meta-learning, the expressiveness of neural networks, and the generalizability of deep learning. In these sections, we have explicitly outlined the connections between these works and the contributions of this paper.

Thanks for the valuable comments and suggestions! We are delighted that you like this paper.

W1. Explore ICL with deep architectures other than transformer

Indeed, as discussed in Section 6, exploring ICL with other deep architectures, such as SSMs and RNNs [1][2], is a worthwhile direction and has been the focus of some works.

In this paper, we focus on transformers as they are the most common model for ICL and are closely aligned with our primary goal: explaining the success of large language models (LLMs).

The results presented in this paper might be generalizable to ICL with other architectures. While the theoretical proof of expressiveness in Section 3 is specific to transformers, similar approaches for proving the expressiveness of label-aware set functions could potentially be extended to other architectures. Additionally, the empirical results in Section 4 and the proposed methods in Section 5 are likely applicable to ICL with other architectures, as they rely on deep-parameterized meta-learners rather than being strictly transformer-based.

[1] Mamba: Linear-Time Sequence Modeling with Selective State Spaces

[2] IS ATTENTION REQUIRED FOR ICL? EXPLORING THE RELATIONSHIP BETWEEN MODEL ARCHITECTURE AND IN-CONTEXT LEARNING ABILITY, ICLR 2024

W2. Typo

Thanks a lot for your careful reading. We apologize for the oversight and have corrected the typos in the revised manuscript.

Q1. Please elaborate why M^2-ICL without adaptation sometimes works worse than ICL

Please note that we cannot conclusively determine whether M^2 -ICL w/o adpt works better or worse than ICL w/o adpt. ICL is trained for the global objective (Eq. 13), while M^2 -ICL is trained for the post-adaptation objective (Eq. 8). Their comparison is analogous to comparing a model trained using standard supervised learning with one meta-trained using MAML on tasks drawn from the same training set. When tested on novel classes (unseen tasks) without adaptation, neither approach is expected to perform well.

When a few labeled data points are available for adaptation, reasonable strategies include:

• ICL w/ adpt: May perform well when sufficient data is available but is prone to overfitting when data is limited.
• ICL w/o adpt: Avoids overfitting but may struggle when more data is available.
• M^2 -ICL w/ adpt: Designed to address overfitting by modifying the pretraining process, making it better suited for scenarios with very few labeled data points.

The meta-level meta-learning approach is specifically designed for M^2 -ICL w/ adpt. M^2 -ICL w/o adpt is only an intermediate product, and its main role is to serve as a baseline for comparison with M^2 -ICL w/ adpt, to verify that adaptation benefits M^2 -ICL even with very limited data.

We have made minor modifications to the explanation of these results in the revised manuscript to improve clarity and understanding (line 475-477, 484-485, in blue color).

Q2. How to define the complexity of other tasks, such as classification tasks? Add some more experiments with regard to few-shot classification tasks (common)

The concern about real-world applications of the proposed strategies is reasonable.

Meta-level curriculum-learning:

For meta-level curriculum learning, there are existing works that demonstrate practical approaches with typical meta-learners on real-world problems. There are various strategies to determine which tasks should be learned in a given episode during meta-training. A common approach is knowledge-based curriculum learning, where task difficulty is defined using expert knowledge specific to the problem. For instance, [2] defines task difficulty based on interactions between semantically similar relations observed in a task sequence for continual relation extraction problems. Another approach is model-based curriculum learning, where task difficulty is determined based on the evaluation performance of the current model, as explored in [3][4]. More generally, to schedule task-sampling strategies in meta-learning, additional factors such as task diversity within a batch or other task-specific information can be considered, as proposed in [5][6][7]. These strategies can be extended to the pretraining of ICL models for real-world problems by replacing the meta-learner with the ICL model. We believe adopting such strategies could help construct an effective curriculum for complex, real-world datasets.

Meta-level meta-learning:

We have evaluated the effectiveness of meta-level meta-learning on a real-world few-shot image classification problem using the Meta-Dataset [1]. This dataset is particularly suitable as it contains multiple sub-datasets, each of which can be used to sample many few-shot classification tasks, making it naturally divisible into multiple domains for meta-level meta-training (Eqn (8)).

Following standard settings, we used the training sets of ILSVRC, Omniglot, Aircraft, Birds, Textures, Quick Draw, and Fungi during training. For testing, we used unseen datasets such as Traffic Signs, MSCOCO, and additional datasets like MNIST, CIFAR10, and CIFAR100. Each dataset is treated as a domain for meta-level meta-learning. We considered 5-way 5-shot tasks at the meta-level and 8/16/32 tasks-for-adaptation per domain at the meta-meta-level. The sampling of classes and images to form tasks, as well as the sampling of tasks-for-adaptation within a domain, was random.

We consider the following baselines:

• ICL w/o adpt: The standard meta-learning setting, where meta-training is performed on all tasks without distinguishing between datasets. During meta-testing, no domain adaptation is performed, meaning that the 8/16/32 tasks are not utilized.
• ICL w/ adpt: The meta-training process is identical to that of ICL w/o adpt. While during meta-testing, the model adapts using 8/16/32 domain-specific tasks by fine-tuning all parameters (step = 5, learning rate = 0.0001, batch size = 8/16/32).
• M $^2$-ICL: Meta-level meta-training of an ICL model using the method introduced in Section 5.1. The domain adaptation process (i.e., the inner loop of meta-level MAML) is configured with step = 5, learning rate = 0.0001, and 16 tasks-for-adaptation per domain. Meta-meta-training and evaluation were conducted for 8/16/32 tasks-for-adaptation, respectively. During testing, the same adaptation process was applied.

Our implementation utilized an 8-layer transformer (without positional encoding) as the ICL model, with input features extracted by a ResNet-18 (512 dimensions). Although this model is relatively simple, it is sufficient to verify the effectiveness of meta-level meta-learning in improving ICL for real-world applications. The pipeline is generalizable, and the architecture can be replaced with more advanced models or adapted for other applications.

The results, presented in the table below and inlcuded in Appendix H of the revised manuscript, demonstrate that M $^2$-ICL significantly outperforms both ICL w/o adpt and ICL w/ adpt across all task settings. Specifically, adapting the ICL model with only 8 tasks often results in overfitting, while using 16 tasks shows limited improvement, and 32 tasks leads to marginal gains. In contrast, adapting M $^2$-ICL with just 8 tasks surpasses the average performance of other methods, and increasing the number of tasks for adaptation brings further improvements. Notably, ICL w/o adpt, ICL w/ adpt(32 tasks), and M $^2$-ICL (8 tasks) show comparable performance. These results highlight the effectiveness of the proposed meta-level meta-learning approach in enhancing few-task domain adaptation.

[1] Meta-Dataset: A dataset of datasets for learning to learn from few examples, ICLR 2020

[2] Curriculum-meta learning for order-robust continual relation extraction, AAAI 2021

[3] Curriculum meta-learning for next POI recommendation, KDD 21

[4] Progressive Meta-Learning With Curriculum, 2022

[5] Meta-Curriculum Learning for Domain Adaptation in Neural Machine Translation, AAAI 2021

[6] Adaptive Task Sampling for Meta-Learning, ECCV 2020

[7] The Effect of Diversity in Meta-Learning, AAAI 2023

Q3. p_f in task generation

The $p_f$ in line 384 should be $p(x)$. We apologize for the notation error and greatly appreciate your careful reading. This has been corrected in the revised manuscript. As introduced in Appendix B.1 and Appendix E, we sample tasks (functions) from $p(\tau)$ and inputs from $p(x)$ to build a task $D_\tau$.

Thanks for the valuable comments and suggestions! We are glad that you find some parts of our paper commendable.

W1. The introduction section could be improved for better accessibility

Thanks for the kind suggestion. We have replotted Fig 1 to make the main idea more intuitive, and would further emphasize in later version of the manuscript. The revised parts in pdf are highlighted in blue.

W2&Q. Could the authors elaborate on whether the proposed methods can be adapted for use with real-world datasets? If so, what modifacations or considerations would be necessary to accommodate the complexities of real-world data?

Yes, our proposed methods can be applied for real-world datasets.

Meta-level meta-learning:

We have evaluated the effectiveness of meta-level meta-learning on a real-world few-shot image classification problem using the Meta-Dataset [1]. This dataset is particularly suitable as it contains multiple sub-datasets, each of which can be used to sample many few-shot classification tasks, making it naturally divisible into multiple domains for meta-level meta-training (Eqn (8)).

Following standard settings, we used the training sets of ILSVRC, Omniglot, Aircraft, Birds, Textures, Quick Draw, and Fungi during training. For testing, we used unseen datasets such as Traffic Signs, MSCOCO, and additional datasets like MNIST, CIFAR10, and CIFAR100. Each dataset is treated as a domain for meta-level meta-learning. We considered 5-way 5-shot tasks at the meta-level and 8/16/32 tasks-for-adaptation per domain at the meta-meta-level. The sampling of classes and images to form tasks, as well as the sampling of tasks-for-adaptation within a domain, was random.

We consider the following baselines:

• ICL w/o adpt: The standard meta-learning setting, where meta-training is performed on all tasks without distinguishing between datasets. During meta-testing, no domain adaptation is performed, meaning that the 8/16/32 tasks are not utilized.
• ICL w/ adpt: The meta-training process is identical to that of ICL w/o adpt. While during meta-testing, the model adapts using 8/16/32 domain-specific tasks by fine-tuning all parameters (step = 5, learning rate = 0.0001, batch size = 8/16/32).
• M $^2$-ICL: Meta-level meta-training of an ICL model using the method introduced in Section 5.1. The domain adaptation process (i.e., the inner loop of meta-level MAML) is configured with step = 5, learning rate = 0.0001, and 16 tasks-for-adaptation per domain. Meta-meta-training and evaluation were conducted for 8/16/32 tasks-for-adaptation, respectively. During testing, the same adaptation process was applied.

Our implementation utilized an 8-layer transformer (without positional encoding) as the ICL model, with input features extracted by a ResNet-18 (512 dimensions). Although this model is relatively simple, it is sufficient to verify the effectiveness of meta-level meta-learning in improving ICL for real-world applications. The pipeline is generalizable, and the architecture can be replaced with more advanced models or adapted for other applications.

The results, presented in the table below and inlcuded in Appendix H of the revised manuscript, demonstrate that M $^2$-ICL significantly outperforms both ICL w/o adpt and ICL w/ adpt across all task settings. Specifically, adapting the ICL model with only 8 tasks often results in overfitting, while using 16 tasks shows limited improvement, and 32 tasks leads to marginal gains. In contrast, adapting M $^2$-ICL with just 8 tasks surpasses the average performance of other methods, and increasing the number of tasks for adaptation brings further improvements. Notably, ICL w/o adpt, ICL w/ adpt(32 tasks), and M $^2$-ICL (8 tasks) show comparable performance. These results highlight the effectiveness of the proposed meta-level meta-learning approach in enhancing few-task domain adaptation.

Thank you to the authors for the response. I maintain my score to support the acceptance of this paper for its valuable empirical studies on ICL and the insights these studies provide into the learning processes involved.

W3&Q1. Define or quantify "optimality"

We have provided a formal definition of optimality in Appendix C of the revised manuscript.

We claim ICL model learns data-dependent optimal learning algorithms (DDOLA), which is different and weaker than (true) optimal learning algorithm (OLA).

OLA: Formally, given a finite training set $D_t=${$(x_i,y_i)$} where each sample is i.i.d.: $(x_i,y_i)\sim p(x,y)$, and a testing set $D_v=${$(x_j,y_j)$} following the same distribution, the optimal learning algorithm (OLA) is $g^*=argmax_{g} \mathbb{E}_{(x_j,y_j)\sim p(x,y)}[Prob[g(x_j, D_t )=y_j]]$. In other words, an OLA is a learning algorithm that makes the most “accurate” prediction given a training set and unseen target inputs from the same distribution. It is possible to know the optimal learning algorithm with prior knowledge of $p(x,y)$. For example, OLS is optimal for linear regression with Gaussian noise. In the paper, three types of tasks are generated in designed ways (see Section 4.1). For example, a MatchNet model with certain parameters (where all modules are identity mappings) is the optimal learning algorithm for Task Type I, and similar analyses apply to the other task types and meta-learners.

DDOLA: However, meta-learners do not have access to the true $p(x,y)$. They only learn to infer $p(x,y)$ from $D_t=${$(x_i,y_i)$} through meta-training, which inevitably introduces variance and bias, making them data-dependent. We define the best that a randomly initialized and meta-trained deep learner can achieve, given a specific meta-training set, as the DDOLA. This can be empirically approximated by meta-training a deep, randomly initialized MatchNet/ProtoNet/CNP on the corresponding meta-training set (for the three task types, respectively).

Thus, we claim that the ICL model learns DDOLA through meta-pretraining, as validated by comparisons with the approximations of DDOLA.

Q2. The meaning of "implicit" in this paper

The terms “implicit” and “explicit” describe the generalizability of a learning algorithm, as introduced in Section 4. An implicit learning algorithm is sensitive to the data distribution, while an explicit learning algorithm is insensitive. Most human-designed algorithms, such as OLS or kNN, can generalize to learn any function within a predefined family regardless of the example distribution. These algorithms can typically be expressed using equations and operations with few parameters left to be determined, which is why they are called “explicit.”

In contrast, implicit algorithms have minimal prior assumptions about the function family patterns and rely heavily on parameters that are determined in a data-distribution-dependent manner. Such algorithms are often difficult for humans to explain, hence the term “implicit.”

Thanks for the valuable comments and suggestions!

W1. Simpler and more intuitive explanations for theoretical background

Thank you for the insightful suggestion. To address this, we have replotted Figure 1 to make the main idea more intuitive. The revised figure emphasizes the core concept: starting with a proof that ICL with transformers is expressive enough to encompass typical meta-learners in the learning algorithm space, this paper demonstrates that ICL possesses meta-level deep learning properties. Consequently, deep learning techniques can be adapted to the meta-level to enhance ICL. The revised sentences in pdf are highlighted in blue.

W2. Lack empirical evaluation on real-world task of proposed method.

Thanks for the suggestion. We have evaluated the effectiveness of meta-level meta-learning on a real-world few-shot image classification problem using the Meta-Dataset [1]. This dataset is particularly suitable as it contains multiple sub-datasets, each of which can be used to sample many few-shot classification tasks, making it naturally divisible into multiple domains for meta-level meta-training (Eqn (8)).

Following standard settings, we used the training sets of ILSVRC, Omniglot, Aircraft, Birds, Textures, Quick Draw, and Fungi during training. For testing, we used unseen datasets such as Traffic Signs, MSCOCO, and additional datasets like MNIST, CIFAR10, and CIFAR100. Each dataset is treated as a domain for meta-level meta-learning. We considered 5-way 5-shot tasks at the meta-level and 8/16/32 tasks-for-adaptation per domain at the meta-meta-level. The sampling of classes and images to form tasks, as well as the sampling of tasks-for-adaptation within a domain, was random.

We consider the following baselines:

• ICL w/o adpt: The standard meta-learning setting, where meta-training is performed on all tasks without distinguishing between datasets. During meta-testing, no domain adaptation is performed, meaning that the 8/16/32 tasks are not utilized.
• ICL w/ adpt: The meta-training process is identical to that of ICL w/o adpt. While during meta-testing, the model adapts using 8/16/32 domain-specific tasks by fine-tuning all parameters (step = 5, learning rate = 0.0001, batch size = 8/16/32).
• M $^2$-ICL: Meta-level meta-training of an ICL model using the method introduced in Section 5.1. The domain adaptation process (i.e., the inner loop of meta-level MAML) is configured with step = 5, learning rate = 0.0001, and 16 tasks-for-adaptation per domain. Meta-meta-training and evaluation were conducted for 8/16/32 tasks-for-adaptation, respectively. During testing, the same adaptation process was applied.

Our implementation utilized an 8-layer transformer (without positional encoding) as the ICL model, with input features extracted by a ResNet-18 (512 dimensions). Although this model is relatively simple, it is sufficient to verify the effectiveness of meta-level meta-learning in improving ICL for real-world applications. The pipeline is generalizable, and the architecture can be replaced with more advanced models or adapted for other applications.

The results, presented in the table below and inlcuded in Appendix H of the revised manuscript, demonstrate that M $^2$-ICL significantly outperforms both ICL w/o adpt and ICL w/ adpt across all task settings. Specifically, adapting the ICL model with only 8 tasks often results in overfitting, while using 16 tasks shows limited improvement, and 32 tasks leads to marginal gains. In contrast, adapting M $^2$-ICL with just 8 tasks surpasses the average performance of other methods, and increasing the number of tasks for adaptation brings further improvements. Notably, ICL w/o adpt, ICL w/ adpt(32 tasks), and M $^2$-ICL (8 tasks) show comparable performance. These results highlight the effectiveness of the proposed meta-level meta-learning approach in enhancing few-task domain adaptation.

[1] Meta-Dataset: A dataset of datasets for learning to learn from few examples, ICLR 2020