Thank you for dedicating your time to give valuable comments on our paper. We especially find [1,2] are reasonable and interesting related works and we would like to discuss and credit to them in later paper.

Contrastive strategy, objective and hyper-parameters.

Reply. Please refer to the Common Reply 2.

Training and inference efficiency.

ConML is specially designed to be efficiently incorporated in the episodic training. As discussed at line 157~158, ConML introduces hardly no additional time-complexity, and the column "relative time" in Table 3,4 provides empirical verification. As for the memory complexity, we provide ablation study varying $K$. Please refer to the Common Reply 2. As for inference efficiency, ConML does not make any modification of the meta-testing procedure, thus as efficient as the original meta-learning algorithm. Thus ConML is feasible at scale.

Thank you for dedicating your time to give valuable comments on our paper.

The paper lacks validation on MetaDataset, which is a common large-scale dataset for few-shot learning tasks.

Reply. As mentioned in Common Reply 2, currently only very few effort has been made to tune hyperparameter settings for performance, so we did not try ConML on large-scale dataset or SOTA methods. According to the reviewer's kind advices, here we provide ConML's empirical results on a common large-scale dataset, MetaDataset [1], and SOTA meta-learning method, CAML [2].

Table 1 shows results on MetaDataset, experimented in the same model, training (on ILSVRC-2012 only) and evaluation setting following [1], and the ConML is introduced with the same setting as section 5.2. It can be observed that ConML also brings consistent improvement on MetaDataset.

Table 1. Few-shot image classification accuracy (%) on MetaDataset.

We also study ConML's performance on SOTA meta-learning method Context-Aware Meta-Learning (CAML) [2]. CAML is based on in-context learning, composed of a feature extractor and an in-context learner (transformer). The feature extractor is pretrained and frozen during meta-training, which ConML does not effect. ConML effect on the training process of the in-context learner following the way introduced in section 4. We follow the same experiment settings in [2], that the feature extractor is a ViT-based model with initial parameters provided by CLIP [9]. The in-context learner is trained on ImageNet-1k, Fungi, MSCOCO, and WikiArt, note that this process is called 'large-scale pretraining' in [8], while it is regarded as the meta-training process in ConML. We introduce ConML with $K=1$, $\pi=$ random half, $\phi$ is cosine distance evaluated by the output of transformer before the readout MLP. The other training and evaluation settings stay the same following the public code from https://github.com/cfifty/CAML. The results are provided in Table 1, where all settings are 5-way (n)-shot. We find that ConML-CAML's performance is consistently better than CAML's, proving that ConML is an algorithm-agnostic approach that can bring universal improvement.

Table 2. Performance comparison of ConML-CAML and CAML (Accuracy %).

There is a need for sensitivity analysis on certain hyperparameters, such as λ and the choice of similarity function for contrastive learning.

Reply. Please refer to Common Reply 2.

When sampling batches of tasks at each iteration, if the same class appears in both tasks, it raises questions about how discrimination is computed between tasks.

Reply. Yes, indeed. As we discussed in conclusion, as tasks are not likely to distribute equilaterally in the model representation space, batch-sampling strategy can certainly be detailed and improved. Currently ConML is statistically effective and left to be improved in many directions. In fact, the sample strategies can be viewed as a hyper-parameter setting about ConML, please refer to Common Reply 2 for more information.

[1] Triantafillou, E., Zhu, T., Dumoulin, V., Lamblin, P., Evci, U., Xu, K., ... & Larochelle, H. (2019). Meta-dataset: A dataset of datasets for learning to learn from few examples. arXiv preprint arXiv:1903.03096.

[2] Fifty, C., Duan, D., Junkins, R. G., Amid, E., Leskovec, J., Ré, C., & Thrun, S. (2023). Context-aware meta-learning. arXiv preprint arXiv:2310.10971.

[3] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., ... & Sutskever, I. (2021, July). Learning transferable visual models from natural language supervision. In International conference on machine learning (pp. 8748-8763). PMLR.

Thank you for dedicating your time to give valuable comments and advises on our paper.

Distinguishing ConML with conventional contrastive learning.

Conventional contrastive learning methods have already provided good insights to the representation learning and deep learning community. It has been successfully used to learn representations or encoders in both supervised and unsupervised learning. However, what can this insight brings to, and how can it perform successfully in meta-learning, have not been explored before. This paper brings the insight and practice of contrastive learning from (un)supervised learning to meta-learning with providing new insight and a practical and universal framework.

On the other hand, ConML does not seek for technical contribution to contrastive learning methods. How to improve meta-leaner's ability of fast-adaptation and generalization to diverse tasks is an essential problem for meta-learning, and ConML is a universal, effective and efficient solution. Inspired by human's learning capacity, ConML encourages meta-learners to perform alignment and discrimination, which is original. This is achieved by minimizing inner-task distance and maximizing inter-task distance respectively. For efficient and universal integration with the general episodic meta-training, ConML is intuitively designed with such a "sample and contrast" implementation. Please refer to Common Reply 1 for the primary contribution.

Comparing ConML with previous efforts to investigate the possible exploration of the task-level information for improved meta-learning.

Thanks a lot for your kind nomination of related works, and we would like to add to the paper to discuss and credit exploration of the task-level information for improving meta-learning. However, ConML is not comparable with existing efforts, but can be adopted beyond those methods, due to its algorithm-agnostic and versatile property. Focusing on the most general meta-learning setting on top of different meta-learning models and algorithms, as illustrated in section 3.0, ConML introduces an algorithm-agnostic objective to improve the episodic meta-training process. Thus ConML can be generally adopted to improve existing meta-learning algorithms. For [1,5,6] and [paper-ref.51,52,55], they constrain on a few-shot classification setting where there is a pool of base classes for meta-training where each class contains abundant data. Though they design different meta-objectives for each specific setting, ConML can be generally incorporated by replacing the naive validation loss in eqn.(4) with their loss. There are also works investigating like meta-training with curriculum [2,3,5] and meta-batch-diversity [7], thus resulting in certain batch sampling strategies in the episodic training. ConML can also be incorporated with their specific sampling strategies as discussed in conclusion.

Integration ConML with SOTA meta-learning models

As illustrated above, ConML focuses on the most general meta-learning setting on top of different meta-learning models and algorithms. ConML is algorithm-agnostic after defining the model representation $\psi(g())$. As the design of $\psi(g())$ is simple and intuitive, we show specification for representative meta-learning methods for above four basic categories. And experiments in Table 3,4 and 5 show the effectiveness for every category. As SOTA meta-learning methods can be viewed as enjoying a combination of the these basic categories, it is easy to integration ConML by specifying $\psi(g())$ for the method, without loss of generalizability of the effect.

Here we study ConML's performance on SOTA meta-learning method Context-Aware Meta-Learning (CAML) [8]. CAML is based on in-context learning, composed of a feature extractor and an in-context learner (transformer). The feature extractor is pretrained and frozen during meta-training, which ConML does not effect. ConML effect on the training process of the in-context learner following the way introduced in section 4. We follow the same experiment settings in [8]. The training of the in-context learner is called 'large-scale pretraining' in [8], while it is regarded as the meta-training process in ConML. We introduce ConML with $K=1$, $\pi=$ random half, $\phi$ is cosine distance evaluated by the output of transformer before the readout MLP. The other training and evaluation settings stay the same following the public code from https://github.com/cfifty/CAML. The results are provided in Table 1, where all settings are 5-way (n)-shot. We find that ConML-CAML's performance is consistently better than CAML's, proving that ConML is an algorithm-agnostic approach that can bring universal improvement.

Table 1. Performance comparison of ConML-CAML and CAML (Accuracy %).

The in-context learning experiments are only on simple synthetic data, lack of significance.

We study the effect of ConML on in-context learning on real data through, ConML-CAML, provided in above Table 1. We apologize for not being able to study on conventional NLP with training a LLM due to the limit of resources.

[1] MELR: Meta-learning via modeling episode-level relationships for few-shot learning

[2] Exploring intra-task relations to improve meta-learning algorithms.

[3] Exploring task difficulty for few-shot relation extraction

[4] Episodic-free Task Selection for Few-shot Learning

[5] Progressive meta-learning with curriculum

[6] Adaptive task sampling for meta-learning

[7] The effect of diversity in meta-learning

[8] Context-aware meta-learning

[9] Learning transferable visual models from natural language supervision

Sorry for the split of our rebuttal due to the character limit.

Why "alignment" is related to fast-adaptation? Isn't fast adaptation related to the number of steps to converge, not the number of shots?

Reply. From the perspective of gradient-based meta-learning methods, "fast" can be interpreted as "few steps" to converge. From a more general perspective of meta-learning, "fast" could be interpreted as "few-shot", for the following reasons: In real world application of machine learning, the accumulation of labeled data takes time; Apart from gradient-based methods, other meta-learning methods do not have the concept of "steps" for adaptation; Early meta-learning works take shots as sequential input (e.g., with RNN, memorization), thus temporal fast means short sequence, corresponding to few-shot.

"Alignment" is mapping the representations of a positive pair together in contrastive learning [paper-ref 48], thus be (mostly) invariant to unneeded noise factors. In ConML, this means aligning model generated by the meta-leaner with different subsets of the same task (line 140~143), by minimizing $D^{in}$ (eqn.(2)). This encourage the model generated with few-shot to be similar with fully-supervised ones. The meta-learner thus can learn noise- and bias-invariant task knowledge from few-shot. Experiment results in Fig.2(c) verifies "alignment" is related to fast-adaptation.

Why "discrimination" is related to the generalization?

Reply. "Discrimination" is maximizing intra-task distance, i.e., distance between negative pairs in contrastive learning (line). Meta-learning aims at improving the performance on unseen tasks. With a natural supposition that different tasks enjoy different task-specific models, discrimination, i.e., meta-learner can learn different models from different tasks, is important for task-level generalizability. Experiment results in Fig.2(f) verifies "discrimination" is related to task-level generalization. This objective also plays the role as an offset of the inner-task distance. In fact, this objective can be not optimized explicitly but through serving as negative pairs to optimize the softmax value in InfoNCE in Common Reply 2 and obtain better performance.

Illustration about experiment results and details.

Reply. Due to the page limit, details about experiment settings have been presented as short as possible (e.g., "following the same settings in [17]", "Following [18]"). We apologize for the difficulty in reading. However, we hope the omission of unmentioned details would not trouble the understanding of the experiment results, and code for all experiments is provided in appendix. We would like to provide full details in appendix later.

As for section 5.1, we provide analysis about statistical results comparing meta-training MAML with different meta-objectives (MAML, in-MAML, out-MAML, ConML-MAML). Result 1: Comparing Fig.2(a) and (d), which are t-sne visualization of model representations, along with the clustering performance in Table 2, ConML do align and discriminate the model representation as expected. Result 2: Fig.2(b) and (e) (x-axis: cosine distance value, y-axis: count) show the distribution of inner-task and inter-task distance respectively, meta-trained by the four variants. We can find that: the alignment and discrimination ability corresponds to optimizing inner- and inter-task distance respectively; the alignment and discrimination capabilities are generalizable; ConML shows the couple of both capabilities. Result 3: Fig.2(c) shows the testing performance varying shots number (x-axis: shot, y-axis: Relative Error (/MAML)). We find alignment is closely related to the fast-adaptation ability of the meta-learner. Result 4: Fig.2(f) shows the testing performance varying the distribution of sine parameter (x-axis: Distribution Shift, y-axis: Relative Error (/MAML)). We find discrimination is closely related to the task-level generalizability.

Ablation studies.

Reply. Please refer to Common Reply 2.

Thank you for dedicating your time to give valuable comments on our paper, especially for valuable advises about improving the presentation.

In the case of optimization-based methods, should the method need to proceed the episode optimization twice? Then why does it only take 1.1~1.5x time only?

Reply. To directly answer 'why does it only take 1.1~1.5x time only?': Though ConML introduces additional inner-update processes using different subsets, the processes are independent with each other thus can be parallel computed.

ConML does not introduce additional episode optimization procedure integrating with any meta-learning methods. We apologize for possibly leading to misunderstanding and want to re-interpret the proposed ConML here. Different meta-learning methods are unified and formulated in an episodic training manner. In each episode, the meta-learner function $g$ generates model $h$ with $\mathcal{D}^{tr}$, i.e., making adaptation according to $\mathcal{D}^{\text{tr}}$, then episode optimization is performed to minimize the validation loss of $h$ on $\mathcal{D}^{\text{val}}$ (eqn.(1)). ConML requires multiple subset samples from $\mathcal{D_{\tau}}^{\text{tr}}\cup\mathcal{D_{\tau}}^{\text{val}}$ to be processed independently by $g$, which are parallel computed inside an episode, to obtain the contrastive objective (eqn.(2)(3)). Then to perform the episode optimization, the contrastive objective and validation loss are added up (eqn.(4)), and minimized by gradient descent. The specifications of meta-learner function $g$ are provided in Table 1. In the case of optimization-based methods, taking ConML-MAML for example (Algorithm 6), the meta-learner function $g$ corresponds to the inner-update. ConML only requires additional inner-update subsets of $\mathcal{D_\tau}^{\text{tr}}\cup\mathcal{D_\tau}^{\text{val}}$, which are independent and parallel computed. Then ConML only need to proceed the episode optimization (global-update) once. Thus ConML introduces no additional time complexity, as illustrated at line 157.

In the case of metric-based methods, if class order is shuffled, the model presentation also changes?

Reply. Yes. Metric-based methods can be viewed as equipped with classifiers with class-order-aware weights. We represent the model with the weights which are class-order-aware. Taking ProtoNet for example, the classifier can be viewed as a linear model with order-aware weights (eqn.(8) in [paper-ref 39]). If class order is shuffled, we do need different model to make prediction, thus the the model presentation also changes.

In the case of ICL, does "consistent value in an episode" mean the same value is used for every task and contrast? Can using a feature after feed-forwarding a random input to the model be also used for the other settings, i.e. meta-learning methods?

Reply. The $u$ for every of the $B$ tasks in a certain episode, is necessary to be consistent to calculate inter-task distance. However, for tasks not in the same episode (batch), no contrastive objective among them would be evaluated. Thus $u$ in different episode can be different, while necessarily being consistent in an episode. Technically, obtaining model representation by input probing can be used for other meta-learning methods. However, we do not recommend this. Because it is less efficient and less expressive than model representation by explicit weights. It is less efficient because requiring additional feed-forwarding. It is less efficient because the difficult to cover the input space. We use explicit weight for other meta-learning methods while input probing for ICL, because we could not obtain task-specific explicit weights of ICL model.

In case of two tasks share almost the same classes (e.g. task 1 consists of class 1,2,3 and task 2 consists of class 1,2,4), should we consider these two tasks to be discriminated much?

Reply. Yes, Good idea. As we discussed in conclusion, as tasks are not likely to distribute equilaterally in the model representation space, batch-sampling strategy can certainly be detailed and improved. Currently ConML is statistically effective and left to be improved in many directions. In fact, the sample strategies can be viewed as a hyper-parameter setting about ConML, please refer to Common Reply 2 for more information.