Learning to Learn with Contrastive Meta-Objective

Thanks a lot for your valuable comments and suggestions. We find most of them very objective, indicating the reviewer's accurate understanding.

W1. Memory consumption

Indeed, ConML introduces additional memory consumption which grows about linearly with K. We want to emphasize that $K=1$ is good enough all our experiment, and the memory consumption (about 2 times) is at the same scale with w/o ConML.

W2. Hyperparameter sensitivity

Our statement "not much tuning effort is required" is because all main results (Table 2 & 3) are obtained with the shared and primitive hyperparameters (lin 254), not even the best results we have witnessed (only corresponding to the red line in Figure 2). And in a large range of choice of hyperparameter, ConML brings improvement over baseline (gray line in Figure 2).

W3. $u$ for ICL with ConML

Please refer to Q1.

W4. Task distribution Problem

we think it is a very rational point and were taking it as future directions. As we have discussed in the manuscript, we mainly intend to provide the idea and investigate the role of contrastive meta-objective, and there are still many directions to further improve the performance, e.g., contrastive function, distance metric, inner-(subset) and inter-(batch)task sampling strategy.

The mentioned problem involves the inter-(batch)task sampling strategy we have discussed. Optimizing inter-task distance with the current random task batch sampling can effect sub-optimally in scenarios pointed out by the reviewer. It just works statistically. This might be improved by advanced task batch sampling strategy with task-level diversity/difficulty/similarity awareness and schedule, which have been studied in literature, e.g., [1,2,3].

[1] Adaptive task sampling for meta-learning. ECCV2020

[2] The effect of diversity in meta-learning. AAAI2023

[3] Towards task sampler learning for meta-learning. IJCV2024

W5. Broader scope in Meta-learning

Thanks for your kind advice. We will try to apply ConML on other meta-learning scenarios including RL as future work.

Q1. Choosing the "virtual input" $u$ for ICL.

First, we want to illustrate why $u=1$ meets ConML's goal of additional supervision by task identity without labeled data for generalization. Let's consider ICL for regression tasks $y=wx$, where the output of Eqn(4) is expected to be $wu$. ConML expect same $w$ can be learned from different subsets of the same task, and distinguishable $w$ can be learned from different tasks. However, $w$ is intractable in ICL models, so ConML use the output of Eqn(4) as an alternative, which is exactly $w$ with $u=1$. Note that above goal only requires $u$ to be constant, where $u=1$ is only an example and choosing $u$ to be any constant is feasible.

Now let's discuss our consideration of keeping $u$ consistent within an episode but sampled independently across episodes, while keeping $u$ consistent globally (e.g. $u=1$) already meets the goal of ConML. Note that in ConML algorithm, the procedure of calculating and optimizing inner- and inter-task distance is complete in each episode, independent to other episodes. So there is no constraint on $u$ to be consistent across episodes. From the principle of "Occam's Razor", we set $u$ free across episodes by sampling independently in each episode. From another perspective, technically we can not define a optimal constant as $u$, and keeping $u$ globally consistent might result in model collapsing with large $\lambda$.

Empircally, we have evaluated setting $u=1$ under the same metric as Table 5. We provide results in the following table (the relative gain of $u=1$ over independent $u$ across episodes is in bracket). We find that though not significant, such independent design makes general improvement.

Q2. Why minimizing $L_c$ makes $P_g$ symmetric

Thanks for your detailed reading! There might be a little misunderstanding. Technically, it is not an "assumption" but a solution to an optimization problem, which can be formally described as $argmin_{p(\tau),p(x|\tau)} E_{i,j,k \sim p^3(\tau), x_\tau\sim p(x|\tau)}[\frac{x_{i,1}^\top x_{i,2}}{||x_{i,1}||||x_{i,2}||}-\frac{x_j^\top x_k}{||x_j||||x_k||}]$. And it is easy to see the solution is $\forall p(\tau), s.t. p(\tau)=p(-\tau), p(x|\tau) =\delta(\tau)$. Then in the manuscript $P(X)=p(x|\tau)p(\tau)$ is symmetric.

Q3. Explanation of performance on Meta-Dataset.

We understand this concern may rise from similar reasons with weakness 4. We agree with such potential problem as discussed in "A reply to Weakness". But we think it does not holds here considering the setting of Meta-Dataset. Following the common setting of Meta-Dataset, we collect all meta-training tasks in ILSVRC, Omniglot, Aircraft, Birds, Textures, Quick Draw, Fungi, VGG Flower datasets for meta-training, then test on meta-testing tasks from both aforementioned datasets (in-domain) and additional Traffic Signs and MS COCO datasets (ood) to perform cross-domain meta-learning. That is to say, we are using the same commonly meta-trained learner when testing on different datasets. So the mentioned reason could not explain well.

We apologize for currently we could not explain the failure of ConML on fo-Proto-MAML testing on Birds. We will analyze this further to see if this keeps occurring with different hyperparameters and vision models if necessary.

Dear Authors,

Thanks for your reply. It has addressed my question to some extent. However, my rating remains the same primarily due to the concern about memory cost.

Best regards,
Reviewer Kgc2

Thanks for you appreciation and valuable comments. We especially appreciate your detailed reading.

Q1. Distance metric and contrastive loss

We would like to discuss the distance metric and contrastive function respectively.

For the choice of distance metric, we believe that the matching between the chosen distance metric and model representation is the key to further success. The model representation is a set of model parameters, whose role in the meta-learner need to be considered. For example, we can find that Euc distance performs much better than cosine in ProtoNet (Fig.3), since ProtoNet makes classification with Euclidean distance, and ConML contrasts the classifier's weights — describing the model's behavior more precisely. Although cosine works generally, it would be interesting to tailor distance metrics for various parameter types (e.g., classifiers, MLPs, CNNs, GNNs, Transformers).

For the contrastive function, InfoNCE generally obtains better performance than the naive $d^{in}-d^{out}$ (Figure 2), which we assume to be generalizable. While InfoNCE exploits same information as the naive loss does. The performance difference might be explained with the difference of gradient magnitudes, where InfoNCE matches the term of lower bound of mutual information between model representations. In most parts of this paper we use the naive $d^{in}-d^{out}$, to illustrate ConML more straightforwardly, and to decouple $d^{in}$ $d^{out}$ for detail investigation (Appendix E). As we have discussed in the manuscript, we mainly intend to provide the idea and investigate the role of contrastive meta-objective, and there are still many directions to further improve the performance, e.g., contrastive function, distance metric, inner-(subset) and inter-(batch)task sampling strategy.

Q2. inner-update steps in MAML in line 198 and 200

We acknowledge the concern appreciate the reviewer's carefulness. In line 198 and 200, we use $\theta-\nabla_\theta L$ just to show that the function of MAML as a meta-learner, is achieved by gradient descents. We only explicitly write one step for notation simplicity. ConML's framework allows, and our experiments in fact do, multiple steps inner-update for optimization-based meta-learners like MAML. Such simplified notation can be found in a lot of related papers, even in "Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks" (MAML) itself (sec 2.2).

Q3. $D$'s identity in line 196-231

Thanks for your kind suggestion. The $D$ in sec 3.3 can be various. For standard mini-batch episodic training (Algorithm 1), all $D$ in this part is $D^{tr}$. For ConML, considering the Algorithm 2 we introduced in sec 3.1, $D$ can be $D^{tr}$, $D^{tr}\cup D^{val}$ or $\kappa_k$. Section 3.3 is intended for specifying the general framework of ConML in sec 3.1 to different meta-learners. We hope readers consider this part based on above information, as a modularized part.

Q4. Appendix line 664-line 668 is repeated.

We sincerely apologize for the error and thank you again for your detailed reading. We will modify it in manuscript.

Thanks a lot for your valuable comments and suggestions.

Q1. Lack of evaluation of robustness under noisy or corrupted data

This is a very good point, as ConML is intended for meta-learning generalization.

We have provided the effect of ConML under distribution shift in Appendix E.

Here, to further evaluate the effect of task interference, we conduct experiments where a portion of tasks during meta-training was 'poisoned' by randomly disturbing the labels of all samples (support and query) independently. After meta-training with 10/20%-poisoned task set, we report meta-testing results on 5-way 1-shot MiniImageNet, and calculate relative performance difference (Relative Diff) as (Clean − Poison)/Clean. Results below show that the relative performance decrease is significantly smaller when using ConML. This shows that ConML improves the meta-learning robustness under noisy or corrupted data.

Q2. ConML's effect when task differences are subtle or overlapping

We would like to discuss by decomposing ConML into optimizing inner- and inter-task distance.

Optimizing inner-task distance benefits generalize to smaller support-size (as shown in Appendix E), would not likely to be effect by task-distribution.

Optimizing inter-task distance helps generalize to ood tasks (as shown in Appendix E). If the meta-training task differences are subtle or overlapping, generalizing to ood tasks becomes more important. This is because if not testing on ood tasks, we can use supervised learning (maybe a little extreme, but the effect of meta-learning itself degrades).

Q3.Computational overhead for optimization-based meta-learners.

In our experiments, we only use the weights in the last layer (linear classifier) as representation of optimization-based meta-learners.

This is a very rational concern. We choose model representation by the ‘essential/influential minimal set’ (EMS) of parameters that explicitly individualises the models. And for optimization-based meta-learners, they are inner-updated model weights. For meta-learners with very large EMS, we can selectively choose a subset. For instance, in MAML models that inner-updates all parameters, limited computational budgets may restrict us to contrasting only the parameters of the last few layers—this approach still achieves comparable performance. We also found using parameters solely from earlier layers fails, likely because the final layers capture more task-specific information. We would add illustration in our manuscript.

Q4.Discrepancy between MAML and other optimization-based meta-learners

In Section 3.3, we select one representative method for each category of meta-learners as example to illustrate how to get model representation. The principle of choosing the 'essential/influential minimal set' is shared in the category.

In line 198, we used MAML as example to show the principle for optimization-based is contrasting inner-updated model weights, which also applies to FoMAML and Reptile in Table 1. We have also provided detail algorithm procedure for incorporating ConML with each specific meta-learner in Appendix C.

Q5. Metric-based algorithms' performance

We assume the fact that Metric-based algorithms slightly down-perform optimization-based ones on miniImagenet, is likely due to their reliance on fixed embeddings functions which may not adapt as flexibly. Note that in Table 1, all metric- and optimization-based algorithms use the same Conv4 backbone as feature extractor, which is a relatively small model that would benefit from task-specific adaptation. Optimization-based ones adapt the backbone but metric-based ones fix it inside each task.

Indeed, metric-based naturally suited for classification tasks. This can also be observed by the strong performance of P>M>F in Table 3, which uses a more deep feature extractor and protonet as classifier. Note that it departures from standard protonet by additionally fine-tuning using support set to achieve task-specific adaptation, which might be an evidence of above explanation.

Q6. Explicitly clarify the relationship between in-context learning and meta-learning.

Thanks for your suggestion. In fact, we have discussed the relation between ICL and meta-learning and reformulate training ICL model as meta-training in Appendix D. Apart from these, we have also noticed a recent paper [1] exactly answers this with "ICL model is meta-learner with minimal inductive bias", we would add this to our manuscript.

[1]Why In-Context Learning Models are Good Few-Shot Learners? ICLR2025

Q7. Related works

We find the two works provided by the reviewer are highly relevant and very good works. We certainly would add them to references.

Thanks for your response. We apologize for previous misunderstanding.

Q2. This is a very rational concern. We would like to address it in two perspectives.

First, though the scenario raised by the reviewer that "contrasting similar tasks as negative pairs" worths special consideration indeed, it is not a special issue faced by ConML, but a common issue faced by general contrastive learning framework. For example, the widely studied standard (sample-level) contrastive learning [0] also draw samples from a i.i.d distribution $p^n(x)$, which could be similar but treated as negative pairs. But such contrastive learning generally works fine, which could possibly be explained as we are optimizing the relative distance where positive pairs should be much similar than negative pairs, and statistically if $\tau_1,\tau_2 \sim$ a continuous $p^2(\tau)$, then $P(\tau_1=\tau_2)=0$.

Second, indeed we can design better sampling strategies and we were taking it as future directions. As we have discussed in the manuscript, we mainly intend to provide the idea and investigate the role of contrastive meta-objective, and there are still many directions to further improve the performance, e.g., contrastive function, distance metric, inner-(subset) and inter-(batch)task sampling strategy.

The mentioned problem involves the inter-(batch)task sampling strategy we have discussed. Optimizing inter-task distance with the current random task batch sampling can effect sub-optimally in scenarios pointed out by the reviewer. It just works statistically. This might be improved by advanced task batch sampling strategy with task-level diversity/difficulty/similarity awareness and schedule, which have been studied in literature, e.g., [1,2,3].

[0] Understanding contrastive representation learning through alignment and uniformity on the hypersphere. ICML 2020

[1] Adaptive task sampling for meta-learning. ECCV2020

[2] The effect of diversity in meta-learning. AAAI2023

[3] Towards task sampler learning for meta-learning. IJCV2024

Q4. Thanks for your detailed reading. Table 1 is describing the meta-learner's behavior, where optimization-based learners are updating model parameter by gradient-descent, and thus to introduce how we obtain model representation. Line 198 is the output of optimization-based meta-learner: a predictive model $h$ with updated parameter, to better align our notations $g$ and $h$. We apologize for the confusing and will modify for better clarity in later version.

We are very glad, and appreciate it a lot to meet such a detailed and professional review.

Q1. Inner-task sampling times K=1

First we want to emphasize the meaning of K=1. Though the reviewer might have understood it correctly, the expression in question "only use one sample subset" might be a little confusing. It does not mean that the subset contains K-shot per class. It means in each episode, we sample K subsets from, and to contrast with, the complete set ($D_\tau^{tr}\cup D_\tau^{val}$), following certain sampling strategy $\pi_\kappa$. With the specification in line 254, in each episode we are contrasting model representation obtained from $D_\tau^{tr}$ with $D_\tau^{tr}\cup D_\tau^{val}$.

We acknowledge the concern about the potential weakness of using only one subset (K=1) for the contrastive objective. In our experiments, Our choice of K=1 was motivated by computational efficiency. However, we agree that increasing K could provide more robust supervision. We observed no significant performance gains when increasing K from Table 4. We infer this is because the large meta-training episode (iteration/step) number, generally over $10^4$, with randomly sampled tasks/subsets, mitigates and compensates for the effect of K in each episode. We would need a larger K for meta-training with only fewer episodes.

Q2. Model identity representation

The ‘essential/influential minimal set’(EMS) that explicitly individualises the models is a very nice summarization! For meta-learners with very a large EMS, we can selectively choose a subset. For instance, in large MAML models that update all parameters, limited computational budgets may restrict us to contrasting only the parameters of the last few layers—this approach still achieves comparable performance. However, using parameters solely from earlier layers fails, likely because the final layers capture more task-specific information.

We believe that the alignment between the EMS's role in the meta-learner and the chosen distance metric is crucial for success. For example, we can find that euc distance performs much better than cosine in ProtoNet (Fig.3), since ProtoNet classifies using Euclidean distance and ConML contrasts the classifier's weights—capturing the learned representations more accurately. Although cosine works generally, it would be interesting to tailor distance metrics for various parameter types (e.g., classifiers, MLPs, CNNs, GNNs, Transformers).

Q3. Grammar and typos

We apologize for the grammatical errors and typos. We will thoroughly proofread the manuscript and use LLM-based tools to ensure clarity and correctness in the revised version.

Q4. Related works

We will expand the related works section to include the suggested contrastive learning papers (Perera et al., Hiller et al., Doersch et al.).

Q5. Proper discussion of limitations

We will modify to add a formal "Limitation" section for proper discussion, including current limitation mentioned in sec 4.3.2 sec 6.

Q6. Checklist

Thanks for your kind reminding. We will modify and expand the justifications.

I'd like to thank the authors for the responses to my questions -- all of them have been appropriately addressed.

I will decide on my final rating once the other reviewers' rebuttal answers have been addressed and discussed.