Is Pre-training Truly Better Than Meta-Learning?

Q1: The paper is solid from the empirical point of view, presenting a large variety of results. However, it is somehow limited in terms of novelty as it does not introduce any new techniques or analyses.

A1: We respectfully differ. For instance, we have introduced a novel method of comparing two algorithms via the effect size, which we believe is more robust to sample size than p-values and confidence intervals (as described in Section 3.2). Furthermore, our paper also introduces a Task2Vec-based diversity coefficient that measures the task diversity of a given few-shot benchmark. We further present a detailed effect size-based statistical analysis of performance that gives substantial support to our claim that meta-learning outperforms pretraining if the data diversity is large enough.

Q2: A significant limitation of the paper lies in its reliance on two methodologies, MAML and PT with fine-tuning restricted to the head, which may not represent the current best practices in the field. Firstly, while MAML is undoubtedly foundational in meta-learning, its relevance has waned over time. Contemporary advancements have introduced more efficient derivatives, such as MAML++ (Antoniou et al., 2018). Incorporating comparisons with these modern variants could have enriched the paper's insights. Secondly, the paper's approach to fine-tuning is notably narrow, focusing only on the head's parameter adjustments. Contrarily, cutting-edge methods today, like BiT (Kolesnikov et al., 20202), fine-tune the entirety of both body and head parameters, while others, like FiT (Shysheya et al., 2022), selectively adjust a subset of body parameters. A juxtaposition against these state-of-the-art techniques would have been insightful. These oversights are critical as the paper's primary conclusions might shift when evaluated against more contemporary, optimized methods.

A2: We see this paper as a direct response to existing work that claims that meta-learning methods are worse than pre-training approaches, using equivalent methods to ours ( https://arxiv.org/abs/2003.11539 ). We propose the use of better experimental methods to allow a fair comparison between the techniques and show a reversal of what are now folklore conclusions. We note that the use of MAML as opposed to other meta-learning methods furthers our claim, since MAML is the simplest meta-learning approach. If MAML can outperform pre-training, better meta-learners should be able to do so too. Please find more details about this in the appendix, section A.6, paragraph 2.

Q3: The presentation of data exclusively in tabular format, though beneficial for transparency, hinders a quick understanding of the trends. I recommend the authors to enhance data representation by incorporating visual aids, such as scatter plots. This would facilitate a more intuitive grasp of the data patterns. The tables could be conveniently relocated to the appendix to maintain thoroughness without overwhelming the main content.

A3: Thanks for the suggestions. We have moved the detailed results to the appendix, leaving only tables providing a concise and easy to understand summary for our results in the main paper. We have additionally added plots indicating the spread of effect size for each of our 2 categories of diversity. Hoping this makes our results clearer and easier to understand.

Q4: There are some formatting issues, e.g. (i) Table 6 should be within the body of the paper, (ii) in the text "Meta-Data set" should be replaced with "Meta-Dataset"

A4: Thanks for pointing this out. We have fixed these issues in the revision.

We thank Reviewer RzB7 for the insightful comments. Below is our response:

Q1. No novel methods are introduced in this paper, which itself is ok if the results are intriguing.

A1. We respectfully differ. For instance, we have introduced a novel method of comparing two algorithms via the effect size, which we believe is more robust to sample size than p-values and confidence intervals (as described in Section 3.2). Furthermore, our paper also introduces a Task2Vec-based diversity coefficient that measures the task diversity of a given few-shot benchmark. We further present a detailed effect size-based statistical analysis of performance that gives substantial support to our claim that meta-learning outperforms pretraining if the data diversity is large enough.

Q2. Presentation of the results is very poor. All results are just listed in tables, and there are no attempts to present the results in a comprehensible/impressive manner. Since I could not find any meaningful insight from the tables, I recommend adding more comprehensive figures which should be contributions of the paper…

A2. Thanks for the suggestions. We have moved the detailed results to the appendix, leaving only tables providing a concise and easy to understand summary for our results in the main paper. We’ve added box plots and violin plots (Figure 1) to address this issue. We have additionally added plots indicating the spread of effect size for each of our 2 categories of diversity. Hoping this makes our results clearer and easier to understand.

Q3. The results of Cohen's d can be caused from (1) meta-training dataset, and/or (2) meta-test dataset, in addition to learning algorithms, but I could not find which factor causes the results of Cohen's d. In other words, which diversity of meta-training datasets or meta-testing datasets makes the difference between the two methods?

A3. We use the meta-training dataset to compute the Task2Vec diversity for each few-shot benchmark, as we believe that the data diversity of the meta-training should be the influencing factor that allows the meta-learner to meta-learn better. Since each few-shot task for meta-training and meta-testing is sampled from the same few-shot benchmark, the diversity of the test and training splits should be more or less the same.

We thank Reviewer myHv for the insightful comments. Below is our response:

Q1. Choice of MAML model: the original MAML model has been developed significantly in recent years. It would be fairer to that class of methods to compare to one of these many developments, given they have demonstrated (in general) better performance.

A1. We aim to counter a rising belief that meta-learning approaches are always worse than pre-training approaches. To do so, we pick the simplest meta-learning approach, MAML, and show that it can perform better than pre-training in high diversity settings. This is sufficient to counter the belief, since if MAML can outperform pre-training, so can the other meta-learning algorithms.

Q2. A discussion of the drawbacks of effect size -- it is useful to understand where this may be inadequate (except the fact that one must choose a threshold level). Relatedly, the paper states that standard effect sizes are 0.2, 0.5, and 0.8, but it is unclear how the choice of 1% relates to these.

A2. We acknowledge the reviewer’s point that it is important to understand the limitations of using any statistical test. One of the major limitations of using effect sizes is that they can be misleading if considered in isolation of other metrics such as meta-test accuracies. We present these values in the appendix, section A.2. We further note that, as with any other statistical test, we need to define arbitrary thresholds for the interpretation of effect sizes. While the common values are 0.2,0.5,0.8, we have used the 1% threshold as a decision rule based on it being common to accept papers due to 1% differences in machine learning. We have provided a detailed description of our choice of the 1% threshold in the appendix, section A.6, paragraph 3. Note that the choice of 1% doesn’t affect our primary results, in which we report average of raw effect size values.

We highlight and answer the aforementioned issues in detail in Section 3.2 of our paper.

Q3. Presentation of experimental setup: given this paper is primarily related to empirical benchmarking, a summary of the experimental setup in the main paper would help contextualise the investigation. This includes summary information about the datasets, abbreviations used (which reflect the results tables), training setup etc.

A3. We have noted the hyperparameters and experimental setup in Section 4 in the revision (re-arranged from the appendix). While the information about datasets and abbreviations are still mentioned in the appendix, we have moved the detailed results section to the appendix, leaving only tables providing a concise and easy to understand summary for our results in the main paper. We have also added plots indicating the spread of effect size for each of our 2 categories of diversity. Hoping this makes our results clearer and easier to understand.

Q4. Clarity in results exposition: when reporting the overall effect size, it is unclear how these are obtained -- my guess is an average over the cases where that hypothesis was determined correct. This is not mentioned ... - it would help to do so.

A4. Thanks for catching that. It is indeed the average over effect sizes that resulted in the same decision. We have added a brief explanation to the caption of table 1.

Q5. Clarity in results table: The tables could be better presented, for example by having the captions clearly specify that MAML5 refers to 5 adaptation steps, and having the same formatting for the heading and the main body (like all uppercase). Also, the detail on seeds in Table 3 is hard to understand -- what are seed1 and seed1 vs seed2 comparisons? The caption mentions 5 layer CNNs but the table refers to Resnet12 and Resnet50? Overall, the results are interesting but the ...

A5. Thank you for pointing these errors out. We have fixed all aforementioned errors in the revision. We have additionally moved the detailed results to the appendix, leaving only tables providing a concise and easy to understand summary for our results in the main paper. We have also added plots indicating the spread of effect size for each of our 2 categories of diversity. Hoping this makes our results clearer and easier to understand.

Q6. Training to convergence: ... If models experience some sort of overfitting, perhaps it makes sense to do early stopping?

A6. All datasets used in this paper have a sufficient number of examples per class for our models to be resistant to overfitting. We further back this claim by noting that we do not see poor test performance as should be expected for overfitting. Since our models do not overfit, it doesn’t make sense to implement early stopping, which would also otherwise hinder our ability to do a fair comparison between the two techniques. In addition, fitting the data perfectly is a standard procedure for ResNet & vision models on classification tasks.

Q7. Minor: ... use vs rapid learning

A7. We have added the citation in, thanks for the catch!

We thank Reviewer ZnA1 for the insightful comments. Below is our response:

Q1: Can the authors elaborate more on why p-value & confidence interval would become zero on larger batch size?

A1: As the batch size increases, our sample size gets larger, and we tend to see that the confidence interval becomes zero (which can sometimes misleadingly suggest that the differences between two groups is significant - see https://www.nature.com/articles/s41598-021-00199-5). As addressed in Section 3.2, effect size is less affected by sample size which alleviates this drawback.

Q2: Please elaborate on the main principle behind a new test (i.e., its underlying assumption regarding the distribution of performance difference between two algorithms + what can we say about the confidence in rejecting the null hypothesis)

A2: Section 3.2 addresses why effect size is a good metric (compared to confidence intervals) to compare MAML and PT. In summary, effect size provides a way of quantifying the difference between the two algorithms - if the absolute value of the effect size is greater than the calculated standardized 1% threshold, we reject the null hypothesis. The effect size also quantifies the magnitude of the difference - the standard values are 0.2, 0.5, and 0.8 for a small, medium, and large effect size difference.

Q3: For high diversity setting, I think the statistic test should be based only on the best model architecture. Please consider revising this part of the evaluation protocol

A3: Our goal in this paper is to provide a fair comparison between pre-training and meta-learning, and to provide empirical evidence to the fact that meta-learning outperforms pre-training if the data diversity is large enough. We define a fair comparison as using the same model, same optimizer and training to convergence. We hence provide comparisons with different models and with different data diversities to conclude that high diversity and small architectures should lead to better meta-learning performance. With bigger architectures, the difference between meta-learning and pre-training is less pronounced due to the possibility of pre-trained models to meta-overfit.