Few and Fewer: Learning Better from Few Examples Using Fewer Base Classes

We would like to thank the reviewer for their positive and constructive comments. In the following, we address their concerns and questions to the best of our ability.

I somewhat doubt that in cases where the domain gap is so large that it (significantly) harms transfer learning, fine-tuning on a subset of these classes would help?

We would like to thank the reviewer for raising this point. Indeed, there exist obvious conditions where the domain gap can be reduced by selecting a subset of base classes, for example if the target task is precisely this subset of classes. In conditions where the target distribution is far from the base distribution, such as Omniglot compared to ImageNet, the intuition behind the approach is not as evident. Nevertheless, our experiments show that, even in the case of this large domain gap, the proposed methodology can significantly improve performance over the baseline, providing at least empirical evidence to support the idea that even in those cases fine-tuning on a subset of classes would help. We enriched the Methodology part of the main paper to reflect these points.

[...] wouldn’t then a reasonable baseline to compare to be the network ‘just’ trained on the subset of classes from scratch?

We would like to thank the reviewer for suggesting this experiment, which indeed makes a lot of sense. We chose fine-tuning rather than training from scratch due to the computational overhead, especially in the Task-Informed setting. We ran a number of experiments to compare with training from scratch in the Domain-Informed setting (Table 3). While Omniglot did show an improvement when trained from scratch, most domains achieved worse accuracy compared to fine-tuning. However, this may simply be because the hyper-parameters were optimized for the large base dataset. In our view, this does not detract from the finding that fine-tuning on a subset of the base dataset improves the representation for a given task. We added these results to the appendix.

The presented results are somewhat constrained to the authors’ own setup [...]

We have run new experiments using logistic regression instead of the NCM classifier on top of our existing backbones. This approach is supposedly close to SOTA according to a recent paper A Closer Look at Few-shot Classification Again by Luo et al. Our results (Table 2) show a very similar trend to those we obtained using the NCM classifier. As such, we see them as strong evidence that the proposed methodology could benefit other more sophisticated approaches, as it is orthogonal to most other works in the domain.

Missing ablation: The authors note that they choose the top 50 classes (AA) in the informed settings to create the subsets, mainly to keep ‘similar size of subsets’ to the uninformed settings; [...]

We added an experiment varying the value of M (the size of the class subset) to see its influence on performance. Results are presented in the Appendix in Figure 2. It can be seen that any value of M between roughly 20 and 80 can lead consistently to improved results on most datasets, showing robustness in the selection of the optimum M. We agree that this value is likely dependent on the base dataset. We observed that clusters of about 50 classes obtained in the UI setting were typically quite coherent. It is not necessarily expected that such clusters would be obtainable on any base dataset. We also showcased the value of M obtained by using a cumulative total (activation) mass reaching a proportion of 90%, to illustrate that this method would indeed lead to a good selection in our setting.

[...] the random subsets seem to be surprisingly better when using AA, FIM and RH – [...] why this might be the case?

We currently do not have an unequivocal explanation. We are investigating this and we think it might be related to the fact that choosing between feature extractors specialized on meaningful subsets (X) can actually lead to worse outcomes than some feature extractors generalist subsets (R).

[...] Semantic features generally provide better ‘upper bounds’ [...]; Do the authors have some intuition whether that’s due to the ‘more powerful’ underlying CLIP training, or whether semantics (text) generally provide richer features for such selection? [...]

This is a very interesting phenomenon. We believe the two hypotheses you propose to be relevant. We think one way to answer this question would be to compare the visual and semantic features of CLIP (which allegedly have features of equal “representational power” ).

[...] However, doesn’t MCS actually have quite high compute requirements? [...]

We changed the text to better reflect what we meant. In short, the UI setting is particularly interesting for cases where latency is constrained, as it does not require finetuning a specific backbone. Yet, among all heuristics, MCS is more costly than others, and as such there is open room for improvements in the heuristics for future work.

I would like to thank the authors for their detailed response and additional information.
While I do think this work provides interesting insights that are somewhat orthogonal to most other works, I do share the concerns that it is difficult to directly generalize/make use of the findings beyond the investigated settings in a robust manner.
$\rightarrow$ I will hence stick with my initial rating.

We would like to thank the reviewer for their comments. In the following, we address their concerns and questions to the best of our ability.

It would be beneficial if the authors could establish the optimality of their method, specifically by showing that it can identify the best 50 classes from a pool of 1000 ImageNet classes. Alternatively, if reaching the optimum is unfeasible, the paper should strive to provide approximate solutions that approach the upper bound.

it would be more insightful if the upper bound, representing the optimal 50 classes for each task, could be explored and discussed

What is the optimal solution? And how to approach it?

The paper, while presenting a method for selecting base classes and achieving better performance with fine-tuning in few-shot scenarios, lacks a strong sense of novelty. It would be beneficial if the authors could establish the optimality of their method, specifically by showing that it can identify the best 50 classes from a pool of 1000 ImageNet classes. Alternatively, if reaching the optimum is unfeasible, the paper should strive to provide approximate solutions that approach the upper bound. Furthermore, while the paper does show improvements over a baseline without fine-tuning, it would be more insightful if the upper bound, representing the optimal 50 classes for each task, could be explored and discussed, if practical. This would provide a clearer perspective on the significance of the achieved results.

We respectfully disagree with the reviewer about the novelty of our approach. To our knowledge, we are the first to clearly hypothesize, and then empirically demonstrate, that fine-tuning to a subset of the base classes can actually result in improved overall accuracy. This finding opens a new line of research, as most papers do not question the base dataset and use it as a whole, indivisible entity.

We sincerely would like to provide results with optimality, but we believe that this is a very challenging question, probably beyond our reach. From a combinatorial perspective, there are 712 chooses 50 possible subsets to create, from which we know no theoretical tool that can predict the performance.

Heuristics, such as greedy selection of the best classes for a target domain, could be added to the paper, but this would inevitably look arbitrary and lacking depth. We think that this question should be considered as future work, keeping our main contribution a demonstration of the ability of improving performance by selecting a subset of base classes.

The approach we introduced and discussed in the paper significantly improves over the baseline, showing that we can improve on the performance in a very competitive field. We think that this should be weighted positively by the reviewer.

We would like to thank the reviewer for their constructive comments and interesting questions. In the following, we address their concerns and questions to the best of our ability.

[...] The theoretical understanding is limited.

We rephrased the motivation in the main paper. In brief, the question is not really about fine-tuning, but rather about the selection of the base dataset for a given target task. There seems to be an implicit belief in the community that the larger the base dataset is, the better will be the backbone, no matter the task. Recent contributions using foundation models as the backbone are going even further down this path. We wrote this contribution because we wanted to question this belief. Our starting point is that there exists a large number of subset of classes, and that some of them may actually help improve performance on downstream tasks, in particular when these downstream tasks are specialised to a subdomain of the base dataset. The main question addressed in our paper is thus: if we increase the number of possible backbones to choose from by artificially creating subsets of base classes, can we find a compromise between having enough of them so that some are going to improve the performance on our tasks, and at the same time few enough so that we can select one that is improving performance using the very limited number of data points on our task?

Nonetheless, we recognise the value of a theoretical framework in this area and hope that future research will explore this issue.

[...] Testing on more complex classifiers would strengthen the results.

We agree that NCM is limited as a representative of existing strategies. We have added experiments using logistic regression. Results (in Table 2), where we see very similar conclusions to the case of NCM. We would like to thank the reviewer for pointing out this concern as it helped broaden our claims.

Better ways to determine class groupings could be developed [...]

We agree with the reviewer that clustering might not be the best approach here. However, since the scope of the paper is mainly to show the existence of strategies that can improve performance, we did not want to dive too much into the optimization of these strategies. (see Appendix for more detail).

[...] Could improvements just come from this distortion rather than better representations?

It is a very interesting question. To investigate whether the improvement is due to mere distortion as opposed to a better representation, we conducted a set of experiments where the backbone was frozen and only an additional linear layer was trained on the class subset. This linear layer can thus distort the feature space without fundamentally changing the representation. Our results (Table 6) show that our method is more effective than a mere distortion of the original representation.

Can the authors provide a rigorous statistical or empirical rationale for selecting M=50 [...]

That is indeed an important question that we spent time investigating during the rebuttal. We ran experiments to observe the effect of varying M in the top-M class selection. (as shown in Figure 2 of the Appendix). We see that the accuracy change is not particularly sensitive to M given M is sufficiently large. As a rule of thumb, any M above 20 classes and below 80 classes is a good fit for every sampling and datasets, justifying the choice of 50. These numbers should be considered highly dependent on the fact we are using ImageNet, on which there exists coherent subsets of classes with dozens of elements each, as displayed in the Appendix. Considering a different base dataset, we expect that the choice of M can be challenging, or require input from an expert.

How do the proposed heuristics for selecting specialist feature extractors address the issue of distributional shifts between the labeled support set and the unseen query set in few-shot learning tasks? [...] How can the method be robustified [...]?

We agree that this is an interesting and important question within few-shot learning, as addressed in recent works such as Bennequin et al., “Bridging Few-Shot Learning and Adaptation: New Challenges of Support-Query Shift”. However, for the purposes of this investigation, we have restricted our scope to the simpler but common setting where the support set and query set can be assumed to be drawn from the same distribution. We have added a few lines in Section 4 acknowledging this limitation and citing a relevant work.

That said, we believe there is potential to investigate a transductive variant of our algorithm that is informed by the query set as well as the support set or domain. Techniques such as transported prototypes (Bennequin et al.) could be used to seek an alignment of the distributions. Alternatively, in the UI setting, one could incorporate the distance between the two distributions as part of a heuristic for selecting a feature transform from the static library.

We would like to thank the reviewer for their perspective. In the following, we address their concerns and questions to the best of our ability.

In general, authors investigate a collection of heuristics to select subsets for training. While some of these heuristics are effective in certain domains, it is difficult to draw a concise conclusion (e.g., Figure 1). It is more like empirical analysis than a novel approach.

We respectfully disagree with the reviewer. We would like to point out that the collection of heuristics that is mentioned in the comment is not our main contribution in this paper. As stated in the paper, the main contribution is to show that there exist subsets of base classes that, when fine-tuned on, can lead to improved performance in few-shot classification of images, and that such subsets can be retrieved even when very limited knowledge of the target task is available. As far as we know, this is indeed a novel approach, and it offers new opportunities for improving few-shot classification. It is true that the paper is mostly empirical, but we still believe it can constitute a significant contribution to the field due to the originality of the described approach.

In Table 1, the performance improvements seem to dependent on the similarities between the classes of target datasets and those of base classes (e.g., Aircraft and Traffic Signs). This may limit the application of the proposed heuristics.

In fact, while class similarity is one important factor, our experiments on a wide variety of domains indicate that it is not the only factor determining the success of the method. For example, while the Fungi and VGG Flower domains have several similar base classes in ImageNet, and the Omniglot domain has none, we observe larger improvements for Omniglot using our method in the DI and TI settings (see Table 1 and pie charts in appendix). This shows that the simple Average Activation strategy is capable of identifying a subset of useful classes even when the distributions of the base classes and target classes do not overlap at all.

We would like to thank the reviewer for their positive and constructive comments. In the following, we address their concerns and questions to the best of our ability.

Some necessary contents, including Alg. 1 and details of heuristics, are put in the appendix. This makes the audience unable to see a rigid, complete version of their method from a main paper, raising concern about page limit circumvention.

We made some minor changes to free enough space so that Alg. 1 is now part of the main paper.

In Sec. 3.2 about informed settings, a method is directly put there without any analysis, reasons, and details. I would like the authors to provide the motivation and some details behind these methods: Why is AA designed in this form? What is the implementation of g, fine-tuning on a pre-trained, fixed ℎ; using a non-learning method on h; or co-train ℎ and g on the pre-trained dataset? Is the same used in the latter testing phase? Does the scale of g (as logits) matter in selecting X ?

We have revised the description of the method and moved the algorithm into the main paper. See revised paragraph below:

To choose a class subset, we need a method by which to identify the base classes which are most useful for a given this set of examples $\mathcal{X}$. Fortunately, the base model already comprises a classifier that assigns a score to each base class. We therefore propose to simply compute the average class likelihoods predicted by the base model on the novel set~$\mathcal{X}$, and then select the $M$ highest-scoring classes. This straightforward selection strategy will henceforth be referred to as Average Activations (AA), and is outlined in Algorithm 1. While it is by no means guaranteed to select the class subset that will yield the optimal representation for the final task after fine-tuning, it is a reasonable proxy for that purpose.

We hope this makes clear that the $g$ used for AA subset selection is precisely that which was learned with the base feature transform $h$, without additional fine-tuning. Fine-tuning is only applied after class subset selection, to adapt the base feature transform to the selected subset.

In Sec. 3.3 about uninformed settings, the authors introduce the method just by listing and citing a number of related works. So, I would like to know what the authors contribute to this part, or whether they are just summarizing or aggregating existing methods.

Section 3.3 describes our procedure for building a static library of feature extractors in the uninformed setting, for which we adopt Ward’s linkage algorithm and CLIP text embeddings. Indeed, the contribution here lies not in any novel algorithm for clustering or text embedding, but rather in the idea of applying these standard tools to build such a library of feature extractors by fine-tuning on class subsets. We designed the algorithm to be as simple and straightforward as possible since our purpose was not to over-optimize the feature extractors for the Meta-Dataset domains, but rather to demonstrate that the approach can lead to consistent improvements in accuracy even when not tailored to a specific target dataset. We clarified the text.

If the proposed method can be applied not only in image classification but also in many other tasks, the contribution of this paper can be even higher.

We would like to thank the reviewer for the interesting suggestion. We designed a new experiment where the DI feature extractors are used to perform semantic segmentation afterwards, using only one image for each considered class. We used Cityscape and adapted the backbones to these tasks by adding a small multilayer subnetworks trained on the few available shots, a methodology similar to Mining latent classes for few-shot segmentation by Lihe Yang et al. As can be seen in Table 5 in the Appendix, we obtained results consistent with classification with significant increase in performance in the DI setting.

We also fixed the typo you found in the algorithm.