Neural Fine-Tuning Search for Few-Shot Learning

Thank you for your review. We understand that you would appreciate further details about the discovered architectures. We hope to address your concerns below.

• Structure and number of parameters: We start with ResNet-18 and ViT, which are well-known architectures with well-documented number of parameters (11M and 22M respectively). These are fine-tuned and/or extended with additive modules: Between 0-16 TSA modules are added in the case of ResNet, and between 0-12 ETT modules are added in the case of VIT. The specific architecture of TSA and ETT modules given in (Li et al 2022, Xu et al 2022) respectively, and already summarised for completeness in Table 1 of our paper. Each TSA module has 100K-2M parameters (depending where in the architecture it is attached), and each ETT module has 200K parameters. The number of parameters increased by adapting is not a fixed quantity, it depends on the result of the architecture search (e.g.,: Fig 2b defines the result of the architecture search by showing which option in Table 1 is used at each layer). With respect to parameter counts of our search space, and of the final models discovered within that search space, we summarise them below - together with a few related methods PMF, TSA, and ETT. Finally, we remark that although NFTS adds some parameters to the base network, our total number of parameters is still much less than the ensemble based methods like SUR (ECCV20) and URT (ICLR21), which are roughly 8x the base architecture size. | | ResNet-18 | ViT-S | | --- | :---: | :---: | | Base Parameter Count | 11M | 22M | | Number of Blocks | 16 | 12 | | Parameter Count per Adapter | +36K-2M | +200K | | Min-Max Possible Parameters Added in Search Space | 0 - 11M | 0 - 4M | | Min Max Possible Parameters Updated in Search Space | 0 - 22M | 0 - 26M | | Number of Actual Parameters after NAS (avg. over N=3 architectures) | 16M (+46%) | 25M (+14%) | | Number of Updatable Parameters After NAS (avg. over N=3 architectures) | 15M | 24M | | Actual Parameters Updated (TSA/ETT) | 11M | 4M | | Actual Parameters Updated (PMF) | - | 22M |
• The most differentiating point from the prior NAS structures: Previous NAS methods focused on constructing architectures that are trained from scratch using large datasets. We develop a NAS approach specifically for searching how to adapt a pre-trained foundation model using a tiny few-shot dataset. The two crucial components of a NAS algorithm are 1) the search algorithm, and 2) the search space. We customize both of these for the few-shot learning problem. We design a search space that consists of the options to fine-tune or not each module (obviously fine-tuning is not applicable in standard NAS search spaces for models trained from scratch), and options to include additional adapters or not. For the search algorithm we extend SPOS into a two-stage algorithm that does most of the searching up-front during meta-train, and selects among a short list of candidate architectures during each few-shot learning episodes. This balances efficacy with runtime computational cost, as well as overfitting with underfitting. The positioning with respect to prior NAS is already explained in Related Work (Sec 4, paragraph “Neural Architecture Search”).
• Ablation study: The requested ablation study is already covered by Tab 5. Please note that “N” in NFTS-N denotes how many candidate architectures are left to be selected at meta-test time.
  • 1-stage vs 2-stage hybrid NAS: This requested comparison corresponds to NFTS-1 vs NFTS-N in Tab 5 (left). NFTS-1 uses the best performing path identified during meta-training. Our proposed NFTS-N uses the two-stage/hybrid approach to select N candidate paths during meta-training, and then pick the best among these N (N=3) during meta-testing, which leads to outperforming NFTS-1 in Tab 5.
  • 2-stage hybrid vs test-time NAS: The third option mentioned by the reviewer (fully conducting NAS during each meta-testing episode) is prohibitively costly to run as it corresponds to NFTS-N with N=2^K. Therefore, this cannot practically be ablated. The closest to this that we were practically able to run is NFTS-N with N=100, shown in Tab 5 (right). Already at N=100, where performance has degraded compared to N=3.
  • The explanation for both of the above comparisons is given in Sec 3.3. To reiterate: Without any NAS at test-time, there is no opportunity to tune the architecture according to the downstream dataset (NFTS-N > NFTS-1). With too much NAS at test-time, there are too many free parameters to fit with the small few-shot support set, and it is possible to overfit the training set and perform badly on the support set (Tab 5, right).

Thank you for your detailed explanations and extensive experiments (clear ablation studies below). I increased my score from 5 to 6. I hope the comments (#parameters and performances) will be included in the future version of the paper.

To be specific, one more ablation study (tables of #parameters and performances) could be prepared to compare the proposed method with others, including SUR(ECCV20) and URL(ICLR21).

Thanks for the revised comments requesting another ablation study for comparison of #parameters and performances between SUR/URL and our method. From the table above, it is clearly seen that our method achieves better performance than several alternatives that have noticeably higher parameter counts. Please consider raising your score if you don't have further concerns.

Thank you for reviewing our paper, and for acknowledging the novelty of our method. Please, find our response to your concerns below.

• Computational cost vs. performance gain: This is a very good point. We agree that MetaDataset is a very popular and highly competitive benchmark, where it is no longer possible to easily make gigantic leaps over the state-of-the-art. However, you raise a valid concern: “Is it worth it to perform an architecture search for 1-2% improvement?”. Our thinking of this is as follows: 1) The main cost of our method is the first stage SPOS architecture search, which is a one-off cost during meta-training. Since the results of this search can be re-used for many subsequent few-shot recognition tasks, it is not unreasonable to pay a large cost upfront. This rationale is used by many other methods with expensive upfront costs such as MAML and others that use expensive second-order gradients during meta-training. 2) Our architecture search cost is not actually “excessive” compared to alternatives. Our total meta-train cost is about 40GPUh, which is less than required for second order MAML on MetaDataset. It is also much less than methods that make heavy use of attention (e.g,: CrossTransformer requires 1300 V100 GPUh on MetaDataset - over 30x more than ours!). What is more, the majority of this cost comes from the meta-training-time search which can be trivially parallelized. 3) Our recurring meta-test cost (even including the small second-stage architecture search) is smaller than methods that use large feature extractor ensembles such as SUR (ECCV20) and URL (ICLR21), and/or cross-attention (URL, CTX). 4) A final benefit is that as a byproduct our method provides insight about your pre-trained architecture that may be valuable. E.g., as you point out in your question, some layers are always adapted, and some are never adapted.
• Architecture blocks consistently adapted/not adapted: The vast majority of adaptation work hypothesizes that foundation models have a lot of knowledge encoded in their latent spaces. Having said that, allow us to answer your question with a toy example: Some architecture blocks may encode the notion of texture, or shape. Some other layers may encode the concept of a fish. The notion of texture is much more relevant to a large number of downstream visual recognition tasks than the concept of a fish, which is very domain-specific. This may be the reason why we see a “fish block” always adapted, but a “texture block” always in its original form, as it can already do its job pretty well. Of course, this is a hypothetical example and it is still an open research question to properly identify which neurons represent which concept, as would be required to interpret our specific discovered architectures in this way. But in the meanwhile, this is exactly why it is helpful to have automated methods to identify which blocks to adapt/not adapt.

Thank you for your response. I will maintain my score. Given that computational cost is a core consideration in NAS-related work, including an analysis, such as the one you pointed out in your previous response, could further enhance the quality of your paper.

Thank you for your review, we are happy that you found our paper well-written, strong, and benefitial to the community. Please, find our response to your concerns below.

• Confidence intervals: Initially, we did not include the confidence intervals for two reasons: 1) They are all quite small; the majority of standard errors are < 0.1%. This should have been explicitly stated in the paper. 2) As you accurately pointed out, the space limitations; since the confidence intervals are not very informative, adding a “±0.002” to every column would make the tables even longer and unreadable. Given the small standard errors on the means, most of the differences between methods are statistically significant. We do, however, have the confidence intervals for our experiments, and are happy to include them in the supplementary material. For example, see the confidence intervals (SEM) reported below (RN18/ViT, MDL): | Dataset | ResNet-18 | ViT-S | | --- | :---: | :---: | | Aircrafts | 90.1±0.013 | 89.1±0.002 | | Birds | 83.8±0.058 | 92.5±0.034 | | DTD | 82.3±0.002 | 86.3±0.002 | | Fungi | 68.4±0.116 | 75.1±0.086 | | ImageNet | 61.4±0.145 | 74.6±0.088 | | Omniglot | 94.3±0.037 | 92.0±0.047 | | QuickDraw | 82.6±0.020 | 80.6±0.041 | | VGG Flowers | 92.2±0.086 | 93.5±0.090 | | CIFAR10 | 83.0±0.001 | 75.9±0.002 | | CIFAR100 | 75.1±0.002 | 70.8±0.001 | | MNIST | 95.4±0.001 | 91.3±0.001 | | MSCOCO | 58.8±0.130 | 62.8±0.063 | | Traffic Signs | 82.9±0.107 | 87.2±0.032 |
• Comparison N=2,4, ResNet/ViT: In practice, N is a hyperparameter that is chosen heuristically, and the results do not change much when N=2,3,4,5, etc. The big difference comes from N=1, N=small, N=large. This is true for both ResNet and ViT architectures. Please, find an analysis of small Ns below (RN18, MDL, N=2,3,4): | | N=2 | N=3 | N=4 | | --- | :---: | :---: | :---: | | Mean acc. (MetaDataset, NumEps=600) | 80.3 | 80.7 | 80.7 | Since the differences between small values of N are insignificant, in our experiments we arbitrarily chose N=3 as a value that is greater than 1 but small enough not to impose a substantial computational burden. We did not attempt to carefully tune this hyperparameter.
• Additional baselines: Thank you for providing these additional references, we have added them to Table 3.