Less is More: Selective Layer Finetuning with SubTuning

Weakness 1 (lack of discussing prior work)

The paper does not mention a lot other prior work that looked into fine-tuning particular layers for specific purposes. For example:

1. DFR [1] is an important work that studies retraining only the last layer to get rid of spurious correlations.
2. The paper discussed linear probing and fine-tuning, but does not discuss LP-FT [2]. There is a complex dynamic behind LP and FT, specially when distribution shift is involved. It is also important to consider pre-trained feature distortion, which can be a reason sub-tuning works better than full fine-tuning.

Finally, the paper’s claim/observation in the abstract is not novel, i.e., it has been recorded in other papers before [7], and should be modified as such.

This paper (abstract): We observe that not all layers are created equal: different layers across the network contribute variably to the overall performance, and the optimal choice of layers is contingent upon the downstream task and the underlying data distribution

Surgical fine-tuning [7] (abstract): This paper shows that in such settings, selectively fine-tuning a subset of layers (which we term surgical fine-tuning) matches or outperforms commonly used fine-tuning approaches. Moreover, the type of distribution shift influences which subset is more effective to tune.

Weakness 2 (formal definition/terminology):

Setting up a formal definition of the problem would make the paper much easier to read. For example, one could define it as: let $S$ be the set of all layers, then sub-tuning is: $S_{best} = argmin_{S' \subseteq S} L(f, S', D_{ft}, D_{val})$

where $L(f, S', D_{ft}, D_{val})$ is the loss after fine-tuning the network $f$'s layers in $S'$ on dataset $D_{ft}$ and evaluating the final performance on $D_{val}$.

With this in mind, how is sub-tuning different than surgical fine-tuning? I think the main difference is not in the formal definition, but in the practical implementation: surgical fine-tuning only considers $S' \subseteq S$ that are contiguous, whereas this paper considers $S'$ that are not necessarily contiguous. If this is the case, this should be proposed as an improved/better way of doing surgical fine-tuning, not a separate problem definition. If not, please explain.

Also a proper explanation of what authors consider as blocks vs layers in a neural network would be important. Ideally, some sort of schematic of ResNet-50, discussing the terminology, in the main paper/appendix would be needed.

Weakness 3 (experiment results):

1. Figure 1: The difference between fine-tuning B1 (94%) vs B14 (95.5%) in figure 1 (left) is negligible. Also denoting the number of parameters in each of these layers would be important.
2. Figure 3: I would interpret the results in a different way: looking at the grid column wise, it seems the columns corresponding to B13, B14, B15 do well in general. I would also guess that the variation of accuracy within these columns is small when adding B2, B4, B8 or B9. How much is adding the second block possibly helping here? It can quite possibly be that fine-tuning B13/14/15 already recovers most of the performance/does most of the heavy lifting, and the improvement that comes from fine-tuning B2 in addition to B14 is negligible.
3. See question 3. I seriously doubt that in the limit of infinite data, subtuning essentially chooses close to all layers, and therefore becomes full fine-tuning effectively. If that is not the case, I am curious why it is not? Given infinite data, and more layers to fine-tune, the neural network should intuitively learn a better classifier, unless only a few layers can solve the problem effectively/the layers not fine-tuned are already close to optimum for the downstream task. For $\textrm{source} \rightarrow \textrm{target}$ dataset with large distribution shift, this should never be the case, so I suspect this result is because of $\textrm{Imagenet} \rightarrow \textrm{CIFAR-10}$ shift. If this is not the case, please explain.
4. Table 1: Missing some very important comparisons, like LP-FT [2], layer-wise learning rate tuning [6], gradual unfreezing [3, 4, 5].
5. Section 3.2: Missing results on some very important benchmark. The section title contains distribution shift, but focuses solely on image corruptions. Some results on natural shifts such as the WILDS [8] (i.e., at least 2 or 3 of these datasets) would strengthen the paper.
6. Section 3.2: “Furthermore, our results challenge the claim made in Lee et al. (2022b) that suggests adjusting only the first layers of the network suffices for input-level shifts in CIFAR-10-C. Interestingly, we found that the ultimate or penultimate block was the first layer selected for all corruptions, resulting in the largest performance increase.” I feel this claim is not well made. For example, even in the reproduced results of this paper, we see surgical L1 > surgical L2 > surgical L3. Note that sub-tuning looks at a much more granular level (i.e., within blocks of a single layer), and finds a better finetuning arrangement. Also, it is possible that the finetuning the last layer performs worse, but the last block within the last layer performs better, hence this is not a contradiction. Furthermore, in figure 5, B12 is the most commonly chosen first block, but B1 is the most commonly chosen second block, so the first block is still important? In what order would sub-tuning pick the parameters, if allowed only to fine-tune layers and not the blocks inside?

Weakness 4 (lack of interpretability):

The authors give a mechanistic way of choosing the subset of layers that needs to be fine-tuned, but there is no understanding of the layers chosen other than it empirically works well. Why would, for a particular downstream task, layers B8 and B14 combine to do the best? Possible things to explore here:

1. How much does adding the second layer help? How much does adding the third layer? And so on.
2. Possibly looking at something like [9]. It is hard to do for vision experiments, but understanding why certain combinations of layers help is important.

Otherwise, this method is computationally heavy (requires multiple fine-tuning rounds) and may not be suitable for use in practice compared to full fine-tuning, LP-FT or surgical fine-tuning for their simplicity.

1. The theoretical guarantee and explanations are not enough. Although theorem 1 provides a generalization bound that depends on the number of stunned parameters, it is hard to link this theory to the algorithm studied in this paper. Maybe the analysis provided in [1, 2] (the analysis on an overparameterized model) would be helpful.
2. The results in Table 1 are based on VTAB-1k, which means the samples used for transfer learning are only 1k. Will the proposed method still work when fine-tuning using the full CIFAR100 dataset? AFAIK, fine-tuning a pretrained ResNet50 on full CIFAR100 should achieve roughly 70%+ accuracy, which is much higher than the numbers provided in the paper. It is easier to compare with other baselines using similar settings.
3. It would be very helpful if the paper could briefly introduce how the VTAB-1k dataset is generated (maybe in the appendix).
4. In Section 4 and Figure 6, the paper claims that subtuning can bring benefits to multi-task learning. However, I cannot find the corresponding experimental results or analysis supporting that.

1. Lack of novelty: Performing layer selection during fine-tuning has been explored by previous work [1, 2, 3]. These works use more advanced selection techniques like policy networks or genetic algorithm. The proposed way of measuring layer importance through simply fine-tuning accuracy on a specific dataset is also not generalizable, i.e., the observed trend only holds for a specific model and a specific dataset and it is hard to deduce any more general and useful rules from that.

2. Related work: The paper briefly mentions related work in parameter-efficient transfer learning and multitask learning. However, the idea of layer selection is also very relevant to layer-wise adaptive learning rate [4] (since a learning rate of 0 is the same as freezing the layers) and neural architecture search for parameter-efficient transfer learning [5]. The idea of layer importance is relevant to the neural network pruning literature as well [6]. A more thorough discussion and comparison with these methods would provide valuable context and insights.

3. Limited models and datasets: The paper only evaluates a few variants of ResNet and ViT, which is relatively small in scale compared to state-of-the-art large vision models. It is not sure whether the proposed method can scale up to these more practical models. Also, only 4 relatively simple datasets are used for evaluation (CIFAR, Flowers102, Caltech101, DMLAB). It would be better to see how the proposed method performs on more datasets with larger distribution shifts.

[1] SpotTune: Transfer Learning through Adaptive Fine-tuning. Guo et al. 2019.

[2] Dynamic fine-tuning layer selection using Kullback–Leibler divergence. Wanjiku et al. 2022.

[3] Automatic layer selection for transfer learning and quantitative evaluation of layer effectiveness. Nagae et al. 2022.

[4] Not All Layers Are Equal: A Layer-Wise Adaptive Approach Toward Large-Scale DNN Training. Ko et al. 2022.

[5] Neural Architecture Search for Parameter-Efficient Fine-tuning of Large Pre-trained Language Models. Lawton et al. 2023.

[6] Layer-adaptive Sparsity for the Magnitude-based Pruning. Lee et al. 2021.

1. I have some concerns about the contributions of this paper -- the key idea that we do not need to finetune all layers and that finetuning only a subset of the parameters can outperform full-feature finetuning was already observed in the "Surgical finetuning" paper of Lee et al. To me, the only new thing shown in this paper (beyond what was already shown in Lee et al), is the fact that we can greedily choose more than just one block of layers to finetune. In Related Work, the authors mention "While our work shares some similarities with Lee et al., the motivations and experimental settings are fundamentally different" -- I'm a little confused by this sentence. As I understand, the experimental settings are exactly the same, and this is also mentioned in the paper later in the data corruption section where the authors say "...we follow the setup proposed in Lee et al". In related work the authors also mention "..we delve deeper into the complex interaction between the appropriate layers to finetune and the downstream task, pretraining objective, and model architecture, and observe that a more nuanced viewpoint is required" -- I'm again very confused by this, what exactly is the nuance and how does it add to what we already know from Lee et al? It'd be very useful if the authors can precisely state their contributions and mention what they add beyond the insights in Lee et al.

2. I like the observations in Section 2 and would've liked a more thorough analysis of how finetuning profiles change with architecture, pretraining, and downstream tasks. Right now I only see (from the first row in Fig 2) how things change on CIFAR10 with different architectures. But then again I notice that all these architectures don't have the same pretraining (ViT is on ImageNet-21k, Resnet18 is ImageNet-1k and ResNet50 is pre-trained using DINO). It might make sense to do this more systematically, one where you hold the pretraining the same and vary architectures and downstream tasks. One where you hold an architecture constant and show different pre-training methods. If this is one part of the "nuance" you add in your paper beyond Lee et al's work, then it requires a proper investigation.

3. Under "The Finetuning Profile" in Section 2, it might help to precisely define what you're profiling for (ie which accuracy are you reporting), only by reading until Section 3.1 I realized that this is cross-validation accuracy.

4. The observation in Section 2 ("Not All Layers Are Created Equal") has been made in prior work as well [1], it's worth citing this paper.

5. In Section 2 where you discuss "The Finetuning Profile", you've already made the assumption that the right unit at which to finetune is a block of layers, can you elaborate on why this was done? Why not layers within a block or even portions of layers?

6. Under "Results" in Section 2 you mention that it's surprising that layers with fewer parameters can outperform layers with more parameters. However, I do not quite understand why # parameters are more relevant than the quality of features? In fact, your own results show that B16 and B8 when finetuned together (Fig 3) outperform the case when either of these blocks is finetuned alone, indicating parameter count probably has nothing to do with the finetuning profile, and that what (probably) matters more is the nature of features being captures by a block. Am I missing something?

7. I'm not sure what the theoretical analysis in Section 2.1 adds to the paper. I again do not understand why parameter count (r) matters more than the quality of features or inductive bias. For example, based on the error bound in Theorem 1, wouldn't you always choose k=1 and choose this such that you minimize r (hence you'll always choose the layer with the least number of params)? But this is obviously not true based on your empirical results. Am I interpreting the theorem incorrectly?

8. Section 4 starts by discussing how SubTuning can be efficient for a multi-task setup, however, all experiments shown in Section 4.1 do not consider any multi-task setup. Additionally, section 4 starts by saying you will focus on FLOPS for compute and bits needed in memory for IO considerations. However, the results in Figure 7 show Inference Time on the x-axis. Why is this the case?

9. I like where Section 4 is going but I feel like it requires a more succinct argument and more convincing experiments. For example, to me it's not clear how you will save on FLOPS -- the moment you choose layer (or block) 0 as a finetuning block in SubTuning then # FLOPS = same as the FLOPS needed with full finetuning.

[1] Are All Layers Created Equal? https://arxiv.org/abs/1902.01996