SEFAR: SparsE-FeAture-based Regularization for Fine-Tuning on Limited Downstream Data

We really appreciate your comments and reviewing. We supply some experiments for this response.

Q1. How do you choose hyperparameters?

A1. We have $\lambda$ (the weight of $L_3$), $t$ (the distillation temperature coefficient) and $s$ (the sparsity of the binary mask). We choose the optimal hyperparameters by a grid search. The detailed configuration of each experiment is shown in our appendix.

Q2. The evaluation dataset is limited.

A2. We conduct experiments on more datasets with ResNet-18:

In this table, SEFAR shows its consistent positive gain for the full fine-tuning method.

Q3. What if sparsity is set to 0 (figure 3a)

A3. When the sparsity is set to 0, on the EuroSAT dataset, the results with different distillation temperature coefficients are:

Results with s=0 are not as good as the result with s=0.3, which illustrates that the sparsity is important.

Q4. How does the sparsity of features affect generalization?

A4. Figure 3 (a) provides an initial insight into the impact of sparsity on linear probing. In our experiments, we conducted a grid search across values {0.3, 0.6, 0.8, 0.9} to identify the optimal sparsity value. In the future, we may explore the detailed mechanism.

Q5. What would happen if the training set is increased?

A5. We conduct experiments with various training dataset sizes and the results are:

The gain in performance with SEFAR becomes more pronounced as the number of training set images decreases. Additionally, as the number of training images increases, the additional generalization can be derived from the incremental data. Therefore, SEFAR is particularly suitable for use when fine-tuning with a limited number of images.

Thanks for your comments. Here are our responses.

Q1. Dropout and Self-distillation are popular. So the contribution is limited.

A1. Our main contribution is we discover that the sparse feature can be a good teacher in the self-distillation process. We theoretically and experimentally prove why it can work well.

Q2. The theoretical analysis (one-class case) with MSE loss is oversimplified.

A2. We think that one-class case + MSE Loss analysis is a common way, as shown in [1]. We also empirically confirmed that the results of MSE loss are consistent with the cross-entropy case in practice, which can be observed in Table 4 of our submitted script.

Q3. t is very strange that the t-SNE of each class in Fig.2c forms as disk. How to understand that ``randomly dropping some dimensions to generate sparse features may result in enhanced discriminative capabilities''? It seems interesting.

A3. We also consider this to be a fascinating phenomenon: when randomly masking certain features, the performance of feature clustering improves. In the future, we plan to conduct more experiments and theoretical exploration to understand the underlying reasons.

Q4. The organization of experiment section.

A4. We think it is a good suggestion and we may reorganize the paper by add a new section for theoretical analysis.

Q5. How Eq. 3 is derived based on the proposed method in Eq.2?

A5. Please refer to A2.

Reference

[1] Hossein Mobahi, Mehrdad Farajtabar, Peter L. Bartlett. Self-Distillation Amplifies Regularization in Hilbert Space. In NeurIPS 2020.

Thanks for your reviewing. Let's answer your comments as possible as we can.

Q1. For the maskgen algorithm and the way to generating the sparse mask.

A1. In this paper, we clarify that the sparse mask is randomly generated, without any learning process. Specifically, we generate an all-one matrix with the same shape of the feature. Then we randomly select a ratio of elements and set them 0. The ratio is the sparsity mentioned as $\lambda$.

Q2. If we have $L_1$, $L_2$, $L_3$ and no mask (i.e. all feature used; or sparsity=0), what is the result.

A2. Here are the results on EuroSAT dataset:

We also explore if L2 itself can work well on EuroSAT, the results are:

According to this table, we can observe that although SEFAR seems to be a combination of DropOut and self-distillation in its implementation, the gains in terms of results are even greater, especially on ViT-small.

Q3. Is ablation study in 4.2 conducted on EuroSAT?

A3. Yes, it is. And the experiment configuration is the same.

Thank you very much for your review and suggestions. Our response to your inquiries is as follows:

Q1. More evaluation on language models and other larger CNN models

A1. Due to constraints in time and computational resources, we conducted experiments using VGG11, VGG11 (with BN), MobileNet v2, LoRA+ViT-B, and LoRA+ViT-L on the EuroSAT dataset. The results are as follows:

It is obvious that SEFAR can work across different network families and scales.

Q2. Whether different masks can lead to different performance and how to choose a good mask. Can we optimize the masks during finetuning.

A2. In our approach, we introduce a randomly generated binary mask, where the sparsity (i.e., the proportion of zeros) is the only variable that needs to be controlled. During the training process, the mask is randomly generated and does not need to be optimized.

Q3. T-SNE results show marginal improvement.

A3. In the T-SNE results, our primary focus is to highlight the differences in clustering outcomes between the original features and sparse features. Specifically, in scenarios with a limited number of training samples, sparse features exhibit superior clustering performance. Therefore, utilizing the classification results generated by sparse features as the teacher in self-distillation proves to be more advantageous.