Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgetting

Thank you for the recognition of our work and constructive feedback. We spent quite some time updating the paper to include the requested experiments and clarifications by all the reviewers. We respond to your specific comments below.

In Fig. 2(c), proxies do not lie in the convex hull of neighbors. How proxies in the convex hull shape the decision boundary?

Thanks for pointing this out. Although Fig. 2(c) is only for illustration purpose, we will modify it to reflect that the generated proxies lie in the convex hull of the corresponding set of positive features ($\boldsymbol{X}\_{i}^{+}$) or negative features ($\boldsymbol{X}\_{i}^{-}$).

Regarding the other question, we conjecture that proxies generated in a convex hull can be key to reshape the decision boundary, especially when there is overlap between the positive and negative feature manifolds — Fig. 1(b) shows how typical such overlap is in the feature space. In case of feature manifold overlap, sparsely sampled feature points won't be able to characterize a fine-grained decision boundary, while we densify feature distributions using proxies to refine that boundary. Note from the performance perspective, it's not enough to generate proxies only around the decision boundary (Fig. 6 and L949-953 show sub-optimal performance). This indicates the key role of proxy diversity in enhancing FDA with both richer data and fine-grained decision boundary.

More explanations of limited data diversity and how hard-class mining alleviates limited diversity.

Due to the batch size constraint, the sampled data in $\boldsymbol{X}\_{i}^{+}$ and $\boldsymbol{X}\_{i}^{-}$ are intrinsically limited in both quantity and diversity. It's more so in few-shot learning tasks where only a couple of training instances can be sampled from each class.

To alleviate the data challenge, we perform hard-class mining within batch (besides generating proxies). Appendix A details the hard mining algorithm that greedily samples related class data. This means $\boldsymbol{X}\_{i}^{-}$ includes negative features that are close to the anchor $\boldsymbol{x}\_{i}$, but still further away from $\boldsymbol{x}\_{i}$ than the positive features $\boldsymbol{X}\_{i}^{+}$. The metric learning community calls such $\boldsymbol{X}\_{i}^{-}$ as semi-hard negatives. They often provide an informative training signal (i.e., relatively large loss), without sampling the vast space of easy negatives with near-zero loss values. Hence, semi-hard negatives alleviate the limited data issue. L260-262 provides such rationale, but we will make it clearer.

How to determine the number of positive proxies $n^{p+}$ and negative proxies $n^{p-}$?

Appendix D (L862-863) sets $n^{p+}$ and $n^{p-}$ proportional to the number of sampled positives $\boldsymbol{X}\_{i}^{+}$ and negatives $\boldsymbol{X}\_{i}^{-}$, respectively, using a scalar $s$ which is fixed across batches. Fig. 5(d) shows the sensitivity analysis for $s$.

Thanks for the recognition of our work and helpful suggestions for improvement. We spent quite some time integrating the requested clarifications into the paper, and the paper is also updated to include supporting ablation studies. Below is our point-by-point response.

Motivation of using kNN feature-based FDA, and differences from knowledge distillation.

As a quick recap, our introduction section (L46-85) motivates that for forgetting mitigation, feature regularization is often better than weight-space regularization, but existing feature regularization methods are point-wise and lack explicit awareness of feature neighborhood structures (hence still sub-optimal). Then for the first time in forgetting mitigation, we propose a structure-wise feature regularization method Proxy-FDA, which preserves structural knowledge by matching kNN feature graphs. As mentioned in L146-152, Proxy-FDA is indeed similar to Knowledge Distillation (KD). Proxy-FDA can be seen as a relational KD method in general, but differs from prior works in distilling knowledge from both neighbor indices and similarities, with an additional proxy learning component. To show the utility of Proxy-FDA in the KD field, we have conducted some KD experiments on ImageNet after paper submission deadline. Please refer to Table 10 and L1005-1018 in the updated paper. Results show the competitive performance of Proxy-FDA and its advantages over related KD baselines.

Explain and compare the weighting mechanism in Eq. (2).

As stated in L209, Eq. (2) defines a re-weighted Sigmoid loss where the weighting is derived by the cosine similarity $\hat{w}\_{ij} = \cos(\hat{\boldsymbol{x}}\_{i},\hat{\boldsymbol{x}}\_{j})$ between pre-trained features $(\hat{\boldsymbol{x}}\_{i},\hat{\boldsymbol{x}}\_{j})$, not between features $(\boldsymbol{x}\_{i},\boldsymbol{x}\_{j})$ being fine-tuned. The goal is to transfer the nearest neighbor graph (with both neighbor indices and similarities) from the pre-trained model to fine-tuned model. We conjecture that the structural knowledge contained in a complete nearest neighbor graph is richer than the knowledge in neighbor indices only. Appendix E (L912-914) and Fig. 6 show that we will see notable performance drop if we only transfer neighbor indices (i.e., setting $\hat{w}\_{ij}=1$).

Thanks for the detailed feedback! We spent quite some time on the requested comparisons and ablations, as well as paper revision to include the new results, analysis and clarifications. We respond to specific comments below.

Discuss CLIPood and PromptSRC in the related work section.

Thanks for the reminder. The discussion is added in L135-137.

Clarifications on some technical details.

Sorry about the missing details. 1) Appendix B details about the proxy generator architecture, including one attention layer with attention mask applied, and two convolutional layers for proxy pooling. Please see text for the feature dimensionality, output channel size and normalization. Appendix D (L898-904) is updated to elaborate on the parameter count and compute cost. We plan to release codes in the future to provide more details. 2) In few-shot settings, when fewer than $n=4$ samples are available for each class, we perform random data augmentation to guarantee 4 samples per class (clarified in L856-858). In this case, FDA aligns the structural similarity between the original and augmented data. 3) We updated Section 3.2 to introduce the scalar $s$ (L266), which is then referenced in Appendix D.

Missing hype-parameters, per-dataset $K$, and implementation details for continual learning.

In response: 1) both temperature $\tau$ and bias $b$ are learnable parameters that are initialized as similarly in (Zhai et al., 2023). Such initialization helps to handle the imbalance between positives and negatives in the loss function. We updated L210-211 to reflect this. While $\lambda$ is the loss coefficient tuned on the validation set of each task, as mentioned in L895. 2) Fig. 5 is updated to include the optimal $K$ for each dataset. 3) We also update Appendix F (L994-1001) to include the dataset splitting and learning details on Split ImageNet-R. The same implementation details enable fair comparisons with other methods.

Comparison with FD-Align and CLAP.

As suggested, we compare Proxy-FDA with FD-Align in the domain generalization setting, i.e., prompt tuning on ImageNet (16 shots per class) before evaluating OOD generalization on ImageNetV2 and ImageNet-Sketch. We use the same ViT-B/32 backbone for fair comparison. The table below confirms our advantage in this setting.

Table 7 in the updated paper compares Proxy-FDA with a couple of recent prompt tuning methods, including the regularization method CLAP. Results show our consistent gains over CLAP in the base-to-new generalization setting and the additional domain generalization setting (see details in L975-989). On the other hand, we acknowledge that CLAP shines in that it doesn't use a validation set which is required by other methods (including ours) for hyper-parameter tuning. One of our future plans is to integrate the high-level idea of CLAP (using the Augmented Lagrangian Multiplier approach) into Proxy-FDA, in order to experiment under the validation-free scenario too.

Ablation on batch size.

Please refer to Fig. 5(a) in the updated paper for ablation results, and to L851-854 for analysis.

Thank you for the constructive feedback on our work. We spent quite some time on the requested experiments, and the paper is updated to include changes as suggested by all the reviewers. Below is our point-by-point response to your questions.

Why does FDA alleviate forgetting? It could collapse back to the original state.

In response, there are two categories of regularization methods to mitigate forgetting in the literature. Weight-space regularization minimizes the change in model weights that implicitly store the learned knowledge; hence such regularization helps transfer knowledge from a pre-trained model to fine-tuned model (i.e., less forgetting or even positive forward transfer). Another type of regularization happens in the feature space that characterizes knowledge of a model in its input-output behaviour. Feature-space regularization is often found to preserve more knowledge due to the explicit constraint on features (i.e., model behaviour).

In practice, either the weight- or feature-space regularization term is added to the task loss with proper scaling. This can keep the fine-tuned model in the desired vicinity that preserves the pre-trained knowledge while still learning the task at the same time (hence moving away from the pre-trained model). Our (Proxy-)FDA belongs to the feature-space regularization methods, but relaxes the feature constraint from point-wise L2 distance to a structural one. Fig. 3 shows the unique advantage of our structural regularization method over point-wise method LDIFS: we have larger L2 distance (i.e., learning task with a farther away model checkpoint) during FDA, but preserves more knowledge and eventually achieves positive forward transfer. Table 1 confirms that (Proxy-)FDA leads to strong fine-tuning performance as well as significantly reduced forgetting.

Include theoretical or empirical evidence to explain why Proxy-FDA outperforms traditional methods.

As a quick recap, we have provided two empirical observations in the paper to justify the advantage of our structure-wise feature regularization method. First, compared to point-wise regularization methods, Proxy-FDA can better preserve the structural knowledge among nearest neighbor graphs in local feature neighborhoods. Fig. 1(b) shows through t-SNE visualization that the structural knowledge is rich enough to go beyond class concepts (e.g., cross-class attributes). Preserving such common-sense knowledge is useful to maintain the generalizability of foundation model, leading to much less forgetting than point-wise regularization methods. Second, Fig. 3 shows a strong correlation between forgetting and a structure-aware distributional distance metric OTDD. As mentioned in L397-398, the strong correlation suggests that having some form of structure-wise feature regularization can mitigate forgetting better than point-wise methods, and our structural method Proxy-FDA is one such instantiation. In other words, the second observation explains our advantage from an optimization perspective. We leave theoretical analysis on how Proxy-FDA improves the fine-tuning generalization error bound for future work.

Compare with recent works from 2024 onward.

Thanks for the reminder. In the submitted paper, we treat LDIFS (published in 2024) as the main competitor for end-to-end fine-tuning (Table 1,4) and continual fine-tuning (Table 3,9) experiments. Now the paper is updated to include 1) comparisons with four 2024 works for few-shot prompt tuning. Table 7 and Appendix F (L975-989) include the experimental details and illustrate our benefits. 2) Knowledge distillation experiments where our Proxy-FDA is quite competitive with many strong baselines including the 2024 one (WTTM). Table 10 and Appendix F (L1005-1018) provides details.

Improve the interpretability of Fig. 1 & 2.

Great suggestion! We plan to split Fig. 1(a) and Fig. 1(b) into separate figures: Fig. 1(a) will be replaced with Fig. 7 in the updated paper to motivate Proxy-FDA in a neat way; Fig. 1(b) will act as a supplementary visualization. Similarly for Fig. 2 , we will focus only on the (Proxy-)FDA loss by replacing Fig. 2(a)&(c) with Fig. 8 that build things up slowly. While Fig. 2(b) will be moved to Appendix to illustrate the proxy generator architecture separately.