Proxy-FDA: Proxy-based Feature Distribution Alignment for Fine-tuning Vision Foundation Models without Forgetting

Thank you for the recognition of our work and constructive feedback. Below is our response to each concern, as well as new comparisons on compute cost.

Q1: Sensitivity to hyperparameters like the neighborhood size K and the scalar s for proxy generation

Given the held-out validation set of each downstream dataset (a reasonable assumption for model fine-tuning), we adjust Proxy-FDA's hyperparameters on that validation set. Fig. 6 shows that results are mostly insensitive to each key hyperparameter in a wide range. We acknowledge there are methods that automatically learn hyperparameters instead of manual tuning, in order to further reduce hyperparameter sensitivity. Examples include meta-learning methods, or recent "validation data-free" methods (e.g., CLAP paper cited in Table 7) that optimize hyperparameters online without even requiring a validation set. Integrating such ideas into Proxy-FDA will be a promising future plan.

Q2: Computational overhead by the proxy generator could be a limitation in resource-constrained settings

This concern is related to our hypothetical claims in the 1st paragraph of Section 3.2: despite incurring some computational overhead, proxy feature generation is still compute- and data-efficient (comparatively) to improve data diversity. Other viable strategies like retrieving external data should have higher cost and suffer from distribution shift. After paper submission, we implemented one type of Retrieval Augmented Fine-Tuning (RAFT) method using external data, and the comparison results support the above claims.

Concretely, to enrich both the positive feature set $\mathbf X_{i}^{+}$ and negative set $\mathbf X_{i}^{-}$ (by the amount controlled by scalar $s$), we retrieve top similar samples from the large-scale LAION-400M dataset, instead of synthesizing proxy features {$\mathbf P_{i}^{+}, \mathbf P_{i}^{-}$} online. Both methods involve the augmented features (retrieved or synthesized) with FDA loss computation, and we call the methods as RAFT-FDA and Proxy-FDA, respectively.

Note for RAFT-FDA to effectively retrieve external data, the same model is needed to extract features for the sampled batch in $\mathcal D_{\text{ft}}$ and external $\mathcal D_{\text{LAION}}$ to measure their feature similarity. It'd be obviously expensive to use the fine-tuned model to repeatedly refresh all the feature representations of $\mathcal D_{\text{LAION}}$. Hence we use the pre-trained model to pre-compute similarities between the entire $\mathcal D_{\text{ft}}$ and $\mathcal D_{\text{LAION}}$, so each sample in $\mathcal D_{\text{ft}}$ has its external neighbor indices that remain fixed. We treat this as a "pre-processing step" for RAFT-FDA whose training time excludes this step. In each fine-tuning iteration of RAFT-FDA, what requires additional feature extraction using the fine-tuned model is now only a limited number of external data (i.e., indexed neighbors from $\mathcal D_{\text{LAION}}$). Overall, the computational overhead of RAFT-FDA is mainly affected by FDA and external feature extraction/augmentation.

The table below compares RAFT-FDA against Proxy-FDA for few-shot prompt tuning based on CoOp.

Observations: 1) Proxy generation is data-efficient in the low-data regime. Our Proxy-FDA outperforms RAFT-FDA when the latter retrieves the same or a double amount of external data. Generated proxies achieve higher data efficiency because they adapt to the fine-tuned feature distributions, while using external data will inevitably suffer from distribution shift and provide less effective feature regularization. However, our performance benefits diminish as the size of external data increases (over 4$s$). 2) On the other hand, external data augmentation is costly and the cost increases drastically with the retrieved data size. This is due to the need of extra feature extraction for each external sample using the large vision model. While our proxy generator is lightweight, and can generate proxy features all at once (not individually).

More comments: 1) We may potentially improve RAFT-FDA by feature fusion strategies (e.g., Mixup) to address the distribution shift issue, but the training cost remains high. 2) It's possible to speedup Proxy-FDA through a more efficient architecture design of our proxy generator, which we leave as future work.

Thanks for the detailed feedback! Below is our response to the main questions and "weaknesses".

Q1: Proxy diversity and novelty lack quantitative metrics. Could FID evaluate synthetic feature quality?

As suggested, many metrics are available to quantify our proxy feature diversity. Also, high proxy diversity often implies a high probability of proxy novelty, especially when the feature distribution is sparsely sampled and has a vast space of unseen data. Hence we focus on quantifying proxy diversity, and the novelty is simply examined through qualitative analysis — for example, one can perform some visual validation by image retrieval (Fig. 4) or even training a decoder on feature representations.

To quantify proxy diversity, we choose the variance loss in Eqs. (3-4), which is widely used in many domains like self-supervised learning to measure feature diversity. Note FID can be used to measure how well our generated proxies maintain the diversity of true features. Our adopted variance loss actually serves a similar purpose, since it is computed in an embedding space that's forced to align with the true one.

Here we report the averaged standard deviation of proxy features in the variance loss: higher value indicates larger diversity. To further aggregate the standard deviation values of the positive and negative proxies, we take their mean and compute its moving average till fine-tuning is completed. The table below compares the aggregated diversity metric of all the proxy generation baselines ablated in Fig. 7.

Our method clearly achieves higher proxy diversity than VOS/NPOS. The latter two methods focus on outlier synthesis in low-likelihood regions, thus miss the chance to encode diverse unseen data that are crucial for improving FDA. Our method also obtains marginally higher proxy diversity than random interpolation. More importantly, our learning-based method improves diversity in a way that best helps FDA: the diverse proxies not only enrich data but also refine the decision boundary between positive and negative feature manifolds. This is not possible with random interpolation, which explains its lower performance in Fig. 7.

Q2: Compare Proxy-FDA’s training time with that of retrieval-augmented methods.

Please refer to our response to Q2 of Reviewer bxDC.

Q3: Lack theoretical guarantees for FDA. Also, the correlation between OTDD and forgetting is observational, not causal.

Please refer to our response to Q2 of Reviewer 3vR2.

Q4: Hard class mining: does random batch sampling degrade performance? Comparing to entropy-based sampling.

As detailed in Appendix A, our batch sampling is performed by hard class mining plus random data sampling within class. Our ablation studies (Fig. 7) already compare with random batch sampling—the "No hard class mining" baseline—where classes are randomly sampled too. The average $\mathcal A_{\text{H}}$ is compared for few-shot prompt tuning, when we apply Proxy-FDA to CoOp/PromptSRC baselines. We see Proxy-FDA obtains $\mathcal A_{\text{H}}$ of 78.13/80.81 with hard class mining, and 75.23/80.35 with random batch sampling. As mentioned in Appendix E (L800-804), the big performance difference shows the hard class mining is crucial — it samples close class distributions, among which we can have meaningful modeling and matching of kNN graphs.

We now compare with the entropy-based batch sampling strategy that is also implemented under our greedy framework in Appendix A (for fair comparison). We simply change the inter-class similarity metric in step 2: our default hard class mining uses FDA loss to select similar class samples, while the entropy-based strategy prioritizes them by low entropy. The entropy-based strategy is found to have moderate decrease in $\mathcal A_{\text{H}}$ (77.46/80.75 vs. 78.13/80.81). This is because entropy cannot characterize similarity adaptively as a function of current feature distribution structure. As a result, the batch sampling criterion is decoupled with the structural FDA within sampled batch. While FDA loss-based sampling adapts to the feature distribution structure, and is coupled with FDA-based regularization in batch. Will add the results in paper.

Q5: Clarifications needed.

Results for 1–2 shot tuning are shown in Fig. 8 (more details in L860-865).

Table 4 (and L853-857) contains results across different foundation models and architectures (including the large-scale one ViT-L/14), where proxy generation (Proxy-FDA vs. FDA) improves $\Delta_{\text{LP}}$ consistently.

Thank you for the constructive feedback on our work. Below is our point-by-point response to your questions.

Q1: Motivation discussion: why Proxy-FDA improves performance.

(Proxy-)FDA is essentially a feature-space regularization term added to the task loss during model fine-tuning. The goal is to keep the fine-tuned model in the desired vicinity of the pre-trained one, so that the tuned model can preserve pre-trained knowledge while still learning the task at the same time. Extensive experiments show that (Proxy-)FDA can significantly reduce forgetting while achieving strong fine-tuning performance (sometimes better).

For better forgetting mitigation, our high-level idea is to extend existing point-wise feature regularization methods that lack explicit awareness of feature neighborhood structures. Proxy-FDA is proposed to fill this gap - it aligns the structural relations between kNN feature graphs, which is further improved by a proxy feature generator that increases feature diversity. Two empirical observations confirm our benefits, and hence reaffirm the motivation behind Proxy-FDA: 1) FDA can transfer the structural knowledge in kNN feature graphs, e.g. visual attribute shared between class concepts (Fig. 4). Preserving such common-sense knowledge is useful to maintain the generalizability of foundation model. 2) There's a strong correlation between forgetting and a structure-aware distributional distance metric OTDD (Fig. 3). Such correlation suggests the need of structure-wise feature regularization in some form to effectively mitigate forgetting, and our structural method Proxy-FDA is one such instantiation. In other words, this observation explains our advantage from an optimization perspective, i.e., optimizing our Proxy-FDA objective is close to optimizing a metric directly related to forgetting.

Q2: Theoretical explanations of the benefits of Proxy-FDA in reducing forgetting

Good suggestion. Note the primary focus of this paper is on empirical evaluation of our new method, along with two empirical observations (refer to our response to Q1) that shed light on the benefits of Proxy-FDA. However, we argue that it's promising to derive theoretical guarantees based on the 2nd observation that forgetting is strongly correlated with OTDD metric. Interestingly, OTDD is computed in an extremely similar way to our FDA loss -- they both use clustering techniques to account for the clustering structure of the underlying space, and hence both can compare feature distributions with rich geometry awareness. In other words, unlike L2 loss, our FDA loss is a good proxy of a metric (OTDD) that itself is directly related to forgetting. Such finding has two key implications: 1) There exists clear advantage of optimizing FDA loss over L2 loss-based optimization for direct forgetting prevention. 2) More importantly, the generalization error (or forgetting effect) could be bounded by some function of our own distance metric FDA (akin to OTDD). We leave such theoretical analysis for future work.

Thanks for the constructive feedback on our work. Below we include the results of requested experiments, and respond to your specific comments.

Q1: Justify motivation of using kNN feature-based FDA, and differences from knowledge distillation.

In response, our introduction section (paragraphs 3 & 4) motivates that we aim to improve over existing feature regularization methods that are often point-wise and preserve limited concepts since they lack explicit awareness of feature neighborhood structures. We propose Feature Distribution Alignment (FDA), a structure-wise regularization method that aligns the structural relations between pre-trained and fine-tuned feature distributions. The structural relations are modeled by feature graphs, and we choose kNN feature graphs because they not only enable efficient graph matching, but also are effective enough to capture the rich knowledge in local feature neighborhoods (e.g. shared visual attribute).

The related work section (L145-152) mentions that Proxy-FDA is indeed similar to Knowledge Distillation (KD), especially to those relational KD methods. Our main difference is that we distill knowledge from both neighbor indices and similarities, with an additional proxy learning component. Appendix G shows Proxy-FDA is directly applicable to KD and is quite performant compared to related KD baselines.

Q2: Improve FDA by kNN clustering with label constraints

During our FDA-based fine-tuning, class labels are mainly used in the task loss $\mathcal L_{\text{task}}$, while $\mathcal L_{\text{FDA}}$ is only treated as a feature regularization term without involving labels. The intuition behind the label-free $\mathcal L_{\text{FDA}}$ is that we aim to preserve a foundation model's general knowledge, which can be much richer than class labels on downstream datasets. More specifically, $\mathcal L_{\text{FDA}}$ matches kNN feature graphs to align their structural relations only based on feature (not label) similarities. Fig. 4 shows this can go beyond class concepts in a feature neighborhood with different classes (e.g., cross-class attributes), which is key to maintain the generalizability of foundation models. On the other hand, matching kNN graphs with label constraints may end up aligning class semantics on the downstream task, thus may risk forgetting the general knowledge embedded in foundation models.

In the table below, we empirically compare with an FDA variant that models and matches kNN feature graphs using both feature similarities $\hat w_{ij}$ and label similarities $w_{ij}^t$. Note we use the text encoder of CLIP to compute $w_{ij}^t$ as the text-text similarity between the class templates "a photo of a {class}". Comparisons are performed under the base-to-new setting for few-shot prompt tuning (average across 11 datasets). Results show that the use of $w_{ij}^t$ may produce comparable or better $\mathcal A_{\text{Base}}$ (i.e., fine-tuning accuracy on seen classes), but always leads to much lower $\mathcal A_{\text{New}}$ (i.e., worse generalization on unseen classes - more concept forgetting).