No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations

Other evaluations (linear probing, k-means, in-context learning)

We conducted further evaluations of FUNGI features for logistic regression, clustering and text classification via in-context learning. The results are discussed in the global response.

What's the dimension of the gradient feature? What's the time cost of random projection and PCA?

Assuming a ViT-B/16 backbone, the weight gradients from the last attention output projection have shape $(768, 769)$, including the bias. We project them to the same dimensionality as the embeddings, in this case $768$. The final FUNGI features (before PCA) will then have shape $768 \times (N + 1)$, where $N$ is the number of gradient features used.

Performing the random projection on the GPU for a batch of $8$ samples results in an overhead of $0.017$ seconds, or $0.0002$ seconds per sample.

As for the PCA, fitting it over the entire training set of EuroSAT ($18,000$ samples) requires $5.18$ seconds, while transforming the training set requires $0.17$ seconds, or $0.00001$ s/sample. For larger datasets an incremental version of PCA can be used.

The reported runtimes were averaged over $1000$ runs ($100$ for the PCA fitting).

What properties does a binary random projection have? What is the advantage?

TRAK [98WT-5] uses a random projection sampled from $N(0, 1)$ to reduce the dimensionality of gradients, which, according to the Johnson-Lindenstrauss lemma, preserves inner products with high probability, given a large enough projection subspace $k$. The main result of [C] is demonstrating that this lemma holds also for matrices whose entries are sampled from either $\\{-1, 1\\}$ or $\\{-1, 0, 1\\}$.

Empirically we show that this holds true by evaluating the k-NN classification accuracy on Flowers102 of raw and projected KL gradients from a DeIT ViT-S/16 backbone.

Table 28. Binary random projections preserve euclidean distances: k-NN classification using raw and projected gradients from a DeIT ViT-S/16 backbone. Projected gradients perform marginally worse, showing that relative euclidean distances are mostly preserved.

One advantage of a random matrix with $\\{-1, 1\\}$ entries is that it can be efficiently stored as a boolean matrix. For example, the projection matrices used for ViT-B models have a shape of $(768 \times 769, 768)$, equivalent to ${\sim}1.8$ GB when encoded as float32 and to $0.45$ GB when encoded as booleans.

We also evaluate the difference in performance given by random projection matrices using different initializations, shown in Table 29, which displays the mean per-class accuracy on Flowers102 of FUNGI features extracted from a DeIT ViT 16/B backbone with respect to the projection initialization. The results show that a binary matrix has a slight advantage over Gaussian and sparse matrices, and has a lower standard deviation as more sub-features are used, but the performance gap is negligible.

Table 29. The random projection initialization has little impact on performance: comparison of the downstream accuracy of FUNGI features built with gradients projected using random matrices with different initializations. We report the mean and one standard deviation measured across three runs using the Flowers102 dataset and a DeIT ViT-B/16 backbone.

PS: We assumed that for this question you were referencing [98WT-5] (TRAK) rather than [98WT-6] (Learning by Reconstruction..), as the former has the exact sentence mentioned in the review. Please correct us in case.

Missing related works

We highly appreciate the suggestion of these works, we discussed them and used some of their results to build an intuition in the global response on why/how gradient features work. We will be include them in the updated version of the paper.

Building on that discussion, our work is also related to [98WT-2], which used the empirical neural tangent kernel (eNTK), i.e. the per-sample Jacobian, to benchmark an estimator of generalized risk, and also shows that kernel regression on the eNTK can achieve a performance close to fine-tuning in vision. Their work is extended by [98WT-4], which shows that eNTK can be used for prompt-based fine-tuning of masked language models, if the task exhibits kernel behavior. In comparison to their work, our method does not require the downstream task to show kernel behavior, and we expect their method to be more computationally intensive, as they compute the full model Jacobian for each sample. [98WT-3] takes inspiration from NTK and propose using gradients from task-specific losses as features for representation learning. Given a pre-trained network, they first train a linear classifier in a supervised manner and subsequently learn a linear model using both activation and gradient-based features. FUNGI differs from this work by not requiring any human labeled samples and not running any training as it simply uses the gradients as features for k-NN classification.

References

[C] Database-friendly random projections: Johnson-Lindenstrauss with binary coins. Journal of computer and System Sciences (2003).

Thank you for the detailed reply.

About evaluation on other tasks: Evaluation on linear probe, clustering and in-context learning are good. It's also great to see that these experiences are carefully conducted in a correct way, e.g., to choose L2-regularization strength for linear probe. From my own experience, the performance of linear probe would be tricky without a proper regularization strength.

About related work and intuition of FUNGI: the explanation that FUNGI can be regarded as update-free fine-tuning on other objectives is interesting.

About comparison to random Gaussian matrix: Yes I'm mentioning TRAC [5] for the projection with random Gaussian matrix. I agree that random binary matrix is more memory-efficient.

About choice of gradient: it seems that you could choose the gradient from [attn_qkv, attn_proj, mlp.fc1, mlp.fc2] as features. From Figure 8 in the paper, gradient in the early layer seems not helpful. In practice, can you explain which part of gradient to choose in order to get best performance? Will the combination of gradients from different layers/modules be helpful (many people just use full gradient of the model, which is very expensive and might not perform very well)?

Overall, I'm glad to see the response from the authors. I'd raise my score to 8 and hope athors would include these changes in the modified version (especially the intuition of using gradient feature) and release their code in the future.

Missing backbones

We extend the evaluation of FUNGI to DeiT-III, MobileCLIP and CLIP and MoCov2 ResNet-50 backbones. For MobileCLIP, we extract gradient features from the attention output projection of the last token mixer, and for ResNet-50 models we use the last convolutional layer, i.e. layer4.2.conv3. For ResNet-50 models we only evaluate KL and DINO gradients, as SimCLR gradients did not seem particularly promising.

Although we find that our method works consistently with ViTs, the results on CNNs or ViT-CNN hybrids are less consistent. While our method improves on MoCov2 ResNet-50, it does not improve on the CLIP ResNet-50 and MobileCLIP-S1. We would like to stress that these are preliminary results, and different data augmentation strategies or a more throughout evaluation of the possible gradient sources (i.e. layers) may yield better results for CNNs or ViT-CNNs. The results, averaged over 11 datasets, are reported in Table 26 and Table 27 for full dataset and few shot setups respectively.

Table 26. FUNGI works primarily for ViTs: accuracy in full dataset k-nearest neighbor evaluation averaged over 11 datasets for FUNGI and embeddings.

Table 27. FUNGI works primarily for ViTs: accuracy in few shot k-nearest neighbor evaluation averaged over 11 datasets for FUNGI and embeddings.

Scalability

In Table 18 of the uploaded PDF we report results for AugReg (labelled as "AR"), CLIP and DINOv2 ViT-L models, and a DeiT ViT-H/14 model, for full dataset and few shot setups. The results are averaged over 11 datasets, except for DeiT ViT-H/14, whose results exclude CIFAR 10 and 100, Food-101 and ImageNet-1K due to computational reasons.

The results show that FUNGI works also for larger backbones, especially in few-shot setups, where it improves by $+4.4\\%$, $+4.3\\%$ and $+2.1\\%$ for CLIP ViT-L/14, DINOv2 ViT-L/14 and DeiT ViT-H/14 respectively.

Does FUNGI extend beyond classification tasks? Can it be used for generative models such as LLMs, VLMs such as Llava, and BLIP?

We're thankful for this suggestion, and yes, in the global response we show that FUNGI can be used to improve the performance of linear probing, clustering and that it can be used to retrieve better example for in-context learning with LLMs.

As for integrating FUNGI in the pipeline of generative models such as BLIP or LLaVA we believe it should be possible, as they both use a frozen vision encoder. But, taking BLIP as an example, the vision features are used by the text decoder via cross-attention, and thus we expect FUNGI features to be out-of-distribution without at least a partial re-training or fine-tuning. We did not have enough time to setup such an experiment during the rebuttal, but we agree that it would be an interesting direction to explore.

Computational cost

Our method does indeed introduce a computational overhead, but we believe that in a retrieval setting, where the embeddings for the database are computed once and only query samples are encoded on the fly, the performance improvement may be very well worth the added computational cost.

In particular, on retrieval on the revisited Paris landmarks dataset [B], our KL loss improves the mean average precision (mAP) of a CLIP ViT-B/16 backbone by $+10.1\\%$ and $+12.4\\%$ on the medium and hard splits respectively, at the cost of a ${\sim}28\\%$ reduction in throughput, including the random projection. The results are also shown in Table 24.

Table 24. FUNGI improves retrieval: retrieval mAP on the Paris landmarks dataset of embeddings and FUNGI features built with KL gradients using a CLIP ViT-B/16 backbone.

Beyond that, we note that the main scope of this work was to demonstrate that gradient-enhanced features can improve performance across several tasks, modalities, and backbones. We acknowledge that our method, particularly with the DINO and SimCLR losses, introduces a computational overhead due to the requirement of multiple views for each individual sample. However, we believe that it is possible to make the computation of these gradient features significantly more efficient, which we leave for future work.

Why are gradients and embeddings complementary?

In the global response we built some further intuition on why SSL-gradient features can improve the pretrained embeddings. Moreover, the second row of Figure 2 in the main paper shows that gradient alone can already provide a significant performance improvement, thus it's not strictly necessary to combine them with the embeddings.

On the other hand, gradient features may be "brittle", as gradients are estimated using a single data point and depend on the local curvature of the loss landscape. Thus, we hypothesize that combining gradients and embeddings may address this issue and provide more stable yet discriminative features, which also result in better performance overall.

What is the role of pre-trained embeddings? Can random or half-trained embeddings be used to extract discriminative information?

Thank you for this interesting idea and question, that can bring new insights into the utility of gradient features. Indeed, it is possible to extract discriminative information from both a half-trained and random embeddings. For the former, as we do not have access to a half-trained backbone, we consider a Masked Autoencoder ViT-B/16, which is known to have bad performance in retrieval and requires (almost) full-fine tuning for peak performance, while for the latter we consider a randomly initialized ViT-B/16.

In Table 25 we show the performance of embeddings and FUNGI features for these backbones, averaged over 11 datasets. The results show that it's indeed possible to extract discriminative information from such embeddings, in particular we improve over a random embedding by $6.25\times$ and by $1.89\times$ on MAE embeddings.

Table 25. FUNGI significantly improves retrieval for random or half-trained backbones: average accuracy, over 11 datasets, in full dataset k-nearest neighbor classification for a MAE and a randomly initialized backbone.

References

[B] Revisiting Oxford and Paris: Large-scale image retrieval benchmarking. CVPR 2018.

Can task specific loss functions help with adaptation?

In this work, we only use self-supervised losses that do not need human labels. If we understand correctly the reviewer's question, task-specific losses such as cross-entropy or a ranking loss are supervised and would require us to have access to the labels. Moreover, if that were the case, we could easily achieve perfect accuracy, as the label information would "leak" into the gradient features, as they are calculated on a per-sample basis.

On the other hand, different losses exhibit different properties, e.g. DINO, being a clustering loss, is particularly helpful in improving the k-means clustering performance, as shown in Table 21 of the uploaded PDF. In particular, using FUNGI features from a CLIP ViT-L/14 backbone, we can improve the cluster/class overlap by up to $+15\\%$.

How much is the method dependent on the PCA? Can other alternatives be used?

The primary role of PCA in our method is to produce features that have the same storage and compute requirements (for retrieval) as the model embeddings. In Table 11 of the appendix (reported here as Table 22 for convenience) we compare the performance of PCA-transformed features to raw features, and find that, on average across 11 datasets, PCA provides a consistent minor performance boost (up to $+0.3\\%$), but it's not essential for good performance of FUNGI features.

Table 22. PCA slightly improves performance, but it's not essential: performance of FUNGI features and embeddings from a DeIT ViT-B/16 backbone transformed (or not) using PCA, averaged over 11 vision datasets.

As for the strategy used for dimensionality reduction, in Table 23 we evaluate the performance of using no reduction, PCA and random projections with different initializations, and find PCA to be the best across the board.

Table 23. PCA is the best dimensionality reduction method: mean per-class accuracy on Flowers102 of embeddings and FUNGI features from a DeIT ViT-B/16 backbone transformed with different dimensionality reduction methods. We report the mean performance and one standard deviation across three seeds.

Can FUNGI features be consumed by downstream discriminative classifiers that are trained with softmax?

Yes, we conducted a thorough evaluation of FUNGI features in logistic regression. See the global response for details.

Are the reported numbers the mean of multiple runs?

The numbers reported in the paper refer to a single run, except for Table 16 in the appendix, where, for 8 datasets and a DeIT ViT-B/16 backbone, we report the mean accuracy over three runs and one standard deviation, which is generally low, falling below $0.3$ in most cases, and being $0.6$ at most, indicating that our method provides consistent results.

We’re thankful to the reviewers for their comments. We’re happy to answer any further questions or clarify our responses as needed.