Towards Seamless Adaptation of Pre-trained Models for Visual Place Recognition

Thanks for your encouraging words and constructive comments.

The results of our SelaVPR on Pitts250k are shown in the table below. MixVPR achieves the previous best results on Pitts250k, and SelaVPR outperforms MixVPR.

The results of re-ranking different numbers of candidates are shown in the table below. Performance roughly reaches saturation when re-ranking top-100 candidates, which is our recommended setting for optimal recognition performance. Re-ranking top-50 (or even top-20) candidates is also a good option for faster two-stage retrieval.

Thanks for your positive recommendation and insightful comments.

[Q1:] Will re-ranking more candidate places achieve better performance or hurt the results?

[A1:] Re-ranking more candidates neither improves nor hurts performance. The results of re-ranking different numbers of candidates are shown in the table below. Performance roughly reaches saturation when re-ranking top-100 candidates, which is our recommended setting for optimal recognition performance.

[Q2:] Can the proposed SelaVPR still outperform R$^2$Former on Pitts30k using only MSLS for training?

[A2:] The results of R$^2$Former and our SelaVPR on Pitts30k using only MSLS for training are shown in the table below. Our SelaVPR also outperforms R$^2$Former. The results of SelaVPR trained only on MSLS are even better than those of SelaVPR only trained on Pitts30k (similar to the case with R$^2$Former), showing that our method has good transferability as well.

For weaknesses: The gap between the model pre-training and VPR tasks primarily refers to the difference in training objectives and data, which further results in the pre-trained models focusing on different objects from the specially trained VPR models. A robust VPR model should focus on the static discriminative backgrounds (e.g., buildings and trees) and ignore the dynamic foreground objects (e.g., pedestrians and vehicles), but a pre-trained model cannot do this, as vividly shown in the attention map visualizations in our paper. The failure cases mainly occur in some natural scenes that lack discriminative landmarks, which is difficult to address for existing VPR methods. It’s the focus of our future work.

Thanks for your positive recommendation and valuable suggestions.

[Q1:] Why is L2 distance used to measure global feature similarity, but dot product used to calculate local feature similarity?

[A1:] To make the first stage (global retrieval) compatible with some one-stage VPR methods and the benchmark [1], the L2 distance is used to compute global feature similarity. For the convenience of finding mutual matches (i.e. fast computing), local feature similarity is based on the inner product. Since local features have been L2 normalized, the inner product is equal to cosine similarity.

[Q2:] Is it feasible to re-rank top-k candidate images directly using coarse patch tokens from ViT? and how is the performance?

[A2:] It’s feasible in terms of program implementation. However, Re-ranking candidates with the coarse patch tokens from ViT does not make sense in two-stage VPR. Although it provides an improvement (but not as good as ours) on Pitts30k compared to global retrieval, it performs significantly worse than global retrieval on MSLS-val, where VPR methods are more susceptible to perceptual aliasing. That is, re-ranking with coarse patch features will bring negative effects and damage the results of global retrieval (i.e. the first stage retrieval).

For the weaknesses, we will improve the clarity in the revised paper and try to use the hybrid adaptation method for other visual tasks in future work. Thanks again for your suggestions.

Reference

[1] Gabriele Berton, Riccardo Mereu, Gabriele Trivigno, Carlo Masone, Gabriela Csurka, Torsten Sattler, and Barbara Caputo. Deep visual geo-localization benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5396–5407, 2022.

The following are Responses for the Weaknesses:

1. About the insights and contributions

Insights: The pre-trained foundation models (e.g. DINOv2) have achieved remarkable performance on many vision tasks given their ability to produce well-generalized representations. The recent AnyLoc work [1] also uses DINOv2 as the backbone, but without any adaptation or fine-tuning. Although AnyLoc uses a larger version backbone (based on ViT-giant), its performance is obviously worse than ours. Our work shows that there is a gap between the tasks of model pre-training and VPR (as we described in the third paragraph of the Introduction section). Model adaptation is a necessary thing to fully unleash the capability of pre-trained foundation models for VPR.

Contributions: The main contribution is the hybrid global-local adaptation method to bridge the gap and fully unleash the capability of pre-trained models for VPR. Our work provides an effective and efficient way to seamlessly adapt pre-trained foundation models to produce both global and (dense) local features for two-stage VPR. Moreover, we propose a mutual nearest neighbor local feature loss to train the local adaptation module, which is combined with global feature loss for fine-tuning. Different from other two-stage methods, the local features produced by our model are directly used in cross-matching for re-ranking, without time-consuming geometric verification.

For different applications: Our method is designed to seamlessly adapt the pre-trained foundation model for the VPR task. However, we think the hybrid adaptation method (to get both global features and local features) may also be useful for other applications.

Would it work for CNN: For CNN models pre-trained on ImageNet, adaptation is not necessary. We can fully fine-tune such a model directly. Most of the adaptation methods are designed for the foundation models (especially these transformer-based models) pre-trained on a large amount of data (larger than ImageNet).

About the performance: The powerful representation ability of the foundation model is the basis for good performance. But more importantly, our method bridges the gap between the tasks of model pre-training and VPR through seamless adaptation, i.e., the implication of doing adaptation is bridging the gap. The adapted model can produce the global features and local features more suitable for VPR to address the challenges in VPR tasks and achieve excellent performance.

2. Design motivations

Please see A2 for Q2. The design of our global adaptation is inspired by previous adapter-based work. However, we design a simple local adaptation method to adapt the ViT-based foundation models to produce dense local features for the local cross-matching in two-stage VPR. Our hybrid adaptation method combines the global adaptation and local adaptation to produce both global features and dense local features for two-stage VPR, which is different from previous adaptation works.

3. Data efficiency

The adaptation method can keep the powerful pre-trained image representations intact. When there is only insufficient training data in the downstream task, it mitigates the risk of over-fitting. This is a benefit shared by most adaptation methods. It is also an advantage of our approach over most other VPR methods. The experimental evidence is in the Appendix.A of our paper.

4. Model parameters

Please see A1 & A3 for the Q1 & Q3.

5. A missing reference

Thanks for the reminder. We’ll mention this work in our revised paper.

Reference

[1] Nikhil Keetha, Avneesh Mishra, Jay Karhade, Krishna Murthy Jatavallabhula, Sebastian Scherer, Madhava Krishna, and Sourav Garg. Anyloc: Towards universal visual place recognition. arXiv preprint arXiv:2308.00688, 2023.

Thanks for your comments and suggestions. We hope the following clarifications help to address your concerns.

[Q1:] How would the adapters work with other (smaller) models?

[A1:] The adapters can work with smaller models in the way described in the paper and achieve good performance. But it only makes sense if this model is pre-trained on a large amount of data and produces well-generalized representations. Otherwise, we can directly fully fine-tune the model. Here we provide the results (only fine-tuning the model on Pitts30k and testing on Pitts30k) of our adaptation method using a distilled version of DINOv2 (based on ViT-Base) as backbone, denoted as DINOv2-B. The DINOv2-L (based on ViT-Large) is the backbone used in our paper. Their performances are close, demonstrating that our adaptation method can be used with smaller models.

[Q2:] What are the reasons why the proposed adapters work?

[A2:] Although visual foundation models can provide powerful feature representations, the objects that need to be focused on may be different for various downstream tasks. Freezing the foundation model can retain the powerful representation capabilities, while inserting the lightweight trainable adapters so that the entire model can be adapted to the VPR task after fine-tuning on VPR datasets, that is, the trained adapter has learned VPR-related knowledge. Our visualization results well illustrate that the adapted model can focus on objects that are helpful for place recognition and ignore dynamic foreground interference, which is the key reason for its good performance. Meanwhile, our hybrid adaptation method combines global adaptation and local adaptation, which not only yields a robust global representation, but also enables the model to output dense local features for spatial matching re-ranking, thereby further boosting performance.

[Q3:] How the adapter parameter space influences the improve of performance, and how does it relate with 'smaller' backbones.

[A3:] The number of parameters of the adapter in our global adaptation is adjustable. Appropriately increasing the parameters of the adapter can slightly improve performance. However, in most cases, we can actually use a very lightweight adapter to obtain good performance. In the Appendix.C of our paper, we have shown the results of using only one adapter in each transformer block or reducing the bottleneck ratio to 0.25, and the performance is still good. Here, we provide the results (on Pitts30k) of using only the parallel adapter and reducing the bottleneck ratio from 0.5 (in the paper) to 0.05, as shown in the table below (SelaVPR** uses the more lightweight adapter. SelaVPR represents the original setup in paper). The SelaVPR** has only 5.19M trainable parameters (less than most methods), and the results are still excellent. For smaller backbones, fewer transformer blocks and lower token dimensions will naturally lead to fewer parameters of the adapter. Overall, the original paper focused more on recognition performance without excessively limiting the number of parameters. In fact, our method can achieve good performance using a significantly smaller number of parameters.