EffoVPR: Effective Foundation Model Utilization for Visual Place Recognition

We thank the reviewer for the positive feedback, acknowledging our SoTA performance, robustness (performing well on datasets from a distinct domain), impressive zero-shot performance, utilizing only reranking during evaluation makes training efficient and fast and having detailed ablation studies

W1: datasets so there’s not as much room for improvement...

WA1: We thank the reviewer for the comment. We compare our method on the common benchmarks of VPR. Additionally, we demonstrate our model’s superior performance on challenging datasets such as SF-XL Night, SF-XL Occlusion, Nordland and others. EffoVPR outperforms existing methods by a substantial margin, achieving improvements of +15.0, +7.9, and +4.3 in Recall@1, respectively.

W2: How much the reranking contributes to performance compared to DINOv2

WA2: Please note that the first row in Table 1 indicates the zero-shot DINOv2 results without re-ranking, while EffoVPR-ZS (zero-shot) indicates DINOv2 with our re-ranking strategy. We show these results below.

W3: R2Former also uses filtered local features from intermediate layers from the VIT backbone...

WA3: To the best of our knowledge, R2Former does not utilize local features from the intermediate layers of the ViT backbone but instead relies on the output layer. Specifically, R2Former leverages the attention map from the final layer to incorporate the most activated output tokens for its classification model. Our approach leverages different feature, K,Q and particularly V-facets from the model’s intermediate layers, that does not require any additional trained modules (in contrast to R2Former).

W4: Memory footprint…

WA4: Thank you for your comment. We provide a breakdown of the memory footprint in the table below, which has been added to the revised version of the paper. For demonstration purposes, we present the memory consumption for 1 million images and the entire SF-XL gallery.

Please note that we specify our ViT-L backbone in the implementation details section (see line 1012, in the original submission).

W5: More manual tuning involved…

WA5: Thank you! While our search process was somewhat manual, we believe there is potential for a better optimization to further enhance performance. We would like to emphasize that throughout all tests and benchmarks our hyperparameters remain fixed and we provide an analysis of the results’ sensitivity to hyperparameters in the ablation study.

Q1: Time evaluation and memory footprint

A1: Thank you for your comment. For the memory footprint in the first stage please see our answer above. To address other concerns, we summarize the runtime comparison with several other methods as reported in their corresponding papers.

Q2: More details for Table S5

A2: Thank you. The table shows the values where the counterpart parameter is set to its selected value. We have added this information to the paper.

Q3: Results on Pittsburg 250K

A3: Thank you. We have conducted this test. Please see below the requested results:

Q4: Train other datasets except SFXL

A4: Thank you for your comment. We discuss this in the appendix, section A.2. We follow the common practice, in recent VPR methods e.g. CosPlace, EigenPlaces, SelaVPR, CricaVPR, SALAD, which utilize a single dataset for training. Our training approach is based on EigenPlaces, which necessitates a dense coverage of 360 panoramas with orientation (heading) metadata. Other available training datasets lack the necessary metadata, preventing us from using them for training.

Q5: How well would the model perform if every layer of the backbone was trained?

A5: Please note that we specify and discuss this point in Table S1 of Appendix B, showing that training all layers is less effective.

Q6: What are the trained final layers

A6: We fine-tuned the last five layers as specified in line 522 (original submission) in the Ablation Study, and further discussed in Appendix B.

Q7: Paper calls itself a “foundational model”, Can the model be used for other tasks like cross view matching or detection?

A7: Please note that we follow the common terminology in the recent foundation based VPR methods, and also refer to our model as a “foundation based model”. This implies that the model backbone is a pretrained foundation model. Similar to other VPR methods such as SelaVPR, CricaVPR, SALAD, our task is for the same-view rather than cross-view.

Q8: Using global descriptor be used also during the reranking... only neighbors that belong in the same predicted cell.

A8: Thank you! This is an interesting idea. However, this approach leverages additional information from the geo-tags in the gallery, which is not typically used/allowed in standard comparisons. Furthermore, in certain competitions, such as MSLS-challenge, the geo-tags are sealed. Nonetheless, since this information can be available in practical scenarios, it presents a promising direction for future research.

Q9: How well does the model perform with anyloc as backbone rather than DINOV2?

A9: Please note that AnyLoc is not a backbone, but a method based on DINOv2 for a zero-shot VPR. We compare and show the superiority of EffoVPR in zero-shot setting over AnyLoc in Table 1.

Thank you for acknowledging our method as simple yet effective, and for finding our paper clear and easy to read.

W1: comparison in a single table

WA1: Thank you for this comment. Below we present this table that we also added to a revised version of the paper (Appendix C).

W2: The notation for Keys, Queries and Values

WA2: Thank you for this constructive remark. We have now omitted the index $l$ in the revised formulation for clarity and explained the dependency to layer in the text.

W3: Failure cases

WA3: Thank you! We have added several failure cases, categorizing and discussing these cases in the Appendix (section D.3) of the revised version.

Q1: Performance for different K values in zero-shot setting...

A1: Thank you for your comment. We have conducted an ablation study on EffoVPR-ZS Recall@1 performance across different K values on four datasets. Notably, EffoVPR's reranking method outperforms AnyLoc, the current state-of-the-art zero-shot method, on two datasets, even with K=5. However, achieving optimal performance with a low K value depends on the quality of the initial ranking success. In the zero-shot setting, the initial ranking is not highly accurate, so reranking performance generally improves as the K value increases.

Q2: ablations in the appendix using EffoVPR-R... confirm...

A2: Thank you for this comment. While most experiments in the ablation study were conducted using EffoVPR-R, a few were performed at the global level (e.g., feature compactness and number of trainable layers). We have ensured that this distinction is clearly stated in the revised version of the paper.

Q3: Re-ranking memory

A3: Thank you. Here we compare the memory requirements of our reranking method with other reranking methods, together with the associated performance on two datasets.

We have added this evaluation to the Appendix of the revised paper.

Thank you for the response. It addresses my concerns. I maintain my already positive score.

Thank you for your feedback. We are encouraged that you find our approach “novel” and with “really good performance”.

Q1a: methodological contribution...

A1a: Our paper highlights two primary contributions:

1. Leveraging ViT’s Intermediate Layers for VPR Matching: Current VPR reranking methods typically rely on training modules atop model output tokens. In contrast, traditional general matching algorithms consist of a simpler approach involving keypoint selection and descriptor extraction. Our exploration revealed that the attention maps from ViT’s intermediate layers encapsulate relevant information for this purpose. Specifically, we found that the Q and K facets serve as high-quality keypoint selectors, while the V facet provides robust descriptors. This straightforward yet effective method enhances performance in both zero-shot and fine-tuning scenarios, outperforming current approaches.

2. Challenging the Need for Complex External Pooling Layers: Many state-of-the-art methods depend on sophisticated, learnable pooling layers that often result in large global features (e.g., NetVLAD and GeM). We question the necessity of these layers and propose a simple yet highly effective alternative. Our approach not only enhances performance but also significantly improves feature compactness. We believe that this perspective offers valuable insights to the VPR community regarding the role of external pooling layers in modern methods.

These contributions enable us to achieve state-of-the-art results across 20 benchmarks, demonstrating exceptional robustness under diverse conditions—day and night, seasonal changes, and occlusions—while delivering a level of feature compactness that surpasses previous methods by orders of magnitude.

Q1b: state that SelaVPR works also without geometric verification.

A1b: Thank you. We have added it to the revised version in line 172.

Q2: training on other datasets

A2: Thank you for this comment. Various VPR methods utilize different datasets for training. Our training approach is based on EigenPlaces, which necessitates a dense coverage of 360 panoramas with orientation (heading) metadata. MSLS, Pitts30k, and GSV-Cities lack the necessary metadata, preventing us from using them for training.

We have added this explanation to the Appendix section A.2.

Q3: GPU memory usage and results on Nordland and SF-XL-Night

A3: Thank you for the suggestion. Indeed, EffoVPR performs exceptionally well even at lower resolutions. Therefore, it is important to also consider memory footprint and computation time when selecting the input resolution. Below we provide the requested analysis and results. We have extended evaluation on three more datasets, Nordland, SF-XL-night and AmsterTime. Both GPU memory footprint and feature extraction time increase as the resolution rises. Our results remain SoTA across all resolution inputs.

We have added these results to the Ablation Study and Table S7 in the Appendix.

**Q4: Computational complexity and memory footprint **

A4: Thank you for your comment. We provide a breakdown of the memory footprint in the table below, which has been added to the revised version of the paper. The table reports memory usage for 1 million images based on the corresponding feature vector size, along with the equivalent memory footprint for a real-world scenario, using the SF-XL gallery (2.8M images) that covers the city of San Francisco.

We compare our method’s reranking runtime and memory footprint with several other methods as reported in their corresponding papers:

We have also conducted an evaluation of GPU memory footprint and runtime, analyzing performance across different input sizes. SelaVPR and CricaVPR used 224px resolution and SALAD 322px. EffoVPR-R achieves SoTA performance over all resolutions.

We have added a special section in Appendix (Section F) presenting details on memory footprint and runtime

Q5: Failure Cases

A5: Thank you! We have added several failure cases, categorizing and discussing these cases in the Appendix (section D.3) of the revised version.

Thank you for your detailed and insightful feedback and finding our paper "easy to follow" and "improving the effectiveness of image retrieval". We address all your concerns below.

Q1: Proposed method based on foundation model... technical novelty incremental...

A1: Please note that most current SoTA VPR methods build upon strong foundation model (DINOv2) [1][2][3]. We just follow this line of work. However, our contribution extends beyond the use of strong foundation models and lies in two key novel aspects.

1. Challenging the prevalent practice in foundation-based models of relying on external aggregation modules and questioning their necessity.

2. Unlocking the encapsulated strength of foundation models, in their intermediate layers. Here we introduce a novel keypoint extraction and descriptor computation module, based on different K, Q and V self-attention facets extracted from intermediate layers. These keypoints and descriptors demonstrate significant robustness, making them highly effective even in zero-shot setting (surpassing the SoTA method of AnyLoc) and excels in reranking after fine-tuning.

We ultimately achieve state-of-the-art performance across 20 benchmarks. We believe these contributions are significant and far from trivial.

[1] Lu, Feng, et al. "Towards Seamless Adaptation of Pre-trained Models for Visual Place Recognition." In ICLR, 2024.

[2] Lu, Feng, et al. "CricaVPR: Cross-image Correlation-aware Representation Learning for Visual Place Recognition." In CVPR, 2024.

[3] Izquierdo, S., & Civera, J. (2024). Optimal transport aggregation for visual place recognition. In CVPR, 2024.

Q2: Design choices... lack clear motivation and justification.

A2: Thank you for your comment. We believe that we addressed this concern in our original submission, in the Related Work section (lines 192-193), where [4] and [5] provide key motivation and intuition for our approach. To elaborate further, we draw attention to AnyLoc, which emphasizes that per-pixel features from off-the-shelf foundation models [1] demonstrate remarkable visual and semantic consistency.

Additionally, prior studies have extensively highlighted the robust properties of SSL-trained models, including their semantic and instance-level capabilities [1,2,3]. For instance, [2,4,5] studies investigate various aspects of feature representation, in appearance and semantic properties, while [5] specifically discusses the roles of K, Q, and V facets in such tasks. Building on these insights, we are the first to analyze and reveal their potential for VPR matching.

Our approach leverages K and Q for keypoint detection and utilizes values (V) as descriptors for these keypoints. A detailed analysis of these design choices is provided in the Appendix (Table S3, referenced in line 304, in the original submission).

To emphasize this motivation more clearly, we have slightly revised the introduction section, dedicating a separate paragraph to address this issue.

[1] M. Oquab, T. Darcet et al., “Dinov2: Learning robust visual features without supervision,” arXiv:2304.07193, 2023.

[2] M. Caron, H. Touvron, I. Misra et al., “Emerging properties in selfsupervised vision transformers,” in ICCV, 2021.

[3] N. Park, W. Kim, B. Heo, T. Kim, and S. Yun, “What do selfsupervised vision transformers learn?” in ICLR, 2023

[4] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep ViT features as dense visual descriptors. ECCVW, 2022.

[5] Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Splicing vit features for semantic appearance transfer. In CVPR, 2022.

Q3: Improving could come from stronger foundation...

A3: Please note that we compare our method with several recent approaches that all utilize DINOv2, including SelaVPR (ICLR 2024), CricaVPR (CVPR 2024) and SALAD (CVPR 2024). Additionally, we incorporated another model formally published after the ICLR deadline – VLAD-BuF (ECCV 2024), which is also based on DINOv2. These comparisons are provided below, and we believe they ensure a fair and consistent evaluation.

Thank you for your time and effort in reviewing our paper and rebuttal. We are happy to have successfully addressed your concerns and appreciate the increased score.