SuperVLAD: Compact and Robust Image Descriptors for Visual Place Recognition

Thanks for your positive recommendation and constructive comments, as well as the recognition of our compact descriptor (especially for 1-cluster VLAD). We will move the limitations discussion to the main paper as you suggested. The following are Responses to the Questions (denoted as Q) and Weaknesses (denoted as W).

Q1. About “class token”:

The “class token” is output by the DINOv2 model, which can also be called ViT class tokens (DINOv2 is a ViT model trained on the large-scale curated LVD-142M dataset). It is also finetuned on GSV-Cities (VPR dataset). That is, all methods in Table 5 have the same settings (backbone, training setting, etc.) except for the descriptor, to compare the performance of descriptors fairly.

Q2. Inference time of CricaVPR:

The inference time of CricaVPR is 17.2ms (without PCA), which is slower than our SuperVLAD (13.5ms). We will add it to Figure 5. Besides, CricaVPR uses the optional PCA operation for dimensionality reduction, which requires more inference time if PCA is added.

Q3. Using the same feature dimensionality to compare via a final FC layer:

We agree with you that adding the final FC layer allows different methods to output features of the same dimensionality. However, the recent DINOv2-based works (e.g., SALAD and CricaVPR) do not do this. If we directly add a final FC layer to retrain the models, it will involve the issue of optimal training parameter setting, which may make the result much worse than the original works. In this case, it is also unfair to the other works. By the way, our method aims at producing compact descriptors without additional dimensionality reduction/ transformation techniques. So, we do not add a final FC layer, which ensures that our aggregation module is lightweight.

W1: Large-scale training dataset vs. the modified aggregation method

We agree with you that the large-scale training dataset can bring more improvements than the modified aggregation method for VPR in most cases. However, a robust aggregation algorithm is important when large training datasets are lacking. It is helpful to improve the performance floor in bad cases.

W2. The novelty of the proposed method:

It's important to note that NetVLAD is a very successful method but its domain generalization performance is still hampered by cluster centers. Our improvement is reasonably motivated and can address this import issue (also gets good results). Unlike previous improvements that usually complicate NetVLAD, our improvements simplify it by directly aggregating local features assigned to each cluster, which no longer computes residuals and is therefore free from the cluster centers (the insight is novel). That is, SuperVLAD is a simple yet effective method (to enhance the generalizability across different domains).

W3. The illustration of typical failure cases:

Thanks again for your suggestion. We provide detailed failure cases in the attached PDF (Figure 2), including cases where the retrieved image is geographically close to the query image but outside the 25-meter threshold, as well as some cases of challenging natural scenes without landmarks. We will add these failure cases to our final paper.

Thanks for your positive recommendation and insightful comments. The following are Responses to the Question (denoted as Q) and Weaknesses (denoted as W).

Q. Inconsistencies in datasets between Table 2 and Table 3:

Sorry to confuse you. We compare our SuperVLAD with other methods on four benchmark datasets, as shown in Table 2. However, considering that we have a large number of ablation experiments, all ablation experiments, i.e., Table 3,4,5,6,7, are only conducted on the two most commonly used datasets (Pitts30k and MSLS) to show the performance more clearly. Besides, Pitts30k and MSLS have their own training and test/val sets, so we can simply use cross-validation (training on Pitts30k and testing on MSLS, and vice versa) to demonstrate the ability of SuperVLAD to address domain shift.

W1. The modification to NetVLAD:

We agree with you that NetVLAD is a mature method. However, the robustness against domain shift is still an important issue to be addressed. Our SuperVLAD is reasonably motivated and gets good results. Different from previous improved methods that usually complicate NetVLAD, our improvements simplify it to enhance the generalizability across different domains (the insight is novel). That is, our SuperVLAD is a simple yet effective method.

W2. The performance improvement:

Although the recognition performance improvement of our SuperVLAD over the previous SOTA method is not very obvious, our method is more lightweight than SOTA methods like SALAD. The number of parameters in the SuperVLAD aggregator (0.0038M) is less than 3/1000 that of the SALAD aggregator (1.4M). The output descriptor dimension of SuperVLAD (3072) is less than 1/2 that of SALAD (8448).

W3. The experiments on the indoor dataset:

Thanks for your suggestion. We use the trained models (on GSV-Cities) to test on the indoor dataset Baidu Mall [1][2] to support the generalization claim. SuperVLAD achieves better results than NetVLAD (DINOv2-based), as well as other SOTA methods on this dataset (see the following two tables), which further supports our claim.

For the suggestion in Limitations:

Thanks again for your suggestion. We provide examples of datasets to show the data domain gap between Pitts30k and MSLS in the attached PDF (Figure 1). We will add it to our final paper.

References

[1] Sun, Xun, et al. "A dataset for benchmarking image-based localization." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.

[2] Keetha, Nikhil, et al. "Anyloc: Towards universal visual place recognition." IEEE Robotics and Automation Letters (2023).

Thanks for your valuable comments and suggestions. We hope the following clarifications will be able to address your concerns.

W1. About the number of clusters:

Sorry to confuse you.

• First, since we did not specifically emphasize that as the capabilities of the backbone model improve (e.g., DINOv2) and the amount and quality of training data increase (e.g., GSV-Cities), both NetVLAD and SuperVLAD can achieve excellent performance even with a small number of clusters, this may lead readers to misunderstand that it is unique to SuperVLAD. We will improve it in the revised paper. However, when using simple models or smaller training datasets, the performance of a small number of clusters is not as good as that of a large number of clusters, and our SuperVLAD can outperform NetVLAD with a small number of clusters, as shown in Table 7. Besides, to our best knowledge, our work is the first attempt to set the number of clusters to be very small (only 4).

• Second, for setting the number of clusters to 1, 2, and 3, we provide the following explanations and experimental results. Setting the number of clusters to 1 will make the softmax layer for soft assignment meaningless. Our 1-cluster VLAD is achieved by adding additional ghost clusters, and there is not just one cluster during soft assignment. In fact, if there is only one cluster, all features are assigned to one category and the weight is "1". So SuperVLAD directly sums up all local features, which is equivalent to global average pooling (multiply by a constant), while NetVLAD subtracts a constant vector from the SuperVLAD vector (equivalent to translating the feature space), which does not affect the similarity calculation, i.e., the retrieval results are consistent. As for setting 2 or 3 clusters, the conclusion is the same as that in Table 7. That is, for using the powerful backbone (DINOv2) and the large-scale GSV-Cities dataset, the recognition performance of SuperVLAD and NetVLAD is nearly equal (but our SuperVLAD is simpler and more lightweight). However, for training on smaller datasets like Pitts30k, or/and using the simpler backbone like CCT, our SuperVLAD can outperform NetVLAD on recognition performance with an obvious margin. The detailed results (R@1/R@5/R@10) are shown in the following tables. Both NetVLAD and SuperVLAD are without ghost clusters and the cross-image encoder for fair comparison.

① When training DINOv2-based models on GSV-Cities, both NetVLAD and SuperVLAD achieve good results without obvious differences on the R@N metric.

② When training CCT-based models on Pitts30k, SuperVLAD outperforms NetVLAD, with an obvious margin on Pitts30k and an even larger margin on MSLS (with 12.3% absolute R@1 improvement on MSLS-val with 3 clusters). This shows that our SuperVLAD performs better than NetVLAD with a simple backbone and small training dataset, i.e., the lower limit of performance is higher than NetVLAD. Besides, it once again verifies that SuperVLAD is more robust than NetVLAD when there is a domain shift in training data and test data (i.e., training on Pitts30k and testing on MSLS). By the way, 2 and 3 clusters get obviously worse results than 8 clusters (see Table 3 in paper).

From the above results, we can summarize the advantages of SuperVLAD as follows:

a. When using a powerful backbone and a large-scale training dataset, SuperVLAD achieves the nearly same performance as NetVLAD using fewer (about half) parameters.

b. When using a simple backbone and a small training dataset, SuperVLAD can outperform NetVLAD by an obvious margin.

c. When there is a domain gap in training data and test data, SuperVLAD is more robust than NetVLAD.

W2. Results of NetVLAD with 4 clusters, DINOv2, and cross-image encoder:

• First, we agree with you that the ghost cluster does not improve the performance when we use the powerful DINOv2 backbone. Considering that the ghost cluster can bring improvement (by eliminating useless information) when using a simple backbone (as shown in Table 4 of our paper), we adopted it in SuperVLAD.
• Second, we also agree with you that showing the results of NetVLAD with 4 clusters, DINOv2, and cross-image encoder can better analyze the improvements from other techniques used in SuperVLAD. The results (trained on GSV-Cities) are shown in the following table. After adding other techniques to both SuperVLAD and NetVLAD (denoted as NetVLAD*), SuperVLAD can outperform NetVLAD on recognition performance, i.e., SuperVLAD is more suitable for adding these technologies. (We have shown in our paper that the recognition performance of DINOv2-based SuperVLAD and NetVLAD is comparable without adding other technologies, in which case the advantage of SuperVLAD is simpler, more lightweight, and robust to domain gap.)

Thanks again for your thoughtful review, and please let us know if you have further questions.

Thanks for your insightful comments and valuable suggestions. The following are Responses to the Questions (denoted as Q) and Weaknesses (denoted as W).

Q1. Results of NetVLAD in Table 2:

Both of our results and the results in the SALAD paper are correct. We use the original version of NetVLAD as described in Appendix G (also can be seen in NetVLAD paper), i.e., VGG16 backbone and trained on Pitts30k. The results of DINOv2-based NetVLAD trained on GSV-Cities (basically the same as in SALAD) are shown in Table 3 and Table 7 of our paper, which basically matches the results in SALAD. Thank you for pointing out our wrong citation in Table 2.

Q2. SuperVLAD perform with ResNet-50:

Our SuperVLAD with ResNet-50 cannot outperform MixVPR. To take full advantage of SuperVLAD we need to use the Transformer-based backbone, which is a limitation of SuperVLAD and has been discussed in Appendix A (Limitations) and Appendix C (Results with VGG16). Considering that more and more VPR methods are using Transformer models and even the DINOv2 model, we can also use the DINOv2 backbone uniformly for fair comparison. There is a DINOv2-based MixVPR proposed, i.e. DINO-Mix [1], and SuperVLAD outperforms DINO-Mix with an obvious margin.

Q3. Total number of parameters and trainable parameters:

We appreciate your attention to the details in Table 2. It's important to note that the methods compared in this table use different backbones, which may impact the direct comparability of the number of parameters. To address this concern, we provide a comparison of SALAD and SuperVLAD that both use the DINOv2-base backbone and only finetune the last four Transformer blocks of the backbone. The value in parentheses is the number of parameters in the optional across-image encoder.

The number of parameters in the aggregator is the most important consideration. The number of parameters in our aggregator is much lower than that of SALAD (less than 3/1000). As far as we know, the SuperVLAD aggregator has a smaller number of parameters than any other common methods except global pooling methods (e.g., GeM).

W1. About the readability of the results:

Thanks for your suggestion to improve the readability of our paper. We will improve it in the final paper. We have introduced the number of clusters in the table caption of Table 3 and we will move it to the table. For the same backbone, the improvement of SuperVLAD mainly exists when there is a gap between the training data and the test data (e.g., training on Pitts30k and testing on MSLS, see the reply to W2 for more details), which is also fits our motivation. In addition, we use fewer parameters than NetVLAD not only because of the smaller number of clusters, but also by removing the cluster centers, which reduces the parameters of NetVLAD by about half even with the same number of clusters. We will emphasize this point as you suggested.

W2. The impact of discarding cluster centers:

Discarding cluster centers can improve robustness when there is a domain gap between the data in training and inference. As shown in Table 3, when the models trained on Pitts30k (only urban images) are tested on MSLS (including urban, suburban, and natural images), our SuperVLAD outperforms NetVLAD. More specifically, discarding cluster centers brings an absolute R@1 improvement of 6.8% (using CCT backbone) and 2.4% (using DINOv2) on MSLS-val. However, for the models trained on the GSV-Cities dataset, there is little difference in results between SuperVLAD and NetVLAD because GSV-Cities covers diverse training data (i.e., no obvious gap between training and test data). Please refer to Lines 312-324 in the paper for more details.

W3. Adding the evaluation on the SPED dataset in Tab 4:

The results on SPED with or without (denoted as “-ng” suffix) ghost cluster are shown in the following table. There is no obvious difference between them. Considering that it can eliminate useless information and bring improvements on some datasets, we still choose to keep it. More importantly, we use it to achieve the 1-cluster VLAD.

W4. Evaluation on Pittsburgh-250k and Tokyo 24/7: The results on Pitts250k and Tokyo24/7 are shown in the following table. SuperVLAD also achieves SOTA results, showing good performance in day-night changes.

W5. a. About MixVPR with ResNet50 and b. training data of models in Table 2:

• a. Please see the response to Q2.
• b. The training data for MixVPR, CricaVPR, SALAD and our SuperVLAD is GSV-Cities (as described in Section 4.3), and for other methods is described in Appendix G.

Thanks again for your thoughtful review, and please let us know if you have further questions.

References

[1] Huang, Gaoshuang, et al. "Dino-mix: Enhancing visual place recognition with foundational vision model and feature mixing." arXiv preprint arXiv:2311.00230 (2023).

Thanks for your positive recommendation and encouraging words. The following are Responses to the Questions.

Q1. Integrate SuperVLAD with visual localization and SLAM systems:

Since SuperVLAD is a general visual place recognition (VPR) method, it can be used for visual localization and SLAM systems like previous standard VPR methods such as NetVLAD. For SLAM systems, our SuperVLAD can be used as the loop-closure detection method to rectify the accumulated mapping error. As you know, our SuperVLAD is more lightweight and can produce compact descriptors, so it is more suitable for resource-constrained environments like mobile robots SLAM. Given the good recognition performance of SuperVLAD, it may not require spatial consistency verification in loop-closure detection, which requires us to further explore its recall performance at 100% precision in future work. For visual localization, it usually only requires the VPR method to retrieve top-N candidate images (obtain location hypotheses), then perform local matching and compute 6-DoF pose [1]. Our experiments have shown that SuperVLAD can achieve good Recall@N performance and is therefore suitable for visual localization.

Q2. Experimental details: inference time, training time, and training data:

The experimental details including training data (i.e., GSV-Cities) and some other settings are described in Section 4.2 and Appendix D. The inference time is 13.5ms for a single image, as shown in Section 4.3 (Figure 5). The training time of our SuperVLAD is 81.6 minutes (training for 7 epochs on GSV-Cities) using two NVIDIA GeForce RTX 3090 GPUs, compared to 210 minutes for CricaVPR [2], which uses the same backbone and training dataset as ours. We will add the training time to the final paper.

Q3. How this application is important to ML, how ML can impact this area and what can SuperVLAD bring:

This is a very meaningful question. We should think more about the impact that the VPR task and our method (SuperVLAD) can bring to the ML community.

• First, the VPR research is a foundational part of studying how machine learning models perceive, understand, and represent geographic space.
• Second, the emergence of better VPR methods/models requires unifying the efforts of the machine learning, computer vision, and robotics communities. Especially when a novel machine learning model is proposed, it usually brings new solutions to the VPR task.
• Last, as a type of aggregation algorithm, VLAD-related methods can not only be used to aggregate image local features, but are also widely used for video sequences aggregation [3][4], speech clustering [5], and multi-modal information processing [6][7]. As a pure aggregation algorithm, our SuperVLAD does not add any prior information related to the VPR task. It also has the potential to be applied to other fields to aggregate other data. From this perspective, it is not just a vision algorithm but can be viewed as a machine learning algorithm. As for the broader implications of improving VPR through our SuperVLAD, we have discussed it in Appendix B.

Thanks again for your meaningful questions.

References

[1] Sarlin, Paul-Edouard, et al. "From coarse to fine: Robust hierarchical localization at large scale." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

[2] Lu, Feng, et al. "CricaVPR: Cross-image Correlation-aware Representation Learning for Visual Place Recognition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3] Xu, Youjiang, et al. "Sequential video VLAD: Training the aggregation locally and temporally." IEEE Transactions on Image Processing 27.10 (2018): 4933-4944.

[4] Naeem, Hajra Binte, et al. "T-VLAD: Temporal vector of locally aggregated descriptor for multiview human action recognition." Pattern Recognition Letters 148 (2021): 22-28.

[5] Hoover, Ken, et al. "Putting a face to the voice: Fusing audio and visual signals across a video to determine speakers." arXiv preprint arXiv:1706.00079 (2017).

[6] Wang, Xiaohan, Linchao Zhu, and Yi Yang. "T2vlad: global-local sequence alignment for text-video retrieval." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.

[7] Wang, Hui, et al. "Pairwise VLAD interaction network for video question answering." Proceedings of the 29th ACM international conference on multimedia. 2021.