EMVP: Embracing Visual Foundation Model for Visual Place Recognition with Centroid-Free Probing

Thank you for acknowledging our motivations and experimental results.

1. The VPR task involves a more fine-grained classification, as two images of the same location may have minimal overlap and require identification based on features from a small region. As discussed in the introduction of this paper, NetVLAD can be seen as a second-order feature, which provides an accuracy advantage over first-order features used by alternative techniques. This perspective is also supported by many experimental results in the field [12; 15; 17; 18; 9].

2. Mainstream VPR datasets (e.g., Pittsburgh250k) use GPS coordinates to determine positive and negative examples. In fact, images with distant GPS coordinates may contain common buildings, while images with nearby coordinates may have no common objects due to different orientations. In other words, the labels in current mainstream VPR datasets are noisy. Consequently, some cutting-edge works have improved the data efficiency by refining the dataset construction methods [12; 19; 20].

3. On one hand, with the advancement of computing power, ViT-based models are increasingly employed in various cutting-edge research areas of mobile robotics, such as localization [9; 21], perception [22], and manipulation [23]. On the other hand, the size of foundation models is continuously decreasing. For example, the 0.23B Florence-2 [11] can rival the performance of the 80B Flamingo. Therefore, it is feasible for us to study VPR tasks based on foundation models.

Thanks for acknowledging our paper, and we appreciate your valuable suggestions.

1. As discussed in our response to reviewer psQJ, LP is a kind of first-order feature and less accurate compared to second-order features in the fine-grained VPR task. MP is proposed solely for coarse-grained classification tasks, with a focus on accelerating the computation of second-order features, which limits the accuracy.

2. This phenomenon was also observed by the authors of SALAD. More thorough fine-tuning can conceptually yield better domain performance, but it needs more data. Therefore, there is a correlation between the amount of data and the optimal number of recalibrated blocks.

3. The baseline in Table 2 is the bilinear form of NetVLAD (with the centroids removed). We implemented it by removing the optimal transport operation in the SALAD code base. After introducing the constant normalization and $\mathrm{DPN_C}$ layer, our CFP achieves a significant improvement over the baseline. For example, it improves Recall@1 on MSLS Val by 2.3%.

Thanks for your insightful comments, we provide the feedback as follows:

Q1&W2. Detailed information on the training procedure. To ensure a fair comparison, we have kept the details unrelated to the innovations of this paper consistent with previous SOTA methods (i.e., Conv-AP [12], MixVPR [15], and SALAD [9]) during the training process. Therefore, the loss function, sampling strategy for batches, and data augmentation techniques remain the same with the above SOTA methods. Specifically, the model is trained by Multi-Similarity loss [10]. We use batches containing 120 places, each represented by 4 randomly sampled images, resulting in batches of 480 images, as shown by Table 4 in Appendix A.1. Training images are resized to $224 \times 224$, and no data augmentation technique is applied.

Q2. Limitation and future work. Current VPR works (e.g., EMVP, SALAD, and SelaVPR) rely solely on extracting more generalized visual features based on VFMs. We have observed a recent trend in large multi-modal models towards lightweight and unified designs, represented by Florence-2 [11], where the size of backbone is only 0.23B, and different image-text understanding tasks are being unified into the Visual-Question-Answering task. We are currently investigating how to guide the model's reasoning ability in adverse weather and ambiguous scenes by constructing multi-modal thought-of-chains.

W1. Thanks for patiently reading our paper. We will reduce the number of abbreviations in the revised version.

W3. Evaluation metrics. We use the standard Recall@k evaluation metric, following [4; 9; 12; 13; 14; 15]. A query image is considered successfully retrieved if at least one of the top-k retrieved reference images is within 25 meters of the query.

W4. Novelty of the paper. We are among the first to explore probing techniques for the VPR task. Specifically, our CFP admits a theoretical and empirical justification for the simplification of NetVLAD, fixing interpretability and performance issues that were present otherwise. Note that, our CFP improved Recall@1 on NordLand by 2.8% compared to NetVLAD, while also reducing the memory for descriptors by 66%. Reviewer M9Bz and psQJ have also recognized our work's novelty, highlighting the value of our contributions.

W5. Time and memory consumption. We compare both single-stage and two-stage methods in the table below. Without using re-ranking, SALAD and our EMVP outperform all other methods while being orders of magnitude faster. Results marked with $^\diamondsuit$ are computed using a RTX 4090 GPU. Memory footprint is calculated on the MSLS Val dataset, which includes around 18, 000 images. The evaluation protocol and code are provided by R2Former [16]. Note that, the table below reports test results of Pytorch models. If TensorRT is used for acceleration in the deployment, the applicability of the proposed method on mobile devices (such as unmanned vehicles) will be greatly improved.. The introduction of VFMs increases the backbone size, but it simultaneously reduces the dependence on re-ranking, saving more latency time and memory.

Thanks for carefully reading our paper and recognizing its writing and novelty. We appreciate the opportunity to address your questions:

1.a. We posit that the second-order statistics (bilinear features) employed in NetVLAD are implemented through the outer product of two vectors corresponding to each location of an image. This interpretation is rooted in the seminal work [1], which pointed out that "VLAD can be written as a bilinear model". Consequently, our paper adheres to the definition of second-order statistics as established in the research line of [1], [2], and [8]. Otherwise, the mentioned LSO-VLADNet [3] applied the "element-wise square operation" to first-order features (residuals), and considered it as a second-order statistic operation.

We also agree that both [3] and [4] consider residuals as first-order features, a definition that does not take into account the soft-assignment. However, if the soft-assignment is seen as another type of first-order feature extracted by MLP Layers, then the features ultimately output by NetVLAD are second-order. Specifically, Eq. (4) in [4] can be understood as computing the outer product of each local residual $(x_i - c_k)$ with the soft-assignment vector $\bar{a}(x_i) = [\bar{a}_1(x_i), \bar{a}_2(x_i),\ldots, \bar{a}_k(x_i)]$, and then using sum pooling to obtain global second-order features. Note that the process is also formulated by Eq. (5) in our paper.

1.b. Softmax in NetVLAD is introduced to ensure that the sum of the probabilities of assigning a local feature to all clusters is a constant, which can be formulated as $soft\\_assign = F.softmax(soft\\_assign, \mathbf{dim=1}), soft\\_assign \in \mathbb{R}^{B \times C \times L}$. $B$, $C$, and $L$ denote batch size, the number of clusters, and the number of local features, respectively. Moreover, NetVLAD adapts intra- and L2-normalization to reduce the effect of bursty image features, as described in paper [5] "intra-normalization fully suppresses burst". The intra-normalization conducted on the features weighted by soft_assign can be formulated as $vlad = F.normalize(vlad, p=2, dim=2), vlad \in \mathbb{R}^{B \times C \times D}$ where $D$ denotes the dimension of local feature.

The motivation of our post normalization method is to remove the cluster centers, as described in [5] ``VLAD similarity measure is strongly affected by the cluster center''. Accordingly, the post constant normalization is introduced to ensure the sum of the probabilities of all local features assigned to a cluster is constant, which can be formulated as $soft\\_assign = F.softmax(soft\\_assign, \mathbf{dim=2}), soft\\_assign \in \mathbb{R}^{B \times C \times L}$. Note that our EMVP retains both intra- and L2-normalization in NetVLAD.

1.c. Thank you very much for pointing out this error. The Nordland dataset exists in two versions, and the version used in our paper is provided by [6]. As shown in the table below, we display the results for the other version provided by [7]. EMVP even surpasses the previous best method (SelaVPR) with a re-ranking stage. In the revised version, we will separately and clearly present the test results for different versions of Nordland dataset.

2. Our DPN module leverages power-law operations to effectively alter the magnitude relationships between local features, surpassing the capabilities of multiplication operations employed in classic adapters. This empowers the DPN module to accentuate discriminative features, enhancing its ability to capture task-specific information as discussed in lines 181 to 187. Note that controlling the preservation of task-specific information through changing the power value is also supported by previous theoretical research [8]. The superiority of the proposed DPN is evidenced by the empirical results in Table 3. Compared to the advanced adapter method PRSP, EMVP achieves notable improvements in the Recall@1 by 0.5%, 3.4%, 0.4%, and 0.5% on MSLS Val, NordLand, Pitts250k-test, and SPED, respectively. Remarkably, these enhancements are accompanied by a substantial 64.3% reduction in parameters, highlighting the efficiency and scalability of the DPN module.

In addition, the implementation of DPN is shown as Algorithm 1 in Appendix A.1. Specifically, the input and output dimensions of the linear layer in $\mathrm{DPN_C}$ are 128 and 64, respectively, and the former corresponds to the output size of $\mathcal{F}_C$. While, the input and output dimensions of the linear layer in $\mathrm{DPN_R}$ are $d$ and $d/2$, respectively, where $d$ is the dimension of the ViT model. In the case of the ViT-B model, $d=768$.

3. SALAD also uses L2-normalization, as described in paper [9] "Following NetVLAD, we do an L2 intra-normalization and an entire L2 normalization of this vector.". SALAD and the baseline do not employ softmax, and they differ from NetVLAD to some extent. Therefore, we will revise the description of the baseline in the updated version to: "the baseline can be seen as a form of bilinear pooling with the centroids removed, and it is implemented by removing the optimal transport operation in the SALAD code base".

4. As shown in the table below, our EMVP achieves the best results under different backbone configurations. For instance, compared to the existing single-stage method based on DINOv2-L, EMVP-L improves Recall@1 on MSLS Val by 6.2%.

Dear Reviewer,

We sincerely appreciate your reevaluation and are especially grateful for the valuable time you have spent to help improve our paper. Please allow us to address your concerns once again.

1.a We agree that the aggregated features in VLAD are first-order. We consider soft assignment to be an extracted feature primarily because NetVLAD uses a separate ("decoupled") layer to predict soft assignment. The literature you provided introduces a new second-order feature production method, which is very enlightening for us. In our future research, we will conduct a more comprehensive discussion on the definition of higher-order features.

1.b Regardless of whether softmax is applied, a cluster will be assigned to all local features simultaneously. This is because NetVLAD uses soft assignment instead of the hard assignment in the original VLAD. Given a cluster, we consider its physical meaning to be implicit, and each local feature containing some degree of this physical meaning, while this interpretation is not immediately intuitive (cons).

As SALAD points out, "some features, such as those representing the sky, might contain negligible information for VPR." Therefore, SALAD introduces a dustbin in the optimal transport operation, allowing the sum of the probabilities of assigning a local feature to all clusters not to be a constant. Instead, the optimal transport operation ensures the sum of the probabilities of all local features assigned to a cluster remains constant. This approach keeps the sum of assignment weights corresponding to each cluster constant, preventing the dominance of any single cluster and making the assignment more "effectively distributed".

Overall, our use of softmax serves a similar function to the optimal transport operation in SALAD, but we provide a theoretical basis for this operation (motivation). Furthermore, we have explained why the cluster centers no longer need to be included in the computation when using our softmax (motivation & pros). Additionally, unlike optimal transport, which requires complex iterative algorithms, our implementation of softmax is much simpler and faster (pros).

1.c Indeed, we should complete the missing results in Table 1. As the author-reviewer discussion period is nearing its end, we will test all the compared methods on these datasets and include the results in the revised version.