Towards Implicit Aggregation: Robust Image Representation for Place Recognition in the Transformer Era

Thanks for your positive recommendation and detailed suggestions. The following are Responses to the Questions (Q), Weaknesses (W), and Misc. Comments (M).

Q. about the term $Q_z K_a^T V_a$

We truly appreciate your attention to this detail. Indeed, this term also represents interactions between agg tokens and the image embeddings, but it is the influence of the former on the latter (unlike $Q_a K_z^T V_z$). Since we leverage the inherent self-attention mechanism within the backbone to jointly process all tokens, bidirectional interactions between agg tokens and image embeddings are unavoidable. When our agg tokens aggregate useful global information from the image embeddings, the image embeddings can also receive information from the agg tokens. This behavior is analogous to the interaction between the [CLS] token and patch tokens in the standard ViT architecture.

We did not explain this item in the method section to avoid confusing the readers. However, in our ablation studies, we explored the impact of setting different numbers of agg tokens. The results show that agg tokens have no negative impact on image embeddings as long as the number of tokens is not too large. When the number of agg tokens becomes excessively large (e.g., 64), a noticeable decline is observed. This indicates that an excessive number of agg tokens can interfere with the contextual information of patch tokens, which is not recommended. It should be noted that an excessively large number of agg tokens will cause serious overhead in model inference, and the high-dimensional descriptors generated by these tokens will also cause serious overhead in retrieval. So, this setting does not meet the actual needs. That is, it is not necessary to view it as a negative impact. Several agg tokens have already been able to effectively aggregate the useful information of patch tokens.

W1. The limitations of the previous method

Thank you for recognizing the novelty and contribution of our approach. We will follow your suggestion and directly state the novel perspective of our proposed method, removing some claims of the limitations of previous methods.

• L81: a) “toward global optimization”. --> We agree with you that this sentence is incorrect. We should claim “unifying feature extraction and aggregation into a more cohesive framework”, rather than “toward global optimization”.

• L81: b) “correction and refinement of image representations”. --> Sorry to confuse you. The image representations here refer to the global image feature (i.e., a representation vector). Although other methods improve the feature representations with each layer, what is improved is the patch descriptors (i.e., feature maps in CNN, patch tokens in ViT). The global image representation is formed after one-shot processing by the aggregation layer and is immediately output (without opportunity for correction and refinement). Our method, however, inserts agg tokens in the backbone's L-th encoder layer. These agg tokens serve as global image representations, and they are subsequently corrected and refined in subsequent encoder layers (synchronously with the refinement process of patch tokens), rather than being formed all at once. This is indeed different from the process of forming global representations in the backbone+aggregator approach. We will explain this in detail in the final paper to avoid confusion.

• L91: ”… overcomes the limitations of the previous works”. --> We agree with you. We will delete “overcomes the limitations” and will revise this sentence as “Our method provides a novel perspective different from the previous paradigm, effectively and efficiently achieving better performance in the transformer era.”

M1. Undefined terms

• "global optimality". --> Please see the response for W1 (L81 a)). We will delete this statement according to your suggestion.
• "correction or refinement". --> Please see the response for W1 (L81 b)). "fine-tuned" is also OK, but we are concerned that it might easily be confused with the training of models.

M2. Unjustified claims

• “The issue with NetVLAD” --> We acknowledge that some improvement efforts in the last two years have made the issue of the loss of position information less of a concern. However, the loss of position information objectively exists in NetVLAD/GeM, but our method does not have this issue. This is indeed a minor issue and is not sufficient to serve as the highlight/motivation of our approach. We will remove these statements from the main paper.
• “How is it proved that these descriptors capture human level semantics?” --> Sorry to confuse you. Although we use unsupervised clustering of patch descriptors during initialization, the transformer backbone used to extract these patch descriptors is pre-trained on large-scale datasets, which can indeed provide high-level semantics [56]. One way to demonstrate this is to visualize the attention weights of agg tokens to patch tokens, observing which patches are most correlated with the agg token. The supplementary materials we submitted (a PDF in the zip file) show that agg tokens from the trained model only focus on objects that are helpful for VPR, which provide vivid and interesting qualitative results. Before training (after initialization), the agg tokens focus on not only buildings and vegetation, but also vehicles and pedestrians, i.e., common visual patterns. However, it should be noted that it does not mean that an agg token exactly corresponds to a human semantic object (e.g., the building or vegetation). From attention visualization, we can see that some tokens contain both buildings and vegetation, and there are also multiple tokens that focus on buildings. We will emphasize this point in the final paper to avoid misleading.

M3. References

We sincerely apologize for citing secondary sources rather than the original work in the two instances you pointed out. For the first instance, we will cite the BoW work you mentioned. For the second instance, we will cite the work [47].

Thanks again for your valuable comments and suggestions. Please let us know if you have any further concerns. We'd be very glad to address them.

Reference

[47] Stephanie Lowry, et al. Visual Place Recognition: A Survey. IEEE Transactions on Robotics, 2015.

[56] Maxime Oquab, et al. DINOv2: Learning Robust Visual Features without Supervision. Transactions on Machine Learning Research Journal, 2024.

Thank you for the rebuttal. The rebuttal has addressed all the concerns, mainly the confusion in some of the writing and to remove some unjustified claims. The rating remains the same i.e. accept

Thanks for your comments to help us improve the paper. The following are Responses to the Questions/Weaknesses.

W1. About the novelty of ImAge

We sincerely appreciate your suggestion of conducting an experimental comparison between ImAge and prompt tuning (the results are in the table below). Regarding the novelty concerns, you might have some misunderstandings about our method. Before explaining the differences between our method and prompt tuning, we need to point out that VPR is an image retrieval task, not a classification task as you said. This might have led to some of your confusion about the sensitivity of our method (please see response to W2). To clarify the differences between our method and prompt tuning, we provide the following detailed explanations.

• We acknowledge VPT is an excellent parameter-efficient fine-tuning (PEFT) method applicable to many downstream tasks. However, it does not perform well in the VPR task as the prior work [1] demonstrated. The PEFT method, including prompt tuning, has been widely studied in the VPR task for as early as 1-2 years ago [2][1]. If our method is prompt tuning, similar performance would have already been achieved by prior work. However, the fact is that our method significantly outperforms previous methods and ranks first on the leaderboard of MSLS place recognition challenge.

• In terms of methodology, our approach differs from prompt tuning. Like several recent SOTA methods, our method adapts a pre-trained model to the VPR task by fine-tuning the last several blocks of the transformer backbone, rather than using prompts. It differs from prompt tuning where "only the task-specific prompts are being updated during fine-tuning, while the transformer backbone is kept frozen" [5]. The added agg token in our method is used to aggregate the key information in the patch tokens, rather than as the prompt.

• In fact, apart from prompt tuning, there are other works that add additional tokens to the pre-trained transformer for purposes other than fine-tuning the model. We have discussed this in detail in the second part of the Related Work. Our approach is more relevant to the other two types of methods than prompt tuning. We also discussed the differences between our approach and other methods in Related Work, Section 3.2 (Lines 187-190), and Appendix C (especially in this part). In fact, our agg tokens are “output-oriented tokens” rather than “prompt tokens”. Meanwhile, our work is inspired by the use of memory/register tokens in DINOv2-register [3] to capture global information.

• Besides, we also conduct fair comparative experiments to demonstrate that our method is different from prompt tuning (VPT). For applying VPT to the VPR task, an aggregation layer is necessary. We use the NetVLAD aggregation layer to obtain the final image descriptor. We trained two model versions of VPT, i.e., VPT (shallow) and VPT (deep). All models are based on the DINOv2-register backbone and trained on GSV-Cities, as in Table 3 of our paper. The results in the table below show that the performance of VPT is limited (basically consistent with [1]). The three methods in the last three rows (including our ImAge) fine-tune the DINOv2-register backbone by fine-tuning the last four encoder blocks (widely used by SOTA methods in VPR), achieving significantly better performance than prompt tuning (VPT). | Method | Pitts30k | MSLS-val | Tokyo24/7 | Nordland | |---|---|---|---|---| | VPT (shallow) + NetVLAD | 83.5/92.5/94.8 | 53.8/67.2/70.9 | 61.6/81.0/84.4 | 23.6/38.0/45.2 | | VPT (deep) + NetVLAD | 90.8/95.7/97.0 | 84.7/93.1/95.0 | 92.4/97.8/98.1 | 52.5/67.8/74.1 | | NetVLAD | 92.8/96.6/97.8 | 91.8/96.5/96.6 | 95.6/98.1/98.7 | 90.5/96.5/97.8 | | SALAD | 92.5/96.6/97.5 | 92.6/96.6/97.0 | 95.6/97.5/99.0 | 86.5/93.6/95.7| | ImAge | 94.0/97.2/98.0 |93.0/97.0/97.2 | 96.2/98.1/98.4 | 93.2/97.6/98.6 |

• In short, the prompt tuning methods use the prompts for fine-tuning the pre-trained large model, rather than as the final representation for downstream tasks. In contrast, our ImAge directly uses the agg tokens as the final global representation, which presents an opposite design perspective. Besides, our proposed insertion and initialization strategies are not the ready-made method used in the prompt tuning research or simple engineering tweaks, but thoughtful designs to enable the added agg tokens to aggregate useful information from the patch tokens. Here, we also use the output tokens corresponding to the prompt tokens in VPT as final image descriptors, and the results are shown in the following table. There is still a significant gap compared to our method. | Method | Pitts30k | MSLS-val | Tokyo24/7 | Nordland | |---|---|---|---|---| | prompt tokens in VPT | 80.5/92.2/94.8 | 62.4/74.2/79.7 | 55.9/73.0/81.3 | 13.0/21.6/26.6 | | ImAge | 94.0/97.2/98.0 |93.0/97.0/97.2 | 96.2/98.1/98.4 | 93.2/97.6/98.6 |

W2. About the ablation studies

We appreciate your concern about the robustness of our method. In standard ablation studies, the common practice is to fix all variables except the one being studied, in order to avoid the influence of other factors. In the prompt tuning research ([4], [5]) you mentioned, they were also fixing the prompt length when exploring the influence of other factors. As for your observation that "the model's performance is highly sensitive to the number of aggregation tokens" in Table 6, we would like to clarify that VPR is a retrieval task, and the performance of the method is directly influenced by the dimension of the image descriptors (the global feature vector). This holds true in all other VPR studies. The different numbers of tokens in our method directly form different dimensions of image descriptors, and this is reasonable in terms of its impact on performance. This is most pronounced on the Nordland dataset (very difficult), which is known to benefit from high-dimensional global descriptors (to provide detailed information). So, as you can see in Table 4, our method does not perform well using low-dimensional descriptors (i.e., only 1 agg token). In fact, this is also true for other methods. Compared with other low-dimensional descriptors, the results of our method on Nordland are actually better, e.g., the baseline (CLS token) in this table is 11.3% lower than our method on R@1. Moreover, on more commonly used benchmarks like Pitts30k and MSLS-val, our ImAge shows much lower sensitivity to the number of agg tokens. In addition, high descriptor dimensions will cause a significant storage burden and computational time consumption for retrieval tasks. Taking all aspects into consideration, we finally chose 8 agg tokens as the default setting. This results in a 6144-dimensional descriptor, which is superior to recent works (SALAD is 8448-dim and BoQ is 12288-dim).

W3. About the initialization strategy

Sorry to confuse you. We appreciate your concern for the rationale behind our initialization strategy. We also agree that NetVLAD constructs global descriptors using residual statistics between patch features and cluster centers, which is different from that our ImAge directly concatenates agg tokens as global descriptors. However, we merely drew inspiration from the initialization of NetVLAD, and this initialization is not unique to NetVLAD. It is actually the widely used k-means clustering algorithm. Since these cluster centers represent common visual categories and provide more visual prior, we use the cluster centers to initialize our learnable agg tokens. Such initialization can provide guidance for agg tokens to learn meaningful global information better. This is also partially discussed in our method section and ablation studies section.

W4. Sensibility to hyperparameter choices

As our response to W2, the performance of all methods in retrieval tasks varies with the descriptor dimensionality (corresponding to the number of agg tokens in our method). Regarding the insertion strategy and initialization method, it is precisely because other methods are not robust enough in the VPR task that we have proposed our insertion strategy and initialization method. It should be noted that in Tables 4 and 5, only the best one is our method. The other results correspond to other methods rather than different Settings of our method. This might be the reason for your misunderstanding. We have explained the reason for the specific design in the Method section. Ablation experiments demonstrate that our method is highly robust to various VPR scenarios, e.g., the Pitts30k (urban scene), MSLS (suburban and urban scene), and Nordland (natural scene) datasets. In addition, we are also happy to apply our method to more scenarios to verify its robustness.

Limitations

As our response to W1, our method is different from prompt tuning. If you agree that our method is a new implicit aggregation approach to obtain global features in retrieval tasks, we believe you will also recognize that it is likely to be equally effective in other retrieval tasks. As far as we know, this implicit aggregation method that does not require an additional aggregator has not yet been used for other retrieval tasks.

Thanks again for your valuable comments and suggestions. Please let us know if you have any further concerns. We'd be very glad to address them.

References

[1] Wu, Rouwan, et al. Investigating the Different Method of Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models on Visual Place Recognition. Available at SSRN 5233632.

[2] Lu, Feng, et al. Towards seamless adaptation of pre-trained models for visual place recognition. ICLR 2024.

[3] Darcet, Timothee, et al. Vision Transformers Need Registers. ICLR 2024.

[4] Lester, Brian, et al. The Power of Scale for Parameter-Efficient Prompt Tuning. EMNLP 2021.

[5] Jia, Menglin, et al. Visual prompt tuning. ECCV 2022.

Many thanks for your detailed feedback. We understand your concerns, and the following responses may help address them.

For R1:

We agree that it is necessary to provide analysis/evidence to explain the role and mechanism of the aggregation tokens. The supplementary material we submitted (a PDF in the zip file) has shown the attention visualization of agg tokens to image patches (patch tokens), which provides vivid and interesting qualitative results to support our claim. While different agg tokens attend to distinct regions (or objects) in the images, they consistently focus on the areas that are important for VPR (e.g., buildings and vegetation), while ignoring variable and useless elements (e.g., cars and sky).

In the main paper (Equation 3 and Lines 180-184), we provide a theoretical introduction about that agg tokens can learn and capture the global contextual information within the patch tokens by agg-patch attention (i.e., the term $Q_aK_z^{\top}V_z$ in Equation 3). The visualization in supplementary material vividly demonstrates that the patch tokens representing the objects highly related to VPR will be assigned the significant attention scores of agg tokens. That is to say, agg tokens aim to capture the key information from patch tokens related to VPR.

The visualization in supplementary material addressed the concerns of Reviewer tSze and Reviewer WgNP, and was considered by Reviewer VZG8 as convincing. We believe this is also useful evidence to address your concerns. We will place it in the paper (rather than as supplementary material) to better demonstrate our method.

For R2:

Thank you for your feedback. Since we leverage the inherent self-attention mechanism within the backbone to jointly process all tokens, bidirectional interactions between agg tokens and patch tokens are unavoidable. When our agg tokens learn and aggregate useful global information from the patch tokens by agg-patch attention (i.e., $Q_aK_z^TV_z$), similarly, the patch tokens can also receive information from the agg tokens by patch-agg attention (i.e., $Q_zK_a^TV_a$), as shown in Equation (3) in our paper. When the number of agg tokens becomes excessively large (e.g., 64), a performance decline can be observed. This is because an excessive number of agg tokens can interfere with the contextual information of patch tokens, which in turn affects the agg tokens in aggregating the information related to VPR.

However, it is not necessary to view it as a negative impact. It should be noted that an excessively large number of agg tokens does not meet the actual needs, as they will cause serious overhead in model inference, and the high-dimensional descriptors generated by these tokens will also cause serious overhead in retrieval. On the contrary, Several agg tokens have already been able to effectively aggregate the useful information of patch tokens. Meanwhile, it will produce descriptors of several hundred to several thousand dimensions, which is in line with practical needs and exactly at the same level as other SOTA works.

In short, a large number of agg tokens are unnecessary, and we don't need to use this setting at all. The experiment conducted in this setting is primarily to illustrate the issue of bidirectional interactions, as described in Lines 376-379. Of course, we can also eliminate this issue by adding an attention mask. We did not do so mainly to maintain the original structure of the backbone and facilitate scalability to other models.

For R3:

We think the key to this issue is that the initialization strategy is actually the widely-used k-means clustering algorithm, and is not unique to residual-based aggregation methods such as NetVLAD. For example, the prior work SuperVLAD [52] removes the cluster centers and directly aggregates local features (instead of residuals), which also employs such initialization in its code implementation to achieve better performance. The common point of these methods is that they all set multiple categories/clusters (note that different agg tokens in our method are responsible for aggregating different categories that are helpful for VPR). Therefore, they can all be based on unsupervised clustering to provide a common visual prior of categories, thereby better guiding aggregation in subsequent training.

Thanks again for your feedback. To ensure the Rebuttal fully addresses all your concerns, we would be grateful if you could share any additional concerns you might have.

Reference

[52] Feng Lu, et al. Supervlad: Compact and robust image descriptors for visual place recognition. NeurIPS 2024.

Thanks for your positive recommendation and valuable suggestions. The following are Responses to the Questions/Weaknesses.

Q1/W1. GPU memory and computational efficiency

When introducing our proposed insertion strategy of agg tokens, we mentioned that "if agg tokens are added at the beginning,…, leading to significant GPU memory and computational overhead." This primarily refers to the advantages of our method during training. We highly agree with your suggestion. Here, we not only compared this method with inserting tokens at the beginning, but also used NetVLAD and SALAD as baselines. The results are shown in the table below. Our method has significant advantages over inserting tokens at the beginning in terms of GPU memory usage and training time, and also outperforms NetVLAD and SALAD.

It is also worth mentioning that we have demonstrated the advantages of our method in inference (inference time and descriptor dimensionality) in the paper (Figure 1 and Table 3). All these results show that our method outperforms previous methods in many aspects.

Q2/W2. transformer-based VPR methods with agg tokens

Thank you for giving this suggestion to improve our paper. In fact, our method can be regarded as a general aggregation method for the transformer backbone. In addition to the experiments in the main paper, Appendix D (Table 7) also shows the results of combining the proposed ImAge method with ViT and CLIP. The R2Former you mentioned uses ViT as the backbone, so it is feasible for our method to improve the performance of its first stage (global retrieval). However, R2Former is a two-stage method with re-ranking using local features. Its training process is elaborate, and the model is trained on 8 Tesla-V100 GPUs, which makes it difficult to combine R2Former with our method during the limited Rebuttal period. Considering that R2Former is an excellent VPR method and ranked first in the leaderboard of MSLS Place Recognition Challenge when proposed (same as our method), and the re-ranking in it can significantly improve the VPR performance, we will discuss the potential of combining it with ImAge in our final paper and try to combine it in future experiments.

To show that our method can replace the aggregation method in other transformer-based VPR methods to achieve better performance, we replace the spatial pyramid GeM pooling feature in the CricaVPR work with our ImAge feature (denoted as CricaVPR+ImAge). The results are shown in the table below. It can be seen that our method significantly improves the performance.

In short, our ImAge is a general image representation method for the transformer-based VPR. In addition, when using the foundation model DINOv2 trained on a large-scale curated dataset, the performance improvement of our ImAge over other methods (such as NetVLAD) is more pronounced than when using ViT pre-trained on ImageNet. This suggests that in the future, as the performance of the foundation model improves (with more training data), the advantages of our ImAge may become more apparent.

W3. Novel for VPR

Thanks for your recognition that our method is novel in VPR. We acknowledge that there have been some works in other fields previously on adding tokens to the transformer, as we discussed in Related Work. Our work provides a new perspective on the method of adding tokens: achieving implicit aggregation of patch tokens. As far as we know, this work is the first to do so not only in VPR but also in visual retrieval. Furthermore, our method achieves the SOTA performance in a concise way.

By the way, we are so happy to see that you appreciate our SOTA performance, real-world applicability, as well as the interesting attention visualizations of agg token (in Supplementary Material).

Thanks again for your valuable comments and suggestions. Please let us know if you have any further concerns. We'd be very glad to address them.

Thanks for your positive recommendation and insightful comments. The following are Responses to the Questions.

Q1 The comment on quality #4

• For quality 4.1: Sorry to confuse you. a) The claim of “global optimality” is indeed inaccurate because other methods like NetVLAD can also be fully differentiable. We will delete this claim and directly state that “our method provides a novel perspective different from the previous paradigm, unifying feature extraction and aggregation into a more cohesive framework”. Unlike previous methods where the backbone is only used for patch feature extraction, our global features based on implicit aggregation can be further corrected and refined synchronously with the patch token extraction process in the backbone, achieving better performance effectively and efficiently. b) Regarding “NetVLAD might suffer from inappropriate cluster centres”, we mainly want to illustrate that it is difficult to artificially design a perfect aggregator, which motivates us to propose a solution that does not require the explicit design of additional aggregators. Since domain generalization is not the main focus of our work, we will remove this statement to avoid causing confusion.

• For quality 4.2: We agree with you. "aggregator-free" and “0 aggregator parameters” are a bit confusing. We will provide further explanations in these places -- there is no extra explicit aggregator, and our aggregation process is completed in the original backbone.

• For quality 4.3: We will add clearer explanations in the table caption according to your suggestion.

• For quality 4.4: You are right. These examples are cherry-picked. In fact, this is a common practice in many great VPR works, e.g., in the main paper or supplementary material of the CosPlace, EigenPlaces, StructVPR, and CricaVPR works. To demonstrate the performance of methods as reasonably as possible, we have included a wide range of scenarios (viewpoint and/or condition changes, dynamic objects, etc.) in Figures 4 and 5. Additionally, Figure 6 presents the failure cases of our approach. Following your suggestion, we will add some cases where ImAge fails while some baselines succeed and provide further discussion.

• For quality 4.5: We will revise it according to your suggestion.

Q2. How clustering works

The clustering process is basically the same as the k-means clustering in the NetVLAD implementation of the VG Benchmark work [1]. We randomly sample some training images (e.g., 500 images), and use the pre-trained backbone to extract patch tokens (features) from these images. Then we randomly sample some patch tokens of each image (e.g., 100 tokens per image). Finally, we perform k-means clustering for the obtained (500*100) patch tokens. The transferring to the new settings is very easy. This is demonstrated by the widely used VG benchmark work (ours is the same).

Q3. Potential limitations

Thanks for your insightful comments. The proposed method is indeed not a good choice when we want to keep the backbone frozen or when the backbone is extremely expensive to fine-tune. We will discuss these two points in the final paper. However, it is worth noting that we only fine-tune the last few blocks of the transformer backbone, which is relatively cheap. This can be seen from the response to the Q1 of Reviewer VZG8. We can also further reduce the fine-tuning cost by reducing the number of blocks fine-tuned, but this may cause performance degradation.

Q4. Why do tokens represent separate categories?

Sorry to confuse you. We guess the word "object category" misleads you. We originally intended to use the analogy of human-level semantic classification to visually illustrate the division of multiple agg tokens, which is indeed not rigorous enough. The supplementary materials we submitted (a PDF in the zip file) have shown the attention visualization of agg tokens to image patches, which provides vivid and interesting qualitative results. The highlight areas of attention for different agg tokens are indeed different, that is, each token is responsible for representing different categories. In fact, both the initial k-means-based initialization and subsequent training will make different tokens represent different categories. If they represent the same category, multiple tokens will be redundant. Only when different tokens represent different categories can the representation ability be stronger. However, it does not mean that a token exactly corresponds to a human semantic object (e.g., the building or vegetation). We can see that some tokens contain both buildings and vegetation, and there are also multiple tokens that focus on buildings. To avoid misleading, we think a reasonable expression could be "each added agg token representing a unique category (but not exactly an object category in human semantics, such as building or vegetation)".

Q5. Are aggregation tokens "output-oriented tokens"

Your judgment is correct—the proposed aggregation tokens are indeed "output-oriented tokens." In fact, we explicitly mentioned this point in the initial manuscript, but it was deleted during the paper’s streamlining process. We only stated that our work was inspired by the DINOv2-register work, not that our method belongs to that category. Since this has caused ambiguity, we will clearly state in the final paper that our method falls into the "output-oriented tokens" category.

Clarity. The paper is very dense

We will make revisions according to your suggestions.

Thanks again for your valuable comments and suggestions. Please let us know if you have any further concerns. We'd be very glad to address them.

Reference

[1] Gabriele Berton, et al. Deep visual geo-localization benchmark. CVPR 2022.