Class-Relational Label Smoothing for Lifelong Visual Place Recognition

The related work, background and competitors in experiments are insufficient. The positioning of providing an improvement over a naive baseline and then competing methods that are not relevant as ideas is problematic. Details follow.

1. CRLS is presented as an improvement over label smoothing, applied to visual place recognition and compared with state of the art (SOTA) methods in the field. The main idea is that the appearance of the input examples is not consistent with the labels, hence the labels are modified. This happens in the particular problem because of evolution over time in the lifelong scenario and the competing SOTA methods are presumably not using any such label modification, although this is not clarified.

   This kind of positioning is problematic, because it treats visual place recognition as a benchmark, with the objective to beat the competition in the benchmark using ideas from anywhere. The proper positioning would be to beat relevant ideas. It would be to provide an improvement over other methods where examples are not consistent with labels, hence discuss such methods in related work and background, then compare with them as competitors in experiments, with visual place recognition being merely the application. Comparison is SOTA in the field of visual place recognition would be a secondary target.

   In this sense, there are several tasks where examples are not consistent with labels, for example learning with weak supervision where labels are noisy, semi-supervised learning where labels are missing, or active learning where examples are selected for labeling. Taking learning with noisy labels as an example, here are some prior works with relevant ideas:

   [*1] Reed et al. 2014, Training Deep Neural Networks on Noisy Labels with Bootstrapping

   [*2] Sukhbaatar et al. 2014, Training Convolutional Networks with Noisy Labels

   [*3] Lee et al. 2017, CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise

   [*4] Li et al. 2017, Learning From Noisy Labels with Distillation

   [*5] Sharma et al. 2020, NoiseRank: Unsupervised Label Noise Reduction with Dependence Models

   [*6] Han et al. 2019, Deep Self-Learning From Noisy Labels

   [*7] Iscen et al. 2022, Learning with Neighbor Consistency for Noisy Labels

   By studying such methods, one realizes that label smoothing is only a very naive baseline. There are many more advanced ideas and some are similar to the proposed method. A quite universal idea is distillation where soft target labels are obtained by a model, possibly the same while being learned. This is used in [*1], [*4] and [*6]. Of those, [*4] also uses a "knowledge graph". This graph plays the same role as the similarity-based affinity in this work, it is just constructed based on class semantics. Beyond pairwise class similarities, [*3] and [*5] use similarities between examples and class prototypes, hence can provide more fine-grained operations per example, while [*7] uses pairwise similarities between examples directly, hence provide even more localized operations per example. The idea of multiple class prototypes would also work well against evolution over time in the lifelong scenario, and there are several such methods in deep metric learning.

   The authors should present such methods as background, also methods from other relevant tasks, then compare with them as competitors. Here, they do not even compare with label smoothing (LS). This could have happened in table 6 for example, which is ablation, but LS is not presented in the absence of CRLS and CSW. There is comparison with other label smoothing strategies (LR and ACLS) in Table 7, but this is treated as ablation study. Instead, more advanced competitors should be chosen and this should be the main comparison. In this comparison, the authors should build on top of the best performing method on visual place recognition and show that adding CRLS works better than adding any other competitor. By doing so, the resulting solution would still beat the SOTA on visual place recognition.

2. For CSW, there are again many strategies by which to assign weights to different loss components based on some sort of confidence, and these weights can be per class like this work or per example, which is more fine-grained. Such ideas can be found in the tasks of semi-supervised learning or active learning. The authors should again study the literature and come up with strong competitors, present them mas background and compare with them.

Post-rebuttal

Let me first present a connection between CRLS and standard distillation that I sadly realized only after rebuttal and discussion.

In eq. (5), let us replace $W_y$ with $x'$, the feature vector of the input image obtained by a teacher model, and let $W_j$ represent the class weights of the same teacher model. In eq. (7), let us replace $(1-\alpha)$ of case $j=y$ by $(1-\alpha) + \hat{A}_{y,j}$. Then, the CRLS loss is the same as $(1-\alpha)$ times the standard classification loss plus $\alpha$ times the standard distillation loss. Thus, CRLS can be seen as a more efficient version of standard distillation in the sense that we represent an image with its ground truth class weight instead of forwarding it through the teacher to get its soft targets, and parameter $\alpha$ controls the trade-off between supervision and distillation (consistency, or self-supervision).

In light of this observation, I am not sure if the comparison to mean teacher (MT) (Tarvainen and Valpola, 2018) performed towards the end of rebuttal is careful enough to highlight the difference to distillation. This is because MT is using MSE loss and it is unclear how classification and consistency losses are weighted. A careful comparison according to the above would reveal if CRLS has similar effect with distillation, thus providing a cheap alternative, also targeting more clustered embedding spaces (because soft targets are all the same for images of the same class).

Concerning the introduction of the class affinity matrix $A$ in the distillation process, it is worth noting that Li et al. (2017), mentioned in my review, is doing the same, thus being equally a cheap alternative of distillation (applied to learning with noisy labels), although the matrix is based on semantic rather than visual similarity. This connection is important when claiming contribution.

Now, I believe a main concern of all reviewers is how to justify that the proposed approach is really connected with the problem of lifelong visual place recognition (VPR). Here, a difficulty is that the challenge of lifelong VPR is not precisely defined. For example, how is a construction in front of a building different from occlusion? Hence, it is equally difficult for authors to argue that the approach is meant for lifelong VPR as for reviewers/readers that it is not.

One aspect of this problem is that in my review I came up with several other tasks that have similar solutions. Another aspect is that other reviews appear to insist on a missing connection between lifelong VPR and the proposed approach, on which the authors do not appear to be able to provide a convincing response.

So maybe, instead of providing more baselines from other tasks or convincing arguments, the authors could just apply their approach to other tasks, as an efficient alternative to distillation. Even if CRLS is not significantly better, it is still a valid choice because it is efficient. Because distillation is ubiquitous, there is a abundance of such tasks. Two or three tasks of the authors' choice would do. If successful, the message would be that it is not very important if CRLS is really meant for lifelong VPR, because it benefits other tasks too.

I believe the paper would benefit a lot from the above suggestions if accepted. I am not sure if such updates are realistic for the camera-ready.

On other concerns:

• The authors did not address my original concern 2 (baselines for CSW).

• I am not sure what is the situation concerning other reviews about fair experimental settings - are all settings equal across all comparisons? It appears that the training set is still different.

1. Although the experimental results seem ok, the advantages compared to SOTA methods are not obvious. For example, on the commonly used Pitts30k/Pitts250k and MSLS val, this method doesn't outperform EigenPlaces with an obvious margin.
2. The work follows the training setting of the EigenPlaces framework. However, this work uses a larger batch size and learning rate than EigenPlaces. The authors claim those do not enhance the performance of the baseline model. So why not use the exact same settings as EigenPlaces? Whether the benefits of the proposed method will become marginal under such a setting.
3. Although the proposed method with the DINOv2 backbone seems to outperform other DINOv2-based methods (SelaVPR, SALAD, CricaVPR), their training datasets are different. The SF-XL dataset used in this work is the biggest one, so this comparison may be not fair. The results of this method seem to be worse than those of EffoVPR, but the latter uses a stronger backbone DINOv2-large. My suggestion is that the proposed method should be compared with EigenPlaces based on the DINOv2-base backbone.
4. Minor error: In some places (Line 939, 940, 1019), EigenPlaces was mistakenly written as EigenPlace.
5. All analysis in this paper is based on the CosFace loss function used in EigenPlaces/CosPlace. However, many recent VPR works are using multi-similarity loss in MixVPR. The contribution that this work can make to the VPR community seems limited.

I have some concerns about the comparisons made in the paper and the rational behind the method.

1. The abstract's claim of "outperforming SoTA across 17 benchmarks" seems to be an overstatement. Current state-of-the-art (SoTA) methods typically rely on pretrained foundation models (e.g., SELA, SALAD, Crica, BoQ [1]). However, the paper includes only a limited number of experiments that compare against these leading techniques. While the paper does demonstrate the method's effectiveness across various architectures and benchmarks, it achieves SoTA results on only 5 benchmarks. To the best of my knowledge, on all the benchmarks used in the paper, SoTA is reached by foundation based models. I encourage the authors to extend their tests with foundation models to all the benchmarks used in the paper to better assess the effectiveness of label smoothing when applied to these SoTA techniques.

2. Foundation models excel at capturing partial matches between images, potentially making label smoothing less impactful or redundant. When part of an image changes, current models, particularly those based on foundation models, are highly effective at identifying matches on the unchanged parts. This is why they also perform well in scenarios involving occlusion. It raises the question of what exactly CRLS captures in such cases—does it identify the changes and recognize what has been altered, or does it simply rely on the unchanged areas like other methods? It seems to me that the benefit of the suggested method is not really due to the moderate changes in the image, but rather due to smoothing of the classification space. I suggest the authors to provide attention maps that may shed light to this issue. I’d be glad to see examples of attention maps with foundation models against for instance SALAD, where the current model succeeds and SALAD fails.

3. Another mechanism used to handle partial changes is re-ranking. It is unclear whether the results presented include re-rankers in the various methods compared. There are multiple approaches that incorporate re-ranking, such as SELA and R2Former. It is necessary IMO, to see results with re-ranking of those methods (and others) and also on benchmarks with occluded benchmarks. Table 4, is particualry important as it includes datasets with more significant changes, such as day vs. nights, weather changes etc. I’d suggest to compare the suggested method against foundation models on these datasets. I believe that most of the results can be found in the literature.

4. Regarding the reported results, the model performs worse on MSLS - R@5 when using foundation models (as shown in Table 11 - Appdx). It shows better performance on SF, which it was trained on. In the single-view category, the results are significantly behind MixVPR, which is attributed to the use of longer features. However, since the authors add extra layers to foundation-based models, I’d suggest to train their model with additional layer to obtain the same feature size, for comparison. It is not clear if the observed differences are solely due to feature size.

[1] Ali-bey etal, BoQ: A Place is Worth a Bag of Learnable Queries, CVPR 2024

Comments:

$l$ is EQ. 1 is not defined. It will be fair to state in the comparison with AnyLoc that it is a zero-shot approach (without training on VPR data).

• 1. The clarify of the second contribution. (L268): The paper describes “min-max” normalization of the magnitudes over all classes”. However, it is not clear what is exactly meant here. Please describe exactly the step-by-step procedure that is used for this normalization. Please include a motivation for each step. Please link this procedure to the analysis shown in the equation 9 and in the supplementary material. This is important to justify/clarify the second contribution of the paper.
• 2. L250: “We assign labels in a sequence that mirrors the visual similarity to the target class, thereby enhancing the model’s ability to generalize across visually similar scenarios, e.g., lifelong variation.” The idea of using the intra-class visual similarity for label smoothing makes sense and is well explained and justified. But it is not clear how it relates to lifelong variation. Why should it improve generalization across lifelong variations? Please clarify this point.