LoCo: Learning 3D Location-Consistent Image Features with a Memory-Efficient Ranking Loss

Thank you for your detailed and constructive review. We appreciate your positive remarks regarding the presentation, efficiency, and potential reusability of our proposed method. We also acknowledge the valuable concerns you’ve raised and address them below.

---

Weakness (a): Limited experimental validation

We understand your concern regarding the focus of our experimental validation on tasks where our model excels. These tasks were chosen to highlight the core strengths of our approach in a controlled setting.

For tasks like surface normal prediction and monocular depth estimation (as reported in, e.g., El Banani et al. [12]), the correct prediction depends not just on the scene geometry, but also on the camera pose. Our work however explicitly aims to make the features location-consistent and thereby invariant to changes in camera pose. The LoCo-trained features therefore lack crucial information for such tasks. In their experiments on optical flow and stereo depth estimation, CroCov2 [44] fine-tune their entire (pre-trained) network rather than training a comparatively small probe. Experiments of that scale were beyond our computational resources.

In the appendix, we provide an evaluation on Visual Place Recognition, which further demonstrates the versatility and utility of the features learned by our method across different applications.

---

Weakness (b): Generalization capabilities

We appreciate the concern regarding generalization and agree that this is an important aspect of any method’s robustness. Due to computational limitations, we were constrained to training relatively small models on indoor scene datasets. In performing well on many unseen diverse indoor environments, the models do demonstrate the capability to generalise to unfamiliar environments. Due to the circumscribed training domain, we do not expect these models to perform well on environments with a larger domain shift (such as outdoor environments), where performance would mostly depend on the degree of their similarity with the training domain.

However, the training method we propose is designed to be fully general and adaptable to a variety of settings. While our current experiments focus on indoor scenes, the methodology itself should be applicable to other scenarios and domains without significant adjustments.

---

Weakness (c): Potential unfair comparison to competitors

We appreciate your concern about the comparison between our method and LoCUS, given the differences in model architecture and parameter count. To address this, we conducted additional experiments, as detailed in our global author response, where we fine-tuned the LoCUS architecture using our memory-efficient loss.

These experiments reveal that while our method does not outperform LoCUS in all tasks in terms of accuracy, it significantly enhances memory efficiency. This efficiency is what unlocks the training of larger models that produce higher-dimensional feature vectors within the same computational budget, leading to substantial overall performance improvements. The ability to scale up model capacity while maintaining computational feasibility is a key advantage of our approach, illustrating its potential beyond the specific settings and model scales we tested. We do however appreciate the importance of having this apples-to-apples comparison in the paper.

---

Typos

We will correct the typo on Line 166 (Eq. 12 → Eq. 3) and address the missing match for the dagger symbol in Table 1 before publication. We appreciate your attention to detail in pointing these out.

---

We hope these clarifications address your concerns and provide additional context for evaluating our work. We are confident that our paper offers valuable contributions, particularly in advancing methods for training location-consistent feature extractors in a memory-efficient manner.

We appreciate your careful consideration of our work and look forward to any further feedback.

Thank you for your thoughtful and encouraging review. We appreciate your recognition of the strengths of our work, particularly in terms of the memory efficiency gains and the analysis of the correction to the loss function. Below, we address your concerns and questions in more detail.

---

Weakness 1: Reliance on existing 3D mesh reconstruction for evaluation

We understand your concern regarding the reliance on 3D mesh reconstructions for the Scene-Stable Panoptic Segmentation task. However, we would like to clarify that this requirement is only necessary for evaluation purposes, not for training or general application of the LoCo features. The primary goal of this task is to highlight the improved location-consistency of our trained LoCo features compared to existing feature extractors.

While it is true that this evaluation necessitates datasets with 3D reconstructions, we believe that this is not a significant limitation. Several existing datasets provide this data, and our method's ability to perform well in this setting demonstrates its robustness and applicability in scenarios requiring high-precision spatial understanding. The improvements in location-consistency, as demonstrated in this task, are transferable to other tasks that do not necessarily require 3D meshes.

---

Weakness 2: Comparison with LoCo at a feature dimension of d=64

We appreciate your suggestion to include a comparison between LoCo and LoCUS at a feature dimension of d=64 for a more direct comparison. In the global author response, we provided additional results for fine-tuning the LoCUS architecture using our memory-efficient loss. While it is true that a direct comparison at a high feature dimension is challenging due to memory constraints, our additional experiments demonstrate that even at a low feature dimension our method offers significant improvements in memory efficiency while maintaining strong performance.

The improvements seen with larger models trained using our method further validate the value of our modifications to the loss function and training algorithm in overcoming memory limitations. This supports the scalability and robustness of our approach across different architectures and feature dimensions.

---

Question: Performance of LoCo with features from different models (e.g., DINO vs. DINOv2)

In response to your question about the performance of LoCo with features from different models, we have provided additional results using a DINOv2 ViT-Base backbone in the global author response. These results show that our method significantly outperforms the original DINOv2 feature extractor in tasks requiring accurate pixel correspondences.

While the performance with the DINOv2 backbone is slightly lower than with the DINOv1 backbone, this likely arises from the differences in the patch size of the backbone, which affects the granularity of the feature map. Nevertheless, the fact that training with our method improves multi-view consistency with different backbone architectures demonstrates its flexibility and effectiveness across various feature extraction models.

---

Limitations: Dataset requirements

We acknowledge the limitation regarding the availability of 3D reconstructions in certain datasets. As discussed, this requirement is specific to the evaluation of certain tasks and not an inherent limitation of our method. The technique we propose is broadly applicable and can be adapted to other datasets and tasks that do not require 3D reconstruction data.

---

We hope these clarifications address your concerns and provide a deeper understanding of the contributions and versatility of our work. We appreciate your constructive feedback and believe that our paper makes a valuable contribution to the field of vision foundation models, particularly in improving memory efficiency and multi-view consistency.

Thank you again for your careful review and for the opportunity to improve our work through your insights.

Thank you for your thorough review and for highlighting both the strengths and weaknesses of our work. We appreciate your feedback, and we would like to address your concerns in detail below.

---

Weakness 1: Fairness of comparisons and generalization of the proposed method

We understand your concern regarding the comparison with existing vision foundation models. However, we would like to clarify that our training and evaluation strategy is designed to test generalization across diverse scenes and datasets, not just on Matterport3D. Specifically, we evaluate on different environments than those used in training, including environments from a different dataset (ScanNet), ensuring that our network is not simply overfitting on the training set, but instead learning transferable and generalizable features.

Regarding the inclusion of a DINOv2 backbone, we have extended our experimental analysis to include results with a DINOv2 ViT-Base backbone (as described in the global author response). The results show clear performance improvements over the original DINOv2 features, which supports the general applicability of our method across different backbone architectures. This new evidence strengthens the claim that our method improves multi-view consistency in a broader context.

---

Weakness 2: Motivation for the saturation threshold $\Delta$

The threshold $\Delta$ is indeed a key hyper-parameter in our method, and its selection is crucial for balancing memory efficiency with the accuracy of the loss function approximation. The chosen value for $\Delta$ represents a trade-off: a smaller $\Delta$ leads to greater memory savings but can also introduce distortions to the loss function, affecting the training dynamics.

We selected $\Delta$ such that the gradient at $\sigma(\Delta)$ is 0.2% of the maximum gradient of the sigmoid function. This choice allows us to achieve significant memory savings (by a factor of approximately 5) while minimizing the distortion of the loss gradients. In this way, we ensure that the memory efficiency gains do not come at the expense of training stability or performance.

The ablations provided in Table 1 of the paper offer insight into these trade-offs, showing how different values of $\Delta$ impact both memory usage and model performance. We agree that the explanation for our specific choice could have been more explicit in the paper, and we appreciate your feedback on this. We believe that our current justification is grounded in practical considerations of training efficiency and accuracy, and we have provided empirical evidence to support it.

---

Weakness 3: Technical contributions and scalability concerns

We would like to emphasize that our approach is more than just an adaptation of the LoCUS architecture with threshold-based pruning. The pruning mechanism we propose is based on a detailed analysis of the gradient behavior, and it includes correction terms to ensure that the loss remains unbiased. We also address the bias in existing batch implementations of the general Smooth-AP loss, deriving the compensating correction terms necessary if using this loss for mini-batches, as is always the case in practice. This addresses a fundamental flaw in this loss function as used in the literature. Overall, our careful loss design addresses the challenges associated with unbiasing the loss signal, improving the memory efficiency and stabilising the gradients, making our method a significant contribution beyond a simple combination of existing techniques.

Regarding scalability, we agree that larger-scale experiments would be beneficial to fully explore the potential of our approach. However, even within the computational constraints we had, we observed consistent performance improvements with increased model size, as evidenced by our experiments with the LoCUS architecture. This suggests that our method scales well, at least within the range we were able to test. We hope that future work, possibly with greater computational resources, will further validate the scalability of our approach on larger foundation models.

---

Questions and Additional Clarifications

1. Computational Requirements: As noted in our paper, our experiments were conducted using a single RTX8000 GPU, with each training run taking approximately 1.5 days. This setup reflects the memory efficiency of our method, making it accessible even for researchers with limited computational resources.

2. Choice of Matterport3D Dataset: The method most similar to ours, LoCUS, trained its models on the Matterport3D dataset only, so we followed their choice to avoid additional confounding factors in comparing the two methods. The Matterport3D dataset is particularly suitable for our task of enforcing multi-view consistency due to its diversity and the way it captures varied viewpoints of the same scene through panorama cropping. We acknowledge that ScanNet is another valuable dataset; however, due to its trajectory-based data collection, there is less viewpoint variation per scene compared to Matterport3D, which influenced our dataset selection.

---

We hope these clarifications address your concerns and provide a better understanding of the contributions and significance of our work. We appreciate your thoughtful feedback and believe that our additional explanations and results strengthen the case for our contributions. We respectfully invite you to reconsider your assessment of our work in light of the additional evidence and explanations provided.

Thank you once again for your time and effort in reviewing our submission.