Towards flexible perception with visual memory

Dear Reviewer xHyy,

Thanks for your review, we’re humbled to hear you appreciated the “extremely well done experiments” and found the work “highly relevant with enormous potential for future extensions”, & “one of the most exciting papers I’ve seen this year in this space” that “could become a foundational paper”.

Falsifiable experiment on interpretability:
We fully agree that interpretability should go beyond visualization. First of all, we don’t mean to imply our system is “fully interpretable”, rather that it is “more interpretable” compared to a standard black-box model (and we will revise the writing to make this distinction clear). Secondly, we appreciate your point that interpretability-related claims and statements are best made through a falsifiable human experiment. We outlined a possible experiment at https://ibb.co/k2jnGH85; it’s aimed at quantifying how much, if at all, a memory-based system improves interpretability as operationalized by helping humans predict model behavior. Before running the experiment, we’re keen to hear your thoughts!

Shortcut learning / bias removal:
We’re happy to add a discussion on the connection to shortcut learning and bias removal. If the image encoder exploited a shortcut during training, this will influence image similarity and thus nearest neighbor selection. There are cases where bias removal is possible, and cases where it is impossible:

• [removal impossible] If the encoder is biased towards textures, a test image of “cat shape + elephant texture” [cf. Geirhos et al. 2019, Figure 1] would pull up elephant nearest neighbors, and removing all elephants from memory would come at an unreasonably high cost (not being able to identify elephants anymore). Here, encoder-level debiasing is necessary.
• [removal possible] If only a part of the memory is biased, memory-level debiasing is feasible. If “fingers” are shortcut predictors for “fish” (due to a dataset bias from proud fishermen holding their catch into the camera, cf. Brendel et al. 2019, Figure 3), then this bias could indeed be rectified by removing the biased “fish+finger” subset from memory. Afterwards, images with “fingers” would no longer lead to “fish” nearest neighbors, demonstrating successful bias removal.

DinoV2 / Could alternative losses, or even different model architectures, be designed to perform even better?*
We agree that DiNO-V2 is a strong vision model, facilitating retrieval. Since our approach is modular, any encoder can be used in a plug-and-play fashion within our framework. Progress on visual representation learning (architectures, losses etc.) can generally be expected to improve retrieval. For example, text-to-image generative models have proven to learn useful representations for open-world, open-vocab tasks and they could become possible image encoders for retrieval-based tasks as well and better losses & architectures developed by the field will directly translate into better visual memory systems.

Adversarial perturbations:
We share your intuition. As long as adversarial attacks can fool featurizers, neighbor selection can be attacked through perturbations as well (e.g., making model think a dog image is instead an airplane, and thus retrieve airplane neighbors leading to misclassification). Neighbor selection is nondifferentiable, but black-box or surrogate-model based attacks (e.g. replacing neighbor selection with differentiable Gumbel-Softmax) will likely work.

Suggestion for discussion: should removal or unlearning occur at the individual data point level or at the concept level?
Concept-level manipulation (adding/unlearning entire concepts) is a really exciting idea that we hadn’t considered so far! Since any similarity space can be used for neighbor selection, including ACE or SAE-based spaces, this is indeed possible. Depending on the use case, both data-based and concept-based manipulation can be desirable: if a single image is corrupted / has licensing issues then unlearning the individual image is the right choice; if a concept is biased then concept-level changes seem preferable. We’ll be happy to add a brief discussion on this exciting possibility.

Data point influence = number of times it's used in the decision?
Yes. In the extreme case, if a sample never shows up as a neighbor, its influence would be zero. Of course, influence isn’t always a good thing - it really depends on whether the sample contributes to (= influences) a correct or a wrong decision. Fortunately, in contrast to a traditional model, that’s very easy to keep track of in a memory model, and can thus be exploited for reliability-based weighting as in Table 3.

Could this score be used to select better quality/diversity on the dataset?
Likely. For ImageNet-A, in ~40% of cases the DinoV2 label was assessed as being better/more suitable than the original dataset label, thus this could be used to identify label issues and thereby improve dataset quality.

Dear Reviewer 4C32,

Thank you for your helpful comments. We’re happy to hear you found our work “well justified”, appreciated the “insightful experiments & valuable insights” (even though some of them were admittedly a bit buried in the appendix), and described it as “a timely analysis in the vision space”.

Beyond classification, which other visual tasks might benefit from a datastore? Ways to leverage for segmentation, detection?
It’s possible to extend the approach to other tasks. One can pool features into multiple embedding clusters instead of a single cluster using classical or learned clustering methods. As a proof of concept, we tested object segmentation based on a visual memory of DinoV2 features; visualized here: https://ibb.co/92dM0B2. This puts features from a single image into memory (a car in the example) based on 8 feature clusters and uses these to identify similar features in the second (test) image, thereby creating a segmentation mask. Such multi-vector representations of images (which can be expanded to image + text) are a natural extension to our work, enabling tasks such as object retrieval or detection from images containing multiple objects, or provide coarse patch-level semantic segmentation. Finer-grained segmentation masks can be obtained with further training of the pooling methods. We hope this provides an intuition for how other tasks can be approached, and we will add the proof of concept example to the appendix. If other tasks are tackled, our main argument about the increased flexibility of a memory-based approach (and the benefits / capabilities enabled by this flexibility) directly translate to those tasks as well (& could be plugged into canonical detection architectures like Fast-RCNN).

Appendix nicely complements the manuscript but provides new insights that would deserve more visibility – esp. Appendix G (& to some extent O & P):
Thanks for the comment, that’s great to know. Since ICML's camera ready version allows for an additional page, we’re able to move the iNaturalist experiment (app. G) to the main paper as well as reference & hightlight the other two sections more than we currently do.

Changing samples might require a rebuild of the index:
Agreed, we will add this to the limitations section (page 8). Regarding the two search approaches described on page 3, adding/removing samples is trivial for approach #1 (GPU/TPU matmul e.g. on ImageNet-scale data; here one can add or drop a row from the num_images x num_features matrix) but for approach #2 (scalable nearest neighbor search index for JFT) this would indeed require adapting the index, though the amortized cost is low.

Related work: suggesting to also express what’s different.
Valid point, we’re happy to incorporate this.

Reliance on a pre-trained encoder:
Indeed, current encoders are limited. Since our approach is modular, any encoder can be used: as soon as more general, open-world encoders are developed they can simply be plugged in; even encoders based on generative models could become a possibility (given that generative models are a lot more open-world, open-vocab). Thus our approach is orthogonal to encoder choice. That said, even with current encoders we see promising success based on adding novel classes (NINCO experiment in Section 3.1).

Innovation:
While we provide technical improvements like RankVoting, we fully agree that we build on well-established methods with a long history in ML and seek to be very transparent about this throughout the paper. Instead, our focus is a broad evaluation of flexible capabilities (including attribution, flexible adding/removal, flexibly increasing granularity like on iNaturalist, …). We believe there is community interest in seeing solid evaluations.

Approximate retrieval:
Great question! Approximate retrieval increases retrieval speed at the cost of recall errors. Based on data from Appendix D, approximate retrieval would not significantly degrade results. Let’s assume approximate retrieval leads to different neighbors (compared to NNs from exact retrieval). In the best case, those neighbors are still from the same class; thus nothing changes. Worst case, they’re from a different class i.e. their label is now misleading. Based on Figure 9, we know that our approach can handle up to 60% label corruption (!) without degradation of performance; thus as long as k>>1 our approach is very robust to approximate retrieval. We’ll add this discussion.

Would an automatic ‘uncertainty’ threshold (e.g. based on distance) be useful to detect generalisation issues?
Based on the analysis from Appendix H, OOD data does indeed lead to higher mean+median nearest neighbor distances; thus a distance threshold would work well. There are a few works on kNN-based outlier detection (e.g. https://proceedings.mlr.press/v162/sun22d/sun22d.pdf).

Kindly let us know if you have any further questions, and thanks again for the great suggestions.

Dear Reviewer Em8N,

Thank you very much for your detailed review. We’re glad to hear you appreciated the “well-designed experiments”, “very well organized presentation / clarity” and “very well written manuscript”.

Abstract: “nearly impossible” is pretty strong
Noted - we’ll change this to “editing the knowledge in a network is hard”.

Generalization to other tasks like segmentation / detection:
It’s possible to extend the approach to other tasks. One can pool features into multiple embedding clusters instead of a single cluster using classical or learned clustering methods. As a proof of concept, we tested object segmentation based on a visual memory of DinoV2 features; visualized here: https://ibb.co/92dM0B2. This puts features from a single image into memory (a car in the example) based on 8 feature clusters and uses these to identify similar features in the second (test) image, thereby creating a segmentation mask. Such multi-vector representations of images (which can be expanded to image + text) are a natural extension to our work, enabling tasks such as object retrieval or detection from images containing multiple objects, or provide coarse patch-level semantic segmentation. Finer-grained segmentation masks can be obtained with further training of the pooling methods. We hope this provides an intuition for how other tasks can be approached, and we will add the proof of concept example to the appendix. If other tasks are tackled, our main argument about the increased flexibility of a memory-based approach (and the benefits / capabilities enabled by this flexibility) directly translate to those tasks as well (& could be plugged into canonical architectures like Fast-RCNN).

Baselines, e.g. Table 1:
We fully agree on the importance of baselines. Due to space limitations, some comparisons were moved to the appendix, like Table 9 (RankVoting 79.9%, CLIP zero-shot 75.3%). We’d be happy to mention and display baselines more prominently in the main paper. For Table 1, we did not include a comparison to linear probes since the goal of the table is to show a lifelong learning evaluation, i.e. understand which performance can be reached on OOD data without re-training anything. If NINCO classes are evaluated with a DinoV2 model and its default linear classifier, the performance would be 0.00% because the classifier is static and cannot transfer what it has learned without further fine-tuning and change of architecture/layer.

Novelty:
Agreed, while we provide technical improvements like RankVoting, we build on well-established methods with a long history in ML and seek to be very transparent about this throughout. Instead, our main focus is a broad evaluation of flexible capabilities as mentioned in your review. We believe there is community interest in seeing solid capability evaluations (as e.g. evidenced by xHyy describing it as “one of the most exciting papers I’ve seen this year in this space”).

How is the reliability factor selected?
It’s directly related to the number of times the training image contributed to a wrong decision on ImageNet-train; see https://ibb.co/2320r1Ld.

What’s the dotted line in Fig 11?
It highlights a chance accuracy baseline (no aggregation, just constantly predicting a single class). For balanced datasets, baseline guessing accuracy can be calculated as 1 / num_classes. Since iNaturalist is unbalanced, this could be misleading. E.g. if a dataset has just two classes, but one of them accounts for 70% of samples, then constantly predicting this class would lead to 70% accuracy. Thus a commonly used “strong guessing” baseline for unbalanced datasets is to always predict the largest class (with the most samples over the entire dataset), without encoder/aggregation. We’ll update the description to make this clear. We could add DinoV2 linear probing as well, though it might struggle to train properly given just a handful of training samples.

Section G aggregation method?
Since we’re adding neighbors to memory starting from 0 exemplars, no aggregation is used here (k=1); we’ll make sure to mention this.

Algorithm 1:
We made a mistake here - Alg. 1 corresponds to an algorithm that we tried, but it didn’t work better than the much simpler kNN-based classifier with k=1. We should have removed the algorithm, apologies for the oversight. Fig. 11 indeed already corresponds to the simple kNN classification approach that you suggested we try instead.

If new samples are added, should soft pruning weights be calculated again?
Since those new samples don’t have reliability weights, their reliability would indeed need to be estimated, unless a “default reliability weight” (e.g. mean reliability of existing samples) is used as a proxy. If the new samples are IID, existing weights for existing samples wouldn’t change systematically, thus those can be kept without recalculating.

Other comments / suggestions:
Excellent points, thank you!