Exploiting Distribution Constraints for Scalable and Efficient Image Retrieval

The novelty is incremental, while important related work and baselines are missing. The setting of learning on the test set is questionable. Comparisons with baselines are lacking, not fair and not informative enough. Section 4 fails to properly separate existing work by Smith et al. (2023) from this work and is partly unclear. More details follow.

1. The paper addresses image retrieval and performs evaluation on standard deep metric learning and place recognition datasets. It misses revelant foundational work on image retrieval, including methodologies, benchmarks and protocols:

   [*1] Gordo et al. 2016, Deep Image Retrieval: Learning Global Representations for Image Search

   [*2] Radenovic et al. 2017, Fine-Tuning CNN Image Retrieval with No Human Annotation

   [*3] Radenovic et al. 2018, Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking

   [*4] Weyand et al. 2020, Google Landmarks Dataset v2: A Large-Scale Benchmark for Instance-Level Recognition and Retrieval

   This work should be repositioned against this work in terms of novelty and protocols used. The related work and experiments sections should be extended accordingly, considering additional training and test datasets and evaluation protocols from these works.

2. The motivation for AE-SVC is scalability. That is, it is challenging to have a labeled dataset-specific training set, hence it is suggested to learn in an unsupervised on the test (reference) set. On one hand, this setting is extremely problematic because it transfers the burden from training to testing, by having to learn something on the test set. The latter is in many practical applications constantly changing, hence the model would need to be constantly updated. It is a kind of overfitting to the test set that has never been used in standard work like the references above. To my knowledge, it has only been used in

   [*5] Liu et al. 2019, Guided Similarity Separation for Image Retrieval

   Hence, this work should be a baseline for comparison if the authors insist in their setting of learning on the test set. On the other hand, if the motivation to learn in an unsupervised way, this can still be done on a separate training set. For example:

   [*6] Kim et al. 2022, Self-Taught Metric Learning without Labels

   Besides, learning in [2*] may be supervised, but with labels obtained automatically without human annotation. Self-supervised learning should not be a reason to learn on the test set.

3. In AE-SVC, I believe variance loss (3) is redundant. It is merely the diagonal part of covariance loss (2). Other than that, the covariance loss (2) and centering loss (4) are inspired by centering, PCA and whitening. Hence, AE-SVC should be compared to a baseline where centering, PCA and whitening are performed as post-processing on features extracted by the foundation model, without learning. This baseline is missing. Whitening has a long history in image retrieval but ignored here. Also very successful is supervised discriminative whitening [*2], which could be adapted for self-supervision to be used as a baseline.

4. In SS2D, I believe the loss is the same as

   [*7] Park et al. 2019, Relational Knowledge Distillation

   The authors cite this paper but do not discuss the actual relation with their work. I believe the only new component here is applying it for dimensionality reduction and learning for multiple target dimensions at once, which are pretty incremental contributions. Also very relevant is

   [*8] Kalantidis et al. 2022, TLDR: Twin Learning for Dimensionality Reduction

   in the sense that it is also learning a kind of autoencoder on top of frozen feature for dimensionality reduction. Instead of a reconstruction loss, it uses the Barlow Twins loss on the output vectors, which is the same as the covariance loss here, but uses cross-correlation instead. Both the architecture and cross-correlation matrix should be compared to as baselines.

5. Section 4 is useful and interesting, but it is presented as new to a large extent while it actually all comes from Smith et al. (2023). The only new part is the experiment of Figure 3. In addition, parts are copied from Smith et al. (2023) without carrying over all relevant information, such that the resulting text is unclear. For example, (8) refers to "discriminative power" but there is nothing to discriminate here, whereas Smith et al. (2023) compare the distribution of target similarities to a "null" or "background" distribution where the term "discriminative" makes sense. As another example, (14) results in a "global minimum" without saying anything about the function being convex, which is shown indeed by Smith et al. (2023).

   In the analysis of Section 4, the data is taken to be centered and nothing is said about the effect of this choice to the distribution of similarities, either here or in Smith et al. (2023). To my understanding, the variance of similarities is a concave function of the mean and is maximized when the mean is zero. This is the opposite of what happens with whitening and raises questions on the motivation of minimizing the variance. To the very least, the experiment of Figure 3 should be extended to also show the distribution of similarities with centered data but without PCA and whitening. It should also include centering, PCA and whitening as post-processing steps rather than being learned with (2), (3) and (4).

6. In the experiments of Figures 4 and 5, the current work (AE-SVC, SS2D or SSD) uses an additional encoder on top of the foundation model, while the "default" and "PCA" are using the foundation model alone, hence are not comparable. These results tell us nothing about whether the benefit is coming from the properties encouraged by the loss functions, the additional learned parameters or the adaptation to the test set. These are three factors that should be evaluated independently in some way, for example by fine-tuning the last layers of the foundation model rather than adding an extra encoder.

   Whitening by post-processing after PCA should be definitely added as a baseline. TLDR or other state of the art dimensionality reduction methods should also be added, because centering, PCA, whitening and VAE are very basic baselines.

Post-rebuttal

The following is an assessment after rebuttal and considering the other reviews.

The contribution is incremental, which is not necessarily a problem if there is a clear and fair positioning with respect to previous work, a clear message and good results. The situation has clearly improved during rebuttal and discussion, but I am afraid that the paper is still weak in these three aspects. In particular:

• Losses. The authors state in the dicussion that there is "stark contrast" with TLDR in that they use three losses rather than one. The covariance loss is very close to Barlow Twins loss used in TLDR, which is cross-correlation. An experimental comparison between the two is not there. The variance loss only refers to the diagonal of the covariance loss and treating it separately makes sense only because of using a different loss weight, which is not mentioned anywhere in the main text. The appendix A6 says $\lambda_{\text{cov}}=1$, $\lambda_{\text{var}}=15$, but the effect of each $\lambda$ is not shown. What if $\lambda_{\text{cov}}=15$ for example? A contribution is meaningful when effective, especially if incremental. The mean-centering loss is something ubiquitous. The relative benefit of each loss component is not shown anywhere. Under these conditions, the contribution claim of three losses is still not adequately supported by evidence.

• Theoretical support. The authors claim in the paper and discussion that providing theoretical analysis in section 4 is a contribution but this study is carried out by Smith at al. (2023) and section 4 does not clarify if there is anything new here except the experiment of Figure 3. This is kind of over-claiming contribution. Minimizing the variance of cosine similarity distribution is still not a clear narrative as motivation since, as a function of the mean, it is maximized. Using the term "discriminative" when there is nothing to discriminate from is still confusing.

• Distillation. The authors state in the paper that their approach "is conceptually similar to SPKD", but this is different from saying "we use RKD (Park et al., 2019)", for example. Again, this is kind of over-claiming contribution.

• Supervision. On one hand, there is a supervised PCA whitening approach that is common in image retrieval, as performed by Radenovic et al. (2017) mentioned in my review. This would be another baseline, plus the authors could use this kind of supervision on their own approach. On the other hand, there are more unsupervised metric learning methods like Kim et al. (2022) that I have mentioned in my review, which could be used for distillation or self-supervision, as competitors or as components of this method.

• Two-step. The authors state in the discussion that their approach is two-step in "stark contrast" with previous work, but they do not experiment with an one-step variant of their approach to show the benefit and they do not explain why two-step is better.

The most important additions during the rebuttal are the experiment of training on a held-out training set, the comparison with TLDR and applying the losses at different layers (Table 2 of rebuttal document). The last one possibly accounts partially for the performance improvement over TLDR but the effect of the losses is not shown. However, as discussed above, several things are still missing in terms of fair positioning with previous work, some rewriting to make the message clearer and more experiments. I believe all of these issues could be resolved in theory, in which case the paper would be interesting enough to publish despite the incremental contribution. In case the paper is accepted, I strongly recommend the authors to revise accordingly for the camera-ready.

• I am somewhat skeptical about the scenario described in L39-L42; in fact, self- or semi-supervised learning might address this issue (see related works [1, 2]).
• I have some doubts about the novelty of AE-SVC, as it combines several common losses. Additionally, there are four losses in Eq. 5, but no explanation is provided on how they are balanced. Experiments or theoretical support in this regard would be beneficial.
• The experimental results are somewhat limited. Is there additional related work beyond PCA for comparison? Also, I noticed a lack of ablation studies, as only two experiments, shown in Fig.4 and Fig.5, are provided.

[1] eProduct: A Million-Scale Visual Search Benchmark to Address Product Recognition Challenges

[2] V2L: Leveraging Vision and Vision-language Models into Large-scale Product Retrieval

1. The theoretical analysis in Sec.4 is only conducted based on the Gaussian distribution assumption, which does not necessarily hold for the embeddings from the visual foundation models. Based on the PDF visualization in Figure 3, the true distribution of cosine similarity between image embeddings is far from Gaussian.

2. In large-scale retrieval, approximate nearest neighbour (ANN) is commonly used to accelerate the retrieval process, so the complexity analysis of image retrieval in the paper is not necessarily right in real-world applications.

3. The figure placement can be optimized. The method details should not be in the first figure, which increases the burden of understanding the introduction, and forces people to switch back and forth when reading the method section.

1. The motivation of AE-SVC is worth discussing: it encourages the network to learn the normal embedding distribution (mean to 0, std to 1), but gt labels of dataset-specific tasks may biased.
2. The authors claim that they are the first to explore representation learning ideas in image retrieval, considering retrieval is essentially a feature learning task, the soundness is not as remarkable as claimed.
3. AE-SVC is only compared with the standard PCA in Sec.5.2, more competitors (e.g. PCA variants or AnyLoc ) are needed to demonstrate the effectiveness of AE-SVC, if possible.
4. Experiment settings in Fig.4 and Fig.5 are not consistent. Compared with AE-SVC, the results of (SS)2D seems incomplete.

1. The detail of the projection function F in L280 may need more explanations. The size of the embedding \hat{z} is adaptive. The change of the embedding size whether influences the projection F. In other words, there are matched functions for different size m.
2. The figures in Fig. (3,4,5) are small. It is hard to distinguish data points of lower embedding dimensions. It could be better that the quantity results are listed in a table.