Self-Organizing Visual Prototypes for Non-Parametric Representation Learning

We greatly appreciate the reviewer’s careful assessment and constructive feedback on our work. Your insights have helped us improving the manuscript. Below, we address each point in turn and provide additional visualizations and tables through anonymous links as requested.

[...] paper does not demonstrate that SOP achieves comparable performance with fewer prototypes [...]

Due to space limitations, we kindly refer the reviewer to Table 10, and to our first response to reviewer KMMU, where we address a similar question.

 Besides, there is no t-SNE/UMAP visualization [...].

As suggested, we provide additional t-SNE visualizations of features using pre-trained SOP and iBOT encoders.

• tSNE Cifar10
• tSNE Cifar100

The FIFO memory might propagate noise [...] lack of explanation on how the model avoids collapse.

We agree with the reviewer on the importance of discussing collapse avoidance. We hypothesize that one key aspect contributing to the stability of our method is the built-in randomization of our SOPs. As quoted in line 67 and 82: “Our proposal organizes the feature space by exploring random local structures within the data”.

We agree that the FIFO updates might introduce biases where newer representations are located at one end of the memory while older ones are located at the other end. However, the SOP's built-in randomization, which selects random anchors at each iteration, breaks this “recency correlation” in the FIFO memory, preventing the bias from affecting the loss function.

To support our hypothesis, we conducted a Collapse Analysis comparing the effectiveness of our method when SOPs are randomly created at each iteration (default behavior) versus a version where SOPs are fixed after the initial random selection, similar to learnable prototypes. As shown, fixing the SOPs lead to collapse.

 [...] does not ablate the SOP-MIM [...] to isolate the contribution of the losses.

We kindly refer the reviewer to Section 4: Online vs. Non-Parametric Tokenizers and Table 7, where we ablate and isolate the contribution of each proposed loss function, as suggested. In summary, training with $L_{CLS}$ alone produces good representations and surpasses competirors in $k$-NN and linear probing. Similarly, training with the non-parametric $L_{MIM}$ alone improves over iBOT (MIM alone) by $+7.3\%$ ($k$-NN).

[...] memory bank $E_C/E_p$ acts as a [...] non-differentiable parameter store. [...] a semantic distinction [..] does not explain how SOP’s design differs [...]

We respectfully offer a different perspective from the reviewer on this point. We believe that learning prototypes from scratch and optimizing a memory of non-parametric representations that self-organize, have important differences, which we highlight below:

1- The optimization task does not involve gradients to update prototypes.

2- In prototypical SSL, centroids are initialized at random locations and move in the space to find an optimal position. The SOP strategy optimizes random regions in the feature space using semantically related embeddings to represent their regions.

3- Our method does not require additional terms in the loss function to avoid collapse, which is a critical component in current prototypical SSL.

Regarding differences in optimization dynamics, we discuss how the proposed approach differs from prototypical SSL in terms of learning fixed prototypes vs. dynamic SOPs, lines 035, 082. About generalization, we discuss the issue of over-clustering that current methods employ and how it can hurt the learned representations (lines 052, 064, and Table 10).

[...] memory size [...] is not tested. [...]

We thank the reviewer for pointing out this flaw. Please refer to Memory Size ablations where we ablated the sizes of the $`[CLS]`$ memory $N_c$ and the size of the patch memory $N_p$. We observed a similar reversed “U” shape in both components. In general, increasing the memory size is beneficial up to a certain point, $N_c=65536$ and $N_p=8192$ for the class and patch memories, respectively.

[...] there is no analysis of prototype collapse [...].

We thank the reviewer for the thoughtful suggestion. However, in our setup, it doesn’t make sense to analyze the prototypes since our working memory is different from the traditional prototypes in existing methods. The memory that we use to compute the SOPs is ever changing due to its FIFO property. Thus, SOPs are dynamic (not fixed) data structures. They are randomly created at each iteration from random locations in the feature space. Hence, we don’t have access to them as we would have access to prototypes in regular prototypical SSL. Please refer to Collapse Analysis where we show that randomly creating SOPs is the key to avoid collapse.

We sincerely thank the reviewer for his/her careful reading and constructive feedback. The insights have significantly contributed to improving our manuscript. Below, we address each point in turn.

2-1 underrepresentation and overclustering [...] no evidence [...]

We argue that over-clustering and underrepresentation of prototypes are related. As noted in lines 036 and 052 (col 2), due to the weak supervision in SSL, prototypical methods tend to learn a large number of prototypes, dividing the feature space into many smaller sections, each with a prototype as its centroid. Consequently, each section contains fewer mages, biasing the prototypes towards learning simpler features, potentially leading to an underrepresentation of its region.

To support our claim, we conducted an ablation study on different values of K (number of SOPs). If our claim is correct, our method should perform well with a small number of SOPs. In Table 10 and line 394, using K=1024 anchors (similar to ImageNet classes), the SOP strategy outperforms iBOT, which learns K=8192 prototypes. Furthermore, the performance difference between K=1024 and our default value of K=4096 is minimal (0.5%). In contrast, iBOT exhibits a 3x larger discrepancy (1.5%) for the same comparison; DINO [2] reports an even greater difference (1.9%).

2-2.[...] no difference one-hot vs. soft approach [...]

In Table 9, we compared two strategies for modeling the contribution of SEs towards their SOPs. We reached the same conclusion as the reviewer and demonstrated that our method is robust to both strategies (line 377, column 2). We believe the "soft assignment" approach has a slight advantage because it handles uncertainty better, which is crucial in SSL. Intuitively, we can relate to the "label smoothing" regularizer commonly used in semi-supervised setups, which improves generalization by softening ground-truth labels, reducing noise and preventing overfitting.

2-3. no performance difference for C=K=100 in [1] [...]

If we understand correctly, the reviewer is asking why SwAV [1] does not report a significant performance difference between learning K=1000 and K=3000 prototypes, which might contradict our claim. We believe there are two reasons. First, the difference between K=1000 and K=3000 may not be large enough to highlight the discrepancy. Second, performance-wise, iBOT and DINO are more effective than SwAV. In fact, DINO is the successor of SwAV (same main author), and one of the reasons why it performs better is over-clustering. This is demonstrated in the DINO paper [2], page 16, where there is a significant difference in performance (1.9%) from learning K=1024 to K=65536 prototypes, supporting our claim. The same pattern occurs in iBOT, (cf. Table 10). We agree that the reference to [1] should be removed from this context.

[2] Emerging Properties in Self-Supervised Vision Transformers (ICCV)

3-1-1. Nc and Np not defined.

Thanks for pointing this inconsistency. Specifically, $N_C$ and $N_p$ are the sizes of the memories $E_C$ and $E_p$.

3-1-2. [...] not possible to infer Y.

Thank you for the feedback. The manuscript now includes the shape of $Y$ in the text. Please note that the shape of $Y \in \mathbb{R}^{K(k+1) \times K}$ is defined in Figure 2's caption, and its practical size is detailed in Sec. A, appendix. Please refer to line 193 for an explanation of $Y$, which encodes the contributions of SEs, towards their SOPs.

5-1."complex backbones, larger performance gains" no intuition [...]

Thank you. The manuscript has been updated with an intuitive explanation. We believe the performance gain from larger backbones is linked to a higher efficiency of the SOP mechanism, which provides richer features to optimize views. Since SOPs use support embeddings (SEs) to represent their region in latent space, complex backbones can better utilize SEs' features, resulting in larger gains (cf. Table 1, line 264).

10-W1. overall improvements [...] seem marginal.

Due to space limitations, we kindly refer the reviewer to our third response to reviewer Rjpz, where we discuss this issue in detail.

10-W2. Table B.1, advantages [...] not clearly evident

While we did not claim advantages in computing and memory, Table B.1 shows that our non-parametric approach does not incur additional computational/time requirements and can be trained on commodity hardware. Please, note that MaSSL does not use a MIM loss, reducing computing/time resources.

12-1. When comparing the case where [...]

The fact that a support embedding (SE) may belong to more than one SOP is a consequence of the k-NN algorithm. Each anchor selects the k closest embeddings as SEs, allowing distinct anchors to select the same SE. To address the reviewer's concerns, we tested a [Non-shared SEs] (https://shorturl.at/iqdmT) version, where an SE cannot participate in more than one SOP. Results show no practical difference, likely due to the small number of such cases.

We thank the reviewer for his/her careful assessment and constructive feedback, which helped us clarify key concepts about our work, improve the readability of the paper, and strengthen our arguments. Below, we address each of the reviewer's points in turn.

Figure 2 lacks some illustration details [...] add explanations [...] for squares, circles, and different colors [...].

We thank the reviewer for the constructive feedback and apologize for the lack of clarity in Figure 2. To improve readability and aid reader comprehension, we have added text labels for each symbol in Figure 2. Please refer to Figure 2. The updated figure now includes labels below each core component, clarifying the meanings of squares, circles, and different colors.

What are the advantages of SOP over learnable tokens? no discussion of trade-offs [...]

In the paper, we highlight the following advantages of our method over existing prototypical SSL approaches:

1- Our work reaffirms the feasibility of non-parametric SSL. Our goal is not to discourage parametric clustering-based approaches. Instead, we sought to demonstrate that our proposed non-parametric method is a viable alternative, a different path that can contribute to the scientific discussion and development of visual unsupervised learning.

2- Our method avoids learning prototypes from random weights and exhibits several key benefits:

• Stability: The method is inherently stable.
• No Extra Regularizers: It does not require additional regularizers to prevent mode collapse (Section 2.2.1).
• Extensibility: It is extensible to various pretext tasks such as MIM (Section 2.3 and Table 7).
• No Over-Clustering: It does not necessitate over-clustering (Section 4, Table 10).
• Transferable Representations: The SOP pre-training strategy produces representations that are transferable across many different tasks, with varying degrees of difficulty (Tables 1 to 7).

3- Moreover, we show that SOP’s performance increases as we scale the model architecture (Section 3.1 and Table 1).

[...] the proposed method shows only marginal improvements over previous baselines, with some results improving by less than one percentage point [...].

We acknowledge the reviewer's concern regarding the magnitude of improvements. However, we believe that scoring well across a wide range of downstream tasks demonstrates the robustness, stability, and effectiveness of the proposed approach, which is particularly challenging in SSL.

As recognized, we conducted thorough evaluations on various downstream tasks, including classification, retrieval, robustness, fine-tuning, transfer learning, object detection, and segmentation. While some improvements may seem marginal, we would like to highlight several non-trivial improvements:

• Image Retrieval (Table 5): mAP improved by +3.2.
• Robustness Against Background Changes (Table 6):
  • OnlyFG (OF) +2.3
  • Mixed-Rand (MR) +2.6
  • Mixed-Next (MN) +2.7
  • Only-BG-B (OBB) +2.3
• kNN with ViT-Large backbone (Table 1): +1.2%
• Semi-Supervised Fine-Tuning with 1% of Labeled Data (Table 1): +1.0%

Additionally, in Table B.2 (appendix), we report an improvement gain of +2.3 in semi-supervised learning (low data regime) on ImageNet using frozen features.

These results collectively demonstrate that our method provides non-trivial improvements and is effective across various tasks and settings.

In previous SSL methods, a view means an augmentation of the input image [...] in the context of SOP, the meaning of “view” is not clear. [...], this description needs more clarity.

We apologize for any confusion regarding the term "view." In this work, views are augmented versions of an image, similar to existing SSL methods. Based on the reviewer's feedback, we have improved the readability of [Figure 2] (https://shorturl.at/hlvGW).

Additionally, we define a view $x^v$ as an "augmented version of an image" in the Notation paragraph (page 094, column 2), and we use multiple views (multicrop) during optimization. The misunderstanding may have arisen from a part of the text where we intuitively suggest viewing SOPs as augmented prototypes. This interpretation emphasizes that, rather than relying on a single prototype, SOPs group features among different SEs, collectively representing the region in space more accurately.

Some typo issues [...].

Thanks, we have corrected the typo issues and the format of double quotes.

[...] does SOP has higher computational efficiency compared to previous methods? The inclusion of an efficiency comparison is recommended.

We agree with the reviewer on the importance of an efficiency comparison. Please refer to Table B.1 (appendix), where we compare the computing and time resources of our approach against existing prototypical SSL methods. Our findings indicate that the computational and memory requirements of SOP are similar to those of existing approaches such as iBOT.

We appreciate the reviewer's fair assessment of our work. We would like to highlight key points from the review, including our "extensive experimental verification" and the accurate summary that captures the core innovation of our method. The review emphasizes our method's ability to mitigate over-clustering and its consistent performance across a wide range of benchmarks. We acknowledge the constructive feedback and address each point in turn.

Training time and memory usage of SOP and baseline methods are not compared. [...] In particular, the efficiency of k-NN search [...].

Regarding the training time and memory usage of SOP against baselines, we kindly refer the reader to Section B.1 and Table B.1 in the appendix. In Section B.1, we discuss the trade-offs and differences between parametric and non-parametric approaches. To address the reviewer’s point, we found that the computing and memory requirements of SOP are similar to existing approaches such as iBOT (Table B.1). As pointed out, our method requires running k-NN to bootstrap the support embeddings. On the other hand, our method does not have the large learnable prototypical layer present in clustering-based methods such as DINO and iBOT, which requires extra computing and memory to compute/store gradients in the backward pass. Our method stores 256-d representations in two separate memories E_C and E_p of sizes 65536 and 8192, which are updated following an efficient FIFO strategy.

Transfer learning on smaller datasets is not very meaningful and can be placed in supplementary materials.

We acknowledge the reviewer's point and share their conclusion regarding the saturation of these benchmarks, as indicated in line 326, column 1. Following the reviewer's recommendation, we will relocate Table 3 (Transfer learning on smaller datasets) to the supplementary material and insert Table B.1 (Training time and memory trade-off) into the main text.

W1 Figure 2 is not very easy to understand, and it is difficult to make a good connection between the pseudo code and Figure 2

Following the reviewer's suggestion, we have updated Figure 2 with supplementary labels to improve clarity. The revised figure can be found in Figure 2 (https://shorturl.at/hlvGW).

W2 The motivation of SOP-MIM is clear [...] it is just an incremental work of MIM. [...].

The majority of work on SSL that optimizes an MIM task focuses on learnable tokens, such as iBOT, BEiT, and DINOv2. Instead, our work presents an alternative approach. We show that a non-parametric strategy also works well for MIM, opening new possibilities for further development.

W3 Is the key to the improvement of this paper due to the increase in the number of anchor points? [...] current method performs best when the number of prototypes K≫C is much larger than the number of actual categories C [...], but the method proposed in this paper also requires larger prototypes.

Thanks for the feedback. We argue that the SOP strategy is a key factor contributing to the performance of our method. As shown in Table 8, overall performance improves with an increase in the number of support embeddings (SEs). This supports our claim that the SOP strategy, which involves bootstrapping semantically similar embeddings to better represent a random region in the feature space, is more effective than learning a single prototype for that region, as prototypical methods do.

Regarding the number of anchors, we used a default value of K=4096. Although this may seem large, Table 10 demonstrates that even with K=1024 (nearly the same number of classes as in the ImageNet dataset), our method still surpasses iBOT, which learns K=8192 prototypes. Additionally, the performance difference between using K=1024 and K=4096 is minimal (only 0.5%). In contrast, iBOT shows a significantly larger gap (1.5%) when comparing similar settings, as detailed in line 397. These results corroborate our claims about overclustering and how the SOP strategy mitigates it.