Clustering Properties of Self-Supervised Learning

Dear Reviewer kGg9,

We sincerely thanks for your time and the valuable suggestions to improve this paper. Here we address each of the comments in detail.

Why encoding and embedding have different clustering properties: Thanks for this constructive question. Due to word count limitations, we kindly ask you to refer to our response to Reviewer 69BS, specifically under the section Why encodings inherently possess better clustering capability. We sincerely appreciate your understanding and attention to this matter.

How to get the ARI values: In Definition A.2 of Appendix A, we provide a detailed definition and formula for the Adjusted Rand Index (ARI). For practical computation, you may also refer to "sklearn.metrics.adjusted_rand_score".

Theoretical foundations of the findings: Thanks sincerely for this insightful suggestion. The theoretical exploration of the distinction between encoding and embedding remains a challenge in SSL. We will follow your suggestion and continue to investigate how ReSA facilitates positive feedback learning, optimizes the model's noise robustness, and whether it offers improved bounds compared to other methods.

$\mathbf{A} _\mathbf{H}$ seems computationally heavy.: We note that the Sinkhorn-Knopp algorithm, which is used to compute $\mathbf{A} _\mathbf{H}$, does not involve gradient propagation and requires only three iterations, making it both efficient and practical as reported in Caron et al. (2020). To quantitatively describe the computation time of the Sinkhorn-Knopp Algorithm, we conducted experiments and found that using an A100-PCIE-40GB GPU, the time for a single computation of a 1024 * 1024 tensor matrix (as used in our experiments) is 0.001578 seconds. This means that when training ResNet-50 on ImageNet for 1 epochs (1251 iterations), the total time for Sinkhorn-Knopp algorithm is about 1.974 seconds. As shown in Table 4, the overall training time is 0.16 × 3600 = 576 seconds, so the proportion of time spent on Sinkhorn-Knopp algorithm is approximately 1.974 / 576, which is about 0.3%. Moreover, when using larger models, the time spent on the Sinkhorn-Knopp algorithm will represent an even smaller proportion of the total computation time.

[1] Caron et al. (2020). Unsupervised learning of visual features by contrasting cluster assignments.

What if an error exists (some samples belonging to different classes are in the same cluster)? Will the proposed method boost this error?: Thank you for this insightful question. In the context of self-supervised learning, the issue of some samples from different classes being assigned to the same cluster is quite likely, especially when there is no label information guiding the process. This is a challenge faced by most clustering-based SSL methods. However, numerous experiments have shown that ReSA exhibits relatively slow performance improvement in the early stages of training, but achieve significantly better results in the later stages compared to other methods, such as contrastive or decorrelation-based SSL, as shown in Figure 9 of our paper and https://anonymous.4open.science/r/long-tailed-evaluation-2843. This suggests that while ReSA may suffer from incorrect cluster assignments in the early training stages, which slows down learning, it is able to gradually address these issues and learn the correct clustering patterns as training progresses, rather than boosting the errors. Furthermore, clustering-based SSL methods like SwAV, DINO and ReSA typically employ clustering assignments with sharp distribution, which help mitigate the impact of incorrect assignments. Investigating the optimization dynamics of clustering-based SSL is an intriguing theoretical direction, and we hope to explore this area in the future.

The assignment matrix and the stochastic matrix: We note that $\mathbf{A} _\mathbf{H}$ is the clustering assignment matrix obtained by applying the Sinkhorn-Knopp algorithm to $\mathbf{S} _\mathbf{H}$. The purpose of the Sinkhorn-Knopp algorithm is to ensure that the output is a doubly stochastic matrix, where each row and column sums to 1. Therefore, $\mathbf{A} _\mathbf{H}$ is a doubly stochastic clustering assignment matrix, and the matrices $\mathbf{A} _\mathbf{H}$ on Line 139 and 146 are exactly identical.

If the encoding replaces the embedding, will Eq. (3) produce better performance?: Thank you for this thought-provoking question. In Eq. (3), SwAV requires learnable prototypes to perform clustering. If the encoding is used in place of the embedding—namely $\mathbf{Q}=sinkhorn(\mathbf{H}^\top \mathbf{C})$—we must ensure that the encoding and embedding output dimensions match. We tested this approach in early experiments on ImageNet-100 and found that, although training proceeded normally, the resulting performance was quite poor. We speculate that this may be due to the difficulty of effectively optimizing the prototypes under this condition.

Dear Reviewer 69BS,

We sincerely thanks for your time and the valuable feedback for this paper.

Why encodings inherently possess better clustering capability: We would like to thank the reviewers for their insightful discussion regarding the distinction between encoding and embedding. To the best of our knowledge, the projector has become an indispensable component of JEA-based SSL. However, the dynamics of its optimization and the reasons behind its success remain an open question within the community. Some works have attempted to explain these principles. For example, Jing et al. (2022) [1] empirically discovered that applying SSL loss either to encoding or embedding led to a significant decrease in the rank of the corresponding features. They argued that this rank reduction indicates a loss of diverse information, which, in turn, reduces generalization capability. This explanation aligns with the hypothesis in SimCLR [2], where the additional projector acts as a buffer to prevent information degradation of the encoding caused by the invariance constraint. Additionally, Gupta et al. (2022) [3] 's null space analysis for the projector posited that the projector might implicitly learn to select a subspace of the encoding, which is then mapped into the embedding. In this way, only a subspace of the encoding is encouraged to be style-invariant, while the other subspace can retain more useful information.

Therefore, the SSL constraint can cause the embedding to lose information, which may include not only clustering-irrelevant features such as background information, but also class-relevant information, making it difficult for us to determine which—encoding or embedding—exhibits better clustering properties in this context. Based on this, our paper empirically demonstrates that the projector indeed filters out information that could support clustering, while encoding achieves better clustering performance.

theoretical analysis on ReSA: We are truly grateful for your expectations and support for this work. As clarified above, the theoretical investigation of the projector remains an open research challenge. The community currently lacks definitive theoretical evidence explaining why the encoding can achieve superior downstream performance compared to the embedding. Building on your insights, we can first conduct interpretability experiments to examine which class-relevant and class-irrelevant features are captured by both the encoding and the embedding, thereby demonstrating how the encoding may possess better clustering properties. We also appreciate your suggestion to quantify this semantic preservation difference through methods such as information bottleneck theory. This could be a valuable approach to theoretically investigate why encodings, as opposed to projector embeddings, may lead to better clustering. We will look into incorporating this analysis in the revised version of the paper to provide a more comprehensive theoretical foundation.

error risk: We agree that the paper could benefit from a more rigorous analysis of how the ReSA loss function handles error propagation. In the context of self-supervised learning, early clustering errors are inevitable, as there is no label information or other prior knowledge to guide the process. This is a challenge faced by most clustering-based SSL methods. However, numerous experiments have shown that ReSA exhibits relatively slow performance improvement in the early stages of training, but achieve significantly better results in the later stages compared to other methods, as shown in Figure 9 of our paper and https://anonymous.4open.science/r/long-tailed-evaluation-2843. This suggests that while ReSA may suffer from incorrect cluster assignments in the early training stages, which slows down learning, it is able to gradually suppresses error propagation and learn the correct clustering patterns as training progresses, rather than amplifying the errors.

projector-free SSL baselines: To the best of our knowledge, the projector has become an indispensable component of JEA-based SSL. Traditional projector-free baselines, such as the MoCo (He et al. 2020), no longer achieve competitive results, while newer versions like MoCo V2 and V3 have demonstrated the projector’s crucial role in enhancing representational generalization.

long-tailed data: Dear Reviewer, please refer to https://anonymous.4open.science/r/long-tailed-evaluation-2843.

Transfer to Downstream Tasks: In addition to the experiments described in Section 5.3, we also perform transfer learning on fine-grained datasets (Table 6) and conduct a low-shot evaluation (Table 10). We will include these details in Section 5.3 as well.

[1] Understanding dimensional collapse in contrastive self-supervised learning.

[2] A Simple Framework for Contrastive Learning of Visual Representation.

[3] Understanding and Improving the Role of Projection Head in Self-Supervised Learning.

Dear Reviewer PEEd,

We would like to begin by sincerely thanking you for the insightful and valuable comments. Below, we address each of the comments in detail.

Relation to Broader...: We have revisited these papers, and will cite and discuss them in the new manuscript.

• We will further investigate the ELBO proposed by Bizeul et al. which may be helpful to derive ReSA's bounds.
• MuGS and MSN are both excellent works that aim to enhance SSL performance in ViTs. In contrast, this work is grounded in traditional SSL benchmarks to demonstrate the effectiveness of encoding-based positive feedback SSL, which is why our experiments primarily use ResNet models. In the past two months, we also explored the effectiveness of ReSA pretraining in ViTs using framework of DINOv2, achieving 77.8% k-NN accuracy with ViT-B after just 100 epochs, compared to DINOv2's 76.0% after 100 epochs and MuGS’s 78.0% after 400 epochs.

087R: Here, we refer to SSL methods that apply clustering constraints on the embedding (i.e., the output of the projector). Since DeepCluster (Caron et al., 2018) does not include a projector, this limitation was addressed and improved as DeepCluster-v2 noted in the SwAV (Caron et al., 2020). We will clarify this point.

086L: In the definition of joint embedding architectures (JEA), it is feasible to use either two identical (i.e., one shared) or different neural networks. In practice, using the same architecture without sharing parameters (i.e. the teacher-student distillation with momentum updates as used in BYOL, DINO, and MoCoV3) often yields the best performance. For generality, we do not emphasize the detailed choices of neural networks here, as different methods may adopt different strategies.

083R: Thanks again for this constructive question. Due to word count limitations, we kindly ask you to refer to our response to Reviewer 69BS, specifically under the section Why encodings inherently possess better clustering capability. We sincerely appreciate your understanding and attention to this matter.

085R: As we clarify above, the invariance constraint imposed by the SSL loss can cause the embedding to lose diverse information, which may include clustering-irrelevant features such as background information, color redundancy, viewpoint variations and so on. In such cases, fewer clustering-irrelevant features contribute to better clustering performance. However, the invariance constraint may also lead to the loss of class-relevant information, making it difficult for us to determine which—encoding or embedding—exhibits better clustering properties in this context.

100R:

• Local clustering capability refers to how well the representation partitions the data into meaningful local clusters. A higher mean silhouette coefficient for the representation indicates better separation and tighter grouping of data points within their respective clusters, which demonstrates stronger clustering capability.
• Local clustering stability is quantified by the standard deviation of the silhouette coefficient. A lower standard deviation signifies that the clustering results are more consistent across the data points, meaning fewer outliers and more stable cluster assignments. This indicates that the clustering representation is robust and less sensitive to variations or noise in the data. We would clarify these more clearly in appendix A.

142L: When we talk about the 'ability' of a set of vectors, we are referring to how well they can represent the underlying structure of the data. In this case, 'Global clustering capability' looks at how well the entire set of data is grouped into clusters as a whole, considering the overall structure of the data. As we use k-mean when calculating ARI, so a higher ARI value also indicates that the relationships between the vectors can be well captured and measured using the similarity.

208L:

• We sincerely appreciate your valuable feedback. We will explicitly revise $\mathbf{S} _\mathbf{Z}$ to $\mathbf{Z}^\top \mathbf{Z}'$ for better clarity.
• Although $\mathbf{S} _\mathbf{H}$ is a symmetric matrix, the iterative row and column normalization inside the Sinkhorn-Knopp algorithm does not preserve symmetry. This is because each row and column can receive different normalization factors during each iteration. As a result, even a symmetric input matrix can become asymmetric after being processed by the Sinkhorn updates.

214L:

• Yes, they indeed represent the same softmax functions. Here we stated them twice to emphasize that these distributions are respectively computed row-wise and column-wise. We appreciate your feedback and will revise it to convey this more clearly.
• $\mathbf{1} _m \in \mathbb{R}^{m\times 1}$ stands for all-ones column vector as we note in (144R) of Algorithm 1. Here, we express the softmax functions in the form of matrix multiplication, where the denominator corresponds to the sum of each row.