Failure-Proof Non-Contrastive Self-Supervised Learning

Thank you for the valuable review. We have included detailed responses following each of your comments.

Choosing $\beta$. $\beta$ is chosen to ensure that both the invariance and matching losses decrease throughout training. Currently, this choice is made manually, but a heuristic could be devised to adapt the hyperparameter and enforce this condition. However, we have opted to keep the approach as simple as possible. Regarding the variability of $\beta$, we did not observe significant differences in its choice across datasets. For instance, for large dictionary sizes, we use $\beta = 0.1$ on SVHN, CIFAR-10, and CIFAR-100, and $\beta = 0.25$ in all experiments on ImageNet100, with selections drawn from the grid $\\{0.01, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10\\}$.

Lemma 1 Thank you for your question. Some clarifications are necessary. Having a lower bound that is computable in closed form is a desirable property of the objective function. This property ensures that attaining the optimal value provides a certificate for avoiding representation and cluster collapse. In practice, when using a finite-capacity or small-capacity backbone network, achieving the global optimum might not be feasible but avoidance can be still ensured. We refer you to Appendix B of the paper, where we have discussed this in more detail. Also, experimental results consistently confirm the avoidance of both representation and cluster collapse. For instance, see Fig. 6 for a conceptual explanation and Figs. 4d–f for further empirical evidence.

Weakness 3. We would like to emphasize that the suggested paper considers a supervised setting, whereas our approach does not rely on labels and is therefore unsupervised. Additionally, the proposed solution is applied in a continual learning context, which is currently outside the scope of our work. Moreover, comparing with contrastive learning is not our primary goal; our aim is to bring together feature decorrelation and cluster-based approaches within the non-contrastive family (as also mentioned by reviewer an4Z).

Question FALCON can outperform other methods thanks to the guarantees on generalization (see Appendix F). Intuitively (and less formally), this happens because the design of the projector and the loss function enforce feature decorrelation, thereby utilizing all neurons in the latent representation and preventing dimensional collapse. Having access to all neurons is crucial for generalization in downstream tasks, as it allows for the selection of relevant information needed to solve the task. The same argument does not apply when the representation is constrained to a smaller subspace, as this leads to a loss of information.

We hope we've effectively addressed your concerns and encouraged a reconsideration of your rating. If you have any additional questions or feedback, please don't hesitate to reach out, we're happy to engage in further discussion.

Thank you for the thoughtful review and the positive feedback. Please find below some more detailed comments.

Code The code will be publicly released upon acceptance.

Question Thank you for the question and insightful comment ! We haven’t thought about this before but we can say that being able to constrain the function class will certainly facilitate and speed up training. This would be therefore a nice future avenue both from a theoretical as well as practical perspective. If you have any additional insights regarding this, we would be happy to discuss it further.

Thank you for the insightful review. Please, find below the answers to the different points.

Scalability Thank you for this comment. We are aware of this, but we don’t currently have the resources to do the analysis on ImageNet1k. We have to rely on external resources to perform such analysis which cannot be achieved in a short timeframe. However, we believe that the theoretical contribution and experimental support of the paper brings new solid insights to the non-contrastive literature.

Limited Training Thank you for the suggestion. Please refer to Table 2 in the revised paper where we have trained DINO and FALCON for 300 epochs. We report the table also here for reference

Novelty Thank you for pointing out the paper on Invariant Information Clustering (IIC). It is important to note that IIC and FALCON are conceptually different. IIC involves using a classification function on the backbone, and its objective is to maximize the mutual information between the class predictions of the two views. Consequently, there is a direct correspondence between prediction outputs and classes. In contrast, FALCON does not establish a one-to-one correspondence between codes and classes, as the dictionary can grow far beyond the number of classes and this is a key contribution of our work, establishing a new connection with hyperdimensional computing. Moreover, while the objective of IIC ensures the avoidance of degenerate solutions like full cluster collapse (i.e., collapsing to a single cluster), it does not prevent other types of failure. For example, as shown in Figure 3 (right) of the IIC paper, the learned solutions suffer from intra-cluster collapse. In FALCON, we avoid all failure modes, including the one mentioned above, due to the conceptual distinction between dictionary codes and clusters. We have cited and discussed the approach at L100 of the revised paper.

Question Thank you for highlighting this important issue. Indeed, 'collapsing' to a set of low-level features such as color is another possible failure mode of non-contrastive learning methods. Since it can typically be avoided effectively using data augmentation, we did not pay much attention to this failure mode. We have added a note at L527 in the revised paper.

We hope we've adequately addressed your concerns and encouraged you to reconsider your rating. If you have any further questions or feedback, please reach out, we’re open for further discussion.

Thank you for appreciating the soundness of our work ! Please find below the answers to your questions:

Setting $\beta$. $\beta$ is chosen to ensure that both the invariance and matching losses decrease throughout training. Currently, the choice is made manually, but one could devise a heuristic to adapt the hyperparameter in order to enforce this condition. However, we have preferred to keep the approach as simple as possible. Regarding the sensitivity of the hyperparameter, we didn’t observe significant variations in performance for small (but non-zero) values of $\beta$. However, it is important to mention that when $\beta$ increases above approximately 0.5, performance decreases significantly. This is reasonable, as the uniformity loss starts to lose its importance, and codes are therefore not used uniformly.

Computation with larger models As you correctly pointed out, the main bottleneck to scalability is the size of the dictionary. There are a few considerations in this regard that could potentially lead to follow-up work. First, while we have a guarantee that performance improves with larger dictionaries, it is important to note that the improvement is not strictly monotonic, and performance plateaus after a certain point (see, for instance, Table 2). Consequently, we don’t need to increase the dictionary size indefinitely. Second, we could better exploit the structure of the dictionary. Currently, we are storing $f\times c$ floats, but one could exploit the bipolar nature of the tensor and develop a more efficient way to store and use it during matrix multiplication. We have not explored this direction yet. Thanks for raising the question ! We have added a paragraph at L530 in the revised paper.

Please let us know if you have any additional questions. We would be happy to answer them !

Thank you for the time you dedicated to understanding and reviewing our paper. We are happy to make the paper more accessible. Please, find below some clarifications to your questions.

Extrema condition To gain some intuition, consider the case with $\epsilon=0$. That condition translates to have one of the p values equal to 1 and the others equal to 0. This can be thought of reaching the extrema of a simplex. We can add a sentence after Lemma 1 to reflect that the notion of extremum is referred to the extremum of a probability simplex. We have added a sentence at L213 in the revised paper.

Decomposition of invariance loss The invariance loss in Eq. 2 can be expressed as a cross-entropy loss. The mentioned decomposition follows directly from the property of the cross-entropy loss, that is $CE(p,p’)=H(p)+KL(p\|p’)$. We have added a clarifying sentence at L228 in the revised paper.

We hope the above explanations, along with the new clarifications in the paper, enhance its understanding. If you have any further questions or feedback, we are happy to engage in discussion.

Thank you for the appreciation of our work and for the suggestions to improve the paper. Please find below more details about the points you mentioned.

Additional Experiments Due to constraints in time and computational resources, we were unable to address all reviewers' requests. We conducted additional experiments with extended training epochs specifically for reviewer R44G.

Batch norm and L2 normalization Thank you for pointing out this intuition. While it is true that both batch normalization (BN) and the L2 norm aid in optimization, they serve different purposes. BN helps condition the optimization to ensure good statistics (zero mean and avoiding variance explosion, as shown in Appendix A). Zero mean is necessary to ensure that the data is spread out on the hypersphere, as you correctly pointed out. However, the L2 norm is crucial for constraining the admissible optimal values of $\alpha$ (as shown in the proof of Theorem 1, L1009–L1024). In other words, the L2 norm enforces a sparse assignment of data to codes. We have added a sentence at L181 in the revised version of the paper.

Prototype collapse Thank you for pointing out the weakness of DINO and referring us to Appendix C of that paper. While this phenomenon is present in DINO, it does not occur in our settings, as the codes are frozen throughout the learning process. Analyzing the singular value distribution directly on the codes of FALCON would be trivial due to the frozen dictionary. We have modified L513 in the revised paper to mention this. This is a nice touch !

Notation Thanks for the suggestion. In our opinion, the apex in $d’$ could overload the notation as it is used for augmentations. $f$ stands for features. While this is an arguable convention, we would prefer to maintain the current notation.

We hope we have effectively addressed your concerns, enhancing the paper's value from your perspective. Should you have any additional feedback, we would be glad to discuss it further.