Collapse-Proof Non-Contrastive Self-Supervised Learning

Thank you for the appreciation of our work and the time dedicated to review our paper. Please find below the answers to your questions:

Comparison with VicReg We agree that VicReg is another non-contrastive self-supervised method. Given the limited amount of time available for the rebuttal, we have decided to provide additional results including experiments with a ResNet-18 on CIFAR10 and add additional baselines, such as SwAV and GEDI no gen, on ImageNet-100 to answer the requests from other reviewers (please refer to the answer to Reviewer KMvq). While we didn’t perform an explicit comparison with VicReg on ResNet-8 and ImageNet-100, it is still possible to observe that VicReg achieves similar performance to Barlow Twins when using a Resnet-18 on CIFAR-10 by looking at the results reported in the solo-learn repository, which is used in our experiments (e.g. Barlow Twins 92.1% vs. VicReg 92.1% acc@1 on CIFAR-10). This is reasonable as VicReg belongs to the family of feature decorrelation approaches like Barlow Twins.

Training Full ImageNet The main bottleneck arises from the fact that we don’t have access to large computational resources to scale up to full ImageNet.

Training convergence We provide a comparison of the linear probing performance over training on the experiments for ResNet-18 on CIFAR10. We visualize the top1 accuracy over training epochs both on linear and logarithmic scales. We observe that CPLearn achieves better convergence rate compared to the other approaches, thanks to the quasi-orthogonality condition of the code matrix W. The plots are available at the following anonymous link and will be added in the Supplementary Material together with the additional experiments done in this rebuttal (that is experiments with a ResNet-18 on CIFAR10 and additional baselines, such as SwAV and GEDI no gen, on ImageNet-100).

Please let us know if you have any additional questions.

Thank you for appreciating the theoretical nature of our paper and providing constructive suggestions.

Baselines for Table 1 and Table 2 Thank you for the suggestion. We did an extra effort in the limited time available to provide additional baselines and make Table 2 more complete. Please find below the table of results with the addition of GEDI no gen (the version without the generative model) and SwAV. The best performance is highlighted in bold for each method. Overall, we observe that CPLearn achieves linear probing performance comparable to the best methods, such as SwAV and Barlow. Moreover, CPLearn significantly outperforms all other approaches in terms of clustering performance. This stems from the unique capability of the model to perform both clustering and feature decorrelation.

Clarity: choice of Rademacher distribution Thank you for the curious question. There are several ways to define random code vectors in hyperdimensional computing (HC). We chose to use the multiply-add-permute encoding, which leverages Rademacher distribution to sample them. We refer you to a recent survey on HC for more details [1]. This is the simplest form of encoding equipped with simple element-wise addition and multiplication operations to perform algebraic compositions. We are going to add this in the paper and reference the survey. It is also important to mention that the exploitation of the compositional properties of HC is beyond the scope of the paper, but we are actively working on this direction.
Also, it is important to clarify that the code matrix W is fixed throughout training. This is necessary to achieve the collapse-proof property as demonstrated by the theory in the paper (cf. Theorem 1 and related Corollaries).

Refer
[1] Kleyko et al. A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations. ACM Comp. Surv. 2022

Time/storage analysis We provide a table for the experiments with ResNet-18 trained for 1000 epochs on CIFAR10 comparing all methods in terms of storage (MB) and training time (minutes). We also provide the linear classification and clustering results for the sake of completeness. The training time does not seem to be impacted much by the larger projector. The size of the checkpoints do increase significantly. However, the relative difference would be much smaller with larger networks such as ViT-S/B/L or larger ResNet architectures as the backbone itself would be much larger compared to the additional parameters in the projector.

Typo Lemma 1 Indeed that is a typo. Thank you for spotting it !

We hope we've effectively addressed your concerns and encouraged you to reconsider your rating.

Thank you for appreciating our work and the constructive review.

Validation of quasi-orthogonality $W^TW=fI$ holds in probabilistic terms, formally governed by Eq. 5 in the paper, namely in expectation we have $E_W[cos(w_i,w_j)]=\delta(i-j)$, with $\delta$ being a Kronecker delta function (equal to 1 iff $i=j$ and zero otherwise), and its variance is determined by $Var_W[cos(w_i,w_j)]=1/f$. For large $f$, the variance quickly approaches zero and the equality holds almost surely.

Baselines on ImageNet-100 Please find below the table of results with the addition of GEDI no gen and SwAV. The best performance are highlighted in bold for each method. Overall, we observe that CPLearn achieves linear probing performance comparable to the best methods, such as SwAV and Barlow. Moreover, CPLearn significantly outperforms all other approaches in terms of clustering performance. This stems from the unique capability of the model to perform both clustering and feature decorrelation.

Experiments on Resnet-18 Please find below the table with the results on CIFAR10 using ResNet-18. We used the solo-learn codebase to run the analysis with the same suggested hyperparameter configurations. Note that we reproduced all baselines ourselves and that their performance are slightly higher than what reported in solo-learn. Also please note that NMI is a standard and well-established metric to measure clustering performance. Overall, the table highlights similar observations to the ones from experiments on ResNet-8, with CPLearn achieving comparable performance to Barlow Twins and significantly outperforming all other approaches in terms of clustering. Interestingly, both Self-Classifier and CPLearn achieve the highest clustering performance for $c=10$, corresponding to the ground truth number of classes.

Clarity about collapses We are going to move Figure 1 (showcasing all collapses) in the 1st page and then reference it at line 15 to introduce it early on.

Design choices. The activation function in $H$ is chosen in a way to ensure the validity of Theorem 1. We have an extra linear layer with batch norm in $H$ to ensure well-behaved statistics throughout training (as shown in Appendix A). $P$ is a standard linear layer with softmax activation.
There are several ways to define random code vectors in hyperdimensional computing (HC). We chose to use the multiply-add-permute encoding, which leverages a Rademacher distribution. We refer you to a recent survey on HC for more details [1]. This is the simplest form of encoding equipped with simple element-wise addition and multiplication operations to perform algebraic compositions. We are going to add this in the paper and reference the survey. It is also important to mention that the exploitation of the compositional properties of HC is beyond the scope of the paper, but we are actively working on this direction.
We are going to explicitly mention that Eq. 2 consists of a sum of two cross-entropy losses (this can be also seen from Algorithm 1 in the main paper).

We hope we've effectively addressed your concerns and encouraged a reconsideration of your rating.

Reference
[1] A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations. ACM Comp. Surv. 2022

Thank you for appreciating the theoretical nature of our paper and for the time dedicated to review it. Please find below the answers to the major concerns.

Experiments on ResNet-18 Thank you for the suggestion. Please find below the table with the results on CIFAR10 using ResNet-18. We used the solo-learn codebase to run the analysis with the same suggested hyperparameter configurations. Note that we reproduced all baselines ourselves and that their performance are slightly higher than what is reported in solo-learn. Overall, the table highlights similar observations to the ones from experiments on ResNet-8, with CPLearn achieving comparable performance to Barlow Twins and significantly outperforming all other approaches in terms of clustering. Interestingly, both Self-Classifier and CPLearn achieve the highest clustering performance for $c=10$, corresponding to the ground truth number of classes.

Experiments with ViT on ImageNet-100 We respectfully disagree with the reviewer. ViT is a standard backbone for benchmarking self-supervised learning algorithms, see for instance DINO.

Typos Thank you for spotting the typos. Indeed, the sentence “whereas SwAV, Self-Classifier and GEDI achieve overall perform better on clustering tasks” should be rephrased to “​​whereas SwAV, Self-Classifier and GEDI perform better on clustering tasks than Barlow Twins.”

We hope we've effectively addressed your concerns and encouraged a reconsideration of your rating.