FroSSL: Frobenius Norm Minimization for Self-Supervised Learning

Thank you for your review! We address your questions below

Point 1 in Weaknesses We agree that the connection between rotational invariance and fast convergence can be made more explicitly. The connection is outlined follows. In Section 3, we choose the Frobenius norm as an operation that is invariant to unitary transformations. In Equation 9, we show that the Frobenius norm enforces the eigenvalues of the representation space to become equal. Finally, in [1], which was heavily discussed in Section 5, it is shown that learning small eigendirections leads to faster convergence. Because the Frobenius norm naturally does this, it thus can lead to fast convergence.

Point 2 in Weaknesses We assume that Figure 4 was meant rather than Table 4. Certainly in scenarios where compute is easily available, this is perhaps not significant. However, for many groups, including us, training for 30 epochs represents a not insignificant amount of training time, roughly 1.5 days on 4 gpus. Thus the significance of our results may be highlighted in scenarios where compute is limited. As pointed out by Reviewer dD2L, experiments in the short training regime would better highlight the advantages of our method.

Point 3 in Weaknesses The referenced figures are explained in Section 5. The loss variants used in Figure 2 are given in Section 5.1. The Figure 3 dataset is STL10, and was specified in the first paragraph of Section 5.2.

Point 4 in Weaknesses We copy our response to Reviewer rGwc for this point. While it is true that FroSSL only achieves competitive performance without beating baseline methods, FroSSL still retains some advantages over the superior baselines. For example, consider the CIFAR-100 column of Table 2. FroSSL is only outperformed by Barlow Twins. However, Barlow Twins has one important hyperparameter to tune ($\lambda$) while FroSSL has none. We consider this to be an important advantage of FroSSL, even if the performance is not quantitatively superior.

Point 5 in Weaknesses Due to the review period time constraints and compute resource constraints, we are unable to provide larger model experiments at this time. However, we have updated Table 2 and Table 3 to include additional results on CorInfoMax and the performance of FroSSL without logs.

Question 1 This question touches on an important property of matrices: the duality between Gram matrices and covariance matrices. In the notation of Section 3.2, these are $Z Z^T$ and $Z^T Z$, respectively. A Gram matrix represents comparisons between samples while a covariance matrix represents comparisons between dimensions. However, they share many properties such as having the same nonzero eigenvalues and having the same Frobenius norm. Methods like FroSSL that only use these shared properties are simultaneously dimension- and sample-contrastive. On the other hand, methods that use nonshared properties can only be either dimension- or sample-contastive. For example, VICReg uses the covariance matrix diagonals which is, of course, not shared by the corresponding Gram matrices.

We also note that FroSSL is not the first method to reconcile both sample- and dimension-contrastive properties. Tico has both properties too, and it was derived in a similar way to ours via Frobenius norms [2].

Question 2 Thank you for pointing out these incorrect references. These should be Figure 2, Figure 3, and Table 3, respectively. We've amended the links in our document.

Question 3 This was done because for CIFAR-10 and CIFAR-100, we cited reported results from the solo-learn library on baseline methods.

Question 4 We thank the reviewer for pointing out this great reference. We were not aware of it before and it certainly merits discussion. As discussed in Section 3.1, one interpretation of FroSSL is the maximization of Renyi $\alpha$-entropy where $\alpha=2$. The I-VNE+ method maximizes the von Neumann entropy which is a special case of Renyi $\alpha$-entropy where $\alpha=1$. One clear disadvantage of I-VNE+ is that its computation requires eigendecomposition. This can lead to slower training as batch size increases. In fact, all $\alpha\neq 2$ requires eigendecomposition. On the other hand, FroSSL avoids eigendecomposition entirely by using the Frobenius norm.

References

[1] James B. Simon, Maksis Knutins, Ziyin Liu, Daniel Geisz, Abraham J. Fetterman, and Joshua Albrecht. "On the stepwise nature of self-supervised learning."

[2] Jiachen Zhu, Rafael M Moraes, Serkan Karakulak, Vlad Sobol, Alfredo Canziani, and Yann LeCun. "Tico: Transformation invariance and covariance contrast for self-supervised visual representation learning"

Thank you for your review! We address your concerns below

Experiments to highlight performance Based on your comments, we agree that more experiments in the short training regime would be interesting future work.

Additional Baseline In our updated document, we have now included comparisons to one of the baselines you mentioned, CorInfoMax. In Table 2 it is shown that FroSSL outperforms this related baseline on both CIFAR-10 and CIFAR-100. However, we were not able to achieve convergence with it on STL-10 and thus cannot compare its early epoch performance in Table 3.

Missing links and citation Thank you for pointing out the missing citation and incorrect hyperlinks. We have fixed these in the document.

Thank you for your review. We address your comments below.

Quantitative Results While it is true that FroSSL only achieves competitive performance without beating baseline methods, FroSSL still retains some advantages over the superior baselines. For example, consider the CIFAR-100 column of Table 2. FroSSL is only outperformed by Barlow Twins. However, Barlow Twins has one important hyperparameter to tune ($\lambda$) while FroSSL has none. We consider this to be an important advantage of FroSSL, even if the performance is not quantitatively superior.

Error Bars We agree that std would be an important addition to Table 2. Due to the time constraints of the review period, however, we are not able to include such results at this time.

Eventually bested by other methods Based on Figure 3, one possible explanation for this is that FroSSL is able to regularize the representation space very early, while other methods take longer. However, once other methods eventually achieve this regularization, they might possibly outperform FroSSL.

Thank you for your review. We briefly address your comments below.

Rotational Invariance You are correct that Barlow Twins representations are invariant to rotations in the input. However, our point is that the Barlow Twins loss is not invariant to rotations in the representations. That is, only representations with an identity cross-correlation matrix achieve optimal Barlow loss. However, intuitively any rotation of identity should be equally applicable to downstream tasks. Encoding this idea into a loss function allows for a less strict optimization problem and a wider class of potential solutions.

Role of Logarithm In our updated document, we include a comparison to FroSSL with no logarithms. This variant has obviously worse performance than FroSSL. Importantly, we do not use any tradeoff hyperparameter between the invariance and variance terms. While such a hyperparameter may improve performance without logarithms, one intuition in Section 3.1 was that the logarithm acts as a natural alternative to tradeoffs because logarithms self-regulate the influence of each of the terms in the gradient of the loss.

Additional Experiments As discussed above, we have included experiments that show the advantage of the logarithm. We also compare to an additional baseline, CorInfoMax. We agree that more experiments would highlight the performance of FroSSL, but it requires computational resources not available to everyone.