Feature Normalization Prevents Collapse of Non-contrastive Learning Dynamics

We are glad to see the reviewer acknowledges our contributions positively. Our responses to your concerns and questions are as follows:

More complex scenarios: Thank you for the suggestion. In the newly added Appendix D.2, we provide the similar experiments to the synthetic case with the ResNet-18 encoder. Overall, we can see a similar trend, and moreover, the regime transition can be observed as well (with properly chosen parameters).

Barlow Twins and VICReg: Compared to the conventional contrastive learning pushing anchors away from negative examples, Barlow Twins and VICReg are still different because their variance regularizer is to regularize the dimension-wise variance, not sample-wise. Thus, their underlying mechanism to avoid the collapse is still an interesting problem to study, particularly under the presence of feature normalization, which has not been dealt with.

Thank you for carefully reviewing our manuscript. Let us discuss your concerns about the assumptions. We hope the discussion sounds reasonable and supports our contribution further.

Assumption of the width-infinite limit: This assumption is relevant to the norm concentration, which is essentially the central limit theorem. Numerically, the concentration can be established with the order of 100 dimensions; this is not unrealistic limit, unlike some generalization analysis of neural nets requiring exponentially more dimensions than samples.

Assumption on the norm constancy: This assumption is only relevant to disentangle the matrix dynamics (4) into the eigenmode dynamics (6). We do not need the norm constancy globally (for all time $t$) if we are concerned about this disentanglement at each fixed time. Globally, the norm values may of course change; however, the shapes of dynamics at each time shown in Figures 2 and 3 remain qualitatively the same.

General: Dynamics analysis generally requires strong assumptions such as (isotropic) Gaussian distributions and linear-algebraic assumptions—they are often stronger than practical scenarios. However, this does not mean that these analyses are pointless; as long as they can serve as proxies to some real scenarios. How about our analysis? Isotropic Gaussian—yes, it is quite strong, yet we can establish the regime transition and go beyond the L2-loss analysis (that fails with excessively strong regularization) at least in this specific scenario. The symmetry assumption of $W$ as well: as our simulation actually observes the eigenmodes converge to the stable interval, this can be seen as corroborating evidence that our model behaves as a reasonable proxy. Everyone anticipates the extension to general setups, but not all assumptions can be lifted all at once—our scientific journey is a continuous step to build a better model little by little. To this end, our work contribute to taking feature normalization into account and relating it to complete collapse in a synthetic setup first.

We are grateful for carefully reviewing our manuscript and providing extremely insightful comments. Please see our response below. Typos have been addressed already.

Previous literature: A good catch! It’s a shame that we had been aware of Halvagal et al. (2023) only one week after the submission… They have successfully established the mechanism of the implicit variance regularization of the eigenvalues, which would not be elicitable from our analysis because we assume the standard normal input, regrettably. By contrast, their implicit regularization holds only for non-collapsed eigenvalues ($\\lambda_k > 0$), as seen in their Eq. (11). Thus, they do not explain how the complete collapse ($\\lambda_k = 0$ for all $k=0$) is avoided; our work contributes to this end. The clarification has been added in the updated draft.

Regimes: Based on your suggestion, we attempted to give a slightly more formal derivation of the regimes in newly added Appendix C, though $p\_j$-dynamics is sixth-order and cannot be solved analytically in general.

Regarding the Acute and Stable regimes, we will not have alternation of them numerically because the Stable regime is conceptual and cannot be attained exactly; the unstable interval (gray in Figure 3) could be arbitrarily small as the norms decrease, but will not completely disappear.

Experiments: Thank you for the suggestion. In the newly added Appendix D.2, we provide the similar experiments to the synthetic case with the ResNet-18 encoder. Overall, we can see a similar trend, and moreover, the regime transition can be observed as well (with properly chosen parameters).

Figure 6c: We improved the visualization by adding smoothing along the time axis and added a description to explain how the regime intervals are calculated: they are calculated theoretically by using the $p\_j$-dynamics (8), but solving the sixth-order equation by numerical root finding. In addition, we expanded the analysis in newly added Appendix D.1 to see not only the largest but also the other eigenvalues. Those eigenvalues converging to non-zero values tend to be in the stable (blue) interval, and all the other converges to zero. More discussion can be found there.

Thermodynamic limit: It is relevant to the NTK regime, but the proportional limit with ratio $\\alpha$ is important. Such a proportional limit has been often used to study the generalization error under the double descent (for controlling random matrices). For example, Pennington & Worah (2017) “Nonlinear random matrix theory for deep learning” leverages it.

Findings about EMA: The most general analysis of non-contrastive learning with the exponential moving average is challenging, so Tian et al. (2021) chose to approximate it by the proportional EMA (their Assumption 1): representing the target representation net by $\\Phi_a(t) = \tau(t) \\Phi(t)$. While we can do the same analysis with our setup, the proportional ratio $\\tau$ is eventually cancelled out due to the feature normalization.

We thank the authors for their detailed response.

Previous literature Indeed, the complete collapse seems possible in their work, which this submission tackles. I may be mistaken but I do not see why the implicit variance regularization is not applicable to this work, since it is due to the $\lambda_m$ term in Eq. 9 of in Halvagal et al., which seems to correspond to the $p_j^2$ term in Eq. 9 in the authors' work, and is not particularly related to the assumption on the input? This may be a misunderstanding of the two works on my part. In any case, I mainly think that the comparison should be discussed in the final version.

Regimes Thank you for the clarifications. These points should be made clearer in the final version.

Experiments / Fig 6c The description added improves the readability of the Figure. The experiments on the ResNet-18 help ground the results in a more realistic setting. However, the claim that the norms remain stable becomes hard to maintain. Similarly, the curve of asymmetry of $W$ is quite surprising, but is less important for the results. (Small mistake: 'W becomes relatively asymmetry")

Thermodynamic limit Thank you for the clarification. The term in itself was a novelty for me, and should maybe be cited from a source using it similarly, such as Pacelli et al., A statistical mechanics framework for Bayesian deep neural networks beyond the infinite-width limit.

The authors have answered some of my concerns and improved their article with some additional experiments and clarifications. Thus, I'm increasing my rating from a 6 to a 8.

Thank you for raising important questions to our work! We address them subsequently. We appreciate it if you could rethink the evaluation of our work based on the updates.

Assumption on the norm constancy: This assumption is only relevant to disentangle the matrix dynamics (4) into the eigenmode dynamics (6). We do not need the norm constancy globally (for all time $t$) if we are concerned about this disentanglement at each fixed time. Globally, the norm values may of course change; however, the shapes of dynamics at each time shown in Figures 2 and 3 remain qualitatively the same.

Importance of the problem studied here: Many recent studies have started to theoretically investigate non-contrastive learning and its implicit bias from the perspective of the L2 loss, which may represent different behaviors from the cosine loss as we show in this work. To elicit more useful insights in the future, we believe that our work can definitely contribute to driving the community into focusing on the cosine loss and providing a cornerstone for the analysis.

6th order vs. 3rd order dynamics: As can be seen in the dynamics (8) and (9), they are 6th- and 3rd-order formula in $p\_j$, respectively, which are called the dynamics with respective orders. One of the practical differences is admissible hyperparameter ranges; for example, while regularization (weight decay) cannot be too large for the L2 loss, it is possible for the cosine loss, as confirmed in our pilot study (Figure 1).