Stepwise Feature Learning in Self-Supervised Learning

Thank you for your thorough and insightful review of our work. We greatly appreciate your time and effort in assessing our paper and providing constructive feedback.

[W1] The primary limitation lies in the reliance on toy models and synthetic datasets, which may reduce the applicability of the conclusions to real-world SSL settings. The practical implications—how to mitigate extent or amplitude bias—are not fully explored. Additionally, the theoretical framework assumes idealized conditions (e.g., feature independence, aligned initialization) that may not hold in complex, modern architectures. More empirical evidence on realistic datasets would strengthen the impact.

[Q3] Broader Empirical Validation The analysis primarily relies on toy settings with synthetic data. To strengthen external validity, could the authors include or comment on experiments using more realistic or semi-synthetic datasets? A systematic study across multiple datasets or SSL frameworks (e.g., SimCLR, BYOL) would help verify whether extent and amplitude bias persist in practice and enhance the relevance of the findings.

[A1, QA3] Our theoretical findings demonstrate strong generalizability to realistic settings, as evidenced by validation across multiple SSL frameworks and practical datasets.

• On the "Idealized Conditions"

  • Feature Independence: It is a widely used assumption.
  • Aligned Initialization : Aligned initialization is generic and is not idealized conditions, which shows the robustness of the findings. To follow the aligned trajectory, all we need is to start from a small initialization, proven in Sec. C of Simon et al. Our experiments (Sec. 4.5, Fig. 10) demonstrate that models initialized with small random weights rapidly achieve alignment in early training stages. The subspace alignment metric quickly approaches 1, confirming that the aligned structure emerges naturally from small random initialization.

• On the Applicability to Realistic Settings: The uncovered principles are broadly generalizable. Our choice of Barlow Twins was motivated by its representative loss structure; as noted by [1], other prominent SSL methods (e.g., InfoNCE, SimCLR and VICReg) also share the same quadratic form $\text{Tr}[WAW^\top]+\text{Tr}[(WBW^\top)^2]$, making it a suitable canonical model for analysis. Our empirical results, including new experiments, validate that our findings are not artifacts of this simplified setup but are indeed a more fundamental phenomenon.

  • [Beyond Barlow Twins]: The phenomenon is not limited to Barlow Twins. In additional experiments, we observed the identical stepwise learning and the extent bias in a 1-layer SimCLR and VICReg model. Furthermore, our experiments on the Colored-MNIST dataset using SimCLR and VICReg reproduced the same results, confirming the behavior holds across different SSL frameworks.
  • [Beyond Single-Layer and Linear Encoder]: The phenomenon is not limited to shallow or linear models. We have verified that the same learning priorities manifest in deep linear networks (with depth = 4). More importantly, as shown in Sec. 4.7, 4.8 and 5.7 of our manuscript, this behavior persists in non-linear networks (e.g., ResNet-18, ViT-B/16). The fundamental learning priority—which features are learned first—remains consistent. This trend becomes even more apparent when evaluating with linear probing accuracy rather than loss.
  • [Beyond Synthetic Datasets]: Colored-MNIST dataset is not "highly simplified" but is a standard benchmark for evaluating model robustness against spurious correlations. We also conduct experiments on Waterbirds dataset, made of Caltech-UCSD Birds-200-2011 and Places dataset. Our findings on these datasets further strengthen the claims.

The consistent results across different SSL methods (Barlow Twins, SimCLR, VICReg), network architectures (linear, deep linear, non-linear), and datasets (synthetic, Colored-MNIST, Waterbirds) provide strong evidence that extent and amplitude bias are fundamental principles of self-supervised learning. We will incorporate these additional results and clarifications into the revised manuscript.

[Q1] The organization of the amplitude-bias section could be improved The transition from frequency to amplitude bias in Section 5 is somewhat abrupt. Although both arise from the same motivation and experimental setup, the narrative jumps between them without first clearly introducing amplitude bias. We suggest restructuring the section to improve clarity.

[QA1] We will restructure the section to provide a clearer introduction to amplitude bias and a smoother narrative flow that better explains how our frequency investigation leads to the key insight that amplitude, rather than frequency, drives learning priority.

[Q2] Interpretability and Practical Implications While the paper provides strong theoretical insights, its practical implications remain unclear. Could the authors summarize key takeaways for practitioners designing or training SSL models? Additionally, although extent and amplitude bias are well-diagnosed, the paper does not propose mitigation strategies. It would be helpful to discuss how the theoretical framework might inform methods (e.g., regularization or augmentation) to encourage learning of more generalizable features.

[QA2] We propose some practical implications based on our findings:

• Extent Bias: We propose data augmentation that partially mask or crop the background, or that increase the size of the object. This would directly manipulate the object's "extent bias" to alter its learning priority.
• Amplitude Bias: We suggest methods to amplify or reduce the amplitude of specific frequency bands where key features are presumed to exist (e.g., for zebra stripes). Alternatively, one could identify high-amplitude bands in a dataset to check for potential shortcuts.

[1] Ziyin et al., What shapes the loss landscape of self-supervised learning?, (arXiv preprint)

Thank you for your thorough and insightful review, and feedback.

[W1] Core issue is its applicability. The current results are limited, both in terms of experimental evaluation in larger networks and complex datasets as well as the theoretical setup. The authors extended the dynamics studied by Simon et al. for the BarlowTwins loss with a high redundancy reduction coefficient. In practice, the redundancy reduction coefficient is significantly smaller, and the resulting dynamics are distinct. Recent work [1] has studied these distinctions, which might be of interest to the authors. Therefore, I would recommend the authors to extend their analysis following the results of Ghosh et al.

[A1] We confirmed that the same phenomenon (e.g., extent bias) occurs even in larger networks, complex dataset, and low redundancy reduction coefficient.

• Large network architecture : We validated our findings using ResNet-18, ViT-B/16, standard deep architectures, rather than analysis to 1-layer linear networks.

• Complex dataset validation :

  • C-MNIST : Fig 2 demonstrates the predicted extent bias behavior across different object pixel ratios (0.05, 0.10, 0.15). Models exhibit rapid initial learning of background features (achieving 70% accuracy plateau), followed by gradual object feature learning, with plateau duration inversely related to object pixel ratio.
  • Waterbirds [4] : We additionaly experiments waterbirds dataset, which is dataset made of Caltech-UCSD Birds-200-2011 and Places dataset. We downsampling 256x256 image size to 128x128. Following the same experimental design on C-MNIST, we set bird-to-image pixel ratio to 0.1 and observed extent bias effects. The background features, having larger extent bias than bird features, showed plateau periods in spurious correlation prediction accuracy (setted by 59.5%), confirming that the phenomenon generalizes beyond synthetic datasets to realistic natural image benchmarks widely used in the spurious correlation literature.

• Low redundancy reduction coefficient $\lambda$ : We confirmed that the same phenomenon (e.g., extent bias) occurs even in a single-layer linear network with a low redundancy reduction coefficient $\lambda$, such as 0.005. Actually, a typical $\lambda = 0.005$ is used in Fig 2 (Colored-MNIST, ResNet-18). Herefore, while the precise learning dynamics might differ from our simplified theoretical model, our empirical evidence suggests the central conclusion holds: features corresponding to higher eigenvalues are learned preferentially, regardless of the specific dynamics.

[W2] It is restrictive to study the feature learning properties in realistic datasets. As a result, I am unsure how useful the results of this study will be to advance practical SSL pipelines. For instance, can the authors provide recommendations on how to improve the feature learning properties by changing the data augmentation strategy or with some dataset preprocessing?

[A2] For practical SSL pipeline, we will expand the discussion section to suggest a potential application of our research. For instance, based on our findings on extent bias, applying preprocessing techniques such as cropping the outer background of an object or masking background—which effectively increases the object's relative extent bias—could encourage the model to easily learn more object-centric features.

[W3] The non-linear network eigenvalue dynamics seem to discrepancy. Recommand moving these results to the main text instead of the appendix. The effect of non-linearity on feature learning in intermediate layers of the network, and the distinctions in learning dynamics of features vs embeddings in BarlowTwins.

[A3] The fundamental learning priority is maintained, where features corresponding to larger eigenvalues are learned first.

[Effect of non-linearity on feature learning in intermediate layers of the network] Eigenvalue overshoot phenomenon is observed in the non-linear MLP models. Non-linear activations cause this discrepancy because non-linear activation in the intermediate layers breaks the symmetry between layers [1].

[Distinctions in learning dynamics of features vs embeddings in BarlowTwins] To understand the differences between embeddings and features, we examine the kernelized perspective. Given input data, $X=[x_1, \dots, x_d]^\top \in\mathbb{R}^{n \times m}$ and kernel matrix $\tilde K = [X;X'][X^\top X'^\top] \in \mathbb{R}^{2n \times 2n}$, the eigenpair $(\lambda, g)$ of $\Gamma$ where $\Gamma g = \gamma g$ has a corresponding kernelized eigenvector $b = \tilde K^{-1/2}[X;X']g$. The kernel matrix $K_\Gamma \equiv \tilde K^{-1/2} Z \tilde K^{-1/2}$ where $Z=[X;X']\Gamma[X^\top X'^\top]$ preserves the duality relationship $(\gamma, g) \Leftrightarrow (\gamma, b)$ as proven in Sec B.2 in Simon et al. (2023). This ensures learning the kernel eigenvector $b$ with $\gamma$ corresponds to learning the feature eigenvector $g$. However, the specific features that achieve higher eigenvalues depend on both architectural inductive biases and augmentation strategies through their invariance properties. Architecture can be assumed as a kernel $K$ that acts as an implicit feature transformation $\phi(x)$, $K(x, x') = \phi(x)^\top\phi(x')$. Features more aligned with the architectural inductive bias - those that remain relatively invariant under the implicit transformation $\phi$ - exhibit larger eigenvalues and be prioritized during learning.

• Features ($x$): Smooth, monotonic eigenvalue increase following theoretical predictions. Direct eigenvalue-based priority (extent/amplitude bias).
• Embeddings ($\phi(x)$): Eigenvalue overshoot phenomenon due to non-linear activations. Same priority maintained but influenced by architectural/augmentational inductive biases.

The fundamental learning prioritization mechanism remains robust, features with larger eigenvalues are learned first in both cases, but non-linear transformations introduce complex intermediate dynamics while preserving the core learning order.

Following your valuable suggestion, we will move the non-linear network experiments from the appendix to the main text to better highlight the robustness of our findings.

[Q1] While the data model in Sec 4.1 and 5.1 are interesting and makes the theoretical analysis easy, how general do you think they are for realistic datasets? Can you provide an example where this can be applied?

[QA1] We believe our findings are quite generalizable to realistic datasets. For instance, in Table 13 of paper [2], SimCLR's test accuracy drops to as low as 19.2%. This demonstrates a real-world scenario where a model prioritizes background cues, not objects, which aligns with our analysis.

[Q2] The notations of $f_h$ and $f_l$ are confusing. Can you change it to $f_{ha}$ and $f_{la}$ to indicate high and low amplitude?

[QA2] I acknowledge that $f_h$ and $f_l$ notation could be misleading. Following your suggestion, this will be corrected to $f_{ha}$ and $f_{la}$ ​ to clearly indicate high and low amplitude.

[Q3] In the derivation in Appendix B.2, is there a missing f in the denominator (after integrating $cos(2fx)$)? I don't think it changes the conclusion, but curious if it's a typo or something that I missed.

[QA3] Yes, there is a missing factor of $f$ in the denominator. The integration should yield $\sin(2fx)/2f$​, not $\sin(2fx)/2$​. Though as you noted, it doesn't affect the final conclusion since the $\sin⁡(2fm)$ term still vanishes in the limit or due to the periodicity argument. This will be corrected.

[Q4] The discussions section talks about the absence of frequency shortcut in BarlowTwins models, as opposed to supervised learning models. Is it possible to talk about this as a pro over supervised learning (or maybe cross-entropy type losses)? Could this be related to work from Balestriero & Lecun?

[QA4] Neural network learning dynamics fundamentally differ across different tasks, reflecting the distinct objectives and constraints inherent to each learning paradigm:

• [Supervised learning (Regression tasks)]: models are known to learn low-frequency components first (frequency bias or spectral bias).
• [Supervised learning (Classification task)]: models often develop a "frequency shortcut," as demonstrated in the paper "What do neural networks learn in image classification?" [3]. This means they prioritize learning distinctive frequency patterns that make it easy to distinguish between classes.
• [Self-supervised learning]: Instead of learning based on frequency, SSL models prioritize features with high amplitude. The key distinction is that classification learns based on which frequencies are most distinguishable for classification, while SSL learns based on which features have the highest amplitude.

Pros over supervised learning (Classification specific) : The model can be less biased towards specific frequency patterns. For example, consider a person wearing a shirt with a zebra pattern. A classification model might use the high-frequency stripes as a shortcut and misclassify the image as a "zebra." However, an SSL model learns based on amplitude. If the the person's outline had a higher amplitude in the training data than the feature for the zebra pattern, the model would prioritize the person's shape, potentially reducing the chance of such a misclassification.

[1] Nam et al., Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking), (ICML 2025)

[2] Qiang et al., On the Out-of-Distribution Generalization of Self-Supervised Learning, (ICML 2025)

[3] Wang et al., What do neural networks learn in image classification?, (ICCV 2023)

[4] Sagawa et al., Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization, (ICLR 2020)

Thank you for reviewing this paper; we really appreciate your comments and questions.

[W1] The authors fall short of providing a “kernelized” version.

[W2] The linear model cannot account for architecture bias, which heavily influences which eigendirections are learned first and the speed of convergence.

[Q1] If kernelized, it is possible to study the bias arising from model architectures (which essentially encodes which eigendirections are preferred). Could the authors relate their findings to Cabannes et al. (2023), Proposition 4 (and related discussions)?

[A1, A2, QA1] While architecture bias can influence the kernel structure and eigenvalue magnitudes, our fundamental learning prioritization mechanism remains across different architectures. The kernelized framework preserves stepwise learning based on eigenvalue ordering regardless of architectural choices. In the kernelized version, given $X=[x_1,\dots, x_d]^\top \in \mathbb{R}^{n\times m}$ and kernel matrix $\tilde K = [X;X'][X^\top X'^\top] \in \mathbb{R}^{2n \times 2n}$, the eigenpair $(\lambda, g)$ of $\Gamma$ where $\Gamma_g= \gamma g$ has a corresponding kernelized eigenvector $b = \tilde K^{-1/2}[X; X']g$. The kernel matrix $K_\Gamma\equiv\tilde K^{-1/2}Z\tilde K^{-1/2}$ where $Z=[X;X']\Gamma[X^\top X'^\top]$ preserves the duality relationship $(\gamma, g)\Leftrightarrow (\gamma,b)$ as proven in Sec B.2 in [1].

This ensures learning the kernel eigenvector $b$ with $\gamma$ corresponds to learning the feature eigenvector $g$. However, the specific features that achieve higher eigenvalues depend on both architectural inductive biases and augmentation strategies through their invariance properties.

Architecture can be assumed as a kernel $K$ that acts as an implicit feature transformation $\phi(x)$, $K(x, x') = \phi(x)^\top\phi(x')$. Features more aligned with the architectural inductive bias - those that remain relatively invariant under the implicit transformation $\phi$ - exhibit larger eigenvalues and be prioritized during learning. Similarly, augmentation strategy $T$ modifies the eigenvalue structure through $\mathbb{E}[T(x,\theta)T(x,\theta')^\top]$, where features maintaining high correlation across different augmentation parameters $\theta, \theta'$ achieve higher eigenvalues.

Therefore, while different architectures may change which features achieve higher eigenvalues through their inductive biases, and different augmentations may prioritize features with specific invariance properties, the core principle that features corresponding to larger eigenvalues are learned first remains.

[W3] The problem of subspace alignment is heavily influenced by the loss function, as well as optimizer hyperparameters. Specifically, I would have appreciated a discussion of the role of the redundancy term in the Barlow Twins loss.

[Q2] Again in the kernel setting, the interplay between data augmentation and the redundancy term (controlled by lambda) of the Barlow Twins loss has been discussed in the literature (Ghosh et al. 2024). Could Theorem 4.5 be adjusted to account for Barlow Twins under weaker orthogonality constraints?

[A3, QA2] We acknowledge the important role of the redundancy term ($\lambda$) in the Barlow Twins loss for subspace alignment dynamics.

Role of redundancy term in $\lambda$: While our theoretical framework assumes a $\lambda=1$ for tractability, our empirical validation demonstrates that the fundamental learning priorities persist even with practical $\lambda$ values (e.g., $\lambda = 0.005$ used in Fig 2 with C-MNIST with ResNet-18). Regarding under weaker orthogonality, stepwise learning behavior remains robust across different $\lambda$ values. While precise dynamics may vary with smaller $\lambda$ (as discussed in [2]), the core eigenvalue-based learning prioritization mechanism is preserved. Practical implications: Lower $\lambda$ values create slower, complex subspace alignment dynamics, but as demonstrated in our Colored-MNIST experiments, the extent bias phenomenon persists, confirming the generalizability of our findings beyond the high-$\lambda$ theoretical regime. We will expand the discussion in the revised manuscript.

[W5] Some results (Theorem 4.2 and 4.3) are direct consequences of Simon et al. (2023), and thus constitute a validation of the prior work’s theory rather than a novel contribution. Such validation could have been moved to the appendix, in favour of additional discussion (e.g. on kernels).

[A5] While Theorems 4.2 and 4.3 build on Simon et al.'s framework, they represent substantial extensions beyond validation. Simon et al. focused on eigenvalue dynamics, while we extend this to shortcut learning and spurious feature analysis in SSL, providing theoretical explanation for why models learn shortcuts.

We apply stepwise theory to specific shortcut scenarios (extent bias, amplitude bias) that plague real-world SSL systems, moving beyond mathematical validation to enable understanding and mitigation of shortcut behavior.

[W6] The validity of the feature orthogonality assumption in practical settings should be discussed and should ideally be supported empirically.

[A6] The core learning dynamics remain robust despite imperfect feature orthogonality in practical settings. C-MNIST experiments (Section 4.8) with ResNet-18, Waterbird with ViT-B/16 show that extent bias persists even when the validity of the orthogonality assumption is compromised due to:

• Overlapping pixel regions where background and digit features share spatial locations.
• Edge blending at digit boundaries where background and object pixels are mixed.
• Spurious correlations between background color and digit labels (70% correlation in our setup).

Despite these realistic violations of feature independence, extent bias remain observable. Feature orthogonality assumption is a theoretical simplification, but our results suggest core mechanism is robust to such practical deviations.

[W4] The role of the noise covariance (data augmentation) is not discussed (see Questions).

[Q3] Data augmentation influences the structure of the data covariance matrix. Several covariance operators have been theoretically discussed in the literature (Lin et al., 2024) beyond the isotropic setting, in a similar toy linear regression model (albeit for supervised learning). How does the noise covariance affect feature learning? Particularly, shouldn’t also depend on the noise covariance in Equation 5?

[A4, QA3] Data augmentation noise covariance does not affect feature learning. Unlike supervised learning using self-correlation $\Gamma_g=\mathbb{E}[g(x)g(x)^\top]$ [1], SSL requires analyzing $\Gamma_g=\mathbb{E}[g_1(x)g_2(x)^\top]$ where $g_1$ and $g_2$ are applied independently. Using $\Gamma=\mathbb{E}[xx^\top]$, $x$ is $x_\text{base}$,

Key Results:

• Additive noise injection (Not necessarily isotropic covariance): $\Gamma_g = \Gamma$
• Multiplicative noise: $\Gamma_g=M \odot \Gamma$, where $M = \mu \mu^\top$ and $\mu = \mathbb{E}[m^{(i)}]$.
• Unbiased random masking: $\Gamma_g = (1-\beta)^2 \Gamma$, where $\beta$ is Bernoulli parameter.
• Salt-and-pepper: $\Gamma_g=(1-\beta)^2 \Gamma$, where $\beta$ is Bernoulli parameter.
• Random cutout: $\Gamma_g=(1-k/p)^2 \Gamma$, where $g_i(x) = \frac{p}{p-k} M_i x$.

1. Independent additive noise $g_i(x)=x+\epsilon_i$ where $\epsilon_i \sim_\text{i.i.d.} P_\epsilon$ and $\mathbb{E}[\epsilon] = 0$.

• Proof: $\Gamma_g = \mathbb{E}[(x + \epsilon_1)(x + \epsilon_2)^\top] = \mathbb{E}[xx^\top] + \mathbb{E}[x\epsilon_2^\top] + \mathbb{E}[\epsilon_1 x^\top] + \mathbb{E}[\epsilon_1 \epsilon_2^\top] = \Gamma.$

2. Multiplicative noise $g_i(x) = m^{(i)} \odot x$ where $[m \odot x]_j = m_j x_j$.

• Proof: $[\Gamma_g]_{ij} = \mathbb{E}[(m^{(1)} \odot x)_i(m^{(2)} \odot x)_j] = \mathbb{E}[m^{(1)}_i]\mathbb{E}[m^{(2)}_j] \mathbb{E}[x_i x_j]$ Therfore, we have $\Gamma_g = M \odot \Gamma.$

[Q4] In the high-dimensional setting, whereby feature covariances exhibit heavy-tailed behaviour (Agrawal et al., 2022), how likely is the assumption that large superpixels correspond to large-eigenvalues to hold in practice, once features are averaged over a large dataset?

[QA4] While feature covariances may follow power law distributions in high-dimensional settings, our eigenvalue-based learning prioritization remains robust. Features with larger eigenvalues are learned first regardless of eigenvalue distribution. Our analysis focused on discrete eigenvalues, producing distinct stepwise learning. With power law distributions, we expect smoother, more continuous learning dynamics with less pronounced steps and reduced feature disentanglement. However, the core ordering principle, higher eigenvalue features are prioritized, remains unchanged.

[Q5] In section 4.8, what is the standard digit-to-pixel ratio of Colored-MNIST?

[QA5] The digit-to-pixel ratio varies per sample. The train dataset ranges from 0.0344 to 0.4377, while the test dataset ranges from 0.0434 to 0.3610.

[Q6] Can Theorem 4.3 predict how long the test accuracy plateaus for in Figure 2?

[QA6] Theorem 4.3 can predict the learning order, but cannot predict the plateau length in non-linear networks. This is due to dependencies on random initialization (affecting $s_j(0)$ values), non-linearity, and hyperparameters like $\lambda$ in Barlow Twins loss and weight decay, creating discrepancies between theory and experiments.

[Q7] The work of Doersch, et al. appears both as citation 10 and 11.

[QA7] Thank you for catching those duplicate citations. We will be sure to fix those in our revision.

[1] Lin et al., The good, the bad and the ugly sides of data augmentation: An implicit spectral regularization perspective, (JMLR 2024) [2]Ghosh et al., Harnessing small projectors and multiple views for efficient vision pretraining. (NeurIPS 2024)

Thank you for your valuable feedback. We appreciate the time and effort you have dedicated to reviewing our paper.

[W1] The scope of the theoretical results is somewhat narrow, being confined to linear encoders, the Barlow Twins method, and highly simplified synthetic datasets, which may limit their generalizability.

[A1] The uncovered principles are broadly generalizable. Our choice of Barlow Twins was motivated by its representative loss structure; as noted by [2], other prominent SSL methods (e.g., InfoNCE, SimCLR and VICReg) also share the same quadratic form $\text{Tr}[WAW^\top]+\text{Tr}[(WBW^\top)^2]$, making it a representative model for analysis. Our empirical results, including new experiments, validate that our findings are not artifacts of this simplified setup but are indeed a more fundamental phenomenon.

• [Beyond Barlow Twins]: The phenomenon is not limited to Barlow Twins. In additional experiments, we observed the identical stepwise learning and the extent bias in a 1-layer SimCLR and VICReg model. Furthermore, our experiments on the Colored-MNIST dataset using SimCLR and VICReg reproduced the same results, confirming the behavior holds across different SSL frameworks.
• [Beyond Single-Layer and Linear Encoder]: The phenomenon is not limited to shallow or linear models. We have verified that the same learning priorities manifest in deep linear networks (with depth = 4). More importantly, as shown in Sec. 4.7, 4.8 and 5.7 of our manuscript, this behavior persists in non-linear networks (e.g., ResNet-18, ViT-B/16). The fundamental learning priority—which features are learned first—remains consistent. This trend becomes even more apparent when evaluating with linear probing accuracy rather than loss.
• [Beyond Synthetic Datasets]: The Colored-MNIST dataset is not "highly simplified" but rather a widely used benchmark for evaluating model robustness against spurious correlations. We also conduct experiments on Waterbirds dataset, made of Caltech-UCSD Birds-200-2011 and Places dataset. Our findings on these datasets further strengthen the claims.

The consistent results across different SSL methods (Barlow Twins, SimCLR, VICReg), network architectures (linear, deep linear, non-linear), and datasets (synthetic, Colored-MNIST, Waterbirds) provide strong evidence that extent and amplitude bias are fundamental principles of self-supervised learning. We will incorporate these additional results and clarifications into the revised manuscript.

[W2] The analysis is based on a carefully selected initialization without quantifying the impact of deviations or error terms around this initialization, potentially restricting the robustness of the findings.

[A2] Our initialization (e.g., aligned initialization) is generic and does not need to be carefully selected, which shows the robustness of the findings. Following the aligned trajectory only requires small initialization, proven in Sec. C of [1]. Our experiments (Sec. 4.5, Fig.10) demonstrate that models initialized with small random weights rapidly achieve alignment in early training stages. The subspace alignment metric quickly approaches 1, confirming that the aligned structure emerges naturally from small random initialization.

[W3] Sec. 4.5 seems tangential (at least in the context of Sec. 4), and Sec. 4.8 employs a supervised learning setup rather than applying Barlow Twins.

[Q5] Sec. 4.8 employs a supervised learning setup rather than the Barlow Twins method. Could the authors explain the relevance of this experiment to self-supervised feature learning and its connection to the main SSL framework?

[A3]

• Sec 4.5: Sec. 4.5 serves a crucial role in validating Assumption 3.1 (aligned initialization) (see [A2]). Since our theoretical results rely on this assumption, it is essential to demonstrate that it naturally emerges in practice from small initialization. However, we acknowledge that the connection to the main narrative could be more clearly articulated. We will clarify this connection and move details to the appendix.

[A3, QA5]

• Sec. 4.8: There is a huge misunderstanding. We train Barlow Twins, and use "the supervised learning setup (linear probing)" only to measure the representational quality of the SSL-learned embeddings. We clarify that Sec. 4.8 uses frozen encoder + linear probing, which is the standard evaluation methodology for SSL representations (widely used in SimCLR, VICReg, DINO, etc.). The "supervised" component is only the linear probe evaluation, not the SSL training itself. Specifically, at each epoch during SSL training, we freeze the encoder and train a linear classification head from scratch for sufficient epochs to purely measure the representational quality of the learned embeddings. This protocol allows us to track how well different features (background vs. object) are learned over the course of SSL training. We will explicitly state "frozen encoder" and "linear probing evaluation" in the text.

[W4] Definition 4.4 raises concerns about potential basis dependence in the subspace alignment metric. The authors should clarify this point or provide a formal proof of its invariance under different basis choices.

[A4] We prove that the subspace alignment metric $\text{SA}(A, B)$ is invariant under different basis choices. For orthogonal change-of-basis matrices $P, Q \in O(d)$ with $A'=AP$ and $B'=BQ$: $\text{SA}(A', B')=\frac{||A'^\top B'||^2_F}{d}=\frac{||P^\top A^\top B Q||^2_F}{d}=\frac{\text{Tr}((P^\top A^\top BQ)^\top P^\top A^\top B Q)}{d}$ $=\frac{\text{Tr}(\underbracket{Q^\top} \underbracket{B^\top APP^\top A^\top B Q})}{d}=\frac{\text{Tr}(\underbracket{B^\top APP^\top A^\top BQ}\underbracket{Q^\top})}{d}=\frac{\text{Tr}(B^\top A\cancel{PP^\top}A^\top B \cancel{QQ^\top})}{d}$ $= \frac{\text{Tr}(B^\top A A^\top B)}{d} =\frac{||A^\top B||^2_F}{d}=\text{SA}(A, B).$ Therefore: $\text{SA}(A', B')=\text{SA}(A, B)$.

[Q1] Could the authors please clarify the precise definition and role of Eq (6), specifically explaining the notation $e_l, e_s$?

[QA1] Feature Alignment (FA) metric in (6) is designed to interpret which features are being learned and how they evolve during training. While singular values ($s_i$) of the weight matrix $W$ indicate the strength of learned components, does not indicate which eigenvalue corresponds to which feature. The FA metric directly measures the model's alignment with specific, predefined feature vectors.

In this context, $e_l$ and $e_s$ are vectors representing distinct types of features:

• $e_l$: A vector for a "background" feature, which spans a large dimension occupied (high extent bias).
• $e_s$: A vector for a "object" feature, which spans a small dimension occupied (low extent bias).

Assuming aligned $W=USV_\Gamma^{(\leq 2)\top}=\sum_is_iu_iv_i^\top$, where $V_\Gamma^{(\leq 2)}=[\frac{e_l}{\sqrt{m_l}}, \frac{e_s}{\sqrt{m_s}}]$, we have $\text{FA}(e_l)=||We_l||_2=||s_1u_1v_1^\top e_l+s_2u_2v_2^\top e_l||_2 \simeq ||s_1u_1v_1^\top e_l||_2=s_1 ||e_l||_2=\sqrt{m_l}s_1$ $\text{FA}(e_s)=||We_s||_2=||s_1u_1v_1^\top e_s+s_2u_2v_2^\top e_s||_2 \simeq ||s_2u_2v_2^\top e_s||_2=s_2 ||e_s||_2=\sqrt{m_s}s_2$

While singular values $s_i$ capture the overall structural properties of the weight matrix $W$, the FA directly measures how strongly the model responds to specific feature directions.

[Q2] The predicted critical-time behavior of the dynamics of s_t doesn't actually hold up if we were to plot the function. Despite this, the empirical curve follows the prediction and Fig.1 seems to exhibit the predicted dynamics. Could the authors comment on this apparent discrepancy?

[QA2] Our theory can predict the discrepancy between the eigenvalue $\lambda_j(\tau)=1/3$ at $\tau$ (where $\frac{d^2\mathcal{L}}{dt^2}(\tau)=0$) and $\lambda_j({\color{red}{\tau_j}})=1/2$ (where $\frac{d^2{\color{red}{\lambda_j}}}{dt^2}({\color{red}{\tau_j}})=0$).