Implicit Bias in Matrix Factorization and its Explicit Realization in a New Architecture

Thank you very much for your thoughtful and constructive evaluation of our work. We appreciate the time and care you dedicated to providing detailed feedback.

We understand the reviewer’s critique regarding the experiment setup, but we would like to clarify that the primary goal of our neural network experiments is to demonstrate that the hypothesized behavior emerges consistently across a wide range of settings. Specifically, we designed these experiments to test the robustness of our observations across different problem setups, which include different optimization algorithms, different parameters (such as learning rate), different network architectures and tasks (regression, classification, transfer learning, fine tuning), different depth and activation functions, etc. The consistent emergence of structure across these diverse setups supports our claim that the proposed $UDV^\top$ reformulation (with constraints) of dense layers promotes a meaningful structural bias. We will revise the manuscript to clarify this intent and better distinguish the goal of these experiments from standard performance benchmarking.

While we agree that exploring the scalability of the proposed factorization in large-scale benchmarks is a natural and important direction, we note that a current limitation is related to the slow early-stage convergence behavior of the $UDV^\top$ model. This pattern is also evident in our matrix factorization experiments, for instance, although $UDU^\top$ converges faster to a high-accuracy solution with $10^{-9}$ residual than the classical BM factorization, it takes longer to reach a moderate-accuracy threshold like $10^{-3}$. This behavior presents a challenge under constrained computational resources; while we have access to capable GPUs, restrictions such as limited session duration and the number of simultaneously available devices limit our ability to conduct extensive large-scale benchmarks. Nevertheless, we can provide some moderately scaled use cases to illustrate practical viability.

That said, this work also serves as the foundation for an ongoing, more comprehensive project on the practical implementation of UDV-based formulations. We are currently exploring ways to improve early-stage convergence without sacrificing the structural bias induced by the factorization. The slow initial progress is linked to transition delays in learning successive columns. Increasing the depth of the diagonal layer (e.g., using a $UDD^\top U^\top$ formulation) appears to shorten these transition phases (tested so far only on matrix factorization). Likewise, incorporating adaptive preconditioning techniques or using signSGD-type methods has shown promise in addressing this concern. In parallel, we are developing a more comprehensive pruning framework based on UDV, for regression, classification, and language modeling tasks. This includes pruning strategies both during and after training. Preliminary results are encouraging: for example, using energy-based thresholding as an adaptive pruning criterion during training, we can prune 96.64% of the parameters in the UDV-factorized linear layers of a ViT model (trained on CIFAR-10) without degrading its generalization performance.

We would also like to emphasize that the emergence of structural bias (such as low-rank tendencies, orthogonality and column sparsity) is not only relevant for practical purposes like pruning, but also valuable to understand in its own right. Even if approximately low-rank structure is sufficient for many applications, uncovering the mechanisms that drive low-rank behavior can provide deeper insight into the inductive biases of neural networks. In this context, the observation of a truly low-rank (rather than an approximate low-rank) representation can make the underlying dynamics more interpretable. In turn, understanding these phenomena may enable more principled architectures and optimization strategies in the future. The goal of this work is therefore to highlight structure-promoting behavior of the $UDV^\top$ reformulation, with the hope that future work can further explain and build upon these dynamics.

Please find below our responses to the specific questions raised:

Q1: This is a great question. We did experiment with very small initializations. Recall that we initialize $D$ as the identity matrix and control the scale via $\xi = ||U||_F$. For instance, with $\xi = 10^{-4}$, the initial matrix $X = UDU^\top$ becomes norm bounded by $||X||_F \leq ||U||_F^2 = 10^{-8}$. In fact, $\xi$ influences early convergence behavior, however this is not immediately visible in Figure 1 due to the wide y-axis range (from $10^{-13}$ to $10^1$). If you zoom in (unfortunately we are not allowed to upload a figure in this year's rebuttal, but you can zoom in the PDF viewer), the influence becomes clearer. This influence is more noticable in the noisy setting in the appendix, so please refer to Figure SM1, where with $\xi = 10^{0}$ the first column is learned as early as iteration 100–200, whereas it takes around 1000–2000 iterations with $\xi = 10^{-4}$. However, interestingly, we observe that larger $\xi$ values tend to introduce longer transition delays before the learning of subsequent components kicks in. This ultimately dissipates the initial advantage of having a large $\xi$, and the asymptotic convergence rates appear quite similar across different $\xi$.

Q2: We understand the reviewer’s concern. Our framework is inspired by this hypothesis, as we introduce constraints to prevent divergence of $U$, and the theoretical insights section suggests that the structure in $U$ emerges as it attempts to grow in norm. That said, we will soften the statement and clarify that this is a motivating hypothesis rather than a claim we rigorously prove or analyze.

Q3: This point is closely related to the previous question. Our intention was to contrast our perspective with the common approach in the literature. We will revise the wording to clarify that this is an informal insight that guided the development of our formulation, rather than a formal claim or strict dichotomy.

Q4: Thank you for your careful reading. We will correct the typo.

Q5: The symbol "$\approx$" is commonly used and widely accepted in the context of supervised learning to denote the goal of fitting a function $\phi(\mathbf{x})$ to the observed outputs $\mathbf{y}$, prior to introducing a loss function that formalizes this approximation. At this point, we have not yet defined the training objective, hence we use "$\approx$" to suggest that the model aims to approximate the target outputs. The sentence preceding equation (6) clarifies that $(\mathbf{x}_i, \mathbf{y}_i) \in \mathbb{R}^d \times \mathbb{R}^c$ are data-points, and equation (6) further clarifies that $\mathbf{x}$ refers to input features and $\mathbf{y}$ to the corresponding target outputs (i.e., responses or labels).

We appreciate the time and effort you dedicated to reviewing our manuscript and your positive evaluation of our work. Below, we address the questions you raised:

Q1: We thank the reviewer for pointing this out and appreciate the opportunity to clarify. We used only two data splits: one for training and one for evaluation. Throughout the manuscript, we referred to the evaluation set as the validation set, but this corresponds to the test set in the standard train/validation/test terminology. That is, the same evaluation split is referred to in both Table 1 and Section 3.1.2, and the metrics reported under “validation” and “test” are identical. Importantly, this split remained strictly unseen during training and was only used for performance reporting. We will revise the manuscript to use consistent terminology and eliminate confusion.

Regarding hyperparameter selection: we did not tune hyperparameters based on this evaluation split. All methods were run using a fixed grid of standard hyperparameters (e.g., learning rates 1e-6, 1e-5, 1e-4, ...), and we report the best performance for each method from this grid. This is a common and practical approach for comparing methods under equal tuning budgets. Since no iterative tuning or model selection was performed using the evaluation set, its role aligns more closely with that of a test set. We will make this explicit in the revised manuscript.

Additionally, please note that Table 1 shows a summary of our results, and the complete results (with all parameter choices) are reported in Tables SM2–SM5 in the supplementary material.

Q2: For simplicity, throughout this response, "number of neurons" refers to the number of neurons in the diagonal ($D$) layer.

• For the regression tasks (HPART and NYCTTD): The diagonal layer was already small, so we pruned one neuron at a time, gradually reducing the number of neurons while monitoring pruning performance. Results are shown in Figure 4 (main text) and Figure SM13 (appendix).

• For the image classification task (MNIST): The classifier layer was wider. We applied a recursive proportional rule: at each step, we retained the top 90% of neurons (by singular value), evaluated performance, and repeated the procedure until only one neuron remained (see Figure SM13).

On a closing note, we would like to highlight that the pruning experiments included in this paper serve as an initial proof of concept. This work lays the foundation for an ongoing, more comprehensive project focused on UDV-based pruning strategies across regression, classification, and language modeling tasks. In this follow-up in progress, we are exploring adaptive and flexible pruning strategies, supported by careful sensitivity analyses.

We sincerely thank the reviewer for their careful reading of our manuscript and for the constructive feedback. Below are the responses for the questions raised:

Q1/Q2: Thank you for this question and the opportunity to clarify this point. We would like to emphasize that the analysis in Section 2.3 is intended as a theory-driven insight, not a full analytical characterization. (We have also noticed some typos in this section, which will be corrected in the revised manuscript.) There are a few key points worth highlighting:

• The fixed-point analysis reveals that there are no fixed points satisfying $||U||_F > \alpha$. This is a fundamental observation because it implies that the set of fixed points for the $U D U^\top$ formulation is equivalent to that of the classical BM factorization, under a simple change of variables (i.e., $U\_{BM} = UD^{1/2}$). Thus, the introduction of the diagonal matrix $D$ along with the constraints does not restrict the fixed-point set, and any structure that emerges is due to the optimization dynamics (rather than the parameterization alone.) However, we have observed a consistent low-rank bias across multiple optimization algorithms (in our neural network experiments). This suggests that the phenomenon is linked to shared features of the optimization process, rather than a specific solver or parameter choice. Hence, fixed-point analysis can be a useful tool for probing the underlying mechanism driving this behavior.
• It appears that the observed structural bias is closely tied to the regime where the norm of $U$ tends to grow and the Frobenius norm constraint becomes active. In fact, we experimented with artificially large $\alpha$ values combined with small step sizes, which allowed convergence to occur without triggering the norm constraint. In such cases, the resulting solutions exhibit no additional implicit bias beyond what is already present in the classical BM factorization. This supports our view that the enhanced bias observed in the $UDU^\top$ formulation is directly connected to the projection step onto the Frobenius norm ball.
• It is also worth emphasizing that we observe more than just a low-rank bias. The columns of $U$ tend to align along orthogonal directions, and $U$ exhibits column sparsity, with many columns converging to zero. Interestingly, even the columns that eventually vanish tend to remain orthogonal during training. These phenomena appear only when the Frobenius norm constraint on $U$ is triggered.
• A closer examination of the fixed-point conditions, assuming that the Frobenius norm constraint is active, offers partial insight into the structural behaviors we observe. In particular, under this assumption, the analysis suggests that the columns of $U$ should either align with the negative eigenvectors of $\nabla f(X)$ or become zero. When combined with random initialization of $U$, this may help explain the empirical tendency of $U$’s columns to align along orthogonal directions. [Importantly, as discussed above, such fixed points do not actually exist: at any true fixed point, the Frobenius norm constraint is inactive. Nevertheless, one can argue that the intermediate steps where the projection becomes active influence the optimization trajectory. These projection steps may push the iterates toward directions that reflect the (hypothetical) solution of the constrained fixed-point system, even if the constraint is ultimately inactive at convergence.]
• Why do we observe low-rankness? In fact, low-rankness emerges as a byproduct of column sparsity in $U$. But how does this sparsity arise? Looking at the update rules, applied columnwise to $U$ and entrywise to $D$, we have: $\bar{u}_j \gets u_j - 2\eta \lambda_j \nabla f(X) u_j \quad \text{and} \quad \bar{\lambda}_j \gets \lambda_j - \eta u_j^\top \nabla f(X) u_j.$ Interestingly, both $\bar{u}_j$ and $\bar{\lambda}_j$ grow rapidly along directions aligned with the negative eigenvectors of $\nabla f(X)$, particularly the dominant ones. Suppose $\nabla f(X) u_j = -\beta_j u_j$ for some $\beta_j > 0$. Then the updates simplify to: $\bar{u}_j \gets (1+2\eta\beta_j\lambda_j)u_j \quad \text{and} \quad \bar{\lambda}_j \gets \lambda_j + \eta \beta_j ||u_j||^2.$ In this regime, $\bar{\lambda}_j$ remains positive by definition, so projection onto the nonnegative orthant for $D$ is ineffective. More importantly, under this dynamic, $\lambda_j$ will continuously increase, as there is no mechanism to decrease it.
• Now, we look at the fixed-point condition for the $D$ update (see condition (d) in the paper): when $\lambda_j > 0$, it must satisfy $u_j^\top \nabla f(X) u_j = 0$. But under the previous assumption that $u_j$ aligns with a negative eigenvector (i.e., $\nabla f(X) u_j = -\beta_j u_j$), this condition reduces to: $-\beta_j ||u_j||^2 = 0 ~ \Rightarrow ~ ||u_j|| = 0.$ This implies that, for such directions, the fixed point dynamics encourage $u_j$ to vanish numerically (hence sparsity). Considering the update scheme, however, this vanishing effect occurs through projection onto Frobenius norm ball, but this happens only if another column $u_j'$ exhibits a faster growth. So the growth one columns (or very few columns) will dominate.
• But then, why don’t we observe a rank-1 solution? If the fixed-point conditions described earlier were the only driving mechanism, one might expect that only the dominant negative eigenvector survives, leading to a rank-1 solution. However, the situation is more nuanced. While the projection step can drive $||u_j||$ numerically toward zero for non-dominant columns, the same condition must be satisfied by all columns with $\lambda_j > 0$, including the dominant one. Crucially, there is no mechanism to shrink the dominant column. Since projection alone cannot enforce the fixed-point condition for the dominant column $u_j'$, the optimization trajectory should push $u_j'$ and $X$ in a way to evolve their alignment such that $u_j'$ falls onto the nullspace of $\nabla f(X)$, i.e., towards $\nabla f(X) u_j' = 0$.
• Finally, as the previously dominant column, initially aligned with a negative eigenvector, transitions toward the nullspace, its growth slows down and eventually halts. At this stage, the algorithm enters a transition phase, where it appears to have nearly converged. However, despite being numerically small, some of the other columns continue to grow, eventually becoming the next dominant column.
• We emphasize that this analysis is of course an oversimplified explanation. In particular, the updates to $u_j$ and $\lambda_j$ occur simultaneously with changes in $X$. Although the analysis presents these effects as if they occur in a sequential narrative, they are in fact all intertwined. Still, the matrix factorization experiments exhibit these phases, as can be observed in part from Figures SM4 to SM9 in the Appendix.

Q3: We indeed plan to extend our approach to model compression across both vision and language tasks. Specifically, we aim to apply the UDV factorization to all (or subsets of) linear projections in multi-head attention (Q/K/V/O) and MLP blocks of Transformer-based models (e.g., ViT for vision and GPT-style models for language). Our goal is to integrate SVD-based pruning both adaptively during training and as a post-training procedure, depending on deployment or fine-tuning requirements. We even obtained some preliminary results after the submission of this paper. For example, on CIFAR-10, we trained a ViT model from scratch using UDV-factorized layers and were able to prune at least 96.64% of the trainable parameters across all UDV-factorized layers, without any observable loss in generalization performance compared to the original ViT.

That said, one current limitation in large-scale models is the slower early-stage convergence of UDV. This pattern is also observed in our matrix factorization experiments: although UDV reaches high-accuracy solutions (e.g., $10^{-9}$ residual) faster than the classical BM factorization, it takes longer to reach a moderate accuracy threshold (e.g., $10^{-3}$). This becomes a challenge especially when dealing with large-scale problems with constrained computational resources. The slower early-stage convergence emerges from the long transitional phases in between learning different columns. To address this, we have ongoing work on improving the practical implementation of UDV. These efforts are directly inspired by the fixed-point insights discussed in the previous response.

We would like to thank the reviewer for careful reading and for pointing out some typos. We will correct these in the revision.

We would like to clarify that the primary goal of our neural network experiments is to demonstrate that the hypothesized behavior emerges consistently across a wide range of settings. Specifically, we designed these experiments to test the robustness of our observations across different problem setups, which include different optimization algorithms, different parameters (such as learning rate), different network architectures and tasks (regression, classification, transfer learning, fine tuning), different depth and activation functions, etc. The consistent emergence of structure across these diverse setups supports our claim that the proposed $UDV^\top$ reformulation (with constraints) of dense layers promotes a meaningful structural bias.

The pruning experiments included in the paper serve as an initial proof of concept. This work forms the basis for an ongoing, more comprehensive project focused on UDV-based pruning strategies across regression, classification, and language modeling tasks, including pruning both during and after training. Preliminary results are encouraging: for instance, by applying energy-based thresholding during training, we are able to prune 96.64% of the parameters in the UDV-factorized linear layers of a ViT model trained on CIFAR-10 without compromising generalization performance.

We address your questions below:

Q1: The nonnegativity of $D$ arises naturally in our matrix factorization framework, where we focus on positive semidefinite (PSD) factorizations. A matrix of the form $U D U^\top$ is PSD if and only if the diagonal entries of $D$ are nonnegative. In the context of neural network training, this choice is largely a matter of convention rather than necessity. Importantly, the search space remains unchanged regardless of whether we impose nonnegativity on $D$, since there is no sign constraint on the entries of $U$ and $V$. For any matrix $U D V^\top$ with a negative entry in $D$, one can equivalently absorb the sign into the corresponding column of $U$ or row of $V$, resulting in an equivalent representation with a nonnegative $D$. We therefore adopted the nonnegativity convention in this case mainly to resemble a classical singular value decomposition, especially given our empirical observation that the columns of $U$ and $V$ tend to become orthogonal. For completeness, we also conducted experiments without the nonnegativity constraint on $D$ and reported them in the supplementary material (see Appendix B.1 and the corresponding figures and tables). The results were similar, although the UDV variant (main variant we used, with nonnegativity constraint on $D$) slightly outperforms the UDV-s (the variant without nonnegativity constraint on $D$) version in most cases; possibly due to the removal of redundant sign ambiguity.

Q2: Yes, there are several practical advantages to having low-rank neural network models. Besides model compression, efficient pruning, and distillation [R1, R2], low-rank solutions enable model deployment on resource-constrained devices and efficient communication in distributed training scenarios [R3]. Empirical studies have demonstrated that low-rank structures can lead to improved generalization and reduced overfitting [R4, R5]. Additionally, they have been linked to increased robustness [R6, R7], enhanced interpretability of learned representations, and improved reasoning performance in large language models [R8], among other benefits.

We would also like to emphasize that the emergence of structural bias (such as low-rank tendencies, orthogonality and column sparsity) is not only relevant for practical purposes like pruning, but also valuable to understand in its own right. Even if approximately low-rank structure is sufficient for many applications, uncovering the mechanisms that drive low-rank behavior can provide deeper insight into the inductive biases of neural networks. In this context, the observation of a truly low-rank (rather than an approximate low-rank) representation can make the underlying dynamics more interpretable. In turn, understanding these phenomena may enable more principled architectures and optimization strategies in the future. The goal of this work is therefore to highlight structure-promoting behavior of the $UDV^\top$ reformulation, with the hope that future work can further explain and build upon these dynamics.

[R1] Denil et al., Predicting parameters in deep learning. 2014.

[R2] Denton et al., Exploiting linear structure within convolutional networks for efficient evaluation, 2014

[R3] Goyal et al., Compression of deep neural networks by combining pruning and low rank decomposition, 2019.

[R4] Huh et al., The low-rank simplicity bias in deep networks, 2021

[R5] Pinto et al., On generalization bounds for neural networks with low rank layers, 2024

[R6] Savostianova et al., Robust low-rank training via approximate orthonormal constraints, 2023

[R7] Langenberg et al., On the effect of low-rank weights on adversarial robustness of neural networks, 2021

[R8] Sharma et al., The truth is in there: Improving reasoning in language models with layer-selective rank reduction, 2023

Q3: This is an interesting question. However, we are not aware of any direct connection between gradient clipping and low-rank implicit bias, either theoretically or empirically. Gradient clipping modifies the optimization process by constraining update magnitudes, primarily to mitigate issues such as exploding gradients. In contrast, our formulation modifies the architecture itself, hence changes the optimization landscape in a structural way, which is fundamentally different than clipping. It might be interesting to apply gradient clipping within the $UDV$ formulation to investigate whether it has any impact (positive or negative) on the structural bias we observe.