Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?

We appreciate the reviewer’s time and valuable feedback. We’re happy to address any further questions or concerns.

Turning-off at different points

We are happy to provide an ablation on turning off the regularization at different points for the vision transformers: We disable weight decay (WD) at different epochs during fine-tuning of Tiny-ViT on ImageNet. We train for 300 epochs with AdamW (lr=1e-4, WD=0.2) and disable WD at epochs [50, 100, 150, 200]. We report the intersection epoch of nuclear norm/Frobenius norm ratio and validation accuracy with/without WD off. If no intersection occurs, we report final validation accuracy and ratio. Results show better performance for similar ratios, supporting our theory. We will include this additional experiment in the revised manuscript.

Theoretical bounds on $T$ and $\alpha$

For underdetermined linear regression (Thm 3.6), we search for the schedule with the fastest convergence given a fixed desirable $a_T = \int_0^T \alpha_s ds < \infty$. This is equivalent to regularizing as much as possible and turning off regularization quickly. Yet, it is known that the saddle point near the origin slows down the training dynamics. Also, this is impacted by a finite learning rate inducing a discretization error. Table for Reviewer Eyxr identifies a solution to the expected trade-off. Yet, the optimal $T$ depends on intricacies of the loss landscape and the training dynamics. For example, for the goal of sparsifying a neural network, gradually moving the implicit bias from $L_2$ to $L_1$ works well [1]. Intuitively, we expect to train with regularization until we hit a loss basin to switch it off soon thereafter. A theoretical bound presents a great challenge that is beyond the scope of this paper with general framework character.

[1] Jacobs, T. and Burkholz, R. Mask in the mirror: Implicit sparsification. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=U47ymTS3ut.

Details of main proofs

$R(x,y)$ is unrelated to the loss function $L$. At the end of Theorem A.7, it is defined by the reparameterization, i.e. $(g,h)$, and the initialization. The second derivative is positive. This follows from the definition of a Legendre function (Definition A.1), which is strictly convex. We will refer to this explicitly. We can express the Hessian of $R$ in terms of block matrices that are derivatives with respect to $x$ and $y$ and then apply the block matrix inversion lemma (see Proposition 2.8.7 in [1]). The first diagonal block matches exactly the inverse of the constructed Hessian for the time-dependent Legendre function. As the inverse of a positive definite matrix is positive definite, it follows from the definition of positive definiteness that also this diagonal block matrix is positive definite. To see this more precisely, denote the inverse of the Hessian matrix with $A$. $A$ is positive definite if and only if for all $z = (x,y) \in \mathbb{R}^{n+1}$, $z^T A z > 0$. Now restricting this to particular $z = (x,0)$ with $x \in \mathbb{R}^n$ yields that the diagonal block matrix that we are interested in is also positive definite. We suppressed the dependency of $\nabla R_{a_t}$ on $x_t$. We agree that it would be more clear to use $\nabla R_{a_t}(x_t)$ instead. In addition, we will separately explain the application of the time-dependent mirror flow relation and the contraction property to highlight how the contraction property is used. We will include these extra explanations in the revised manuscript. Moreover, we will provide the block matrix inversion calculation according to Proposition 2.8.7.

[1] Bernstein, D. S. (2005). Matrix Mathematics: Theory, Facts, and Formulas with Application to Linear Systems Theory. http://engineering.nyu.edu/mechatronics/Control_Lab/ToPrint.pdf

Experiment interpretation

We agree that the improvement for the LORA experiment is lower, although significant. However, for the vision transformers, the increase in validation accuracy is substantial for the same ratio. The additional experiments above support this claim. Furthermore, we would like to emphasize that our experiment is not designed to maximize performance but to confirm the prediction of the theory. The desirability of a particular ratio is problem-dependent. For example, a lower rank (smaller ratio) might be desirable in cases where efficiency or interpretability is a priority. Moreover, the framework could be used to inspire and help design new regularization schedules that are tuned for better performance.

We would like to express our gratitude for the reviewer’s time and effort in providing valuable comments on our manuscript and appreciate the acknowledgement of the work's relevance and novelty. We would be happy to discuss any open questions or concerns, if there remain any.

Background knowledge

We appreciate the reviewer’s observation regarding the significant background knowledge needed to fully understand the details. Given the initial page limit, we focused on presenting the main storyline.

In the current manuscript, we elaborate on the implications with concrete examples in Section 4. Moreover, the main theorems are accompanied by detailed descriptions of the key proof steps, along with characterizations of the regularization and a geometric interpretation. These findings and concepts have been formally defined, highlighting important details of the analysis. As we would be allowed to use an additional page for the main manuscript upon acceptance, we would be happy to make adjustments. To this end, we will include more definitions from the appendix into the main manuscript like:

1. The definitions of a regular and commuting parameterization in Section 3 (Introduction part).
2. The definition of the Legendre function in Section 3 (Introduction part).
3. The definition of the Bregman function in Section 3.2.

With these additions, we believe the main storyline would be accessible to a broader audience, with the key idea being that explicit regularization changes the implicit bias.

The 3 effects

We believe that the introduction provides a precise definition of the three effects without requiring the explicit introduction of the parameterized Legendre function. In Section 4, we present concrete theoretical examples of all three effects. However, we are happy to further clarify these effects by including the definition of the parameterized Legendre function at the beginning of Section 4:

1. Type of bias: The shape of $R_a$ changes with $a$.
2. Positional bias: The global minima of $R_a$ changes with $a$.
3. Range shrinking: The range of $\nabla R_a$ can shrink due to a specific choice of $a$.

Experimental implications

The experiments are there to verify the theory and to show that the principles hold in more practical settings. A practical implication is the potential utility of dynamic regularization schedules. For instance, in the case of quadratic reparameterizations, to get the best generalization accuracy for a desired sparsity ratio, it is beneficial to regularize more during the first half of training and then turn off weight decay in the second half. Moreover, our work provides a stepping stone for further research to derive better regularization schedules that take the implicit bias into account, as mentioned in the discussion.

Miscellaneous

We agree that it is more clear to use $R_{a_t}$ than $R_a$. The intent was to make a more direct reference to the previous definition of the parameterized Legendre function.
In addition, $x_*$ is a solution of the objective $f$ as the regularization is turned off eventually. In other words, after turning off the regularization the iterates move to a solution of the objective $f$. We revise our manuscript accordingly.

We would like to express our gratitude for the reviewer’s time and effort in providing valuable comments on our manuscript and appreciate the acknowledgement of the theoretical contributions. Below, we address the raised questions and concerns, but would be happy to extend the discussion on request.

How the 3 effects are distinct

We provide some examples that highlight how the 3 effects are distinct from each other and also can occur independently from each other:

1. The range shrinking means that the actual range of $\nabla R_a$ changes during training; this does not occur for quadratic reparameterizations, for example.
2. The positional bias is distinct. When we initialize a quadratic parameterization at zero, the positional bias stays zero. So it does not change, while the type of bias changes still from L2 to L1.
3. The type of bias can be different in all these scenarios and even changes dynamically in most of our examples.

Commuting condition and eigenbasis

We will restate the commuting condition such that it is clear that indeed $(g,h)$ is commuting in Theorem 3.2. Moreover, we will mention the connection between an appropriate eigenbasis choice and the commuting condition explicitly after the quadratic parameterization result.

Figure 2, L1 setting

The training dynamics for the linear parameterization are different from that of the quadratic reparameterization. Therefore, it is hard or impossible to match the same performance in the experiment. The main takeaway is that the type of implicit bias of the linear parameterization has not changed. In general, what would occur when the L1 regularization is turned off is that the training dynamics go to the L2 interpolation solution. To alleviate any concern, we have run an additional experiment for the L1 setting with higher regularization ($0.2$) and turned it off at the same time, this also leads to similar performance, i.e. it goes to the L2 interpolator with a reconstruction error of $0.32$. Therefore, the amount of regularization does not change the type of implicit bias. We will include this additional experiment in Appendix C of the revised manuscript.

Figure 3, ratio

We do not make claims about the desirability of these results; this would depend on the specific application. A lower rank (smaller ratio) might be desirable in cases where efficiency or interpretability is a priority, for example. For the practitioner it is relevant to understand how regularization impacts the rank and thus the solution. This insight is the purpose of our theory. The main message of this Figure 3 is that we can obtain similar ratios by turning off weight decay. Appendix F furthermore demonstrates that this can achieve better generalization. Specifically, at similar ratios, the validation accuracy of a ViT on ImageNet can be improved by more than 1%. This illustrates the utility of controllable implicit bias in practice, but the main purpose of our experiments is to verify our theory. We still believe our theory could inspire better algorithm design in future, as mentioned in the discussion.

Different schedules for matrix sensing

Consider the family of schedules with constant regularization strength $\alpha_i$ up to specific time T_i such that $\int_0^{T_i} \alpha_i ds = T_i \alpha_i = C$ for $C >0$ a constant. We choose $\alpha_i = [0.02, 0.2, 2, 20]$. In addition, we consider a linear and cosine decay schedule for the regularization with the same total strength (i.e. same integral), but the regularization is switched off after half of the training time to ensure convergence. To compare with the effect of turning off (t-o) the regularization, we include the constant schedule with regularization strength $\alpha=0.01$. Note that this schedule is also reported in Appendix C. We observe that all schedules with decay or turn-off (t-o) converge to a solution with the same nuclear norm of the ground truth, confirming Theorem 3.6, while the constant schedule does not reach the ground truth. We would be happy to include this additional experiment in Appendix C.