Optimization Insights into Deep Diagonal Linear Networks

Unfortunately, both the technical and theoretical contributions of this paper are rather limited, which I will discuss as follows.

1. The ultimate goal of this paper is to reveal the implicit regularization effect of GF for deep diagonal linear networks. However, the explicit form of the corresponding entropy function for the induced mirror flow dynamics is completely absent. There is even no suggestion about possible properties that the entropy function should have.

   In addition, the derivation of the mirror flow form is a direct application of results in Li et al., 2022, and the first point of Proposition 1 can be a direct application of the Euler’s theorem for homogeneous function. Thus I think the technical contributions of this paper are rather limited.

2. As a comparison, Yun et al., 2021 already explicitly characterized the implicit bias of deep diagonal linear networks by using the tensor network formulation developed in their paper.

   Specifically, they established the optimization problem with an explicit form of the entropy function that the GF dynamics of deep diagonal linear networks (note that they did not require the parameterization $u^{\odot L} - v^{\odot L}$) aims to solve. They also established the convergence of the dynamics. The only possible weakness of their result is the additional requirement of the initialization, which the authors in this paper are able to relax at the cost of the characterization for explicit form of entropy function. But I cannot view such relaxation as a significant theoretical contribution that is sufficient for this paper to be published in its current version.

Reference

Yun et al., 2022. A Unifying View on Implicit Bias in Training Linear Neural Networks.

This paper lacks novelty in the following ways:

1. Theorem 1, showing that GF on $L$-layer diagonal linear networks induces a mirror flow, as authors have acknowledged, is an application of Li et. al., 2022. So the contribution of this theorem is rather weak.

2. Theorem 2 is not new. Min et. al., 2023 (See their Section 4.2) have shown the exponential convergence of GF under the same condition as described in equation $(\mathcal{A})$ with a better lower bound on the rate.

References:

Z Li, et. al., Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descent, NeurIPS, 2022

H Min, et. al., On the Convergence of Gradient Flow on Multi-layer Linear Models, ICML, 2023

• The major weakness is the mirror potential is not explicitly defined - even if it not explicitly defined the limiting behavior in the case of large depth or small initialization are not discussed or analyzed which is a major drawback.
• The implicit bias of optimization benefits/drawbacks of the depth is not discussed and this weakens the motivation for studying the deep diagonal linear networks.