Understanding Deep Neural Networks as Dynamical Systems: Insights into Training and Fine-tuning

1. The function $d$ which signifies the distance between neurons, appears to be used to denote the space’s dimension. While the majority of the paper consistently employs $D$ for this purpose, there are instances where the author inadvertently mislabels it as $d$.
2. In the legend of neural network visualization (Fig.2(b)), the “sub-model” should be changed to “dynamical neurons”
3. This paper focuses exclusively on the network's trainable weights and does not address the pre-defined operations within the network.

I was not able to understand this paper at all. The authors appear to be using mathematical terms with very different meanings than they usual have. For example, the definition of a dynamical system on page 3 is bizarre. For example, a (continuous time) dynamical system is normally simply any system where the time derivative is a deterministic function of its current state. Not all dynamical systems have a Lagrangian, and certainly they don’t all have the specific form (1). Another example: on page 12, d(q_1,q_2) is described as a “distance function”. This would normally mean that d(q,q)=0 for all q. But equation (17) then implies that \phi(q)=0 for all q. So it is really not clear what they authors mean by distance.

Many other aspects were unclear. For example, does the dynamical system refer to the change of weights over learning, or the change in neurons’ activity over time? If the former, then a “dynamical neuron” is actually a weight, rather than a neuron.

Equation (7) seems to be stating that one can interpret gradient descent as the movement of a dynamical system. Which is trivially true - what do you gain by saying it this way other than more terminology?

First of all, the notation of dynamical system in this paper differs greatly from the common understanding of a dynamical system, which is a well-defined concept in the mathematical and machine learning communities. Therefore, I suggest that the author use an alternative term to avoid confusion.

Secondly, the reviewer also finds that the paper is not well-written and difficult to read, as there are several places that require further explanations. I would like to provide the following examples:

1. In equation (1) on page 3, it appears that the definition of $E_{s_1}$ requires $V_{11}$, and similarly, the definition of $V_{11}$ also requires $E_{s_1}$, this makes both these two notations not well-defined.

2. In Lemma 1, since $m_l$ and $n_l$ are large integers, the fraction in the bound of $D$ provided in Lemma is a positive number smaller than 1. As a result, $D$ will always have a value of 1. This implies that every network can be embedded into $R^1$, which seems impossible according to Lemma 1. Moreover, although Lemma 1 claims to provide an upper bound for $d$, it does not appear in equation (5).

3. In the definition of dynamical discrepancy (equation 11), it seems that $l$ is used as a subscript to denote summation. Therefore, in equation (11), having $N^{(l)}$ in the denominator without summation is confusing. As dynamical discrepancy is a fundamental concept of this paper, this mathematical error cannot be ignored.

The paper is extremely poorly written. For one, the basic claim that a "dynamical system embedded in a high-dimensional Euclidean space can effectively represent [a deep neural network]" as stated is trivial. As an example, the standard approach would be to create Wilson-Cowan equations the steady-state of which give the activation of a given unit, to wit:

a_i = \Phi(W*a) =>  da/dt = -a + \Phi(W*a)

I assume then that there is a more precise claim the authors intend, but it is not clear. What the authors do specifically claim is that there is a map based on set of Lagrangian dynamics that gives rise to the network activity, which is a mysterious claim. Equation 4, for example, represents a conservative physical system (when training W is frozen) and will not necessarily settle into states that provide the usual network output (as given above for example). From this the construction seems to fail at the outset. Perhaps I have misunderstood the nature of the construction, but these equations will not mimic the action of any deep network in common use. They authors even claim that equation 7, for example, "is equivalent to gradient descent", which would be true were equation 7 not second order.

The algorithms and proofs are unclear (and notation seems to change from time to time without explanation). The authors continually reference "path integrals" without defining what is being discussed and how they are formulated (e.g. an input to Algorithm 1 is "Path integrals d": what does this mean?).

The verification of their approach is driven by what appears to interpreting T-SNE plots and a section with next to no details of how numerical experiments were carried out.

-the paper discusses at length how to model DNNs as dynamical systems, but offers little to no insight on what we actually gain by their framework. The main conclusion is that a newly introduced notion, that of "dynamical discrepancy" correlates with some properties for representation of the network, but the reviewer believes that to be not very informative.

-The authors justify the usefulness of "dynamical discrepancy" as a way to measure the networks' representation capacity and as they state "this solely relies on the model's dynamical representation without the requirement of auxiliary datasets". However, there are several ways to determine the networks representation capabilities that do not depend on auxiliary datasets. I don't understand why this is particularly interesting or particularly a unique advantage of "dynamical discrepancy"

-The main theoretical contribution states as Th. 3.1 and Th. 3.2 seem very weak conclusions.

-Overall, reading the paper I agree with the modeling aspect that it proposes, but I can't see what we gain by this interepretation.