Nature-Inspired Local Propagation

The encouraging comment of continuing this line of work is very much appreciated. We have been currently working on further validation ``on more mature learning problems'', but we believe that before presenting a professional application we need to prove that proposed theory is technically sound. Hence, this paper is primarily intended to provide the foundation of the theory. The experiments are primarily conceived to disclose the main principles more than for showing advances in application domains. We also fully understand the criticism on the presentation style. However, since this is the first paper which presents the theory, we decided not to run the risk of a rejection of the paper due to arguable statements. A comprehensible criticisms could have been that a new theory must not be claimed only by conjectures. In the near future, the Reviewer's comment will be carefully considered especially for reaching a broader scientific community.

Q1. Our method reduces to Backprop and GD, as explained in the manuscript when we choose a directed acyclic graph and take the limit as $c \to \infty$ (classic feedforward nets used in deep learning). BPTT is not local in time and can only work for sequences of limited length, since we need to store neural activations and delta errors for all unfolded nets. The distinctive feaure of the neural propagation scheme that arises from Hamiltonian equations is that such a limitation is overcome!

Q2. This is a crucial question that we have been considering of primary importance. We believe that the primary challenge in appropriately evaluating the proposed method lies in defining an experimental setting that does not distort the online/continual nature while still factoring out the effects introduced in this framework by forgetting phenomena. This because what we propose opens the doors to a way of conceiving experiments which can perfectly be understood when thinking of time-series data (exactly what the Reviewer was considering). Other applications in other domains of Machine Learning (e.g. vision, speech, and language) can be naturally considered.

Q3. From a computational point of view we can find a strict relation between the computations required for BP/GD and our method. In particular with reference to Eq. (14) we have that:

• The forward phase of BP is replaced by the residual-like computation of the state equation (the first equation of the system);

• The backward phase that is responsible of the computation of the delta error in BP is here replaced by the computation of $\dot p_x$ (third equation) that are the equivalent of the updates of the delta errors;

• The computation of the gradients done with the BP factorization here is instead replaced by the computation of $\dot p_{\boldsymbol{w}}$ (forth equation);

• Finally the gradient step is replaced by the update rule for the weights (second equation).

Hence the time complexity of our proposal is the same as the BP algorithm.

The Reviewer's comment on the presentation of the paper is understandable. We agree with the comment that the paper might not reach a broad audience. However, we are proposing a new approach that we believe to be relevant in the lifelong learning scenario. As such, we need to fully convince the scientific community of the technical soundness and, in particular, that the Hamiltonian equations yields a local spatiotemporal process which replaces Backpropation. While we considered to expose our results by adopting a more qualitative descriptions, we decided to give up so as not to run the risk of a rejection of the paper due to arguable statements. A comprehensible criticisms could have been that a new theory must not be claimed only by conjectures. However, in the near future, our ambition is exactly what the Reviewer was asking: to present the paper to a broad audience and, especially to explore relationship to reinforcement learning.

Q6. While the reduction for $c\to\infty$ can be formally done in general as suggested for instance in Corollary 1, it only describes a meaningful and viable computational scheme only when we have a directed acyclic graph (which correspond to the FNN case). So the reduction indeed makes sense only under the additional hypothesis that are the one that makes possible to apply the standard Backpropagation algorithm.

Q7. Consider the last equation in the system of Eq. (16) with all $c_i=c$, and group them as follows $\lambda^i_x -\sigma'(a_i)\sum_{k\in{\rm ch}(i)}\lambda^k_x w_{ki} - L_{\xi_i}(x,t)\sigma'(a_i)= \frac{1}{c} \dot \lambda^i_x -\frac{1}{c}\biggl[-\frac{\dot\phi}{\phi}+\frac{d}{dt}\log(\sigma'(a_i))\biggr] \lambda^i_x,$ now the rhs of this equation as $c\to\infty$ formally goes to $0$ and we are left with the desired equation.

Q8. This implies that, in general, it is not possible to simply define learning/optimization theories that operate in a temporal environment solely in terms of optimization problems for integral functionals (like the one in Eq. 8), unless the problem is small enough to be solved in a batch-mode fashion. This observation is also related to the fact that the solution of calculus of variations typically describes elliptic problems, and they are incompatible with hyperbolic problems, which is usually the a class of problems that are suitable to describe temporal evolution (like learning in this context). This is why the discussion proposed in Section 4, addressing boundary conditions, is instrumental to enable the use of such tools within the context of learning.

"Nits" We have address them directly in the revised manuscript.

Many thanks to the Reviewer for his humble approach. When looking at the constructive his criticisms, it looks like his/her expertise in reinforncement learning led to understanding the paper much more than what mentioned in the "Disclaimer part." We have tried to use some of the comments exposed in the weakness section to improve the presentation. In particular we tried to clarify some terms like `"pre-algorithmic algorithm" and to immediately comment on the role of loss function of the term $\ell$ right after Eq. (8). We furthermore changed the notation for the general variable that indicate the $\boldsymbol{w}$ component of the state that was previously denoted as $\boldsymbol{u}$ and now it is named $\boldsymbol{\omega}$ to further disambiguate it from the input variable $u$. We would also like to remark that the ${\rm IN}$ function is defined on right after Eq. (5). In what follows we will answer to the precise questions that the Reviewer formulated in the corresponding section of his/her review.

Q1. Yes, in this work, we are mainly concerned with online computations where there is a strong connection between the time that captures the dynamics of the sequence we are processing, expressed in our notation by the function $t \mapsto u(t)$, and the dynamics of the parameters of the model defined by the map $t \mapsto \boldsymbol{w}(t)$. Many learning protocols can be cast in this framework, ranging from reinforcement learning, lifelong and online continual learning. The specific strong emphasis in this paper is that we look for a learning framework which can work without accessing large data collections, but simply by processing and properly representing the information as it is comes from sensors over the given temporal horizon (agent life span).

Q2. We have added the references and commented how our approach contributes to the mentioned problem.

Q3. Perhaps we understood the comment, but we invite the Reviewer to ask again if we didn't grasp the essence of the question. Cycles like $A\to B\to A$ amongst non-input units are permitted since our assumptions correspond to typical recurrent nets. However, the consistent computational scheme that we adopt does require that the processing does not take place "within the same time step"! In general, in the discrete setting of computation, we need a delay, while in the continuous framework this correspond to the introduction of temporal derivatives of the state.

Q4. The term $\int_0^T \ell(\boldsymbol{w}(t), x(t;\boldsymbol{w}),t))\phi(t)\, dt,$ comes from the ergodic transposition of the functional risk inspired by classical mechanics as discussed in (Betti et al., 2019). The basic idea is that when dealing with time and with a stream of information the expectation of the loss that defines the empirical risk can be rewritten as a sum over time. In our case however, differently from previous works in this directions, we are considering recurrent computations and, therefore, the term $x(t;\boldsymbol{w})$ is a solution of the recurrent dynamics expressed by Eq. (4). As correctly noticed by the Reviewer, function $\ell$ can itself contain a regularization term on the weights (like the classical weight decay). However it is important to add an additional temporal regularization term that links the weights at different temporal instant of the form \int_0^T mc/2|\dot\boldsymbol{w}|^2\, \phi(t) dt for at least two reasons: firstly, it promotes convergence and the smoothness of the solution and, secondly, it enables the explicit calculation of the Hamiltonian in as in Eq. (11); hence, there is always a classic regularization term on the weights and also an appropriate temporal regularization term on the derivative of the weights that is consistently considered in Eq. (8) and in Theorem 2.

Q5. This is because the fact that we are considering the particular set of differential constraints in Eq. (10) allows us (by using the techniques of optimal control) to describe the stationary point of (9) in terms again of local differential equations (Hamilton Equation). In this equation, in addition to the spatial locality, the presence of time derivatives gives rise to local temporal propagation.

We thank the reviewer for his fairness in the evaluation and for having posed many specific question that will surely help to improve the quality of the manuscript.

Q1. Corollary 1 shows that in the limit of infinite speed of propagation ($c\to+\infty$) the differential equation for the rescaled costates, the $\lambda_x^i$, described in Eq. (16) (last equation) satisfy exactly the classical equation for the delta error terms used in the Backpropagation algorithm. This reduction, which in Corollary 1 is formally expressed in the general setting, make sense only in the case when the graph that we consider is acyclic which is an assumption required by Backprop. In this case, which correspond to the classical architectures of FNN we can choose, as it is usually done, the set of output nodes to be the one at the "end" or at the "top" of the architecture, i.e. the ones that have no childrens ($i\in O$ implies ${\rm ch}(i)=\emptyset$. Then if we look at Eq. (17):

we can make the following observations: i. if $i$ is an output neurons, $i\in O$, then the sum $\sum_{k\in{\rm ch}(i)}$ is empty and we get $\lambda^i_x = L_{\xi_i}(x,t)\sigma'(a_i)$ which is exactly the statement that the delta error on the output neurons is simply the gradient of the loss with respect to their activation, ii. when $i\ne O$ it is the second term that vanishes since, as we remarked in Theorem 2 $L$ only depends on the value of the neurons in the output and reduces to $\lambda^i_x= \sigma'(a_i)\sum_{k\in{\rm ch}(i)}\lambda^k_x w_{ki},$ that again is exactly the classical way in which the delta error is computed in Backprop.

For what concerns Proposition 4 connections with GD with momentum can be better understood as follows. The second formula in Eq. (20) has the form $\ddot w_{ij}+\theta \dot w_{ij} + \gamma \nabla_{u_{ij}} E(w,t)=0$ where $E$ is a generic compound loss. This is the case since the term $V_{u_{ij}}(\boldsymbol{w},t)$ is an explicit gradient of a regularization loss with respect to the weight $w_{ij}$ and $\lambda^i_x x_j$ is the Backprop factorization of the gradient of the loss that involves the outputs of the NN with respect again to the weight $w_{ij}$. If you consider an Euler discretization of this ODE you end up with

Q4. The algorithmic description of the proposed rules basically relies on the explicit Euler approximation scheme of an ODE. [As a reference you can look at the classical book Richard, L., Faires J Douglas, and M. Annette. Numerical analysis. sengage, 2016.] For instance in the case of the ``sign-flipping algorithm'' the once you choose a temporal discretization step $\tau>0$ the algorithm looks like this

1. Randomly initialize the state and costate variables and set the sign term in Eq. (23) equal to $1$: $s\gets 1$;

2. Compute the update terms following Eq.(14):

   • $\Delta x_i\gets c_i\Bigl(-x_i +\sigma\Bigl(\sum_{j\in{\rm pa}(i)} w_{ij}x_j\Bigr)\Bigr)$

   • \Delta w_{ij}\gets -p^{ij}_\boldsymbol{w}/(mc)

   • $\Delta p^i_x\gets c_i p_x^i-\sum_{k\in{\rm ch}(i)} c_k \sigma'\Bigl(\sum_{j\in{\rm pa}(k)} w_{kj}x_j\Bigr)p_x^k w_{ki} - c L_{\xi_i}(x,t)$

   • $\Delta p_{\boldsymbol{w}}^{ij}(t)\gets - c_i p^i_x \sigma'\Bigl(\sum_{m\in{\rm pa}(i)} w_{im}x_m\Bigr)x_j - c k V_{u_{ij}}(\boldsymbol{w},t)$

3. Update all the variables according to the implicit Euler scheme:

   • $x_i\gets x_i +\tau s \Delta x_i$

   • $w_{ij}\gets w_{ij} +\tau s \Delta w_{ij}$

   • $p^i_x \gets p^i_x +\tau s \Delta p^i_x$

   • $ p_{\boldsymbol{w}}^{ij} \gets p_\boldsymbol{w}^{ij}+\tau s \Delta p^{ij}_\boldsymbol{w}$$

4. Recompute $s$ according to Eq. (24).

5. If this is not the last iteration go back to step 2.

Q5. The regularization on the velocity of the weights it is important for at least two reasons: firstly it promotes convergence in the parameters of the models, as well as the smoothness of the overall learning process, and secondly enables the explicit calculation of the Hamiltonian in as in Eq. (11); without this term the Hamiltonian would have been defined in terms of another implicit minimization problem.

Q6. In the classical LQ problem the objective is to have bring the state variable as close as zero as possible, while also keeping small values of the control parameters, hence the more the term $x^2/2$ is weighted (by taking higher value of $q$) the more the solution is expected to accurately favour the zero, constant solution. In the experiment done with the sinusoidal signal there is indeed a misprint in the text the functional that we are considering is $G(v)=\int_0^T q (x-z)^2/2+rv^2/2 + r_w^2\, dt$ so that $q (x-z)^2/2$ has a direct interpretation as accuracy parameter. This has been now corrected in the revised version.

Q7. In this figure $x$ is the value of the output neuron, $z$ is the target and $u$ is an input as in Eq. (5) (notice that however in this experiment it is just a null signal). As explained in Q6, the actual accuracy term that is used in the experiments is $q (x-z)^2/2$ which should clarify the Reviewer's legitimate question.

Q8. The proposed theory works in general for a directed graph with the properties described in Section 2, which basically correspond with recurrent neural network architectures. However as we remarked also in Q1., while the reduction for $c\to\infty$ can be formally done in general as suggested for instance in Corollary 1, it only describes a meaningful and viable computational scheme only when we have a directed acyclic graph (case for which Backprop can be used). So the reduction indeed makes sense only under the additional hypothesis at the basis of standard Backpropagation algorithm. In all other general cases, that are the one of interest for this study, the expression we found in the limit are just formal expressions. However, what is really important is that the proposed theory clearly leads to a spatiotemporal local propagation scheme which, to the best of our knowledge, is an open problem for recurrent neural nets.