Nature-Inspired Local Propagation

We thank the Reviewer for having appreciated our work. In what follows we will address the comments and questions raised.

I appreciate that the author/s states in the conclusion that more work needs to be done to apply this to more applicable machine-learning problems, however, as a machine-learning practitioner I am not sure where I would start. I think this is something that could be cleared up in the paper. [...]

Thanks for pointing this out—it’s a really important observation! The learning framework we have in mind while developing these ideas, and one of the main motivations for this work, is lifelong learning from a continuous, possibly infinite, stream of data. We decided to give this work a more technical style because, as we expressed in the conclusions, we feel it is necessary to establish some precise results to build upon. One prototype of the learning problems we believe this approach is best suited for is learning from a very long video stream. In this case, the input stream $u$ will be the video stream, and a simple Euler method for Hamiltonian equations will readily provide an “optimizer” for the weights of a recurrent neural network (RNN). We will do our best to incorporate explicit statements on the desired outcomes of the work in the revised version of the paper.

Which elements of your derivation depend on control theory? [...]

You are perfectly right. One of the main results of the paper, Theorem 1, is derived as an application of standard control theory. Indeed, what we are trying to do is find optimal trajectories for the weights of a continuous-time RNN by controlling the velocity of the weights. This is expressed in the first paragraph of Section 3.1. In particular, the control variables are denoted by v in Equation (10). The adoption of this theory makes it possible to define a new spatiotemporal neural propagation scheme that generalise backprogatation.

What is the overall conclusion regarding data requirements? [...] I missed how the derivation ties back to this, perhaps you can make that more clear to me here.

Yes, precisely as you are saying, data enters the evolution equations through the input signal $t\mapsto u(t)$. For instance, when processing a video stream, $u$ is the video itself, and $u(t)$ is the frame of the video at time $t$. Notice that this signal is used in the computation of the outputs of the neurons (see Eq. (5)), as one would expect: $u$ updates the input neurons, which in turn update their children, and so on. For this reason, as noted in the introduction, this theory is applied in the context of lifelong/continuous learning, where the main assumption is that data is handled as a stream of information instead of a static training set. This means that at each time $t$ we only need to access $u(t)$ and we neither need to store nor to access data collections.

Hi,

Thank you for further clarifying. I appreciate your time on the replies.

1. Great!
2. That was helpful, thank you.
3. Very cool idea, and I like the way you summarised it here. I could probably ask questions about this particular point for a while, but at some point, it is just for my own interest.

I do think some things about the paper could be made clearer, and perhaps you will have addressed that by the time the paper comes out. But I really appreciate the discussion in the reviews, it has helped me gain a clearer picture of the work and its possible significance.

I am happy to bump your score up a bit on my side. I will recommend, though, that you add some of these concrete statements in the paper regarding things like theoretical applications and certainly addressing memory. The reason is that the points didn't come across easily in the actual manuscript, which is a shame.

We thank the reviewer for acknowledging the novelty of our approach. We will address the comments and questions raised.

It would also be much better if the pseudo-code of the final algorithm were presented directly, [...]

We agree that the presentation of the final algorithm that we are using could be improved. We will add an appendix with the pseudo-code that you can read in the attached PDF (at the end of the overall rebuttal).

Lack of experiment [...] is necessary for readers to understand its border applicability.

This paper proposes a natural learning scheme that promotes environmental interactions. It covers theoretical foundations, and the experimental results are expected to validate the consistency of the theory rather than achieving high performance on classical benchmarks. We acknowledge the importance of conducting similar experiments. The proposed test can be conducted for instance by generating an appropriate data stream from MNIST, where each pattern is presented at the input for a fixed amount of time. However, this requires a considerable amount of experimental work and is beyond the scope of this paper. It is worth mentioning that the MNIST experimental setting described above is not the primary focus of the theory and that experimental work in this area is still in its early stages. We also would like to point out that one of the topic from the call for paper is "theory (e.g., control theory, learning theory, algorithmic game theory)", which we believe is coherent with the spirit of this paper.

Related work [...]

We will add a “Related Work” section either after the introduction or as a subsection of it. In the overall rebuttal comments, we have included a draft of this section.

Am I correct that the proposed algorithm is based on (23) with zero initial conditions and s(t) defined in (24)? [...] How could the weight be updated when the label is not accessible before the final time step?

Yes, you are correct but with random initialisation on the state. If supervision is not provided until the end of the temporal interval, changes in the weight during the middle of the interval will not be informed by the label but only by regularisation terms; all the Hamiltonian equations are the same.

Explanation of the graphs and experimental setup We utilize a fully connected recurrent network with five neurons (see Figure 1 in the PDF rebuttal), each using $\tanh$ activation functions. One neuron, designated as the output unit, aimed to approximate a given reference signal through a quadratic loss in the Lagrangian. Figures 1–3, demonstrate the effectiveness of the local spatiotemporal neural propagation algorithm and the sign flip policy. Figure 4 displays the behaviour of the Lagrangian and Hamiltonian over time. In the figure caption “Track a highly-predictable signal” should be “Track a highly-unpredictable signal.”

Assumptions on temporal dependence See Q5 below

Assumption in line 8 We assume for the derivation of the Hamilton equation that the state satisfies Equation (4). This equation is associated with multiple approximating sequences at discrete time, of the form in Equation (3). We emphasize the importance of causality because we aim to develop an algorithmic scheme that can be easily computed forward in time. However Section 3 does not rely on this implicit assumption, it only assumes~(4).

Defining a simple computational graph Yes, this facilitates understanding and will be included in the final version of the paper. In the attached PDF file we have illustrated the structure of the network used in the experiments, along with the main variables.

Metrics for HSF policy effectiveness Our goal was to demonstrate the stabilizing effect and tracking capabilities of the HSF policy. We monitored the effectiveness of our approach using a quadratic loss function, which is commonly used in regression.

Appendix C Thank you again for the suggestion. We will expand the derivations in the proof in the final version of the paper.

Line 165 From the last line of (16) with $c_i=c$ we can divide both sides by $c$. In the limit $c\to\infty$ the only surviving terms are the ones that do not have $1/c$ in front; these are exactly the terms in Eq (17). We will expand the proof in the final version of the paper.

Suggestions on referencing (in order)

• Bertsekas, Dimitri. Abstract dynamic programming. Athena Scientific, 2022, pag. 2.
• Francis Crick. The recent excitement about neural networks. Nature, 337:129–132, 1989.
• Stork. "Is backpropagation biologically plausible?." International 1989 Joint Conference on Neural Networks. IEEE, 1989.
• Timothy P. Lillicrap Geoffrey Hinton, Backpropagation and the brain, Nature Reviews - Neuroscience, 2020
• Evans, Lawrence C. Partial differential equations. Vol. 19. American Mathematical Society, 2022. Pag.597

Answers to questions

Q1. By “processed information,” we refer to the data stream $u$. The internal memory representation in our model is given by the state variables (outputs and weights). Some justification for our assumptions may be found in the way representational systems are realized from a neurobiological perspective.

• Byrne, John H. Learning and memory: a comprehensive reference. Academic Press, 2017. Pag. 228–230.

Q2. We address two issues: the “update locking problem” and the often-overlooked issue of infinite signal propagation speed in neural networks. A clarifying sentence will be added.

Q3. We aim to develop a causal learning algorithm that processes information forward in time. Confusion may arise from continuous-discrete transitions in Section 2. We will make efforts to address it further.

Q4. The assumption is not necessary; it was used for simplification. We only require the condition $c_i/c \to 1$ as $c \to \infty$ for all $i$ if we want a reduction to backpropagation.

Q5. We assume $t \mapsto u(t)$ given function of time. Outputs, while also functions of time, are subject to dynamic constraints Eq.(5), and indeed depend on variables (neurons, inputs, and weights). Our proposal emphasizes the explicit temporal dependence of the parameters, modeling the continual adaptation necessary for continuous learning. We advocate shifting from finding a set of optimal parameters to searching for optimal weight trajectories. This approach reflects the ordinary cognitive perspective where learning and inferential processes are not separate.

Q6. All the results holds if the spacing between points in the temporal partition is non-uniform, so $\tau\to\tau_n$.

Q7. Yes, we agree that we need to be more clear. At any rate the $u$ in line 52 is the input signal while the bold u in line 92 and 93, is a generic vector that lives in the parameter space. ${\rm IN}^i(w)$ selects the components on the weights $w$ associated to arcs that point to neuron $i$. We will change the notation.

Q8. No, they are not related. The function in line 89 is a capital $\Phi$ and it specifies the structure of the neuron. The one on line 111 is an overall factor to the lagrangian that weights the contributions of different time steps to the overall cost. We will use more different symbols.

Q9. It is a generic vector that has the same dimension of $x$. You are right, we need to add a definition.

Q10. Since in the literature the term activation is not always used consistently the best way to answer your question is to say that $a_i$ are the quantities defined in Eq (15), that may be what you refer to as pre-activations.

Q11. From a computational point of view the approach is equivalent to GD + backpropagation and so it can in principle be used on networks of the same scale that are now used in deep learning.

Q12. Thanks, for the question. In the plots $x$ is the output of the output neuron (in the new attached figure it is $x_1$), $u$ is the input signal, and $z$ is the reference signal. The parameter $q>0$ weights the term of the lagrangian that enforces the fit between $x_1$ and $u$. The purpose of the plots is to show that the usage of Hamilton’s equation with the described sign flip prescription is effective in solving an online tracking problem.