Only Strict Saddles in the Energy Landscape of Predictive Coding Networks?

We thank the reviewer for their feedback and are glad that they share our excitement about the work. Below we address each point and question raised by the reviewer.

The reviewer points out that our analysis of deep linear networks (DLNs) is a weakness of our theory. We would like to highlight that DLNs are the standard model for theoretical studies of the geometry of the loss landscape (see our review of related work). This is in part because of the mathematical challenges of analysing non-linear networks. Moreover, and perhaps more importantly, DLNs can be seen as a good minimal model for understanding the optimisation properties of their non-linear counterparts. While non-linearities clearly affect the geometry of the landscape, as we note in the introduction DLNs retain all the most important properties of non-linear models including non-convexity and non-linear learning dynamics.

We would also like to highlight that our work goes significantly beyond previous theoretical studies on PC (reviewed in the related work section). In particular, previous theories make either non-standard assumptions or loose approximations. For example, both Alonso et al. (2023) and Innocenti et al. (2023) make second-order approximations of the energy to argue that PC makes use of Hessian information. However, our results clearly show that PC can leverage much higher-order information, turning highly degenerate, $H$-order saddles into strict ones. We now add this point of discussion to the appendix.

We appreciate the point that we performed experiments only around the origin saddle. As we explain in the global rebuttal, we now include two additional sets of experiments (see attached PDF), and we refer the reviewer to that for a full explanation. In brief, the first set of experiments (Figure 13) tests another zero-rank saddle than the origin covered by Theorem 3, and the second investigates higher-rank saddles which our theory does not address (Figure 14).

Whether “PC uses only local information available to each neuron to update weights” depends on what one means by “local” and “available”. We acknowledge that we were being loose with words in this particular sentence and now remove it from the paper as not critical to the main message. Nevertheless, to answer the more specific question, the reviewer is correct that standard PC needs both the transpose of the weights and the derivative of the nonlinearity, though we note that Millidge et al. (2020; https://arxiv.org/pdf/2010.01047) showed that on similar datasets one can remove these constraints without significantly harming performance. How PC could be implemented on alternative (neuromorphic) hardware is largely unknown, and we know of no published work on the topic. This is clearly a very important question for the future.

The reviewer asks whether there is a consensus on how the difficulty of an optimisation problem maps onto the strict vs non-strict saddle distinction. The short answer is “yes”. We summarise the results in the introduction (paragraph 4) and provide an in-depth review in the related work section. At a very high level, strict saddles are benign for even first-order algorithms like GD as long as one does not initialise too close to them or uses a particularly small learning rate, while non-strict ones can effectively trap (S)GD in the phenomenon of vanishing gradients since they are very flat. As we note in the paper, adaptive gradient methods like Adam have been argued to better deal with vanishing gradients and curvature in this work, but and their behaviour near non-strict saddles is much less understood than that of (S)GD.

The reviewer asks whether there is a straightforward way of seeing that the gradient of both the MSE and the PC energy vanish at the origin ($\theta=0$). The simplest case where this can be seen is for a network with two weights, where the loss is $\mathcal{L} = ½(y- w_2 w_1 x)^2$ and the (equilibrated) energy is $\mathcal{F}^* = \mathcal{L} / 1+w_2^2$. Because the residual $r = (y- w_2 w_1 x)$ is quadratic in the weights, the gradient is zero. Specifically, $\partial \mathcal{L} / \partial w_1 = -w_2xr$ and $\partial \mathcal{L} / \partial w_2 = -w_1xr$. For the energy, the expression of the gradient is a little more involved and has basically 2 terms. One is the gradient of the loss scaled by $1+w_2^2$ which will therefore also vanish at 0. The other term depends on the derivative of the rescaling, in this case $\partial (1+w_2^2)/ \partial w_i$. The rescaling does not depend on the first weight so this vanishes, and for the second weight is also zero because it is again quadratic in that weight.

The reviewer asks whether the origin can be a non-strict saddle for other loss functions than the MSE. This is an interesting point, and it turns out to be true for any generic convex, twice-differentiable cost as soon as we have two or more hidden layers as shown by Jacot et al. (2021, last paragraph before Section 5.1; https://arxiv.org/pdf/2106.15933).

The reviewer asks about the relationship between PC and equilibrium propagation (EQ). As shown by Millidge et al. (2022; https://arxiv.org/pdf/2206.02629), PC can be seen as an EQ algorithm where the free phase is minimised with a feedforward sweep (i.e. the squared energy of the last layer is equivalent to the nudging term). This avoids the need for two phases and is arguably more biologically plausible. It is not clear to us whether the theory could be generalised to the case of negative nudging or other energy-based networks, but this could be an interesting future direction. We thank the reviewer for directing us to reference [2] which we agree could potentially open interesting connections.

We thank the reviewer for their feedback. Below we address each point raised by the reviewer.

Given that the equilibrated energy turns out to be a rescaled version of the MSE loss, the reviewer asks if PC could be interpreted as having a different effective learning rate than BP. Though pointing in the right direction, we think that this is an insufficient explanation. First, the rescaling depends on the weights and so it will change at every weight update. In this sense, the gradient magnitude (or effective learning rate) of PC could be seen as adaptive, specifically higher for small weights and smaller for bigger weights. This is easy to see for the simple case of a network with two weights, where the equilibrated energy is just $\mathcal{L}/1+w_2^2$. More to the point, however, the rescaling not only affects the magnitude of the gradient, but also the curvature and other higher-order derivatives of the energy. To see this, consider an initialisation near the origin, where the weights are approximately 0 as can be seen from the toy model above, and the rescaling is therefore close to the identity or 1, $\mathcal{F} \approx \mathcal{L}$. If PC had the same effective learning rate as BP, then it would also show slow training dynamics near the origin (for networks with more than one hidden layer where the saddle is non-strict). However, this is not what we observe, theoretically and empirically.

Changing the learning rate of BP to match the “effective learning rate” of PC would therefore not work, in the sense that there is no learning rate for which the two algorithms will show exactly the same (S)GD dynamics. Nevertheless, we agree with the reviewer that it is an interesting question whether higher learning rates would allow BP with SGD to escape the saddle faster than PC. However, as we explain in our global response, such an analysis was beyond the scope of this paper, and we refer the reviewer to that rebuttal for our argument.

We agree with the reviewer that we did not particularly motivate our work from a neuroscience perspective and we discussed only superficially its implications for the field. There are many excellent reviews of PC we cite (the most recent being Salvatori et al., 2023) in the first sentence of the paper that cover the neuroscientific origins and connections of PC.

We now expand on the neuroscience implications of our work based on Song et al. (2024; https://www.nature.com/articles/s41593-023-01514-1), who proposed that energy-based algorithms like PC rely on a fundamentally different principle of credit assignment for learning in the brain called “prospective configuration”—where essentially weights follow activities (rather than the other way around). Based on a range of empirical results, Song et al. (2024) argued that PC confers many benefits for learning including faster convergence. Our theoretical results suggest that this latter claim may have been overstated. In particular, our results clearly show that PC and BP operate on two different landscapes and that therefore convergence speed will depend on many factors including network depth, width, dataset (specifically output covariance), initialisation, learning rate, and optimiser–all of which affect the landscape geometry or the learning dynamics. Any universal claim about faster convergence of PC should therefore control for all these factors. Song et al. (2024) claimed that PC learns faster than BP mainly based on an experiment with a deep ($L=15$) but narrow ($n_\ell=64$) network controlling for learning rate. However, our work reveals the importance of the initialisation of the weights, which interacts non-trivially with the network width. In particular, Orvieto et al. (2022) showed, in brief, that the narrower the network, the closer one will start from the origin saddle for standard initialisations. This insight likely explains the speed-up observed by Song et al. (2024). We add the above point to the discussion and thank the reviewer for highlighting this deficiency and allowing us to improve the reach of the paper.

We agree with the reviewer that it is not very clear why $W_L = 0$ in order for the gradient of the equilibrated energy to be zero. We can see no simple or intuitive explanation, as it simply turns out to be a property of the energy gradient. In particular, this is because of the rescaling term $S = I_ +W_LW_L^T + …$, which as we highlight contains a term quadratic in the last weight matrix. The derivative of the rescaling $\partial S / \partial W_L$ needs to be zero in order for the gradient of the equilibrated energy to be zero. Because as just noted the rescaling is quadratic in $W_L$, the derivative will be linear in $W_L$ and so the last weight matrix will have to be zero for the gradient to be zero. We now clarify and expand on the explanation of this property in the appendix, which as pointed out by the reviewer was previously only noted as a side remark.

The gradient $g_f$ is now defined before being used, and we thank the reviewer for pointing out this imprecision.

The ½ in the MSE loss (and energy) is often used as a theoretical convenience to cancel out the factor of 2 when taking derivatives of squared losses. It does not affect any of the derivations or theoretical results.

We thank the reviewer for their feedback. Below we address the points made by the reviewer one by one.

We tried to be very careful to not overstate our claims based on the results. In the abstract, introduction and conclusion, we state that “we provided theoretical and empirical evidence that the effective energy landscape of PC networks has only strict saddles”. Nevertheless, we agree with the reviewer that this could be misinterpreted to overstate our results, so we removed this sentence from the abstract and edited it more conservatively in the introduction and discussion. To avoid overstating our results, we also note that throughout the paper we used word conjecture to emphasise that we do not prove that all the saddles of the equilibrated energy are strict. For the same reason, we included a question mark in the title of the paper.

The reviewer is correct that when we say “other non-strict saddles than the origin” we are referring to the zero-rank saddles. It is true that the origin is one of these saddles as we point out. However, there is no a priori reason to believe that these zero-rank saddles (other than the origin) should also be strict in equilibrated energy. Nevertheless, we agree with the reviewer that we did not study other types (higher-rank) of saddles, theoretically or empirically. Therefore, as we explain in our global response, we now also include two additional sets of experiments (in the attached PDF), one looking at another zero-rank saddles than the origin and the other supporting the strictness of higher-rank saddles of the equilibrated energy.

The reviewer points out that our “experiments are primarily with simple MLPs”. This is true, although we note that we also performed experiments on convolutional networks too as noted in the caption of Figure 5.

We completely agree with the reviewer that the convergence results were not at all tuned. As we explain in our global rebuttal, our main goal was to characterise the intrinsic geometry of the equilibrated energy landscape, which had not been done before. Studying under which conditions, such different learning rates and optimisers, the energy landscape might be faster to optimise than the loss landscape is beyond the scope of our work. We refer to the global rebuttal for our argument and note that our work facilitates the study of such conditions.

We strongly disagree that Section 3.2 is a corollary of Singh et al. (2021). Singh et al. (2021) derived expressions for the Hessian blocks of arbitrary deep linear networks with MSE loss. They did not focus on characterising the landscape. We simply use the notation employed by Singh et al. (2021) to derive the Hessian of the equilibrated energy, which is non-trivial and significantly goes beyond this work. The reason why we include a re-derivation of the Hessian (with slightly different notation) in the appendix, and its structure at the origin in section 3.2 (which as we note was first derived by Kawaguchi, 2016, with a different Hessian derivation) is for intuitive comparison with that of the equilibrated energy.

We agree with the reviewer that we do not provide a full picture of the landscape, including all saddles and minima. However, we strongly disagree that “the technical part is largely based on Achour et al and Singh et al”. As explained above, we use mainly the notation of Singh et al. and re-derive the Hessian of the loss for comparison only. Achour et al. characterised all the critical points of the MSE for DLNs to second order, and we simply use their characterisation for comparison (i.e. to check what happens to these critical points in the equilibrated energy). Importantly, we would like to emphasise that showing whether the critical points of the MSE loss (characterised by Achour et al.) are also critical points of the energy and, if so, what kind of points, are additional questions that require a completely separate technical analysis of the gradient and Hessian of the equilibrated energy, which as our calculations show are non-trivial to derive. As we also review in the related work section, it is worth emphasising that the landscape of DLNs for the MSE loss took many papers before being understood to the degree that it is today. Our paper can be seen as a first step into understanding the landscape of the equilibrated energy. Indeed, our novel derivation of the equilibrated energy enables further studies into the landscape.

We agree with the confusions in terminology pointed out by the reviewer and made changes to the text accordingly.

We agree with the reviewer that the Bengio et al. (1994) paper is the first to introduce the vanishing gradient problem. We made reference to the other paper (Orvieto et al., 2022) because this was the first (to the best of our knowledge) to connect the phenomenon of vanishing gradients to vanishing curvature and therefore the non-strict saddle at the origin addressed in part by our work. We now refer to both papers to also acknowledge Bengio et al. (1994).

We agree with the reviewer that our original proof of Theorem 3 was loosely written. We now update the paper with a full proof including all steps and clearly separating the equalities from the approximations. We thank the reviewer for pointing out this imprecision.

Thanks for the rebuttal. Please be more precise in the claims in the text about the kind of results proved. Also, don't take me wrong in comparisons with Achour/Singh et al. Please see the qualifications in my original comments. Then there is definitely novelty in looking at the landscape of equilibrated energy, my comment is more on eqns 6 and 7 and not 3.2 as a whole and then it's broadly the way the discussion comes off in the main text. And, likewise, for the discussion around Achour et al. Just make it clear what is known from before, what is evident from prior work even if unsaid, and what you are introducing.