A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks

Unfortunately, all of the results are derivative of Neal and Hinton's incremental EM where they showed that both the expectation and the maximization step can be done at the same time and is a theoretically sound way of doing decent for model parameters. There is nothing that this paper contributes conceptually beyond this well known result. The hierarchical Gaussian generative model is also very well known. All in all, the paper lacks sorely in novelty of contribution.

We agree that iPC is mainly obtained by applying incremental EM to the original formulation of PC, and it is hence a mix of two existing techniques. We believe, however, that this should not be a reason to completely dismiss a research project, without first considering the impact that it could potentially have in the relevant literature. Let us highlight the reasons why we believe this work could be impactful in this field:

Going back in time, the first work introducing the weight update of PC is Rao and Ballard’s in 1999. From Rao and Ballard’s paper to today, countless works use the original rule to update the weights, which is much slower than ours, and less performant in the tasks that we have proposed. Researchers performing experiments have certainly experienced and struggled with the bad efficiency and stability of PC models. Figuring out a new update rule for the weights that addresses this problem, without altering bio-plausibility or adding additional drawbacks, is a problem of vital importance in the field. Our proposed iPC hence addresses an existing practical problem that a large community has had for years. We foresee that many colleagues working in PC or similar fields could now use this update rule, instead of the standard one, as it results in a big gain of efficiency, as well as theoretical guarantees in terms of convergence, with (according to our research) no drawbacks.

The fact that it is much more stable than the original formulation of PC, and constantly outperforms it by converging to better minima is not a negligible detail, as it allows PC to reach a performance comparable to those of BP on complex tasks, such as training deep convolutional models, or language models. Such results are novel in computational neuroscience. This is important, as it allows researchers to improve the performance of their experiments. We have given evidence that this is the case via a large number of experiments: generative models trained on complex datasets such as CIFAR10 and Tiny ImageNet, and discriminative models of different kinds (convolutional and feedforward) on a large number of datasets.

We also believe that our work has the potential to impact research in computational neuroscience, as it does not alter the biological plausibility of the original formulation of PC in any way. Actually, the lack of requirement for a control signal that updates the weights makes the model even more plausible biologically. Hence, this will allow our algorithm to be used in theoretical research in computational neuroscience, where the goal is to simulate brain behaviors and not improve the performance of machine learning models. All in all, we believe that our contribution has the potential to be impactful enough in the community of neuroscience-inspired learning, regardless of the simplicity of its update rule.

We thank the reviewer for the thorough read, and for all the pointers, spotted typos, and suggestions. They have all been addressed in the updated manuscript. We believe that this definitely improves the clarity of the manuscript. More generally, every time that we discuss accuracies, they are test accuracies (now addressed everywhere in the text).

About the weight transport, we have added the following, where pointed out by the reviewer:

Weight Transport While predictive coding is designed to model information processing in various brain regions, the current formulation still exhibits certain biological implausibilities. A notable example is the weight transport problem, which posits that the synaptic weight responsible for forwarding information is identical to that which conveys error information backward, as highlighted by Lillicrap et al, 2016. This issue has garnered substantial attention, contributing to the evolution of deep learning models that achieve performance levels on par with backpropagation in large-scale image classification tasks, as discussed in Xiao et al. 2019. Within predictive coding research, it has been shown that the removal of these features does not markedly impair classification performance highlighted in Millidge et al.2020. While it is important to emphasize that our proposed algorithm remains functional in the weight transport framework described in the aforementioned study, testing it goes beyond the scope of the paper. To conclude, other works have introduced predictive coding-like algorithms that do not rely on weight transport (Ororbia & Kiefer, 2022).

Questions:

why PC with smaller T values converges faster (in terms of Energy in Figure 2). Could you give an insight about it, or elaborate it?

We believe that the reason is that out-of-equilibrium updates are be better (and of larger magnitude) than updates performed to models closer to convergence. This has been shown to be consistent in a wide range of experiments.

What is the meaning of the notation on page 17?

Thank you again, this was a typo: the correct notation is O(max(n^l,n^{l+1}). This has been updated in the manuscript.

We thank the reviewer for the feedback and the time spent on the manuscript.

the key algorithmic innovation is simply to alternate weight and hidden layer activations every step (without running either to convergence). It's quite hard to believe this hasn't been tried before. Is there something new here that makes this work?

This is completely novel and has never been tried before in the field of predictive coding networks (to our knowledge). It is important that you mentioned the alternation of the updates of weights and activations, as well as the fact that the algorithm does not run until convergence, as this is key to make iPC work: running iPC until convergence on every batch of data would result in a strong overfitting of the last batch presented, as well as a forgetting of the previously learned information. It is the combination of both that allows the model to generalise well on unseen data (as well as making it much more efficient). This is different from other works, which tend to run every iteration for a lot of time steps, or until convergence (for example, in the paper Predictive coding approximates backprop along arbitrary computation graphs the authors use T=200 [1]).

The analysis is limited to Gaussian networks, which limits the range of applicability of the learning rules in (6) and (7).

Note that predictive coding is defined on Gaussian networks only: linear prediction errors arise naturally from this formulation (they are the derivatives (or scores) of the variational free energy of a generative model formed by a hierarchy of Gaussians). Having different probability distributions, such as categorical, would result in update rules that do not have a structure of neuron-synapses-neuron, as well as prediction errors.

That being so, this assumption is sometimes relaxed, to allow the algorithm to work on more complex generative models. An example of that is the experiment on Transformer networks that we presented in the last section, where we have added categorical distributions after the attention mechanism, hence showing that iPC naturally generalises to non-Gaussian distributions as well.

While the experiments are good, there is not much insight provided as to why this new approach works better.

Incremental updates mitigate the risk of overfitting by continuously adjusting the model parameters, preventing it from becoming too fixated on the specificities of the initial data that it was trained on. Standard predictive coding, on the other hand, tends to overfit on single data points. In terms of convergence guarantees, incremental methods are good in escaping local minima and reach more globally optimal solutions, in which non-incremental ones such as standard PC can get stuck. This is due to the continuous update of the parameters when the neural activities have not yet converged, making the dynamics more “noisy” [3].

Questions:

The analysis in the paper is focused on feedforward Gaussian networks, but can these ideas be extended to networks with substantial recurrence? In the original PC formulation, it's these recurrent inputs that carry error signals, but I'm curious as to whether the "out of equilibrium" method proposed here would also be able to dynamically balance multiple kinds of feedback.

It can naturally be extended to recurrent cases, such as [4]: note that the final update rule is strictly local. Hence, it is not aware of the structure of the model: they will work without any modifications in models with cycles, loop and recursions.

how much biological plausibility is lost if the Gaussian and feedforward assumptions are relaxed to mirror more cortical-like networks.

If by cortical-like networks you mean entangled and cyclic structures that resemble brain regions, our method can be applied with no modification (for examples of predictive coding models that use such structures, see [4]).

[1] Millidge, Beren, Alexander Tschantz, and Christopher L. Buckley. "Predictive coding approximates backprop along arbitrary computation graphs." Neural Computation 34.6 (2022): 1329-1368.

[2] Friston, Karl. "A theory of cortical responses." Philosophical transactions of the Royal Society B: Biological sciences 360.1456 (2005): 815-836.

[3] Karimi, Belhal, et al. "On the global convergence of (fast) incremental expectation maximization methods." Advances in Neural Information Processing Systems 32 (2019).

[4] Salvatori, Tommaso, et al. "Learning on arbitrary graph topologies via predictive coding." Advances in neural information processing systems 35 (2022): 38232-38244.

Dear reviewer,


First of all, we thank you for your time and for the positive comments and kind words about our work. Second, we would like to state that the policy submission of ICLR does allow for the ArXiv submission of our work, as highlighted in the rules, that we copy below:

Dual Submission Policy: Submissions that are identical (or substantially similar) to versions that have been previously published, or accepted for publication, or that have been submitted in parallel to this or other conferences or journals, are not allowed and violate our dual submission policy. However, papers that cite previous related work by the authors and papers that have appeared on non-peer reviewed websites (like arXiv) or that have been presented at workshops (i.e., venues that do not have publication proceedings) do not violate the policy. The policy is enforced during the whole reviewing process period. Submission of the paper to archival repositories such as arXiv is allowed during the review period.


And again, in the double blind review:



Double blind reviewing: Submissions will be double blind: reviewers cannot see author names when conducting reviews, and authors cannot see reviewer names.  Having papers on arxiv is allowed per the dual submission policy outlined below.

The above texts have been copied and pasted from the official “call for papers” webpage of ICLR 2024, that you can find here:

https://iclr.cc/Conferences/2024/CallForPapers .

Hence, we did not violate any rule, as the preprint of our work on arXiv is allowed.

To this end, would it be possible for you to raise the score in a way that reflects your true opinion about our work?

Many thanks,

The authors.