We thank the reviewer for their insightful comments. The point-to-point answer is provided below.

---

Weaknesses

1. We thank the reviewer for appreciating the value of our theoretical results, beyond simple contributions in performance and accuracy.

Experimental evaluation:

2. Our work extends previous work that impeded training deeper networks with the original PEPITA method, and so improves on the complexity of networks that PEPITA can work with. Overall, many local or biologically plausible training algorithms in present-day literature are not yet ready to be tested on challenging large-scale datasets and highly complex network architectures.

3. It is unfortunately common in the literature to use image classification tasks as the initial benchmark of new training algorithms. We share this choice with Hinton’s FF paper and others in the same area (e.g. [Ororbia and Mali, 2023]). We do think it would be interesting to extend this work to other types of tasks, particularly to time-dependent datasets, where a forward-only algorithm can be especially useful for analyzing incoming data in real-time. However, this is best treated in a separate paper, where we can focus on the specific needs of such a case.

4. At this time, we are not aware of any implementation of Forward-Forward which utilizes feedback connections to generate the negative samples, as proposed by [Hinton, 2022]. Given the significant difference, we decided not to compare PEPITA to Forward Forward with externally generated samples. Despite this, for completeness, we have added the results that FF achieves on MNIST as reported in [Hinton, 2022].

5. As was pointed out above by reviewer sGVa, the performance of forward-only algorithms has been catching up in recent years, but what is lacking is the theoretical understanding of their learning dynamics. Our aim is not yet to apply PEPITA to real-world tasks but to understand which tools may or may not lead to performance improvements, and why, in order to guide further development. In this respect, small improvements are also informative, and even negative results are valuable lessons.

Minor issues:

6-7-8. We thank the reviewer for these suggestions, we are currently working to implement them in the paper.

9. We thank the reviewer for noticing this mismatch. We have now correctly referred to Fig.2d at the end of Sec. 3.2

10. We thank the reviewer for noticing this. The table now reports Lillicrap 2016 instead.

Spelling:

11. We thank the reviewer for catching the typo. We have corrected this in the revised version of the manuscript.

12. We thank the reviewer for the suggestion. In the revised version you will now find “GPUs”.

---

Questions

1. We thank the reviewer for the fair question. We consider our primary contribution the theoretical analysis of the nonintuitive learning dynamics of PEPITA, the study of training techniques that lead to improved performance, and the connections with FF and DFA. Overall, we believe that these different characterizations help us build a common framework to study forward-learning rules. The reason why we included PEPITA-TL in Table 1 is to have a complete comparison. Figure S1 shows that training with PEPITA-TL is possible, but we leave the study necessary to make its performance competitive for future work. Moreover, we have added PEPITA-Hebbian and PEPITA+WM in Table 1.

2. We thank the reviewer for the question. To find the best hyperparameters for training with activation normalization we run a grid search, which includes searching for the optimal value of weight decay. In all cases with normalization, the grid search returns no weight decay, for this reason, we did not include the combination of normalization and weight decay in the table.

3. We thank the reviewer for the question. The fact that at the start of the training, the angles look identical is due to the small learning rate for WM that the grid search has returned (Table S3). This causes pre-mirroring to have a small effect on the angle, although the effect is noticeable in terms of slowness (Table S2). For MNIST and CIFAR-100, the grid search returns higher learning rates for WM (Table S3), and the effect is more noticeable both in the angles (Figure S6) and in the slowness (Table S2).

We thank the reviewer for their insightful comments. The point-to-point answer is provided below.

---

Weaknesses

1. A key difference between DFA and PEPITA/FF is that DFA requires the computation of gradients of activation explicitly. This means that because the feedback is added after the non-linearity, an additional backward computation is needed. In FF and PEPITA, instead, the update only depends on activations computed during the forward pass. We also note that in [Hinton, 2022], the author mentions the possibility of negative samples constructed using top-down connections, which is the case for PEPITA. Moreover, we note that in forward learning algorithms, the error does not travel by itself throughout the network and that the networks continuously respond to inputs (original or modulated/negative), unlike in DFA where a forward pass is followed with a backward pass containing the error info.

2. We thank the reviewer for the questions. First, we want to clarify the difference between spatial and temporal locality. Spatial locality refers to rules that use only local information (unit activation). In [Nokland, 2016], the author mentions that the update is “almost local”. This is due to the fact that the feedback is added after the nonlinearity, and the update needs to compute the derivative of the activation, causing the update to be non-local (it would be local with linear activation). Instead, PEPITA does not require any computation that does not depend on local signals (unit activation). Temporal locality refers to rules that do not need to store the activations in memory, and are able to compute updates on only temporally-local information. This is the case for PEPITA-TL, which updates the weights on the fly. We note that PEPITA-TL is the only temporally-local rule in Table 1. Second, in DFA - similarly to BP - activity freezing is a consequence of alternating a forward pass (the network responds to an input, and the nodes are “active”) and a backward pass (the error information travels through the network from the top to the bottom layers, but the network does not process any inputs, thus the nodes are “inactive” or “frozen”). In PEPITA and in FF, the error never travels through the network “by itself” to drive the updates. The network always responds to an input in a forward fashion - the input is modulated through the error in PEPITA, the negative samples in FF. Last, we would like to note that this paper is not proposing the PEPITA algorithm, but rather an analytical characterization of the latter. Locality and activity freezing are explained thoroughly in the work by [Dellaferrera & Kreiman, 2022].

3. As specified at the beginning of Sec. 3, we focus on two-layer networks for our theoretical results. Sec. 3.1 is aimed at providing an analytical understanding of the alignment phenomenon observed in the paper introducing the PEPITA algorithm [Dellaferrera & Kreiman, 2022] that considered two-layer networks. Our approximation suggests that a perturbative approach could be insightful to understand the PEPITA learning rule in deeper networks. The analytical characterization of the dynamics of deep non-linear networks is to a large extent still an open problem even for networks trained with back-propagation and goes beyond the scope of our work.

4. We thank the reviewer for pointing out the work of [Launay et al 2020] and [Aktrout et al 2019], which we now cite. The reviewer raises valid points. In general, it has been hard to get better results than using back-propagation in the field. We usually refer to approximations to back-propagation. As one example of many, DFA [Nokland 2016] also started with relatively shallow networks and worse performance than back-propagation and it took a few years for the community to extend it to deeper networks [Launay et al 2020, as cited by the reviewer], and even in these deeper networks, back-propagation generally outperforms DFA (though DFA is a very reasonable approximation, [Laynay et al 2020]). Such approximations can still be interesting due to their biological plausibility, theoretical understanding of learning, and the possibility of implementing them in neuromorphic hardware.

We thank the reviewer for the positive comments on our work. We address the concerns and questions below.

---

Weaknesses

1. Generally, achieving superior outcomes compared to back-propagation in the field has proven challenging. Typically, we resort to utilizing approximations to back-propagation. For instance, DFA [Nokland 2016], among various examples, initially employed relatively shallow networks and exhibited poorer performance than back-propagation. It took a few years for the community to enhance it for deeper networks. We believe that our work provides the insights necessary to improve the performance of these types of algorithms.

2. In [Dellaferrera & Kreiman, 2022], the authors show that the algorithm can work well on CNNs too. In this work, we focused on other aspects, in particular on the theoretical analysis, which can be performed on fully connected networks. We leave the extension of our results to future work.

---

Questions

1. We have run experiments with PEPITA using the same hyperparameters as the best ones for PEPITA-TL. With this choice of hyperparameters, standard PEPITA also reaches around 40% accuracy, due to the low learning rate. Though these parameters were chosen using a grid search for PEPITA-TL, we conclude that the hyperparameter choice is not sufficient to explain the performance gap.

2. We thank the reviewer for the interesting insight. We provide an answer from two different perspectives. From the hardware implementation perspective, we say that a rule is temporally local if the updates are computed without the need to store any information in memory. This is the case for PEPITA-TL, which computes updates on the fly without any need to keep activations in memory. Instead, PEPITA-Hebbian needs to store the activations of the first forward pass and then use them for the update. From the biological perspective, we believe that the reviewer’s insight is very interesting: on the timescale of neural plasticity, we agree that PEPITA-Hebbian might be temporally local, which would also mean that PEPITA is.

Small comments:

1. We agree with the reviewer, and we added PEPITA-Hebbian and PEPITA+WM in Table 1.

We thank the reviewer for highlighting the strengths of our paper. Below, we provide detailed answers to the reviewer's concerns.

---

Weaknesses

1. The reviewer is right in observing that our theoretical analysis is limited to one-hidden layer networks. Our goal is to elucidate the alignment phenomenon initially observed in the paper introducing the PEPITA algorithm [Dellaferrera & Kreiman, 2022], which considers one-hidden layer networks. Further work would be required to study the multi-layer case, but our approximation suggests that a perturbative approach could be useful in this direction. We stress that the analytical frameworks to study the dynamics of learning in deep networks are still very limited, and – to the best of our knowledge – tracking analytically the average performance of deep nets with non-linear activations is still an open problem even for networks trained with back-propagation. One possible preliminary step to elucidate the alignment phenomenon in the multi-layer case would be to consider deep linear networks. We added a comment in Sec. 7.

---

Questions

1. As shown in Table S1, the performance decreases for 3-hidden-layer networks for MNIST (95.07% accuracy) and CIFAR-100 (drops to random chance). For CIFAR-100 weight decay allows us to train the 3-hidden-layer network (23.13% accuracy). We run additional experiments to test PEPITA Hebbian in MNIST and CIFAR-100 with 1-, 2-, and 3-hidden layer networks and we observe less than 0.15% difference (statistically insignificant) in accuracy between PEPITA and PEPITA Hebbian.

2. In the work by [Dellaferrera and Kreiman, 2022], the authors show that the presynaptic term ($h_{\ell - 1}^{err}$) can be replaced with the same term computer in the first forward pass ($h_{\ell - 1}$) without changing the performance. For this reason, we expected the Hebbian learning rule in eq. 8 to perform equivalently to the original learning rule, which is empirically the case. In the time local version, the updates are computed separately, meaning that the update computed during the second forward pass is operating on the weights that have already been updated in the first forward pass. We suspect that separating the Hebbian and anti-Hebbian updates changes the loss landscape, although we leave this analysis for further work. With the same choice of hyperparameters, standard PEPITA also reaches around 40% accuracy, due to the low learning rate. Though these parameters were chosen using a grid search for PEPITA-TL, we conclude that the hyperparameter choice is not sufficient to explain the performance gap.