Towards training digitally-tied analog blocks via hybrid gradient computation

We thank Reviewer pevw for their detailed review and we are happy that they've appreciated our work.

i) The paper does NOT consider a detailed analog computing device model.

We fully agree with Reviewer pevw that this is a current limitation of our work, as explicitly acknowledged inside the “Limitations and future work” paragraph (L.294-301). This is why moving towards hardware realistic ff-EBM training is one of the main thrusts of our research trajectory - see section 3 of the proposed roadmap to Reviewer kDhg as well as Fig. 3 inside the pdf attached with the global rebuttal).

This being rightfully pointed out, there are still good reasons not to start the investigation of ff-EBM training by EP with realistic analog computing device model but instead Deep Hopfield Networks (DHNs). DHNs, while being remotely inspired by physics [1] and without “analog-realistic” noise, serve as a useful model baseline to start with in the context of EP for at least three reasons:

• In all past studies investigating the scalability of EP training and motivated by analog computing, DHNs were used as EB models [2, 3, 4, 5].

• DHNs are already difficult to train by EP. Compared to feedforward models trained by backprop, training DHNs by EP comes with extra critical and subtle hyperparameter tuning, most importantly: a) the root find algorithm at use to compute equilibria [6, 7, 8]; b) the number of iterations needed to converge in the first, second and third phases of EP; c) the value of the nudging parameter $\beta$; d) the way nudging is applied in the output layer [9]; e) weight initialization statistics, which implicit learning more generally is highly sensitive to [6, 7]. For instance, our study reveals that initializing the weights of the EB blocks as Gaussian Orthogonal Ensembles (abbreviated as “GOE weight initialization” in our paper) was instrumental in having ff-EBM training by EP working properly.

• Finally, simulating energy-based analog resistive networks comes with even more hyperparameters compared to DHNs [8, 10], making in turn EP training harder to debug.

ii) How much can ff-EBMs tolerate significant variations on the analog devices?

We would like to clarify the scope of the question by defining two broad classes of analog device imperfections which Reviewer pevw may have in mind:

• Device imperfections affecting the weight update given a gradient value. In analog systems, weights are typically encoded as conductance values of resistive devices [10]. Given a gradient estimate, programming a resistive device to update its conductance by this gradient estimate precisely is generally very hard, if not impossible because conductance updates suffer from device-to-device (“static”) variability, cycle-to-cycle (“dynamic”) variability, conductance drift, non-linearity with respect to the applied programming voltage, asymmetry between device potentiation and depression, etc. [11]. However, these imperfections apply to any gradient computation algorithm since they only pertain to the conductance update given a gradient estimate. As highlighted in the most recent study of “analog SGD” [14], the gradient estimate is given. Therefore, all conclusions regarding “analog parameter optimization” given a gradient value hold in particular for ff-EBM training by EP.

• Device imperfections affecting the gradient estimation. Once an analog nonlinear resistive circuit is instantiated with some fixed conductance values, it may still be subject to static variability (i.e. two analog devices do not share the exact same input-to-output characteristic) and dynamic variability (i.e. a single analog device may not react exactly the same when exposed twice to the same input). However, as highlighted in the global rebuttal and that addressed to Reviewer Jt5z, EP-based training of EB blocks is inherently tolerant to this static variability since this very same circuit is used for both inference and gradient computation [12]. Therefore, ff-EBM training by EP may only be impacted by dynamic variability. Investigating quantitatively the tolerance of ff-EBM training to dynamic variability and coming up with strategies to mitigate its potential negative effects is a project of its own that naturally falls in our research agenda (see task 3c) inside rebuttal to Reviewer kDhg and in Fig. 3 inside the pdf attached to the global rebuttal.

iii) What is the relationship between the convergence speed and the device variations? We assume that Reviewer pevw has in mind the convergence of the learning dynamics of ff-EBMs in the parameter space when trained by the proposed algorithm. The “device variations” may refer in this case to those involved in the conductance updates which, as mentioned above, are agnostic to how gradients are computed. It is known that the learning dynamics of "analog SGD" [14] are severely impacted by such non idealities even on simple tasks as MNIST [13]. Therefore, any such observation, or theoretical prediction [14] as well as any heuristic to mitigate non idealities of the conductance updates, e.g. Tiki-Taka [15], are also directly applicable to ff-EBM training by EP.

iv) What is the energy saving for the energy-based models? What is the energy saving of skipping the accurate gradient computations? Please, see our global rebuttal.

[1] Hopfield, Science, 1986 [2] Laborieux et al, Frontiers in Neuroscience, 2021 [3] Scellier et al, 2022 [4] Laborieux & Zenke, NeurIPS 2022 [5] Laborieux & Zenke, ICLR 2024 [6] Bai et al, NeurIPS 2019 [7] Agarwala and Schoenholz, ICML 2022 [8] Scellier, ICML 2024 [9] Scellier et al, NeurIPS 2023 [10] Kendall et al, 2020 [11] Burr et al, Advance in Physics X, 2017 [12] Yi et al, Nature Electronics, 2022 [13] Burr et al, IEEE Transactions on Electron Devices, 2015 [14] Wu et al, ArXiV, 2024 [15] Gokmen & Haensch, Frontiers in Neurosciences, 2020

We thank Reviewer kDhg for their detailed review and we are happy that they've appreciated our work.

i) Energy efficiency claims, detailed comparisons with existing systems to substantiate these claims. Please see our global rebuttal.

ii) Why only ImageNet32?

ImageNet32 was used for two reasons: a) we wanted to align our work with the most recent EP literature evaluating algorithms on vision tasks only with dimensionality at most 32 [1, 2, 3]; b) The reason we restricted ourselves to such a low-dimensional dataset is because ff-EBM training simulations are already relatively slow on simple datasets – see below for a more detailed discussion.

iii) Explicit discussion on the current limitations and a roadmap for future work.

We herein detail the roadmap for future work which addresses three major limitations of our paper: 1) Scalability to complex tasks; 2) Generalizability to other datasets and architectures; 3) Hardware realistic training of ff-EBMs.

Please find the associated Fig. 3 inside the PDF attached to our global rebuttal.

1. Scalability to complex tasks, i.e. higher input dimensionality (e.g. going from ImageNet32 to ImageNet224) and deeper and larger architectures. As we wrote to Reviewer k7RS, we insist that what bottlenecks the scalability of our work is not the proposed algorithm itself but the ability to efficiently simulate ff-EBM inference, because of the lengthy root finding algorithms needed to compute equilibria inside the EB blocks. As highlighted by Table 2, training a ff-EBM of 15 layers on ImageNet32 takes at least 40 hours, regardless of the training algorithm used. Therefore, “scaling” either requires:

• a) maintaining the current architecture, i.e. with Deep Hopfield Networks (DHNs) as EB blocks, on ImageNet224, with more GPUs and more compute time, or:
• b) changing the EB block, i.e. use EB blocks with better convergence properties than Deep Hopfield Networks (DHNs). A compelling class of models is called Deep Resistive Nets (DRNs) [4] which have the double advantage to be more hardware plausible and faster to simulate owing to their convex energy function.

2. Generalizability to other datasets and architectures. As pointed out by Reviewer 9JF8, our approach has yet to be proved on transformer architectures and associated datasets, which subsumes:

• a) Defining a ff-EBM counterpart of standard transformers. To map a transformer into a ff-EBM architecture, we need to break it up into feedforward / digital and energy-based / analog blocks. As illustrated in Fig. 2 inside the PDF attached to our rebuttal, the ff-EBM counterpart of a transformer encoder block would comprise: i) a feedforward block with normalization and attention layers, ii) an energy-based block with the two fully connected layers. We denote below this ff-EBM counterpart of a transformer architecture “TF-ff-EBM”.

• b) Testing a TF-ff-EBM with DHN blocks. As we did in this study, we would use DHNs as EB blocks in the proposed TF-ff-EBM architecture and test it on a relatively simple dataset to start with (e.g. the PTB dataset).

• c) Testing a TF-ff-EBM with DRN blocks. See 3a) below.

• d) Scaling hardware-realistic training of TF-ff-EBM to more complex tasks. We would scale the previous approach to deeper TF-ff-EBMs on the Wikitext datasets, with several additions to the architecture depicted in c) to make it more hardware realistic (see 3c) below).

3. Hardware realistic training of ff-EBMs. Moving towards more realistic proof-of-concepts of mixed-precision training of ff-EBMs, the following milestones need to be hit applying to both vision models considered in this paper as well as language models to be investigated:

• a) Using DRNs as EB blocks. As mentioned earlier, DRNs are exact models of nonlinear resistive analog circuits, made up of resistors, diodes, operational amplifiers with actual voltage nodes as neurons.

• b) Using quantized activations and gradients within feedforward blocks. A realistic assumption for feedforward transformation would be to employ quantized activations, weights and gradients. Techniques such as Quantization-aware Scaling (QAS) could be employed to compute meaningful quantized gradients inside feedforward blocks [5].

• c) Taking into account analog non-idealities. As pointed out by Reviewer pevw, we need to consider a “detailed analog computing device model”, for instance taking “device variation” into account or any other device imperfection which may affect gradient computation. Taking these into account is crucial to assess how much extra digital circuitry it would take to mitigate these as it entails energy consumption.

• d) A comprehensive energy consumption analysis of hardware realistic ff-EBM training. Finally, quantitatively and precisely assessing the energy efficiency gains of ff-EBM training with the more hardware-realistic assumptions introduced above is crucial.

We propose to add this detailed research roadmap inside the supplementary material of the camera-ready version of our paper in case of acceptance

iv) Should the first-order optimality conditions be strictly guaranteed for the BP-EP chaining to work?

In practice, meeting optimality conditions can be quantitatively assessed with the static gradient analysis presented inside Fig. 3 and Fig. 4 (since $\lambda_\star$ can be typically be computed by automatic differentiation) and we observe that the “more optimal” $s_{\star}^{\pm \beta}$ are (in the sense of satisfying the KKT conditions) , the better the resulting training performance by EP. For a further mathematical analysis of approximate first-order optimality in implicit models, we refer to [6, 7].

[1] Laborieux & Zenke, NeurIPS 2022 [2] Laborieux & Zenke, ICLR 2024 [3] Scellier et al, NeurIPS 2023 [4] Scellier, ICML 2024 [5] Lin et al, NeurIPS 2022 [6] Zucchet & Sacramento, 2022 [7] Meulemans et al, NeurIPS 2022

We thank Reviewer 9JF8 for their comments and we are happy that they've appreciated our work.

i) Why must we combine an analog forward circuit with a digital one for EP-based training?

We first want to clarify that per our modelling choice, analog and digital parts are respectively modeled as energy-based and feedforward models. Therefore, we do not assume that feedforward models are mapped onto analog. Modelling analog parts as energy-based models is a plausible assumption as it was shown that arrays of analog resistive devices minimize an energy function at equilibrium called the “pseudo-power” function [1, 2]. We kindly invite Reviewer 9JF8 to read details about these aspects inside the global rebuttal.

Additionally, EP-based training does not by itself require the combination of analog (i.e. energy-based) and digital (i.e. feedforward) circuits. EP only applies, by default, to models which are fully energy-based, and that would map to a fully analog system. Rather, this combination is a realistic hardware design constraint, as analog hardware can currently only easily support vector-matrix multiplications and certain non-linear functions, with all other operations (e.g. max pooling, batchnorm) carried out in digital (see Fig. 1 of the PDF). Therefore, as such a system may not be fully energy-based but only by parts, it was unclear, before this work how EP training would extend to such systems.

Lastly, there is also an algorithmic advantage to use ff-EBMs as abstraction of such systems over EBMs: they are faster to simulate. Please read Reviewer k7Rs's rebuttal for further details, along with new results inside Table 1 of the PDF attached to our global rebuttal.

ii) What are the benefits of incorporating an energy-based model, compared to using forward-forward only or zeroth-order optimization?

This is an excellent question. Assuming Reviewer 9JF8 refers to the “Forward-Forward” (FF) algorithm [3] when mentioning “forward-forward only” optimization, both FF and zeroth-order (ZO) techniques apply to any model, be it energy-based or not, and estimate gradients through multiple forward passes through the same circuit, which is the holy grail for analog substrates. So why considering energy-based models at all?

FF and ZO algorithms, while being mechanistically appealing for analog hardware [4], do not match the performance of automatic differentiation (i.e. backprop for feedforward models) on equal models, of around the same size as those studied in our work.

• ZO. When using random weight perturbation [5] (WP, the best-known variant of ZO optimization, also known as “SPSA” [6]), resulting gradient estimates, although unbiased, have variance which scale cubically with the model dimensionality [7], yielding a significant gap in resulting model performance compared to backprop. This issue can be mitigated by architectural changes allowing the use of layer-wise, or even patch-wise losses [7,8], effectively reducing dimensionality and therefore the variance of ZO gradients or using small auxiliary networks to compute good “guesses” in the weight space (instead of random ones) [8]. These heuristics, while being effective at nearing backprop performance when training feedforward models from scratch, still don’t entirely close this performance gap on equal models. For instance, training a ResNet-18 by weight perturbation yields in the best case 45.8% top-1 test accuracy on ImageNet32 (using in this case auxiliary networks) while BP achieves 53.7% on the same architecture [8]. In contrast with ZO optimization, the use of EP on EB models or ff-EBMs always match the performance of automatic differentiation on equal models, in a principled fashion and without any heuristic, as observed in this work and past EP works [9, 10, 11]. Indeed, the energy-based requirement guarantees that EP gradients match automatic differentiation gradient in the limit of small nudging [12, 13]. This being said, we are aware that ZO techniques may however be sufficient to fine-tune pre-trained models [14], as pre-trained models somehow behave as small models (i.e. their Hessian has small rank). Therefore, neither first-order optimization techniques as EP, nor energy-based models, may be necessarily needed in the fine-tuning context, depending on the difficulty of the downstream task.

• FF. The FF algorithm is endowed with the same high-level features of WP or ZO order algorithms, namely with the use of multiple forward passes with the same circuit and of local losses. However, it is a heuristic algorithm with no theoretical guarantees: the weight update does not estimate the gradient of the loss of interest, nor is it guaranteed to decrease the loss. Instead, the algorithm is designed to increase (resp. decrease) the “goodness” of each layer on positive (resp. negative) samples, with the goodness function and negative samples being heuristically motivated [3]. As a result, the resulting model accuracy trained by FF on CIFAR-10 does not exceed 55% on CIFAR-10 [15].

In summary, both ZO and FF algorithms, while possessing certain qualities and advantages similar to EP, are as yet not known to scale comparably to BP, whereas our algorithm (in principle) does.

We propose to add this detailed discussion around FF and ZO in the related work section

[1] Johnson, 2010 [2] Kendall et al, 2020 [3] Hinton, 2022 [4] Oguz et al, 2023 [5] Fiete et al, 2010 [6] Spall, 1992 [7] Ren et al, 2022 [8] Fournier et al, 2023 [9] Laborieux et al, 2021 [10] Laborieux & Zenke 2022 [11] Scellier et al 2023 [12] Ernoult et al 2019 [13] Scellier & Bengio 2019 [14] Malladi et al, 2023 [15] Aminifar, 2024

We thank Reviewer k7Rs for their comments and we are pleased they've appreciated our work.

i) Accuracy performance on the ImageNet dataset

As acknowledged inside the “Limitations and future work” paragraph (L.302-305), ff-EBM training by EP remains to be proved at scale on deeper models and more complex tasks (see Fig. 3 of the PDF about our research roadmap). Therefore, we fully agree that the resulting performance of ff-EBMs trained by EP on ImageNet “lags behind” SOTA performance achieved by modern deep learning architectures, such as transformers or deeper CNN models, trained by BP on the same dataset.

However, when comparing different models (ff-EBMs and transformers) trained by different training algorithms (EP and BP respectively), it is hard to disentangle improvements that come from the model or the training algorithm taken separately. Our results displayed inside Tables 1 and 2 of our original submission clearly show that EP always performs as well as the automatic differentiation baseline on equal models. This suggests (we believe) that the scalability of our approach is not limited by the proposed algorithm itself, but by the ability to efficiently simulate large ff-EBMs. To further support this, note that (as seen in Table 2) training a ff-EBM of 15 layers takes at least 40 hours, regardless of the training algorithm used. These simulation times come from the lengthy fixed-point iteration used to compute equilibrium states inside EB blocks.

Therefore, we envision two paths towards achieving better performance on ImageNet (see Reviewer kDhg's rebuttal and Fig. 3 of our PDF for further details):

• using more GPU resources for longer times on deeper ff-EBMs, whilst preserving the core architectural components of the ff-EBM used in the present paper;
• using EB blocks which converge faster to equilibrium. As mentioned in L. 297, using Deep Resistive Networks [1] instead of Deep Hopfield Networks as EB blocks, or even employing techniques such as Anderson Acceleration for root-finding, would yield a significant speed-up which would allow to train much deeper ff-EBMs.

We propose to make it clearer, in the “Limitations and future work” paragraph, that our simulations are bottlenecked by the ability to simulate large-scale ff-EBMs rather than the proposed algorithm itself, and will accordingly mention the two research paths mentioned above

ii) There is a lack of compelling reasons to use this method given its comparatively lower performance

• Clarification. We would like to clarify that the goal of this work is only to bring a proof-of-concept to guide the design of mixed-precision training accelerators with analog and digital parts. As ff-EBMs remain inefficient to simulate on GPUs, we do not advocate any practical use of ff-EBM over standard modern feedforward architectures, nor training them by EP instead of automatic differentiation, to achieve the best possible performance when using GPUs. For lack of widely accessible real mixed precision architectures to test our training algorithm against, ff-EBMs are high-level abstractions thereof which we simulate using GPUs.

• ff-EBMs are much faster to simulate than their EBM counterparts. With regards to simulating hardware systems as mentioned above, a compelling reason which did not appear clearly enough in our original submission is that ff-EBMs are much faster to simulate than their single-block EBM counterpart of same depth. To demonstrate this, we re-ran experiments of Table 1 (the "splitting experiment", section 4.3) with a slightly different method to compute equilibria inside EB blocks. We contrast these two settings below:

  • Former setting of the original submission (Table 1 of the paper). Our initial goal was to demonstrate that starting from a standard ("single-blocked") EBM that ff-EBMs of same depth retained the same expressive power across all possible even splits, per the resulting performance on CIFAR-10. In this case, a fixed-point iteration scheme was applied inside each EB block, with the same fixed (and possibly large) number of iterations across all splits and throughout training. However, EB blocks of smaller size are expected to converge faster and therefore may require fewer fixed-point iterations [2-4].
  • New setting (Table 1 of the PDF attached to the rebuttal). In order to take advantage of the faster convergence of smaller EB blocks and ensure the fairest comparison of wall-clock times across all splits for a given depth ($L=$6 or 12), we employ a relative convergence criterion to compute equilibria inside EB blocks. Namely, denoting $s(t)$ the state of a given EB block at the t-th fixed-point iteration, the fixed-point dynamics (Eq. 13 of our submission) are executed until $|(s(t + 1) - s(t))/ s(t)| < \epsilon$ where $\epsilon$ denotes some threshold. In spite of its simplicity, this trick was never employed in past EP works. Using this criterion, we can observe from Table 1 of the PDF attached to the rebuttal that ff-EBM training can be up to 4 times faster that with equivalent EBM, while maintaining and often improving performance.

To summarize: instead of the superlinear scaling of the convergence time of DHNs (with respect to the number of layers) empirically observed in past EP works [2--6], the convergence time ff-EBMs made up of DHNs are EB blocks are guaranteed, by construction, to scale linearly with the number of blocks, the convergence time of a single block decreasing with its size.

iii) How much energy savings does your method offer?

We kindly invite the Reviewer to read our global rebuttal for this question.

[1] Scellier, ICML 2024 [2] Ernoult et al, NeurIPS 2019 [3] Laborieux et al, Frontiers in Neuro., 2021 [4] Laborieux & Zenke, NeurIPS 2022 [5] Scellier et al, NeurIPS 2023 [6] Laborieux & Zenke, ICLR 2024

We thank Reviewer Jt5z for their honest feedback. We consider these clarifications important to highlight our contribution for readers.

i) Difference between this research and previous studies that have conducted training on analog in-memory systems using EP or backpropagation?

We detail two research trends and contrast them with our work:

a) Training analog systems using BP. In this context, analog hardware sustains feedforward models, with two main techniques:

• On-chip BP training: BP is directly executed on the analog hardware itself alongside inference. In general, forward and backward passes are necessarily executed on two distinct dedicated circuits. However, weights and activations must be transported from the inference circuit to the gradient computation circuit. Since weights/activations are encoded as noisy analog quantities (upon read and write operations), transporting them yields a device mismatch between these two circuits affecting in turn the resulting training performance [1].

• Off-chip BP training: a digital proxy of the analog system is trained by BP on GPUs and the resulting weights are mapped onto the analog system, yielding in turn an analog inference-only engine [2]. This approach circumvents the difficulty of on-chip training of analog systems.

b) Training analog systems using EP. When considering energy-based (EB) instead of feedforward models, the same circuit can be used for both the forward and backward passes. Therefore, when mapping an EB model onto a single analog circuit, the resulting system no longer suffers from device mismatch [1], with all the quantities required to compute gradients locally available at the same location, unlike BP. See global rebuttal and PDF for greater details.

c) This work. While it is clear how to map models small enough to fit single analog chip and with standard weight stationary operations such as fully connected layers onto energy-based, analog compute engines trainable by EP [3, 4], models spanning multiple analog chips alongside many feedforward operations (e.g. batchnorm) to be maintained in digital with high precision result in systems which may not be readily trainable by EP – see Fig 1 of our attached PDF. No existing work has tackled the extension of EP to this setting where EP may only be applied within some subparts of the model, with BP applied elsewhere, in a principled fashion.

We accordingly propose to add an extra related work section related to this literature.

ii) Examples of situations where ff-EBMs are needed?

See our global rebuttal and attached PDF. Additionally, the architecture considered for our experiments (L.213) is a very concrete example of this where it is unclear how to map batchnorm onto an energy-based, analog piece of hardware, and which may therefore be maintained in digital. Another example (mentioned L.304) would be transformer architectures – see Reviewer’s kDhg rebuttal and Fig. 2 of the PDF.

iii) Which parts of the overall system are digital and which are analog when using ff-EBM?

As explained in L.71-72, we “model digital and analog parts as feedforward and EB modules respectively”. This is further illustrated on Fig. 2, with “yellow and pink blocks denoting EB and feedforward transformations” and using the exact same notations, Eq. 12 explicitly defines the energy function of EB blocks and the feedforward transformations used for our experiments.

We propose to extend the color code used inside Fig. 2 to algorithms and equations to better emphasize which parts of the model and of the algorithm would be sustained in digital and analog

iv) Can existing training techniques can be used in these scenarios and what advantages ff-EBM offer compared to traditional methods?

• As explained in L. 229, ff-EBMs can be trained by traditional automatic differentiation (AD) through the root finding algorithm used to compute the equilibrium state of the ff-EBM (Eq. 13), which here reduces to Implicit Differentation (ID) – its implementation is detailed in Algorithm 12 below L. 605 inside the Appendix.

• About the advantages of EP-based training of ff-EBMs compared to ID or AD: we want to emphasize that AD is only used here for simulation benchmarking purposes to demonstrate that our algorithm achieves the best possible performance and would not be practical to deploy onto analog hardware for all reasons mentioned above. Namely, exactly as for BP as described previously, ID would require two separated circuits to compute the steady state of the ff-EBM on the one hand, and the associated Lagrangian multipliers on the other hand.

• Another advantage of ff-EBMs alone: they are faster to simulate than standard EBM counterparts (see Table 1 of the PDF and details inside Reviewer's).

We propose to add a more detailed mathematical definition of AD and ID when applied to ff-EBMs to better emphasize the benefits of EP-based training of ff-EBMs

v) Disregarding energy minimization, the proposed algorithm appears very similar to BP and therefore seems trivial

We assume that Reviewer Jt5z refers to the edge case where each EB block consists of a single layer such that the resulting ff-EBM is purely feedforward (“Recovering a feedforward net”, L.160), with our proposed algorithm reducing BP in this case (“Recovering backprop”, L.198, corollary A.5.1, Alg. 3 and Alg. 5 in appendix A.2). If so, we agree with Reviewer Jt5z on this particular edge case, and this is why we discarded it upfront for our experiments (L.201). However, in any other situation where EB blocks don’t reduce to a single layer, the proposed algorithm, deeply rooted in a Lagrangian-based approach thoroughly described in Appendix A.2, doesn’t trivially reduce to any known algorithm in the literature.

[1] Yi et al, Nature Electronics, 2022 [2] Wright et al, Nature, 2022 [3] Kendall et al, ArXiV, 2020 [4] Scellier, ICML 2024

Thank you very much for your careful and detailed response.

Figure 1 and Figure 2 in the attached PDF file clearly illustrate the hardware set-up and the proposed layer design. I acknowledge that this paper is pioneering in addressing the training algorithm for hardware composed of multiple analog chips, and I agree that this is a realistic configuration.

However, I still have a few questions regarding the novelty of this paper. The ff-EBM model architecture comprises a chain of the FF module and the EB module. Given that the training algorithms for the FF module and EB module are established conventions (backpropagation and equilibrium propagation, respectively), I find it challenging to identify the difficulty in designing a training algorithm for ff-EBM. Since the training algorithms for the FF and EB modules are predetermined, the primary task for ff-EBM is to ensure the proper passing of gradients between the FF and EB modules. Nonetheless, passing gradients between these modules appears straightforward by following the chain rule.

Therefore, I would like the authors to highlight any algorithmic challenges related to configuring the FF and EB modules within a single network architecture. More specifically, I would appreciate it if the authors could provide more detailed explanations on the challenges involved in passing gradients between the FF and EB modules and how these challenges are addressed.

Dear Reviewer Jt5z,

We thank you very much for engaging promptly in this discussion, and are happy indeed to read that the PDF file brought some satisfying clarifications about the relevance of the problem tackled.

If we understand you correctly, since BP and EP are well established algorithms in their own right, chaining them inside a given architecture appears straightforward “by following the chain rule”. In fact, we are very pleased that you have drawn attention to this issue, as we are increasingly recognizing that some of the terminology we use could prove misleading, and to some extent confusing for readers. Therefore, as requested, we highlight below the specific “algorithmic challenges” pertaining to this chaining, “how [they] are addressed” and more broadly the “novelty of this paper”.

• EP-BP chaining is intuitive, but not trivial to derive. You write that chaining EP and BP inside a given architecture “appears straightforward by following the chain rule”. We would like to focus attention on this question: what is meant by “following the chain-rule” in the context of ff-EBMs?

  • Traditionally, “chain-rule” refers to gradient chaining inside feedforward models. Namely, let us assume a computational graph of the form $s^1 = F(x) \to s^2 = G(s^1)$. Assuming an error signal $\delta^2 = \partial_{s^2} L$, then the “chain rule” prescribes that the error signal at $x$ reads $\partial_{x}L = \partial_x F(x)^\top \cdot \partial_{s^1}G(s^1)^\top \cdot \delta^2$.

  • Now let us assume instead the following “hybrid” computational path: $s^1 = F(x) \to s^2: \nabla_{s^2}E(s^2, s^1) = 0$. Given that $s^2$ in this case is an implicit function of $s^1$, i.e. there is no explicit G mapping as before between $s^1$ and $s^2$, how would one directly apply the above “chain-rule” here? To put it differently, how do we rigorously route error signals backward through this computational graph? This is the first “challenge” we addressed.

  • Hence the need to derive gradient chaining inside ff-EBMs rigorously, from first principles by: 1) stating the learning problem as a multilevel constrained optimization problem (Eq. 8), 2) writing the associated Lagrangian (Eq. 21), 3) solving for the associated KKT conditions (Eqs. 22-34) for the primal variables (i.e. the steady states of the blocks) and associated Lagrangian multipliers (i.e. the error signals inside blocks). Therefore, this is how “we addressed” the above challenge.

We emphasize that this derivation (namely Theorem 3.1 about the rigorous chaining of EP and BP gradients) and the resulting explicit and implicit chaining algorithms (Alg. 4 inside Appendix A.3, Alg. 2 in the main) are novel. Also, note from the above that our algorithm comes into an explicit and implicit variant (see paragraph “Proposed algorithm: implicit BP-EP chaining” L.187 and Lemma A.4), the latter appearing as a “pure” EP implementation. As such, the fact that EP-BP chaining can be cast into a non-trivial generalization of EP, as appearing in Alg. 2 and Lemma A.4, is not self-evident neither.

• The experimental demonstration of this algorithm is also novel. Finally, to further highlight the “novelty of this paper”, our algorithm was never tested in practice before this work and having it succeed on ImageNet32 came with lots of “challenges” as well, the most important one being the simulation time. We addressed this problem in two ways:

  • As explained in the global rebuttal and in greater details inside Reviewer k7Rs’s rebuttal, splitting an EBM into several EBM blocks tied by feedforward modules results in an architecture that is not only more hardware realistic, but also easier to simulate. Indeed, instead of the superlinear scaling of the simulation time with respect to the number of layers observed in past EP works, our approach guarantees, by construction, that this simulation time scales linearly with the number of blocks, each of these blocks converging much faster than the full EBM counterpart. See our new table of results inside our PDF attached to this rebuttal. While the ff-EBMs trained are still relatively shallow, they are twice as deeper as the deepest EBM trained by EP in most recent related works.

  • Finally, using Gaussian Orthogonal Ensembles (GOE) to initialize weights inside EB blocks was instrumental in having ff-EBM training experiments work.

Finally, we would like to propose the addition of a few sentences to our introduction highlighting this novelty, and particularly the degree to which our algorithm is derived by exploiting an intimate theoretical connection between energy based learning and implicit differentiation, rather than literal "backprop" as applied in standard feedforward nets where the standard "chain rule" applies. Hence our claim that this contribution belongs to the realm of EP, and implicit learning more broadly. Again, we thank you for drawing our attention to this.

Thank you for your detailed explanation.

Your response has helped me better understand the proposed work, and I have made every effort to assess its value.

Firstly, I am increasing my score to 5, as the novelty of your work is now clear to me. From my understanding, EP is designed to be applicable to any network architecture, including those with feedforward blocks, as the minimization of energy can be aligned with the minimization of the objective function (as shown in Figure 1 and Chapter 3 of the BP paper [1]). Therefore, I still view Eq. (8) as a straightforward integration of FF and EBM. However, I acknowledge that the gradient calculation starting from Eq. (8) is not trivial, and the detailed derivation of these gradients is a key novelty of your work.

While I believe this work is highly significant in the field of analog-based AI accelerator systems, I am hesitant to raise my score further because the presentation of your work could be improved to better highlight its true value.

In my opinion, the design of networks for analog-based computing is heavily constrained by hardware considerations, as there are significant challenges in scaling fully analog systems. Analog computing is known for its energy efficiency compared to digital computing, while digital computing offers greater scalability. To build an efficient yet scalable AI acceleration system, a hybrid design is essential. In this context, I believe the true value of your work lies in extending the scalability of EP-based models for analog computing by integrating FF modules. For this reason, I think that when adopting ff-EBM, the key concern should be scalability. The scalability of ff-EBM, as compared to using EBM alone, is a critical point that should be thoroughly explored. For example, with EBM alone, only a single analog-based unit in Figure 1 of attached PDF could be used for a single network. Obviously, ff-EBM should be much better than EBM, but I believe this paper needs detailed discussion on the accuracy of this EBM-based model and how it compares to the accuracy of a larger model designed with ff-EBM that fully utilizes the entire system depicted in Figure 1 of attached PDF, as this is an important aspect of this work.

[1] B. Scellier and Y. Bengio. Equilibrium propagation: Bridging the gap between energy-based models and backpropagation. Frontiers in computational neuroscience, 11:24, 2017.

We are very grateful to Reviewer Jt5z for spending time to understand the value of our paper, subsequently increasing our score and engaging in this discussion which will tremendously benefit the presentation of our work. Thank you so much!

In the light of Reviewer’s Jt5z last answer, we would like to clarify some essential points they raised:

• "EP is designed to be applicable to any network architecture, including those with feedforward blocks". As mentioned in Section 2.3 of our paper, EP only applies to energy-based models. Fig. 1 of the seminal EP paper [1] is indeed misleading: when writing “Equilibrium Propagation applies to any architecture”, one should understand “any architecture topology so long as it derives from an energy function”. Indeed, as indicated by the title of the section 3 of this paper, EP really is a “Machine Learning Framework for Energy-Based models”: their Figure 1 is only meant to emphasize that EP applies to any energy-based models, not necessarily layered energy-based models. In this context: layered does not meant feedforward. As Scellier & Bengio write themselves: “In particular, the [EP learning rule] holds for any architecture and not just a layered architecture (Figure 1) like the one considered by Bengio and Fischer (2015) [which is also an EB model]”.

• "Therefore, I still view Eq. (8) as a straightforward integration of FF and EBM". Given the clarification above, this conclusion may no longer hold, especially when noticing that Eqs.17-18 of the seminal EP paper [1] (in the section 3 mentioned by Reviewer Jt5z), i.e. the bilevel program the EP algorithm solves, is an explicit particular case of the Eq. 8 of our paper, i.e. the multilevel program that our algorithm solves.

• "I think that when adopting ff-EBM, the key concern should be scalability". We totally agree! Investigating the scalability of ff-EBM training by our algorithm on deeper ff-EBMs, more complex tasks and exploring new datasets and architectures is part of our research roadmap. See our detailed answer to Reviewer kDhg and associated Fig. 3 inside the PDF.

• "The scalability of ff-EBM, as compared to using EBM alone, is a critical point that should be thoroughly explored". We are happy Reviewer Jt5z mentions the importance of this comparison since our “splitting experiment” (Section 4.3) goes exactly in this direction. This experiment reveals that a single EBM block performs comparably to an ff-EBM of equal depth with various block sizes.

[1] Scellier, B., & Bengio, Y. (2017). Equilibrium propagation: Bridging the gap between energy-based models and backpropagation. Frontiers in computational neuroscience, 11, 24.