strengths: contributes a new perspective, covers a set of failure modes and extensions.

We would like to thank the reviewer for the possitive assesments about the novel perspective and the contributions of this article.

Weaknesses 1. significant concern about comparison to Clark et al. 2021. provide a genuine comparison, the same degree of parameter searching/sweeping, at least against BP

We understand the importance of ensuring that comparisons between different training methods are conducted under equivalent conditions to provide a fair and accurate assessment of their relative performance.

To address your concerns, we have taken the following steps:

• We re-implemented the existing methods, specifically Backpropagation (BP) and Direct Feedback Alignment (DFA), under the same training conditions as our proposed EBD algorithm.
• This includes using the same learning rate schedulers, number of epochs, regularizations, and performing a comparable degree of hyperparameter optimization for all methods.
• In our original submission, we noted that Clark et al., 2021 trained all their models for 190 epochs. In our experiments, we trained the MLP models for 120 epochs and the CNN and LC models for 100 epochs on MNIST and 200 epochs on CIFAR-10. For the revised experiments, we ensured that all methods (EBD, BP, and DFA) were trained for the same number of epochs specific to each architecture and dataset (for MLPs 120 epochs and for CNN/LC 100 epochs for MNIST and 200 epochs for CIFAR-10).
• We performed extensive hyperparameter tuning for BP and DFA, similar to what was done for EBD.
• Detailed hyperparameter settings and learning curves are provided in the appendix of the revised manuscript.
• The results of these new experiments are included in the updated Tables 1 and 2 of our manuscript, which are also available below:

MNIST Dataset: [x]: values from Clark et al., 2021 [ours]: our numerical experiments DFA+E: DFA with correlative entropy regularization

CIFAR-10 Dataset: [x]: values from Clark et al., 2021 [ours]: our numerical experiments DFA+E: DFA with correlative entropy regularization

• Based on these results, we observed significant improvements in the performance of BP and DFA compared to the results reported in Clark et al., 2021, due to the optimized training conditions.
• Despite these improvements, our EBD algorithm still demonstrates competitive performance: On the MNIST dataset, EBD achieves slightly better accuracy than DFA under the same conditions. On the CIFAR-10 dataset, EBD shows a more pronounced improvement over DFA.

We hope these new numerical experiments answers the concerns of the reviewer in terms of setting the optimization conditions of the algorithms to the same grounds.

Weakness 3. scalability of simulations

Our main contribution lies in offering a fresh theoretical backing for the error broadcast mechanism, which is believed to be a key process in biological learning, especially through the involvement of neuromodulators. While it is true that most biologically plausible frameworks face scaling issues, recent work by Launay et al. (2020) demonstrates that algorithmic enhancements can enable error broadcast approaches like Direct Feedback Alignment (DFA) to scale to large-scale, complex problems.

Our framework is essentially similar to DFA, with the crucial distinction that our broadcasting weights are adaptive rather than fixed. Given these similarities with DFA and the demonstrated scalability in Launay et al.'s work, we believe that our proposed method can likewise be extended to train more complex network structures effectively.

Ref: Launay, Julien, et al. "Direct feedback alignment scales to modern deep learning tasks and architectures." Advances in Neural Information Processing Systems 33 (2020): 9346-9360.

First, thank you for the detailed response - it is clear that you have put significant work into improving this work. Below I try to respond to all three sections of comments, first at an overview level, and then at a point-by-point level.

Overview: The work has been improved significantly in terms of empirical results. As an example, the results in Table 1/2/3 are now far more appropriate for comparison. However, the written form of the work leaves much still to be desired, especially the complete absence of a serious discussion or explanation of the drawbacks of this work and considerations that a reader should make when drawing conclusions. I remain believing that this idea is an important contribution to the field, however there remain a number of issues with the presentation that I believe it should be improved further before acceptance (i.e. the form does not yet meet the quality level in my opinion). Based upon current changes, my score has improved by one point but not more.

Point by Point

• The results of Table 1 are much improved by a re-implementation and are now more comparable. They are however presented in an unorthodox way: bold values generally indicate the best performing model (with underscores used for second best). In this Table, BP should be bold and in many cases your methods as underscores. This should be corrected for reader clarity.
• Scalability is ignored in favour of DFA's application in existing work to other domains (Launey et al. 2020). However, I do not believe that this applies to the current work. There has been a great proliferation of bio-plausible learning rules in recent years, and based upon the work by Bartunov et al. (2018) I believe that we should test all proposals against truly high-dimensional tasks such as ImageNet to verify that they are capable of credit assignment in such a regime. I believe that this would take some time to implement but is important.
• The addition of an appendix to better illuminate how entropy updates can be computed without an inverse is helpful. However, given the complexity of this argument on biological plausibility, I believe that the main text could be much more clear in outlining how and why this implementation of CorrInfoMax is indeed more biologically plausible. At present this is rushed and unclear.
• Modifications to the explanation of EBD are appreciated and it is now a clearer read.
• The implications for an alternative loss function (CCE) are now present in an appendix, however this appendix is never referred to in the main text. This is a good demonstration of the way in which important considerations for application of this rule are not fully discussed in the main text. I am aware that the page limit is ... well ... limiting. But it is important to provide such context to make this paper of high quality.
• Thank you for the addition of runtime comparisons, these are additionally useful.

Future Score Improvements: To be clear about what I would need for a higher future score, I have included this section. Most importantly, a wider perspective on the explanation, implications, and drawbacks of this work is necessary. This would mean a re-writing of the narrative of the work, with more focus on bringing the reader along through both the positive contributions (more details on bio-plausibility and a clearer overall narrative), but also balanced with the potential remaining open-avenues (e.g. a real discussion of the implication of other loss functions) and remaining open questions. This would complete the written side of this work to a standard of having an acceptance score. To go further beyond this (in terms of score), I would request an application of the algorithm to a much more challenging task than those presented currently (specifically ImageNet classification would suffice) and for an excellent score an application to even other more general tasks, e.g. language models/transformers or such.

• Limitations: Lastly, we revised the Limitations section and added the final sentence, highlighted in bold, to more clearly articulate our paper's position on scalability. This addition frames scalability as an important area for future exploration:

  "Limitations. The current implementation of EBD involves several hyperparameters, including multiple learning rates for decorrelation and regularization functions, as well as forgetting factors for correlation matrices. Although these parameters offer flexibility, they add complexity to the tuning process. Additionally, the use of dynamically updated error projection matrices and the potential integration of entropy regularization may increase memory and computational demands. Future work could explore more efficient methods for managing these components, potentially automating or simplifying the tuning process to enhance usability. Furthermore, while the scalability of EBD is left out of the focus of the article, we acknowledge its importance. Launay et. al. (2021) demonstrated that DFA scales to high-dimensional tasks like transformer-based language modeling. Since DFA is equivalent to EBD with frozen projection weights and without entropy regularization, we anticipate that EBD could scale similarly. However, this remains unvalidated empirically. Examining EBD’s scalability and streamlining its components to improve usability are important tasks for future work."

---

In conclusion, we appreciate the reviewer’s feedback and have carefully addressed the concerns about narrative. The revised manuscript aims to present a more balanced perspective, clearly outlining the scope and limitations of this work. By framing scalability as a future direction rather than an immediate claim, we hope to make the paper’s contributions and positioning clearer while setting the stage for future research that builds on these findings.

References:

• Clark, et. al. "Credit assignment through broadcasting a global error vector.", Neurips, 2021.
• Bozkurt, et. al. "Correlative information maximization: a biologically plausible approach to supervised deep neural networks without weight symmetry.", Neurips, 2023.
• Kao et. al., "Counter-Current Learning: A Biologically Plausible Dual Network Approach for Deep Learning", Neurips, 2024.
• Dellaferrera et al., "Error-driven Input Modulation: Solving the Credit Assignment Problem without a Backward Pass", ICML, 2022

Thank you for your constructive feedback on our manuscript. We genuinely appreciate your continued engagement and the opportunity to clarify and improve our work. We understand your concerns regarding scalability and the clarity of our article’s narrative on this aspect.

Our primary goal in this article is to present a novel theoretical framework for learning. This framework provides principled underpinnings for both error broadcast-based learning and three-factor learning—mechanisms that are believed to play crucial roles in biological networks. By rooting our approach in a major orthogonality property from estimation theory—the nonlinear orthogonality principle—we aim to lay down the foundational aspects of this new learning framework. This approach is in line with several recent works that focus mainly on deriving principled methods for biological and artificial learning mechanisms (e.g., Clark et al. (2021), Bozkurt et al. (2023), Dellaferrera, et al. (2022) and Kao et al. (2024)).

To address your concern about the clarity of our claims regarding performance and scalability, we have carefully revised the manuscript to ensure it accurately reflects the intended scope. We emphasize that while scalability is an important future direction, our present work is centered on establishing a new learning mechanism based on the nonlinear orthogonality principle.

In response to your feedback, we have revised key parts of the abstract, introduction, and conclusion to clarify that the current article does not focus on scalability. Instead, we highlight that scalability is a valuable extension of our proposed theoretical framework.

Below, we describe the specific changes made in the latest revision to address your concerns:

• Abstract: We have revised the abstract and added the bolded sentence and phrase:

  "We introduce the Error Broadcast and Decorrelation (EBD) algorithm, a novel learning framework that addresses the credit assignment problem in neural networks by directly broadcasting output error to individual layers. The EBD algorithm leverages the orthogonality property of the optimal minimum mean square error (MMSE) estimator, which states that estimation errors are orthogonal to any nonlinear function of the input, specifically the activations of each layer. By defining layerwise loss functions that penalize correlations between these activations and output errors, the EBD method offers a principled and efficient approach to error broadcasting. This direct error transmission eliminates the need for weight transport inherent in backpropagation. Additionally, the optimization framework of the EBD algorithm naturally leads to the emergence of the experimentally observed three-factor learning rule. We further demonstrate how EBD can be integrated with other biologically plausible learning frameworks, transforming time-contrastive approaches into single-phase, non-contrastive forms, thereby enhancing biological plausibility and performance. Numerical experiments demonstrate that EBD achieves performance comparable to or better than known error-broadcast methods on benchmark datasets. The scalability of algorithmic extensions of EBD to very large or complex datasets remains to be explored. However, our findings suggest that EBD offers a promising, principled direction for both artificial and natural learning paradigms, providing a biologically plausible and flexible alternative for neural network training with inherent simplicity and adaptability that could benefit future developments in neural network technologies."

---

• Introduction: In the concluding paragraph of the introduction, we added a sentence to highlight scalability as an open question:

  "We demonstrate the utility of the EBD algorithm by applying it to both artificial and biologically realistic neural networks. While our experiments show that EBD performs comparably to state-of-the-art error-broadcast approaches on benchmark datasets, offering a promising direction for theoretical and practical advancements in neural network training, its scalability to more complex tasks and larger networks remains to be investigated."

---

• Related Work and Contribution: In the final paragraph of Section 1.1, "Related Work and Contribution," we softened the language to temper claims about the advantages of our approach:

  "In summary, our approach provides a theoretical grounding for the error broadcasting mechanism and suggests ways to its effectiveness in training networks."

Strengths: well written, interesting theoretical block, certainly deserves attention, of value to both ML and neuroscience, approach well-thought out, numerical results apper impressive

We really appreciate the positive assessment by the reviewer especially about the significance of our article's contributions. We would like to also thank you for the detailed review and the constructive comments.

Weakness 1: dense paper harming accessibility, section 3.2 difficult to grasp, why CorInfoMax-EBD more bio-plausible, and what is new in CorInfoMax-EBD

We have revised article to address the concerns of the reveiewer to improve its accessibility. For this purpose,

• We moved the EBD extensions section, which was plased after the section on the bioplausible CorInfoMax-EBD, to Section 2 where the EBD method is introduced.
• We also moved some details about forward broadcasting to the appendix to improve readability.
• We added a new appendix (Appendix D) explaining bio-plausible CorInfoMax-EP model of Bozkurt et al., 2024. This new appendix section helps understanding the derivation and the operation of the CorInfoMax network dynamics as well as the equilibrium propagation based learning dynamics applied to the CorInfoMax network. This would clarify the Section 3.2 on the application of the EBD approach to reduce two phase learning to a single phase.
• In the main article, we have provided additional explanations and clarifications in each section, as much as the page limit allowed. (Please see the common response above for further details.)
• We have included cross-references to the new appendix sections to guide readers who wish to delve deeper into the technical details.

Regarding CorInfoMax-EBD being more biologically plausible than the MLP based method in Section 2:

• Online Operation vs. Batch Mode: The MLP-based implementation in Section 2 requires batch mode and doesn't ensure biologically plausible entropy-gradient updates. In contrast, the CorInfoMax network operates in an online setting, can work with batch sizes as small as 1, and integrates entropy gradients into lateral weights, making the updates more biologically realistic.
• Biologically Realistic Architecture: CorInfoMax includes lateral (recurrent), feedforward, and feedback connections, mirroring biological neural networks. It employs a three-compartment neuron model—soma, basal dendrite, and apical dendrite—which better captures biological reality compared to standard artificial neurons. In this model, apical dendrites receive feedback, basal dendrites receive feedforward information, and prediction errors are represented by differences in membrane potentials.
• To clarify these points, in Section 3.2, we added "The CorInfoMax-EBD scheme proposed in this section is more biologically realistic than the MLP based EBD approach in Section 2 due to multiple factors: Unlike the batch-mode operation required by the MLP-based EBD, CorInfoMax operates in an online optimization setting which naturally integrates entropy gradients into lateral weights, resulting in biologically plausible updates, whereas the MLP approach uses entropy regularization without ensuring biological plausibility. Besides, it employs a neuron model and network architecture that closely mirror biological neural networks."

We were not able to go into further details due to space limitations.

We believe these revisions enhance the paper's clarity and better highlight our main concepts and contributions.

Thank you authors for your comprehensive reply.

Performance wise, I'm impressed with the additional simulations that have been carried out and am convinced by the emperical results. I also think that the fact CorInfoMax-EBD can operate in a batch size = 1 does improve its bio plausibility. As I originally stated, I also repeat that I think the paper's central concepts are interesting and should be encouraged in the field of computational neuroscience.

Based on this I will increase my score. That said, even after various notation fixes and the addition of individual sentences (which I do appreciate), I do slightly worry that the structure/delivery of the paper makes it difficult to grasp for a non-expert, and I do agree with VBft's comments that future efforts should prioritise the overall narrative/story of the work.

Dear Reviewer, We would like to thank you for the comprehensive review. Before drafting our response, we want to ensure we fully understand the reviewer’s points. Therefore, we kindly request clarification on the following:

In Weakness Point 2 and Question 2, the reviewer suggests that the performance gains reported in our article may result from 'orthogonalization of layer representations' and/or 'orthogonal representations of different classes.' We would be grateful if the reviewer could clarify precisely what is meant by these terms and indicate any mechanisms within our method that might be contributing to this effect. We note that in our framework, 'orthogonality' refers to being uncorrelated in a statistical sense. However, we believe the reviewer’s use of orthogonality pertains to vector orthogonality with respect to the Euclidean inner product.

Additionally, could the reviewer provide more details about the exact implementation of the “comparative baseline” that we are asked to compare against: for example, what is meant by and how is local class orthogonalization implemented in each layer? Is there an available article/codebase that the reviewer can point us to for this comparison baseline.

Strengths: Compelling Biological Motivation, significant promise of error-broadcasting, improved performance, intriguing theoretical foundation, showcase practicality

We would like to thank the reviewer for all of these positive comments about the novel contributions of our article.

Weakness 1 and Question 1: problematic reverse use of orthogonality principle.

We appreciate the reviewer's comment, which give us an opportunity to clarify the main contribution point of our paper. In linear Minimum Mean Square Error (MMSE) estimation, the orthogonality principle states that the estimation error is orthogonal to the observations and their linear functions. Mathematically, this is expressed as $E(\epsilon_* \mathbf{x}^T)=\mathbf{0}$. Using this orthogonality condition in reverse to derive linear estimators is a standard practice in the field (see, for example, Kailath et.al. 2000) . Techniques like Kalman Filtering are based on this principle, which is firmly grounded in the Hilbert space projection theorem.

For nonlinear MMSE estimation, the orthogonality condition is even stronger: the estimation error is orthogonal to any nonlinear function of the input. Exploiting this stronger condition to construct nonlinear MMSE estimators is an open problem, primarily because it raises questions about which nonlinear functions to choose and how many are needed.

In our work, we model the neural network as a parameterized nonlinear MMSE estimator and seek as many equations from the orthogonality principle as possible to determine these parameters. This is exactly the same principle as how the orthogonality condition is used in reverse to find parameters for linear estimators. To address the challenge of selecting nonlinear functions that yield informative equations for determining network parameters, we choose the activations of the hidden layers in the neural network as these functions. This choice is natural because these activations are directly related to the network's parameters through differentiation. By enforcing orthogonality between the estimation error and the hidden layer activations, we create a framework that effectively guides the learning process. Our empirical results support the feasibility and advantages of this approach in training neural networks.

Further refinement of this methodology—such as integrating additional nonlinear functions of the layer activations and inputs—not only holds significant potential for enhancing the model's capabilities but also represents an exciting avenue for future research.

To enhance clarity on this matter, we have revised Section 2.2 to include the clarifications discussed above. We hope that, based on the reviewer's suggestion, the article now better positions the proposed method.

Reference: Kailath, Thomas, Ali H. Sayed, and Babak Hassibi. Linear estimation. Prentice Hall, 2000.

Weakness 1 part 2: the hidden layers is very large, a solution to be orthogonal to a single direction is weak

We believe there is a misunderstanding regarding our use of the term "orthogonality" and how it applies within the context of our methodology, especially in high-dimensional hidden layers.

In our paper, "orthogonality" refers to the statistical condition where two scalar random variables are uncorrelated. When we state the orthogonality between a hidden layer vector and the output error vector, we are implying that each unit in the hidden layer is uncorrelated with each component of the error vector. This is not a constraint to a single direction in a high-dimensional space. Contrary to that, it is a set of multiple orthogonality conditions applied element-wise between hidden layer activations and error components. In fact, for a hidden layer with $𝑛$ units and an error vector of dimension $𝑚$, there are $n\times m$ orthogonality consraints. As the hidden layer size scales, so does the number of orthogonality constraints, ensuring that the learning process remains robust and well-defined in high-dimensional settings. By imposing these multiple orthogonality conditions, we are not facing dimensional degeneracy or a lack of directional information. Instead, we are enhancing the network's ability to learn meaningful representations by ensuring that each hidden unit contributes uniquely to reducing the output error.

The success of our numerical experiments substantiates the effectiveness of our methodology. The performance improvements we observe are directly attributable to the enforcement of these orthogonality conditions across all hidden units and error components.

Thank you for your kind and detailed follow-up. We very much appreciate your thoughtful critique.

Point-1 and Point-2: Applying the orthogonality principle in reverse

Thank you for restating your question; it has helped clarify your perspective on our side. We indeed agree that in neural networks, the number of parameters typically exceeds the number of samples, leading to a system that is underconstrained in a classical sense. To directly address your question, our method does not assume that the number of samples matches the number of parameters.

In our paper, we define a decorrelation condition involving the number of neurons multiplied by the number of outputs. The orthogonality principle in our method is defined as the uncorrelatedness of a given hidden layer neuron’s activation with all components of the output error separately. Specifically, for the $i^{th}$ neuron of layer-$k$ and the $j^{\text{th}}$ component of the output error, $\epsilon_j$, the orthogonality condition is expressed as, $R_{h_i^{(k)}\epsilon_j}=E(h_i^{(k)}\epsilon_j)=0, \hspace{0.2in} j=1, \ldots m, \hspace{0.1in} i=1, \ldots n \hspace{0.2in} (\text{Eq.A})$ where $m$ is the number of output components, and $n$ is the size of activations for layer $k$.

Based on the (Eq.A) above, for each hidden layer, there are $m$ x $n$ orhogonality constraints. Therefore, even if the hidden layer dimensions and/or the network depth increase, the total number of constraints also increases, making our system less underdetermined. We use constraints for different neurons separately to adjust their corresponding weight/bias parameters. In other words, the constraints in (Eq. A) are used to update the $i^{\text{th}}$ row of $\mathbf{W}^{(k)}$, i.e. $\mathbf{W}_{i,:}^{(k)}$, and the bias compoent $b_i$. To show this numerically, for a linear layer with 1024 units and error dimension of 10, there are 1024x10 = 10240 orthogonality conditions, and for a linear layer with 4096 units and error dimension of 10 there are 4096x10=40960 orthogonality conditions.

Note that, more generality of the orthogonality condition for nonlinear estimators offers potential to increase the number of constraints per hidden layer neuron: We can increase the number of the orthogonality conditions per neuron even further by considering the fact that uncorrelatedness requirement is for any function $g$ of hidden layer neuron activations, i.e., $R_{g(h_i^{(k)})\epsilon_j}=E(g(h_i^{(k)})\epsilon_j)=0, \hspace{0.2in} j=1, \ldots m \hspace{0.2in} (\text{Eq.B})$ Therefore, the number of uncorrelatedness (orthgonality) constraints per hidden layer/output neuron can be increased by introducing multiple $g$ functions. (However, in our numerical experiments we haven't pursued this path.)

Although the orthogonality conditions scale with the increasing parameter size, the total number of parameters in the network is in general larger than the number of decorrelation conditions. This results in fewer constraints than parameters, leading to an overparameterized system, where a unique optimal estimator cannot be determined solely based on these conditions. Your concern about infinite samples or learning time in overparameterized networks is valid, but our results show that the learning rule converges effectively within practical timeframes. Particularly in the case of using Locally Connected Networks (LC) (Refer to Table-1,2 in the paper), which are highly overparameterized, the improved performance and generalization observed strongly validate the practicality of our approach to successfully train in the overparametrized regime.

Importantly, this issue of overparameterization also exists in standard backpropagation, where the number of parameters often exceeds the number of training samples, leading to an overparameterized and underdetermined system. In both cases, this overparameterization does not hinder learning; rather, it is a fundamental characteristic of deep learning. Research has demonstrated that the implicit bias in gradient descent introduces a regularization effect, steering the optimization process toward solutions that generalize well to unseen data (Soudry et. al, 2018).

Reference: Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S., & Srebro, N. (2018). The implicit bias of gradient descent on separable data. ICLR 2018

Similarly, in our method, while the number of orthogonality constraints is smaller than the total number of network parameters, the system is guided by the statistical properties of the error and activations. While we cannot claim to fully characterize the implicit regularization effect in our method, we suggest that these statistical constraints play a role similar to the implicit regularization observed in regular backpropagation. This helps ensure that the learned parameters are not arbitrary but are shaped by the decorrelation principles inherent to our framework, contributing to the model’s generalization capabilities. We believe that investigating the inherent implicit bias in Error Broadcast and Decorrelation (EBD) opens the door to further understanding how this framework naturally regularizes the learning process.

We would also like to highlight that our method imposes even more constraints than Direct Feedback Alignment (DFA), where the feedback weights are fixed matrices. In contrast, EBD learns these matrices directly from the data as the cross-covariance matrices between the error and hidden layers, making it more scalable and data-dependent, even in the finite data regime. Considerig our EBD mechanism as a generalization of the DFA framework, it becomes clearer that the orthogonality condition is neither simplistic nor underconstrained. Because when we turn off the update on the cross-covariances, and leave it to initialization, it becomes exactly equal to DFA method (apart from entropy max. and power norm.).

To further adress limited-data problems, our method incorporates several regularization techniques:

• Entropy regularization: Encourages the network to utilize the full feature space by spreading activations.
• Sparsity regularization: Enforces sparse activations to reduce redundancy.
• Weight decay: Prevents overfitting by penalizing large weights. These regularizers supplement the orthogonality-based learning rule, particularly in the limited-data regime, improving generalization and stability.

Based on your suggestions, to clarify this point, in the new revision of our article we included a new appendix section (Appendix E.9), with the title "On the scaling of the orthogonality conditions", discussing how the number of orthogonality conditions increase with the increasing model size, and the possible implicit and the existing explicit regularizations on our algorithm.

Point-3: The use of entropy and power regularizers

Thank you again for raising this important point. We think clarifying this also in the main paper increased the clarity of our main contributions.

Point-4: On the vagueness of biological plausibility

Thanks for this comment. It is true that the term biologically plausible is somewhat vague; this partly stems from the fact that what constitutes a biologically plausible learning rule is not fully settled in neuroscience. Here, we adopted the following principles which are largely accepted among neuroscientists:

• Local Learning Rules: Our method uses the orthogonality principle to enforce local relationships between the estimation error and hidden layer activations. This locality mimics the way biological systems may rely on local interactions for learning, as opposed to requiring global error signals.

• Three Factor Learning Rule: There exists several experimental resutls supporting the existence of three factor learning mechanism in biological networks. Furthermore, our EBD framework provides a principled derivation of this rule.

• Training without weight symmetry: Our method avoids the biologically implausible assumption of symmetric feedforward and feedback weights. Instead, it dynamically learns feedback matrices, mirroring the independence of connections seen in biological neural networks, unlike regular backpropagation.

• Lateral, feedforward and feedback connections: Experimental studies indicate that biological neural networks incorporate feedforward, feedback, and lateral connections among neurons. Our CorInfoMax-EBD framework provides a principled method for designing neural networks with these connectivity patterns, where output errors are propagated through adaptable feedback connections.

• More Realistic Neuron Model: Our method parallels the three-compartment neuron model, where dendrites receive input signals, the soma processes these signals, and axons transmit output signals. In our method, the hidden layer activations represent the dendritic computations, while the feedback connections act as modulatory signals akin to axonal inputs in biological neurons, just like our CorInfoMax-EBD model.

• Online learning: As you also pointed out, batch learning does not closely mirror learning in biological systems, which typically process data in an online and sequential manner.

Our contribution, as the reviewer correctly points out, is not solving learning in the brain, but to provide a learning rule that is both mathematically principled and aligned with widely agreed-upon biological principles as much as possible.

We would like to thank the reviewer for their active engagement during the discussion period. Below, we provide our responses to the reviewer's comments:

Applying the orthogonality principle on a per-neuron basis appears to be incorrect. Therefore, the assertion that you have $m \times n$ conditions is flawed, and the problem of finding the optimal estimator remains highly under-constrained.

We would like to clarify that applying orthogonality condition per-neuron basis is correct:

(i). First, we need to clarify the terminology: in this context, "orthogonality" refers to statistical uncorrelatedness. The geometric term "orthogonality" is defined within a Hilbert space of random variables, where the inner product between two random variables $a,b$ is defined as

$\langle a , b \rangle \stackrel{\Delta}{=} E(ab)$. (Eq. A)

That is the inner product, which is defined as the expected value of their product, i.e., their correlation. Two random variables $a,b$ are said to be orthogonal if their inner product is zero:

$\langle a , b \rangle {=} E(ab)=0$. (Eq. B)

Therefore, two random variables are called orthogonal if their correlation is zero (see, for example, (Kailath et al., 2000), Chapter 3.)

In our framework, each individual neuron in a given layer and each error component are modeled as random variables, with realizations varying across dataset samples. Defining the correlation on a per-neuron basis is thus appropriate and well-grounded in statistical theory.

---

(ii). Second, we clarify the orthogonality theorem related to nonlinear minimum mean square error (MMSE) estimation. As stated in our article (and proved in Appendix A), the MMSE orthogonality condition can be expressed in more detail (Papoulis & Pillai, 2002):

Let $\hat{\mathbf{y}}\in \mathbb{R}^p$ be the optimal nonlinear MMSE estimate of the desired vector $\mathbf{y}\in \mathbb{R}^p$ given the input $\mathbf{x} \in \mathbb{R}^m$. Let $\mathbf{e}_*=\mathbf{y}-\hat{\mathbf{y}}$ denote the corresponding estimation error. Then, for any poperly measurable function $\mathbf{g}(\mathbf{x})\in \mathbb{R}^q$ of the input, we have

$E(\mathbf{g}(\mathbf{x}){\mathbf{e}_*}^T) = \mathbf{0}\_{q \times p}$. (Eq. C)

In other words, the cross correlation matrix of the nonlinear function of the input, $\mathbf{g}(\mathbf{x})$ and the output error for the best MMSE estimator, $\mathbf{e}_*$ is equal to a $q \times p$ zero matrix. The matrix equation in (Eq. C), can be more explicitly written as $q\cdot p$ orthogonality conditions:

$E(g_i(\mathbf{x}){e_*}_j)=0$ for $i=1, \ldots, q$ and $j=1, \ldots, p$. (Eq. D)

In other words, the correlation between each component of $\mathbf{g}(\mathbf{x})$ and each component of $\mathbf{e}_*$ is equal to $0$. Using the Hilbert space terminology from item (i), the random variables $g_i(\mathbf{x})$ and $e\_{*j}$ are "orthogonal" to each other for any choice of $i\in \{1, \ldots, q\}$ and $j \in \{1, \ldots, p\}$.

---

(iii). In our framework, we pose a neural network as a MMSE nonlinear estimator. Let $\mathbf{e}\in \mathbb{R}^p$ represent the output error (when there are $p$ outputs, e.g., $p=10$ for CIFAR 10) for the neural network that corresponds to the optimal MMSE estimator. Then we can pick the arbitrary nonlinear functions of the input as hidden layer activations $\mathbf{h^{(k)}}\in\mathbb{R}^{N^{(k)}}$, where $N^{(k)}$ is the number of neurons in hidden layer $k$. Then by (Eq. D) above

$E(h^{(k)}_i(\mathbf{x})e_j)=0$ for $i=1,\ldots, N^{(k)}$ and $j=1, \ldots, p$. (Eq.E)

In other words, the correlation between the random variables $i^{\text{th}}$ neuron of layer $k$, i.e., $h_i^{(k)}$ and the $j^{\text{th}}$ component of the output error $e_j$ is zero. So, using the terminology defined in item (i) above, we call that the random variables $h_i^{(k)}$ and $e_j$ are "orthogonal" to each other (for any choice of $i \in \{1, \ldots, N^{(k)}\}$ and $j \in \{1, \ldots, p\})$.

Consequently, for each hidden layer neuron activation $h_i^{(k)}$, there exists $p$ orthogonality conditions, which we can write

$E(h^{(k)}_i(\mathbf{x})e_j)=0$ for $j=1, \ldots, p$. (Eq.F)

As a result, for the hidden layer $k$, where there are $N^{(k)}$ neurons, with $p$ orthogonality conditions per neuron, there are $N^{(k)} \cdot p$ orthogonality conditions in total. Therefore, there is no error or flaw in our arguments in our paper or responses.

---

We believe the potential misunderstanding may stem from confusing the orthogonality of random variables (defined via the correlation inner product in item (i)) with the orthogonality of Euclidean vectors in $\mathbb{R}^{n}$, defined by the standard Euclidian inner product

$\langle \mathbf{j},\mathbf{r} \rangle=\mathbf{r}^T\mathbf{j}$. (Eq. G)

If the reviewer still believes there is an issue, we kindly request a precise indication of which item, equation, or statement above is of concern so we can provide further clarification.

Strengths: novel idea, the orthogonality property, avoids weight symmetry, better theoretical ill. & performance than DFA.

We would like to thank the reviewer for the positive assessment about the main contributions of our article.

Weakness 1: critical issue of scaling up, as most of non-BP learning frameworks exist

Our main contribution lies in offering a fresh theoretical backing for the error broadcast mechanism, which is believed to be a key process in biological learning, especially through the involvement of neuromodulators. While it is true that most biologically plausible frameworks face scaling issues, recent work by Launay et al. (2020) demonstrates that algorithmic enhancements can enable error broadcast approaches like Direct Feedback Alignment (DFA) to scale to large-scale, complex problems.

Our framework is essentially similar to DFA, with the crucial distinction that our broadcasting weights are adaptive rather than fixed. Given these similarities with DFA and the demonstrated scalability in Launay et al.'s work, we believe that our proposed method can likewise be extended to train more complex network structures effectively.

Ref: Launay, Julien, et al. "Direct feedback alignment scales to modern deep learning tasks and architectures." Advances in Neural Information Processing Systems 33 (2020): 9346-9360.

Weakness 2. experiments: slightly better than DFA, not comparable with SOTA Hebbian

While the performance on a machine learning tasks is an important indicator, it may not be the only or most significant metric in evaluation of learning frameworks. In biologically realistic models, capturing more realism and providing mathematical explanations for the existing phenomena are also important criteria. The proposed error-broadcast decorrelation may not achieve state of the art performance, with the existing choice of architecture and parameter settings, however, numerical experiments demonstrate that it performs on par or better than the existing biologically plausible error-broadcast approaches (Note that Journe et al,, 2023 is not a error broadcasting approach.). Due to its groundedness on the estimation theoretic orthogonality principle, its potential expressive power for the mathematical modelling of error-broadcast based learning through neuromodulators holds a signifcant value. Furthermore, the proposed bio-plausible CorInfoMax-EBD approach captures several features of the biological neural networks such as lateral connections, feedforward and feedback connections, multi-compartment neuron models and three factor update rule as a result of the application of the EBD method.

Question 1. why low BP performance in Table 2

For Table 2, we used results (including BP) from Clark et al. (2021) for all algorithms except our EBD algorithm. To clarify this point and ensure consistency, we have now performed our own BP and DFA training implementations. For the revised experiments, we ensured that all methods (EBD, BP, and DFA) were trained for the same number of epochs specific to each architecture and dataset (for MLPs 120 epochs and for CNN/LC 100 epochs for MNIST and 200 epochs for CIFAR-10). We performed extensive hyperparameter tuning for BP and DFA, similar to what was done for EBD.

The following is the updated version of Table 2 containing new simulation results for DFA, DFA with entropy regularization (DFA+E) and BP.

Table 2: CIFAR-10 Dataset: [x]: values from Clark et al., 2021 [ours]: our numerical experiments DFA+E: DFA with correlative entropy regularization

According to this table, the BP values from our new experiments with optimized hyperparameters are higher than those reported in Clark et al. (2021). In this article, we used these new values in Table 2.

---

• Introduction: In the concluding paragraph of the introduction, we added a sentence to highlight scalability as an open question:

  "We demonstrate the utility of the EBD algorithm by applying it to both artificial and biologically realistic neural networks. While our experiments show that EBD performs comparably to state-of-the-art error-broadcast approaches on benchmark datasets, offering a promising direction for theoretical and practical advancements in neural network training, its scalability to more complex tasks and larger networks remains to be investigated."

---

• Related Work and Contribution: In the final paragraph of Section 1.1, "Related Work and Contribution," we softened the language to temper claims about the advantages of our approach:

  "In summary, our approach provides a theoretical grounding for the error broadcasting mechanism and suggests ways to its effectiveness in training networks."

---

• Limitations: Lastly, we revised the Limitations section and added the final sentence, highlighted in bold, to more clearly articulate our paper's position on scalability. This addition frames scalability as an important area for future exploration:

  "Limitations. The current implementation of EBD involves several hyperparameters, including multiple learning rates for decorrelation and regularization functions, as well as forgetting factors for correlation matrices. Although these parameters offer flexibility, they add complexity to the tuning process. Additionally, the use of dynamically updated error projection matrices and the potential integration of entropy regularization may increase memory and computational demands. Future work could explore more efficient methods for managing these components, potentially automating or simplifying the tuning process to enhance usability. Furthermore, while the scalability of EBD is left out of the focus of the article, we acknowledge its importance. Launay et. al. (2021) demonstrated that DFA scales to high-dimensional tasks like transformer-based language modeling. Since DFA is equivalent to EBD with frozen projection weights and without entropy regularization, we anticipate that EBD could scale similarly. However, this remains unvalidated empirically. Examining EBD’s scalability and streamlining its components to improve usability are important tasks for future work."

---

In conclusion, we appreciate the reviewer’s feedback and have carefully addressed the concerns about narrative. The revised manuscript aims to present a more balanced perspective, clearly outlining the scope and limitations of this work. By framing scalability as a future direction rather than an immediate claim, we hope to make the paper’s contributions and positioning clearer while setting the stage for future research that builds on these findings.

References:

• Clark, et. al. "Credit assignment through broadcasting a global error vector.", Neurips, 2021.
• Bozkurt, et. al. "Correlative information maximization: a biologically plausible approach to supervised deep neural networks without weight symmetry.", Neurips, 2023.
• Kao et. al., "Counter-Current Learning: A Biologically Plausible Dual Network Approach for Deep Learning", Neurips, 2024.
• Dellaferrera et al., "Error-driven Input Modulation: Solving the Credit Assignment Problem without a Backward Pass", ICML, 2022

We appreciate the reviewer's constructive feedback and for acknowledging that our work presents an interesting idea. We understand the concerns regarding the narrative of our paper. In response to the reviewer's suggestions, we have revised the manuscript to better address the limitations and open questions of our method.

I raise the example of Journe et.al on Hebbian learning is because my intuition is that an error broadcast method should perform better than such a pure local learning like Hebbian as the former has a (implicit) global error to guide the learning, maybe the authors could have different explanations?

Currently, unfortunately, we lack a rigorous explanation of how a feedback-free Hebbian approach—where hidden layer weights are trained using an unsupervised Hebbian mechanism and then frozen, and only the final layer is trained with a supervised mechanism—can achieve better performance than error feedback methods.

Overall, as other reviewers mentioned, either the authors should provide more evidences on the scale up capability of this method, or the authors should revise the current manuscript in a better narrative on the limitations and open-questions of this method.

Our primary goal in this article is to present a novel theoretical framework for learning. This framework provides principled underpinnings for both error broadcast-based learning and three-factor learning—mechanisms that are believed to play crucial roles in biological networks. By rooting our approach in a major orthogonality property from estimation theory—the nonlinear orthogonality principle—we aim to lay down the foundational aspects of this new learning framework. This approach is in line with several recent works that focus mainly on deriving principled methods for biological and artificial learning mechanisms (e.g., Clark et al. (2021), Bozkurt et al. (2023), Dellaferrera, et al. (2022) and Kao et al. (2024)).

To address your concern about the clarity of our claims regarding performance and scalability, we have carefully revised the manuscript to ensure it accurately reflects the intended scope. We emphasize that while scalability is an important future direction, our present work is centered on establishing a new learning mechanism based on the nonlinear orthogonality principle.

In response to your feedback, we have revised key parts of the abstract, introduction, and conclusion to clarify that the current article does not focus on scalability. Instead, we highlight that scalability is a valuable extension of our proposed theoretical framework.

Below, we describe the specific changes made in the latest revision to address your concerns:

• Abstract: We have revised the abstract and added the bolded sentence and phrase:

  "We introduce the Error Broadcast and Decorrelation (EBD) algorithm, a novel learning framework that addresses the credit assignment problem in neural networks by directly broadcasting output error to individual layers. The EBD algorithm leverages the orthogonality property of the optimal minimum mean square error (MMSE) estimator, which states that estimation errors are orthogonal to any nonlinear function of the input, specifically the activations of each layer. By defining layerwise loss functions that penalize correlations between these activations and output errors, the EBD method offers a principled and efficient approach to error broadcasting. This direct error transmission eliminates the need for weight transport inherent in backpropagation. Additionally, the optimization framework of the EBD algorithm naturally leads to the emergence of the experimentally observed three-factor learning rule. We further demonstrate how EBD can be integrated with other biologically plausible learning frameworks, transforming time-contrastive approaches into single-phase, non-contrastive forms, thereby enhancing biological plausibility and performance. Numerical experiments demonstrate that EBD achieves performance comparable to or better than known error-broadcast methods on benchmark datasets. The scalability of algorithmic extensions of EBD to very large or complex datasets remains to be explored. However, our findings suggest that EBD offers a promising, principled direction for both artificial and natural learning paradigms, providing a biologically plausible and flexible alternative for neural network training with inherent simplicity and adaptability that could benefit future developments in neural network technologies."

1. Bio-plausibility of Entropy and Power Normalizations : One common concern was whether the regularizations introduced in extensions of EBD are bio-plausible.

• Power Regularization Even when a single sample is used for power calculation, the power regularizer remains effective due to the averaging effect across samples over time. Since all hidden layer activations have decoupled power terms, the gradient-based implementation is biologically plausible in the single-sample mode. The implementations for MLP, CNN, LC, and the CorInfoMax-EBD are biologically plausible with batch sizes of $1$.

• Entropy Regularization As introduced in Section 2.4.1, entropy regularization corresponds to the term $\frac{1}{2}$ $\log\det$ $(\mathbf{R}\_{\mathbf{h}}+\varepsilon \mathbf{I} )$. The derivative of this expression with respect to activations involve the inverse of the correlation matrix $\mathbf{R}_\mathbf{h}$.

  • MLP,CNN,LC Models: The current batch based implementations can be converted into more bio-plausible form.

  • The CorInfomax-EBD implementation of Section 3: As described in Appendix D (new appendix), the entropy regularizer is an integral part of the correlative information maximization objective in the CorInfoMax framework, and its online maximization is implemented in a biologically plausible manner. In this implementation, the inverse of the layer-correlation-matrix $\mathbf{B}=\mathbf{R}_\mathbf{h}^{-1}$ manifests as lateral (recurrent) connections of the CorInfoMax network. The CorInfoMax objective gradients for this regularizer function can be implemented based on rank-1 updates on the $\mathbf{B}$ matrix (in addition to the three-factor learning updates from EBD). Therefore, The CorInfoMax-EBD algorithm with a batch size of $1$ updates is bio-plausible. To address concerns about batch (with sizes greater than 1) based operation of CorInfoMax-EBD, we have repeated numerical experiments with a batch size of $1$. The following is the updated form of Table 3, including the CorInfoMax-EBD results for a batch size of $1$:

Table 3: Accuracy results (%) for EP and EBD CorInfoMax algorithms. Column marked with [x] is from reference Bozkurt et al. (2024).

2. The use of nonlinear MMSE orthogonality condition: In linear MMSE estimation, it is the most fundamental approach to utilize the orthogonality condition, i.e., the uncorrelatedness of errors with input, to obtain the parameters of the estimator. Well known estimators such as Kalman Filters as well as adaptive algorithms are derived based on the use of this orthogonality principle (Kailath et al., 2000). Building upon this approach for linear MMSE estimator, we pose a neural network with MSE loss as a parameterized nonlinear MMSE estimator and use the nonlinear orthogonality condition—that is, the uncorrelatedness of the output error components with any nonlinear function of the input—to obtain its parameters. While choosing which nonlinear functions of the input to use is an open problem, for deriving biologically plausible update mechanisms for layer weights and biases, the layer activations are natural choices. The proposed layer-dependent decorrelation objective functions are formed based on this choice. The experiments performed with these objectives confirm the functionality of the corresponding learning method.

Reference: Kailath, T. et al., . Linear estimation. Prentice Hall, 2000.

3. The EBD framework for different loss functions Our framework utilizes the nonlinear orthogonality property unique to MMSE (Minimum Mean Square Error) estimators. While MMSE is especially suitable for regression, we acknowledge that other tasks may benefit from different loss functions, such as cross-entropy. Although we are currently unaware of similar theoretical properties for cross-entropy, this presents an intriguing area for future research. Notably, our numerical experiments (Appendix F.2, "Correlation in Cross-Entropy Criterion-Based Training") show that even with cross-entropy, a similar decorrelation between errors and layer activations occurs, suggesting that decorrelation may be a general feature of the learning process. Thus, while MMSE may not be ideal for all tasks, decorrelation could still play a critical role across various loss functions. Further research could deepen our understanding of this learning mechanism.

C. Concerns of reviewers, in reviews/discussions, and how they are addressed in the revised article and responses.

• biological plausibility of entropy and power regularization:

Reviewer P7w2: "The paper’s claim of biological relevance is weakened by the batch learning requirement., ..., Is there a potential for an online version of your algorithm that eliminates the need for batch learning?". Reviewer JH3K: "To what extent does EBD depend on batch size? It seems like it would require large batches to get a good correlation estimate." Reviewer VBft: "a power normalizing and entropy encouraging mechanism is added ... not discussed whether these are reasonable mechanisms within a biologically plausible context."

In our response and the revised article, we clarified that CorInfoMax architecture’s lateral connections provide entropy regularization and even operate in batch of size 1, increasing its biologically plausibility. In the revised article we included CorInfoMax-EBD numerical examples with batch-1.

The reviewers found the explanations in the revision/response and the new example as the satisfactory answer: Reviewer JH3K: "I also think that the fact CorInfoMax-EBD can operate in a batch size = 1 does improve its bio plausibility."". Reviewer P7w2: "Thank you for your explanation about the biological plausibility of batch learning.".

---

• result comparability with other error-broadcasting mechanisms:

Reviewer P7w2: "Why does BP in Table 2 show such a low performance?". Reviewer VBft: "the original work by Clark et al. has no learning rate scheduler and far fewer training hyperparameters in general. This suggests that the comparison is entirely inappropriate.".

To address these concerns, we re-implemented the compared methods—specifically Backpropagation (BP) and Direct Feedback Alignment (DFA)—ensuring they were trained under the same conditions as our proposed EBD algorithm. This adjustment standardizes the optimization settings across all methods, enabling a fairer and more accurate comparison. We also added runtime comparisons in appendix, for better comparison.

By aligning the training conditions, we enhance the validity and soundness of the results presented in the paper, addressing the reviewers' critiques and improving the reliability of our findings: Reviewer JH3K: "Performance wise, I'm impressed with the additional simulations that have been carried out and am convinced by the emperical results.". Reviewer VBft: "The results of Table 1 are much improved by a re-implementation and are now more comparable... Thank you for the addition of runtime comparisons, these are additionally useful."

---

• presentation flow of the article:

Reviewer JH3K: "The paper is rather dense, and I worry that it harms its accessibility ... structure/delivery of the paper makes it difficult to grasp for a non-expert". Reviewer VBft: "The description of this work’s method is comprehensive but requires a reader to go back and forth to understand it well ... the paper in general is too heavy on the methods aspects".

To address these concerns, we have reorganized the article's sections to improve its readability and ensure a smoother flow. Specifically, we relocated the variations on MLP to Section 2 and moved detailed explanations about forward broadcast to the appendix, making the main text less dense. Additionally, we expanded discussions and added clarifying comments, particularly in Section 3 (CorInfoMax-EBD), the conclusions and limitations to enhance the accessibility of the content. We included a more thorough discussion of the biological plausibility of various components of our algorithms, such as the regularizations and the structure of the CorInfoMax-EBD network. These changes aim to make the paper more comprehensible, even for readers who are not experts in the field.

As appreciated by the reviewers: Reviewer VBft: "Modifications to the explanation of EBD are appreciated and it is now a clearer read."

---

• discussion of alternative loss functions:

Reviewer qRs4: "Is optimal MMSE estimator the best objective of an arbitrary defined network? (since different tasks might have different loss functions)". Reviewer VBft: "The implications for an alternative loss function (CCE) are now present in an appendix, however this appendix is never referred to in the main text."

For this, we mentioned that our proposed framework exploits the nonlinear orthogonality property specific to MMSE (Minimum Mean Square Error) estimators. However, our numerical experiments in Appendix F.2 reveal that even with cross-entropy loss, errors and layer activations decorrelate, suggesting this phenomenon is a fundamental aspect of learning, not limited to the MMSE objective. We also included an additional paragraph regarding this in the conclusion, discussing possible extensions to different losses.