Stochastic Forward-Forward Learning through Representational Dimensionality Compression

Thank you for your valuable commets. A point-by-point response to the weaknesses and questions are as follows:

Weakness1: Despite its conceptual elegance, the paper's empirical validation remains relatively limited. The model underperforms compared to equilibrium-based learning and backpropagation, particularly on more complex datasets, and it is tested only on shallow architectures.

Response: While we agree that scalability to arbitrary architectures is important, our current experiments show that simply stacking more blocks does not improve performance. In fact, using deeper networks or the same architecture as in EBL leads to worse performance with our ED-based layerwise training compared to shallower models.

This performance drop is likely not due to depth alone. We suspect that this is due to that samples from the same class may not consistently activate the same subset of neurons. As network depth increases, this issue may be amplified, distorting representations and deviating from task-relevant goals. In contrast, EBL benefits from directly clamping outputs to desired labels, which enforces task alignment. We will revise the manuscript to clarify this performance gap compared to EBL. We will revise the manuscript to clarify this performance gap compared to EBL.

Nevertheless, we believe our method can be scaled for deeper networks with enhancements like residual connections.

---

Weakness 2: The reliance on manually chosen hyperparameters (e.g., the projection dimensionality and trade-off weight alpha) and the lack of large-scale or real-world benchmarks weaken the claim of generality.

While our experiments show the method works across various projection dimensionalities (Fig 4c), we currently use hand-tuned settings because Farrell [c,f] revealed intrinsic ED patterns in BP-trained networks. The optimal configuration for projection dimensionality and $\alpha$ remains an open question. We acknowledge the need for more rigorous benchmarks and will clarify the scope of our claims regarding generality.

Weakness 3: Additionally, while the theoretical motivation is sound, the exposition of experimental results lacks a clear narrative, and some of the key concepts, like the compression ratio and its link to performance, are underexplored in the broader context of learning dynamics.

Response: We acknowledge the lack of theoretical analysis in the original manuscript. Further investigation of the learning dynamics during the weight update process is needed.

Weakness4: A more direct comparison to EBL and a clearer explanation of what is gained relative to prior methods would further strengthen the contribution.

We will include a more thorough literature review to cover the necessary backgrounds in the revision.

Reference
[c] Farrell, M., Recanatesi, S., Moore, T., Lajoie, G. & Shea-Brown, E. Gradient-based learning drives robust representations in recurrent neural networks by balancing compression and expansion. Nat Mach Intell 4, 564–573 (2022).
[f] Farrell, M. Revealing Structure in Trained Neural Networks Through Dimensionality-Based Methods. PhD thesis, Univ. Washington (2020).

Thank you for your valuable commets. A point-by-point response to the weaknesses and questions are as follows:

Weaknesses 1: - It is unfortunate that the loss function is not tested (with identical architecture and with similar analysis) against alternative loss functions. For instance, I wonder how the result compares to the same architecture but compared with the (more common) InfoNCE loss function. Methods like InfoNCE, which are usually applied end-to-end, can also be applied block-wise [1] and layer-wise [2,3], and vice versa, ED could be applied ETE. So it would be useful to know how ED compares to these methods.

Response:

Our initial focus was on forward-forward learning, so we primarily reviewed works published after 2022, when the original FF paper [a] was introduced. As a result, we unfortunately overlooked the follow-up work [1].

To address this, we conducted the same experiments using the proposed Greedy InfoMax method, following its architecture and training protocol. The only minor change was resizing input images to 64×64, allowing us to use the authors' hyperparameters without further tuning.

The results are shown below:

Due to time and computational constraints, we only ran each experiment once. Nevertheless, the results suggest that our method has the potential to achieve performance comparable to Greedy InfoMax.

While we can also train a network end-to-end using ED, this field has been well studied that there are many successful solutions for unsupervised ETE learning. So we limit our discuss in non-BP methods.

---

Weakness 2: - One of the motivations is to make a "non-contrastive" algorithm. But I suspect that it is secretly a contrastive algorithm: the term ED_d applied on the batch dimension performs comparison on the batch dimension. In fact, it seems that the algorithm does not work for alpha=1, where this term is dropped. So I suspect that the contrastive component is needed. I do not think it is crucial for the message of the paper to claim that the method is non-contrastive, though.

Response:
You're correct that $\text{ED}_d$ implicitly encourages separation between samples, which could be viewed as a contrastive mechanism. However, our key contribution is not about being "non-contrastive" but rather proposing an alternative unsupervised feedforward learning approach that avoids explicit negative sample construction by leveraging noise. We will revise the introduction and Section 3 to clarify this and remove any misleading claims about non-contrastiveness.

---

Question：I wonder what the specificity of the chosen loss function is in comparison with other learning algorithms. The algorithm is prominently compared with EBL, but EBL provides an alternative to gradient descent as a whole and is a supervised algorithm. What is the high-level intention of doing this comparison ? References like [1,2,3] are in the sense more similar to the proposed algorithm. If the key question is to provide a bio-plausible learning algorithm, how would he gradient descent step of the ED look in a simple case, would this indeed be apparent to a bio-plausible learning mechanism?

Response: Why Compare with EBL?
We emphasize connections to Energy-Based Learning (EBL) for three reasons:

1. Forward-Forward (FF) learning has roots in Boltzmann Machines and Hopfield Networks, making EBL a natural comparison.
2. EBL does not have to be supervised. It can optimize an energy function that similar to FF's layer-wise objective if we restrict the "whole network" as a single layer. Moreover, biological plausible rules such as Hebbian and anti-Hebbian learning, can also be framed as energy minimization by stabilizing correlated activity.
3. There are many evidience suppprt that both the mean firing rate and the firing variability can affect neural coding and neural computation [b,f]. In our method, it is naturelly to suggest using $E[X^2]$ as the metric for classication instead of $E[X]$, which means we should jointly consider the mean and variance. Interestingly, this fomula also is regard as the energy term in EBL.

Although an explicit weight update rule derived from ED would have non-local terms such as calculating the trace of covariance matrix, we would like to provide the following high-level rationale to support its biological plausibility:

1. Plausibility of optimizing ED within local circuits:
   • Minimizing $\text{ED}_c$ naturally leads to a sparse activation pattern, which can be implemented through a biologically plausible winner-takes-all (WTA) mechanism. In such a circuit, only a small subset of neurons respond strongly to a given input, resulting in a low effective dimensionality at the population level due to widespread silencing.
   • Maximizing $\text{ED}_d$is more subtle but still achievable in plausible circuits. One scenario involves excitatory and inhibitory neurons operating on different timescales. When an input is clamped, the most active excitatory neurons drive activity in associated inhibitory neurons to maintain network balance. Upon switching from input A to B, the slower inhibitory dynamics can transiently suppress previously active excitatory neurons, allowing other neurons to become active via a renewed WTA competition. This mechanism effectively reshapes the population response to new inputs, thereby increasing $\text{ED}_d$.
2. Experimental relevance: The effect of tuning curves [d] and higher-order statistics [e] on neural coding and representational capacity has been extensively studied in neuroscience. The optimization of effective dimensionality offers a unifying theoretical principle that could be tested against neural data in future works.

We will incorporate these points into Section 3 and the Discussion to better explain our motivation and why the proposed objective is compatible with biologically plausible learning mechanisms.

Reference
[a] Hinton, G. The Forward-Forward Algorithm: Some Preliminary Investigations. ArXiv (2022).
[b] Qi, Y. et al. Toward stochastic neural computing. ArXiv (2024)
[d] Kriegeskorte, N. & Wei, X.-X. Neural tuning and representational geometry. Nat Rev Neurosci 22, 703–718 (2021).
[e] Panzeri, S., Moroni, M., Safaai, H. & Harvey, C. D. The structures and functions of correlations in neural population codes. Nat Rev Neurosci (2022)

Thank you for your valuable commets. Due to space limitations, we have focused our response on addressing Weaknesses 1-3, select aspects of Weakness 4, and Questions 1-4 in this rebuttal (while retaining reference numbering). We have also corrected all typographical and grammatical errors mentioned in Weakness 4.

Weakness 1.

Response: We understand the mathematical notation in the original manuscript was a bit messy and may be ambiguous or inconsistent at times. Below is the definition of the learning objective with cleaner mathematical notations and a theoretical explanation of why it should work.

Denote $X^{(l)}\in \mathbb{R}^{B\times F}$ the neural response in block $l$ to a batch of noisy input samples (B is the batch size and F is the number of features or neurons). The covariance of the neural responses across random realizations of different inputs is $\Sigma^{(l)}=\mathbb{E}[X^{(l)T}X^{(l)}]$ where $\mathbb{E}[\cdot]$ denotes expectation over random realizations of the inputs. Note that in the noise-free setting the covariance is simply $\Sigma^{(l)}=X^{(l)T}X^{(l)}$ which captures the data variability. The ED is then defined as
$\text{ED}(X^{(l)}) = \frac{\text{tr}(\mathbb E[X^{(l)T}X^{(l)}])^2}{\lVert\mathbb E[X^{(l)T}X^{(l)}]\rVert_F^2} = \frac{(\sum_{i=1}^{d} \lambda_i)^2}{\sum_{i=1}^{d} \lambda_i^2},$
where $\lambda_i$ are the eigenvalues of the uncentered covariance matrix $\mathbb E[X^{(l)T}X^{(l)}]$.

Let $x\in\mathbb{R}^{B\times F}$ be a batch of data samples and use dropout to create its noisy copies, $X$, by randomly setting the elements in x to zero with probability p.
For the $l$-th block, we denote its output (i.e., the corresponding neural responses) as $X^{(l)}$. The input block is therefore $X^{(0)}=x$. Further denote the neural responses to a particular input sample in the batch as $X_{i}^{(l)}$, that is, the i-th row of $X^{(l)}$ .
The goodness function for each block is then partioned into two terms, a consistency term and a diversity term, defined as:

$\text{ED}_{c}=\frac{1}{B}\sum_i^B\text{ED}\left(X^{(l)}_i\right)$,

and

$\text{ED}_{d}=\text{ED}\left(\mathbb{E}[X^{(l)}]\right)$

where $B$ is the batch size, $\mathbb{E}[X^{(l)}]$ denotes averaging over noisy copies of the input batch. The consistency term describes the average ED of neural responses to noisy realizations of individual input samples and captures the noise variability, whereas the diversity term describes the ED of response distribution across different input samples and captures the data variability.

To understand why our method works, we examine an ideal scenario modeled by a Gaussian mixture model (GMM) with two modes, as in Fig. 1b of the main text. The blue and orange dots represent noisy responses for inputs belonging to classes a and b, respectively. The covariances of class a and class b are visualized by the corresponding blue and orange ellipses, while the overall non-centered covariance of the GMM is depicted by the dashed gray ellipse. In this case, $\mathrm{ED}_c \approx 1$ is computed based on the non-centered covariance within each class, whereas $\mathrm{ED}_d \approx 2$ is computed from the total non-centered covariance across all data.

Furthermore, minimizing $\mathrm{ED}_c$ promotes anisotropy in the statistical moments—i.e., it increases the difference in variance across different dimensions (as shown in Fig.1c). For example, given an input from class $a$, the corresponding response becomes more concentrated along one dimension and more dispersed along another. This asymmetry facilitates linear separability between the two classes.

Weakness 2:

Response: Due word limitation, please see the response to the Question mentioned by Reviewer ck6c, where we explain the plausibility of optimizing ED within local circuits.

Weakness 3

Response: We appreciate the reviewer's concern regarding the performance gap between our method and other biologically plausible (non-BP) algorithms. While our method improves upon the original Forward-Forward (FF) algorithm, we acknowledge that it does not yet outperform the best existing non-BP methods (such as EBL).

However, we would like to offer two clarifications:

1. Performance among non-BP methods remains comparable: If we compare only among existing biologically plausible algorithms (excluding our method), their performance generally falls within a similar range. This suggests a broader challenge: non-BP methods as a class still struggle to match the performance of standard BP, likely due to the constraints imposed by biological plausibility (e.g., locality, lack of weight transport).
2. Finding what works is currently more important than finding what works best: At this stage, the priority is to identify learning principles that allow biologically plausible models to function reliably across a variety of tasks. Our contribution lies in demonstrating that optimizing effective dimensionality offers a promising direction that enhances performance within the FF framework while remaining compatible with biological constraints. We believe this lays the groundwork for future improvements in efficiency.

We will revise the Discussion to more explicitly acknowledge this limitation and clarify our intended contribution.

Weakness 4.4

Response: Correct. It refers to the dashed circle, though in general it is an ellipse - depending on the components of the gaussian mixture. Its center represents the mean of the gaussian mixture and the ellipse itself represents the uncentered covariance of the gaussian mixture. Note that this does not imply that the gaussian mixture itself has a unimodal shape.

Weakness 4.6

Response: We understand the original manuscript lacks a precise definition of the tuning curve. In our case, if we let $X$ be the noisy neural response to an input sample $x$ with label $c$, then we can write the tuning curve (in mean firing rate) as $\mu(c)=\langle\mathbb{E}[X]\rangle_c$. Here, we use $\mathbb{E}[\cdot]$ to denote expectation over random realizations and $\langle\cdot\rangle_c$ to denote averaging across all data samples with label $c$. One may also write out these expectation explicitly in terms of conditional distribution but we find that would be too cumbersome and it does not improve further the clarity.

Weakness 4.8

Response: The ED in response to Weakness 1 is now more clearly defined. If we restrict the sample data to the same class, then the ${\rm ED}$ formula in the revised notation would measure the effective dimensionality of that class.

Questions:

Question 1 Section 3, Line 99: " a population of neurons where each 100 neuron is selectively responsive to a particular type of stimuli" is this linear separability condition envisioned for neurrons at all layers, or they are mostly for the final layer neurons?

Response: The statement refers to all layers, not just the final one. This passage provides high-level intuition for why it is possible to define a goodness function suitable for Forward-Forward (FF) learning.

The key idea is that since the final classification depends on the linear separability of the output representations, improving linear separability at each layer—through layerwise optimization—can cumulatively lead to good overall performance. While we do not enforce strict linear separability at every layer, encouraging this property throughout the network helps guide learning in the absence of backpropagation across layers.

We will clarify this point in the text to avoid confusion.

Question 2 Figure 1 (c): Is ED in these figures for the unconditional distributions, or class condition distributions? why should ED of class conditioned distributions depend on the center locations of class clusters?

Response: In this panel, what variable the distribution is conditioned on is irrelevant. It is simply a graphical illustration of how ED would change based on parameters of a distribution - in this case the mean, variance, and correlation of a 2d gaussian.

Question 3: What does ED_c in (2) measeure and what does ED_d in (3) measure? why should they be a goodness of fit functions? how do they guarantee that the class conditioned distribution domains have low effective dimension?

Response: For the first two questions, please refer to our responses to Weakness 1 and Question 2. Regarding the third question, while we cannot guarantee that class-conditioned distributions will achieve low effective dimensionality (since optimization is unsupervised), we hypothesize that samples from the same class share similar structures, activating the same subset of neurons and thus reducing ED.

Our experiments support this: (a) the method achieves reasonable classification performance, and (b) after training, the ratio $\text{ED}_d / \langle \text{ED}_c\rangle$ increases, where $\langle \text{ED}_c\rangle$ is the average class-conditioned ED and $\text{ED}_d$ is the unconditional ED.

Question 4 Does the proposed loss lead to local learning rule? if not what is its advantage over the backpropagation algorithm?

Response: No. The gradients are still calculated using BP, though there is no gradient flow across blocks. However, this approach could help guide biologically plausible learning rules in future work. For more details, please see response to the Question mentioned by Reviewer ck6c, where we explain the plausibility of optimizing ED within local circuits.

Reference
[d] Kriegeskorte, N. & Wei, X.-X. Neural tuning and representational geometry. Nat Rev Neurosci 22, 703–718 (2021).
[e] Panzeri, S., Moroni, M., Safaai, H. & Harvey, C. D. The structures and functions of correlations in neural population codes. Nat Rev Neurosci (2022).

Thank you for your valuable commets. A point-by-point response to the weaknesses and questions are as follows:

Weaknesss 1: - The method effectiveness is demonstrated only on shallow architectures, especially considering that the non-BP methods from the literature consider much deeper networks.
Weakness 2: Equilibrium Propagation: The authors note their performance is behind to that of equilibrium propagation, commenting that it might be a result of deeper networks and increased computation. Based on this the comparison to equilibrium propagation is quite indefinitive and should ideally be tested in a control to compare on equal grounds
Question 1: Can the approach remain effective and competitive for deeper architectures?

Response: Weaknesses 1 and 2, along with Question 1, primarily concern the scalability of our method, so we address them together. While we agree that scalability to arbitrary architectures is important, our current experiments show that simply stacking more blocks does not improve performance. In fact, using deeper networks or the same architecture as in EBL leads to worse performance with our ED-based layerwise training compared to shallower models.

This performance drop is likely not due to depth alone. We suspect that this is due to that samples from the same class may not consistently activate the same subset of neurons. As network depth increases, this issue may be amplified, distorting representations and deviating from task-relevant goals. In contrast, EBL benefits from directly clamping outputs to desired labels, which enforces task alignment. We will revise the manuscript to clarify this performance gap compared to EBL.

Nevertheless, we believe our method can be scaled for deeper networks with enhancements like residual connections.

---

Question 2: - Why are the authors not comparing to layer-wise contrastive methods such as successors of [1]?

Response: Our initial focus was on forward-forward learning, so we primarily reviewed works published after 2022, when the original FF paper [a] was introduced. As a result, we unfortunately overlooked the follow-up work [1].

To address this, we conducted the same experiments using the proposed Greedy InfoMax method, following its architecture and training protocol. The only minor change was resizing input images to 64×64, allowing us to use the authors' hyperparameters without further tuning.

The results are shown below:

Due to time and computational constraints, we only ran each experiment once. Nevertheless, the results suggest that our method has the potential to achieve performance comparable to Greedy InfoMax.

---

Q3: - There is no indication about the number of orthonormal basis vectors employed for the pre dimensionality reduction. This is a bit misleading and never is mentioned to sufficient capacity, which could be misleading to readers.

Thank you for pointing this out. The number of orthonormal basis vectors used for projection was 30-20-10 for MNIST and CIFAR-10 and 90-150-100 for CIFAR-100, as mentioned in the Supplementary Material S1.2.

We will explicitly state the number of orthonormal basis vectors used in the experiments in Section 4 of the main text to avoid any confusion. In Fig. 2, we will add a dimensionality reduction module to clarify the preprocessing step before computing the ED loss.

---

Q4: - Can the method be applied to biologically more realistic models both on the neuron and architecture level (recurrent)?

Yes. To explore the applicability of our method to more biologically realistic models, we conducted experiments using the Moment Neural Network (MNN) [b], which captures the dynamics of firing statistics of spiking neural network (SNN) up to second order moments. Given noisy inputs with first and second moments $(\mu,\Sigma)$, the MNN produces corresponding spike count moments $(\hat\mu,\hat C)=\phi(\mu,C$) through a nonlinear mapping.

In this framwork, $ED_c$ can be computed directly. For $ED_d$ , we define the non-centered total covariance over a batch as:
$\mathbb E[XX^T] = \frac{1}{B}\sum_{i=1}^B (C_i + \mu_i\mu_i^T)$
We tested this setup on MNIST using a one-hidden-layer MNN and achieved 92% accuracy. Although this is below state-of-the-art, it demonstrates that a biologically realistic model (SNNs) can potentially be optimized in a similar fashion due to the close correspondence between MNN and SNN.

Regarding recurrent networks, we were unable to conduct experiments due to time constraints. However, prior empirical findings by Farrell et al. [c] show that ED in RNNs increases and then decreases during training with backpropagation. This suggests that optimizing ED in a principled way could yield competitive results in RNNs as well.

Weakness 3 ：- The dependency on noise could also be harmful. Indeed, what if the neuromorphic device that would be considered offers a lower level of noise than what is needed? - What would be the effect of changes in the employed noise distribution introduced by dropout?
Question 4: - What would be the effect of changes in the employed noise distribution introduced by dropout?

To better understand the effect of noise, we conducted experiments by varying the dropout probability (from 0.1 to 0.5) and the sampling size (from 4 to 20) to empirically investigate how the noise level affects performance. The results are summarized below:

MNIST

CIFAR10:

CIFAR100

These results demonstrate that both noise strength (dropout probability) and sampling size significantly influence model performance. In general, moderate noise levels and appropriate sampling sizes yield optimal results.

Beyond the numerical findings, we offer further commentary on this issue. Currently, digital computing hardwares often aim at eliminating noise. However, neuromorphic chips are designed to closely mimic the computational mechanisms of the brain, one of which is the highly irregular and stochastic nature of spiking activity. This inherent variability—often regarded as noise—has been shown to play a functional role in biological computation. Consequently, many studies explore how the brain harnesses such stochasticity to perform useful work. In the long term, we anticipate that neuromorphic hardware will likely evolve toward analog computation, embracing rather than suppressing intrinsic noise. In this context, the noise-dependent optimization paradigm may not be a limitation, but a reflection of biological realism and a design opportunity.

---

Reference
[a] Hinton, G. The Forward-Forward Algorithm: Some Preliminary Investigations. ArXiv (2022).
[b] Qi, Y. et al. Toward stochastic neural computing. ArXiv (2024).
[c] Farrell, M., Recanatesi, S., Moore, T., Lajoie, G. & Shea-Brown, E. Gradient-based learning drives robust representations in recurrent neural networks by balancing compression and expansion. Nat Mach Intell 4, 564–573 (2022).