Counter-Current Learning: A Biologically Plausible Dual Network Approach for Deep Learning

[W1] Algorithm selection.

We appreciate the reviewer's question regarding concurrent work. A performance comparison is provided in general response 2 [GR2], demonstrating that our CCL algorithm outperforms other methods. While SoftHebb achieves similar results, it requires 24 hyperparameters compared to our 3, making our approach more efficient. We acknowledge the importance of comparing with other emerging biologically plausible approaches and will include more rigorous comparisons to methods like forward-forward and deep SoftHebb, as suggested.

[W2] More validation.

We appreciate the reviewer's questions regarding dataset and model scalability. However, due to the size and complexity of ImageNet, one typically requires batch normalization and other tricks in model architectures, which may not be biologically plausible. Additionally, recent work on biologically plausible learning algorithms (e.g., FA, DTP, DRL, FW-DT, and DRTP) has not evaluated these methods on large datasets like ImageNet, as datasets such as CIFAR-10 and CIFAR-100 demonstrate sufficient complexity for current studies in this field. We will include this discussion in the revised manuscript, acknowledging the need for further research on scalability to more complex datasets like ImageNet. Nonetheless, to showcase the capabilities of the CCL algorithm, we have applied it to an autoencoder task, which, to our knowledge, is the first time a biologically plausible algorithm has been scaled to an image autoencoding paradigm.

[Q1] Explanation for the result.

For a detailed discussion, please see General Response 3 [GR3] and Figure 1.a in the rebuttal document.

[Q2] Comparison to the forward-forward algorithm.

Quantitative comparisons are provided in the general response [GR2]. We discuss algorithmic differences here:

• Compared to Section 3.3 in [4], CCL does not embed the label in the image. CCL adopts a feedback network to process and project the one-hot label, while in FFA the label is embedded in the input data, e.g., for MNIST training the first 10 pixels of an image is replaced with a one of N representation of the label.
• Compared to Section 3.4 in [4], CCL does not need recurrent architecture to propagate the supervised signal backwardly: FFA adopts a recurrent network architecture to send the supervision signals in a top-down manner, i.e., propagate the signal one-layer by one-layer for 3 - 5 steps. In contrast, our CCL scheme seeks to propagate the signal directly, thus attaining more computational efficiency. Rolling out the signal transmission step-by-step would make the algorithm hard to scale to deep networks.

[1] Lee, D. H., Zhang, S., Fischer, A., & Bengio, Y. (2015). Difference target propagation. ECML PKDD.

[2] Meulemans, A., Carzaniga, F., Suykens, J., Sacramento, J., & Grewe, B. F. (2020). A theoretical framework for target propagation. NeurIPS.

[3] Shibuya, T., Inoue, N., Kawakami, R., & Sato, I. (2023). Fixed-weight difference target propagation. AAAI.

[4] Hinton, G. (2022). The forward-forward algorithm: Some preliminary investigations. arXiv preprint arXiv:2212.13345.

[5] Nøkland, A. (2016). Direct feedback alignment provides learning in deep neural networks. NeurIPS.

[6] Launay, J., Poli, I., Boniface, F., & Krzakala, F. (2020). Direct feedback alignment scales to modern deep learning tasks and architectures. NeurIPS.

[Q1] In-depth discussion of bIological plausibility for counter-current learning

We thank the reviewer for the opportunity to further elaborate on the biological plausibility of our method from weight-transport, locality, freezing of neural activity, and update locking perspectives. Although the explanation has been provided in the manuscript Line 41-50 and Line 124-129 on how the model architecture and algorithm design mitigates these problems, we discuss more explicitly below

• Weight-transport. We leverage a feedback network architecture for processing feedback signals, which uses a different parameter set from the forward network. The weights for feedback and forward networks are updated independently, mitigating the problem of weight transport by construction.
• Locality. We leverage local layer-wise loss functions and gradient detach (i.e., stop gradient operation) to ensure that the parameters are updated by a local loss function, not a global error signal that propagates through the network.
• Freezing of neural activity. The back propagation method requires keeping the neural activations for the global error back propagation since the process involves element-wise multiplication of the error signal with the derivative of the neural activations and thus it has to remain static. Our CCL algorithm, like all the algorithms in the Target Propagation Family, also requires freezing the neural activity for the loss computations. However, the duration of freezing time is halved with our CCL scheme since the two networks can process the signals simultaneously.
• Update locking. In back-propagation, The latent activations of the forward network are kept and re-used for the backward phase. Moreover, the backward phase can only begin after the forward phase is finished, causing the update locking problem. In CCL, the backward process is performed through the feedback network, which is independent of the forward phase, and thus the forward and feedback processes occur simultaneously.

[Q2] Literature review and algorithm comparison

Quantitative comparisons are provided in the general response [GR2]. We discuss algorithmic differences here:

• Forward-forward algorithm, FFA [1]

  • Compared to Section 3.3 in [1], CCL does not embed the label in the image. CCL adopts a feedback network to process and project the one-hot label, while in FFA the label is embedded in the input data, e.g., for MNIST training the first 10 pixels of an image is replaced with a one of N representation of the label.
  • Compared to Section 3.4 in [1], CCL does not need recurrent architecture to propagate the supervised signal backwardly: FFA adopts a recurrent network architecture to send the supervision signals in a top-down manner, i.e., propagate the signal one-layer by one-layer for 3 - 5 steps. In contrast, our CCL scheme seeks to propagate the signal directly, thus attaining more computational efficiency. Rolling out the signal transmission step-by-step would make the algorithm hard to scale to deep networks.

• Error-driven Input Modulation, PEPITA [2]

  • PEPITA requires two forward passes. In PEPITA, the parameter update function requires computing the difference between the two forward passes. This causes the update locking problem
  • PEPITA requires the neurons to track two states. As PEPITA requires two forward passes,it requires every neuron to keep track of two neural activities, which is more biologically implausible.
  • PEPITA does not follow locality. The global error signal is directly transformed and added to the latent feature; this does not follow the locality learning concept where synaptic changes depend on the correlated activity of the pre-and postsynaptic neurons.

• Channel-wise Feature Separator and Extractor, CFSE [3]

  • CFSE is a layer-wise training method that contains structural inductive bias. CFSE groups input channels in the forward stage, which can potentially provide inductive bias preferable for training.
  • CFSE works with two different normalization methods. CFSE adopts batch normalization and group normalization methods to work, while CCL attains similar results without group normalization.
  • CCL is validated on auto-encoder experiments, which CFSE does not.

[Q3] Explanation for the result on CIFAR-100.

For a detailed discussion, please see General Response 3 [GR3] and Figure 1.a in the rebuttal document.

[1] Hinton, G. (2022). The Forward-Forward Algorithm: Some Preliminary Investigations. Technical report.

[2] Dellaferrera, G., Kreiman, G., and Kreiman, G. (2022). Error-driven Input Modulation: Solving the Credit Assignment Problem without a Backward Pass. ICML.

[3] Andreas Papachristodoulou, Christos Kyrkou, Stelios Timotheou, and Theocharis Theocharides. (2024). Convolutional channel-wise competitive learning for the forward-forward algorithm. AAAI.

[4] Shaw, N. P., Jackson, T., & Orchard, J. (2020). Biological batch normalisation: How intrinsic plasticity improves learning in deep neural networks. Plos one, 15(9).

I thank the authors for their response! I have gone through the manuscript and reviews multiple times.

About the biological plausibility, I think there are still several very unclear areas. In particular, it is not clear how exactly the training is performed. I have also read the answer to other reviewers' questions, still I have trouble understanding the details and the details are where things become clear (e.g., is the definition of L_feature accurate?).

1. Weight-transport: The authors mention that "The weights for feedback and forward networks are updated independently, mitigating the problem of weight transport by construction." However, from the loss function you define (in Equation 2), is it possible that you end up with W^T (as in BP) for the backward pass? That is, you don't introduce the weight-transport explicitly, but implicitly the optimization is formulated in such a way that it ends up with the solution of BP. (The definition of the L_feature makes me wonder even more.) Also, what does this objective (minimize ||$\hat{a}_l-\hat{b}_l$||) mean really? Why should this optimization happen in the cortex or any biological system?
2. Locality: how do you optimize the loss function (in Equation 2)? It seems the loss function depends on all layers. From Figure 2, it seems that the optimization problem in Equation 2 is solved all together, isn't it?
3. Update locking: the update locking problem is defined in the literature (e.g., PEPITA): "Input signals cannot be processed in an online fashion, but each sample needs to wait for both the forward and backward computations to be completed for the previous sample." Isn't this also the case in CCL? In Figure 1/2 of CCL, it seems that although the forward and feedback passes can be simultaneously run, the first layer's update need to wait for the completion of the backward pass; the last layer's update need to wait for the complete of the forward pass.
4. Freezing of neural activity: I see that the authors acknowledge that CCL requires freezing the neural activity.

One might argue that the whole motivation of this work is biological plausibility. If we want to sacrifice biological plausibility in any way, then we use BP, because it has generally the best performance.

About the comparison with the state of the art, I remembered a new paper, how does your approach relate to this work: Ororbia, A.G. and Mali, A., 2019, July. Biologically motivated algorithms for propagating local target representations. In Proceedings of the aaai conference on artificial intelligence (Vol. 33, No. 01, pp. 4651-4658).

About [GR3], could the author show that this is indeed due to overfitting of the BP or use common techniques to avoid overfitting?

I'd appreciate it if the authors could clarify these point in their next response.

I thank the authors for their response!

Weight Transport: We acknowledge the reviewer's concern about implicit weight alignment. In Figure 1.b (in rebuttal material), cosine similarity between the forward layer weights and the corresponding feedback layer weights are computed for the model trained on MNIST using a five-layered MLP architecture. Our analysis reveals a weak alignment between forward and feedback weights, with a maximum cosine similarity of 0.25 after training. This low similarity suggests that our method does not enforce weight transport; instead, even with weak alignment, the models attain good performance.

I think the original issue was the extra biologically implausible assumption/constraint that the weights in the backward pass were the same as the forward pass. Now, while this is not the case here (at least not explicitly), with the objective function, it seems that the extra biologically implausible assumption/constraint is still introduced. One way to go around this issue is to say that this is actually the optimization problem that takes place in the cortex, which we discuss in the following.

Loss Function: The loss function (minimize $||\hat{a}_l -\hat{b}_l ||$) aims to align neural activities between corresponding layers in the feedforward and feedback pathways. This approach is inspired by local learning principles in neuroscience. It is related to Hebbian learning [1], where synaptic connections are strengthened when neurons on either side of the synapse have correlated activity such that the activity correlation would increase.

I am familiar with the principle of "fire together, wire together." However, I don't see what is the reason for this cost function to be optimized by the cortex at one single-layer level locally. One could say, why should they fire together at all at one single-layer level? The objective to minimize $||\hat{a}_1 -\hat{b}_1 ||$ does not seem to be relevant at a higher level (e.g., for ultimately reducing the classification loss) for the cortex.

Locality: Using Figure 1 (Counter-Current Learning subplot) as an example, one local loss function is $||\hat{a}_l -\hat{b}_l ||$. This means: (i) ..., and (ii) ..., (iii) ...

Do (i), (iii), (iii) happen in order in time? That is, first you do (i), then you do (ii), and finally (iii)? Or they are formulated as one joint optimization problem solved in python?

Update Locking and Activity Freezing: Our approach halves the time for activity freezing and partially mitigates the update locking problem.

It seems there is some confusion about weight updates and passes. According to Nøkland2019: "The hidden layer weights cannot be updated before the forward and backward pass has completed. This backward locking prevents parallelization of the weight updates'', and Huo2018 ''Backwards Locking – no module can be updated before all dependent modules have executed in both forwards mode and backwards mode''. In CCL, again, the first layer's update needs to wait for the completion of the backward pass; the last layer's update needs to wait for the completion of the forward pass.

On the other hand, the authors mention that "Our approach halves the time for activity freezing". My question is: is that important? and, if so, why?

Nøkland, A., & Eidnes, L. H. (2019). Training neural networks with local error signals. ICML.

Huo, Z., Gu, B., & Huang, H. (2018). Decoupled parallel backpropagation with convergence guarantee. ICML

I'd appreciate it if the authors could clarify these points in their next response.

I thank the authors for their response and for bearing with my many questions!

I try to summarize what we discussed below about biological plausibility:

1. Weight transport: As the authors mention, "We acknowledge the reviewer's concern about implicit weight alignment." They also provide nice insight that "CCL is demonstrating that solving local problems can lead to global loss reduction."
2. Locality: As the authors mention, "the local loss functions are formulated as a joint optimization problem solved simultaneously in our implementation." Therefore, I am not sure if the locality principle holds. Even if they stop gradient propagation the problem, in my understanding the problem is still not solved (entirely) locally.
3. Update locking: As the authors mention, they address this issue partially: "We acknowledge that while our method partially mitigates this issue..." and "We acknowledge that CCL does not fully mitigate the backward locking problem, as each module still needs to await both forward and feedback passes for updates."
4. Freezing activity: As the authors mention, they address this issue partially: "Our approach halves the time for activity freezing".

Biological plausibility is one of the main motivations and in the title of the paper. As such, there is still several unclear areas around the biological plausibility of CCL. Therefore, as much as I think the paper has interesting contributions, my confidence in the biological plausibility of the approach is still low. Therefore, I slightly raise my score, since the discussion with the authors clarified a few points about the results and comparison with the state of the art, while the biological plausibility part remains largely unclear, at least to this reviewer.

Once again, I thank the authors for their response and patience.

[W1] Discussion of biological plausibility of the regularization mechanisms.

We thank the reviewer for pointing out and allowing us to elaborate on this.

• Normalization: Our use of activation normalization during loss computation aligns with divisive normalization observed in biological neural circuits [1]. This process helps maintain neural activity within a functional range, similar to the divisive normalization phenomenon observed in animals in the early 1990s in the primary visual cortex.

• Modulation. The additional loss provides signals for surround modulation [2], a foundational property of visual neurons at many levels of the visual system (e.g., retina, visual cortex), auditory system, and so forth. Surround modulation describes that dissimilar stimuli between inside and outside neuron's receptive field would evoke stronger responses for a neuron, compared to similar stimuli. This is similar to the regularization methods used to prevent the model from representation collapse.

• Flooding method: This approach, preventing weight updates when the sample-wise difference is small, can be likened to the concept of activation thresholds in biological neurons, where small input changes do not trigger action potentials.

Implementation details:

• Modulation. We implemented a technique to prevent feature collapse for each latent activation $X ∈ R^{b×d}$, where $b$ represents the batch size and $d$ represents the feature dimension. We added an L2 loss term to minimize the difference between norm($X$)norm($X$)$^\top$ and the identity matrix. The loss is computed as: L_feature = $||norm(X)norm(X)⊤ - I||_F$
• Avoid large gradient norms. We adopted gradient centralization [3] to mitigate the issue of large gradient norms. This technique centralizes the gradient vectors by removing their mean values before updating the model parameters: $g_{c} = g - mean(g)$, where $g$ is the original gradient and $g_c$ is the centralized gradient.
• Focus on major differences. We implemented the flooding method [4] to focus the model on cases where predictions significantly differ from the ground truth. The modified loss function is expressed as $L_{flooded} = max(L, b)$, where $b$ is set to 0.2 as the original paper suggested.

[Q1] Why is there no diagonal trace in the figure 6.

The observed pattern in Figure 6 arises from the functional asymmetry when comparing forward and backward networks because their input signals are different. If we were to compare two independently trained forward networks using CCL, we would expect to see diagonal traces in the CCA analysis because both networks perform similar compression and learning functions. However, forward and backward networks exhibit functional asymmetry (e.g., one for feature extraction and another one for reconstruction), leading to rank disparity (e.g., the backward activations have a lower rank). This reduced detail and compressed information in the feedback activations result in lower pair-wise CCA values between forward and backward layers, manifesting as a non-diagonal trace in Figure 6.

[Q2] What is the origin of initial alignment?

Since MNIST is a relatively small and simple dataset, digits of the same class are relatively similar and are more likely to be projected to neighboring features in the latent space, forming several loose clusters. This would cause a relatively positive alignment in terms of CCA.

[Q3] Checking stop gradient implementation.

We followed your advice and checked the gradient implementation. We checked that the weight detachment is implemented correctly such that optimization over a local loss only involves the nearest layers.

[1] Carandini, M., & Heeger, D. J. (2012). Normalization as a canonical neural computation. Nature reviews neuroscience.

[2] Angelucci, A., Bijanzadeh, M., Nurminen, L., Federer, F., Merlin, S., & Bressloff, P. C. (2017). Circuits and mechanisms for surround modulation in visual cortex. Annual review of neuroscience.

[3] Yong, H., Huang, J., Hua, X., & Zhang, L. (2020). Gradient centralization: A new optimization technique for deep neural networks. ECCV.

[4] Ishida, T., Yamane, I., Sakai, T., Niu, G., & Sugiyama, M. (2022). Do We Need Zero Training Loss After Achieving Zero Training Error?. ICML.

Response to Reviewer Comments on Theoretical Derivation

We appreciate the reviewer's insightful comments, which have led us to refine our theoretical derivation. Based on their feedback and further consideration, we have revised our approach to better align with the CCL algorithm.

Problem with Previous Derivation

In our previous derivation, we incorrectly represented the relationship between variables. For example, in the term $E_{q(z_1, z_2 | y)} \left[ \log \frac{q(z_2 |y)}{p(z_2 | z_1)} \right]$, the $z_1$ in $p(z_2 | z_1)$ depends on $x$, not $y$.

Key Changes

1. We now explicitly account for the relationship between $x$ and $y$ in the dataset $D$.
2. We have revised the ELBO derivation to more accurately represent the CCL process.
3. We have introduced an important approximation in the derivation that maintains consistency with the CCL algorithm's implementation.

Updated Derivation

The revised ELBO derivation is as follows:

$E_{(x,y) \sim D} \log p(y|x) \geq E_{(x,y)\sim D, q(z1,z2|y)}[\log (p(y,z_1,z_2|x) / q(z1,z2|y))]$ $= E_{(x,y) \sim D, q(z_1,z_2|y)}[\log p(y|z_2,z_1,x)] - E_{(x,y)\sim D, q(z_1, z_2 | y)}[\log \frac{q(z_2 |y)}{p(z_2 | z_1,x)}] - E_{(x,y)\sim D, q(z_1,z_2 | y)}[\log \frac{q(z_1 | z_2, y)}{p(z_1 | x)}]$

Term-by-Term Analysis

• The first term can be understood as $E_{(x,y) \sim D, z_1\sim p(\cdot|x), z_2\sim p(\cdot|z_1)} [\log p(y|z_2)]$. This term represents the reconstruction of $y$ given the latent variables $z_1$ and $z_2$ sampled from the forward model, not the inference model.
• The first latent alignment term can be rewritten as $E_{(x,y)\sim D, z_1\sim p(\cdot|x)} [KL(q(z_2 |y) | p(z_2 | z_1))]$. This term encourages alignment between the distribution of $z_2$ inferred from $y$ and predicted from $z_1$ (which is sampled from $p(\cdot|x)$, not from $q(z_1|y)$).
• Likewise, the second latent alignment term can be rewritten as $E_{(x,y)\sim D, q(z_2 | y)}[KL(q(z_1 | z_2) | p(z_1|x))]$. This term encourages alignment between the distribution of $z_1$ inferred from $z_2$ and predicted from $x$.

Consequently, the ELBO is $E_{(x,y) \sim D, z_1\sim p(\cdot|x), z_2\sim p(\cdot|z_1)} \log p(y|z_2)] - E_{(x,y)\sim D, z_1\sim p(\cdot|x)} [KL(q(z_2 |y)||p(z_2 | z_1))] - E_{(x,y)\sim D, q(z_2 | y)}[KL(q(z_1 | z_2) || p(z_1|x))]$

Connection to CCL Implementation

In our practical implementation of CCL, we approximate the last two KL divergence terms using L2 losses between latent features from forward and feedback networks. This approximation captures the essence of aligning the forward and backward pathways while providing a computationally tractable solution.

[1] Luo, C. (2022). Understanding diffusion models: A unified perspective. arXiv preprint arXiv:2208.11970.

[2] Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. NeurIPS.

[Q1] Generalization of models

A: While MLPs and CNNs have established biological plausibility, the biological relevance of Transformers and State Space Models remains unclear and under-explored. For vision tasks, CNNs have been shown to match Transformer performance when computational resources, datasets, and techniques are equalized [1,2]. Recognizing the need for generalization, we've extended our research to auto-encoding tasks. To our knowledge, this represents the first application of a biologically plausible algorithm to an auto-encoder architecture, broadening the scope of our approach beyond traditional supervised learning paradigms.

[1] Liu, Z., Mao, H., Wu, C. Y., Feichtenhofer, C., Darrell, T., & Xie, S. (2022). A convnet for the 2020s. CVPR.

[2] Liu, S., Chen, T., Chen, X., Chen, X., Xiao, Q., Wu, B., ... & Wang, Z. (2023). More convnets in the 2020s: Scaling up kernels beyond 51x51 using sparsity. ICLR.

[Q2] Literature review and comparison

We appreciate the reviewer's suggestion to compare our Counter-current Learning (CCL) approach with recent work on dual-stream information processing in the brain. Here's a detailed comparison of CCL with BurstCNN/BurstProp, Sacramento, et al. (2018), and Predictive Coding:

Network Structure:

• CCL uses two separate networks: a feedforward network and a feedback network.
• BurstCNN/BurstProp and Sacramento et al. (2018) use a single network with two firing modes.
• Predictive Coding uses a hierarchical structure with prediction and error units at each layer.

Error Representation:

• CCL doesn't use global error signals; differences between forward and feedback local activations guide the learning.
• BurstCNN/BurstProp encodes errors implicitly in burst firing patterns.
• Sacramento, et al. (2018) allow local error signals to be generated by comparing different sources of neural input activity; the axonal output is a corrected target signal.
• Predictive Coding uses explicit error neurons.

Biological Plausibility:

• CCL mitigates the update locking problem, allowing for more flexible and potentially more biologically plausible learning dynamics.
• BurstCNN/BurstProp is based on observed burst firing patterns in neurons, enhancing biological plausibility. However, it still suffers from update locking problems.
• Sacramento, et al. (2018) does not resolve the update locking problem, nor does it elaborate on multi-layered structures, limiting its biological plausibility in complex networks.
• Predictive Coding aligns with theories of how the brain processes information, but explicit error encoding and propagation remains debated in neuroscience.

Computational Complexity:

• CCL allows for parallel processing in forward and feedback networks, potentially offering computational efficiency.
• BurstCNN/BurstProp and Sacramento, et al. (2018) require distinct forward and backward phases, which may result in slower processing.
• Predictive Coding involves iterative refinement, which can be computationally intensive.

Our CCL approach offers several advantages in this context. By using separate forward and feedback networks, we avoid the update locking problem that affects BurstCNN/BurstProp and Sacramento et al. (2018). This separation allows for more flexible learning dynamics that could better reflect the parallel processing capabilities of biological neural networks. Moreover, CCL's approach to error representation - using differences between forward and feedback activations rather than explicit error signals - offers a unique perspective on how learning might occur in biological systems without the need for specialized error neurons or complex temporal coding schemes.

We will add the above discussion over comparison in the camera-ready version.

[Q3] Testable prediction

We appreciate the reviewer's thought-provoking question. Based on our observations of the learning dynamics in our model, we propose the following testable predictions:

• Organized increase in cross-layer neural activity correlation: As illustrated in Figure 6 of our manuscript, we observe an organized increase in cross-layer neuron activity correlation (highlighted by red and green boxes). This suggests a reciprocal and complementary learning dynamic among neurons. We predict that this phenomenon would be more pronounced when the network is exposed to novel, unseen patterns. This could be tested by measuring cross-layer neural correlations in biological networks before and after exposure to new stimuli.

• Low similarity in local error signals across layers: Unlike backpropagation or related algorithms where error signals are globally coordinated, CCL's architecture suggests that error signals at different layers should exhibit low correlation or similarity. This prediction could be tested by comparing the similarity of local error signals across layers in biological neural networks, particularly during learning tasks.

[W1] We provide a theoretical extension for the existing framework

We provide an analytical framework for understanding CCL in General Response 1 [GR1]. In short, we show that CCL can be considered as optimizing an ELBO for a hierarchical model.

[W4] Validation on large-scale dataset/models

We acknowledge that the ImageNet dataset poses significant challenges due to its size and complexity. It typically requires batch normalization, whose biological plausibility is under-explored and often necessitates extensive architectural changes. Additionally, recent work on biologically plausible learning algorithms (e.g., FA, DTP, DRL, FW-DT, and DRTP) has not evaluated these methods on large datasets like ImageNet, as datasets such as CIFAR-10 and CIFAR-100 demonstrate sufficient complexity for current studies in this field. We will include this discussion in the revised manuscript, acknowledging the need for further research on scalability to more complex datasets like ImageNet. Nonetheless, to showcase the capabilities of the CCL algorithm, we have applied it to an autoencoder task, which, to our knowledge, is the first time a biologically plausible algorithm has been scaled to an image autoencoding paradigm.

[Q1] What is the loss for the final layer?

We thank the reviewer for reminding us of this. We adopt cross-entropy loss for the final layer, which is not explicitly mentioned in the paper. We will modify the content accordingly.

[Q2] Suggestion on reporting weight alignment

We appreciate this insightful suggestion. Please refer to General Response 4 [GR4] for discussion.

[Q3] Suggestion on caption for Figure 1(a)

We thank the reviewer for checking the details. The main idea is that signal processing cannot increase information, which corresponds to the data processing inequality (DPT) concept in information theory [1]. From a more intuitive perspective, due to dimensional reduction and the random initialization of the weight parameters, the information of the input signals can be lost during data processing. We will revise the sentence accordingly. Also, we agree that the sentence “Notably, … operator” should move to “(b) During training,” and we will also revise this accordingly.

[1] Tishby, N., & Zaslavsky, N. (2015). Deep learning and the information bottleneck principle.

[Q4] Do we see a similar organization of representations in other layers?

Yes, similar organization is presented in other layers, especially for deeper layers. In Figure (2) in the rebuttal document, we show the results of a five-layered CNN model trained on CIFAR10. The model is trained for 3000 steps using CCL, and the t-SNE for layers 1, 3, and 5 with different time steps are shown.

[Q5] Why does CCL utilize ELU?

We find that ELU empirically gives better results than Tanh in CCL training when we perform exploratory analysis on MNIST in the early stage of research. We posit the main reason is that Tanh is a symmetric function and thus would induce feature collapse when trained with CCL. For example, at the beginning of training, say the first entries of $a_l$ (i.e., activation from forward network layer $l$) and $b_l$ (i.e., activation from feedback network layer $l$) are 1 and -1 respectively, then solving the local loss function, the weight would be updated such that the first entries of $a_l$ and $b_l$ becomes 0. This would lead to feature collapse for both the forward and feedback networks. We appreciate the reviewer for the insight and will add this to the Supplementary Section.

[W3, Q6] Result inconsistency from Shibuya, et al.

First of all, we thank the reviewer for noting the error, where the citation for DTP should be corrected. We will revise it in the camera-ready version.

Second, we would like to kindly remind the reviewer that the statistics Shibuya et al. [2023] reported are the test error (%), not the test accuracy (%) shown in Table 1.

Third, we would like to identify the sources that can potentially lead to different empirical results:

• (a) We note that the original implementation for the code is incorrect, where the original codebase from Shibuya, et al. (2023) normalizes FMNIST, CIFAR10, and CIFAR100 using the mean and standard deviation of MNIST. We fix this by using each dataset’s mean and standard deviation;
• (b) We proceed with performing a hyperparameter grid search again using five random seeds because directly using the paper’s best hyperparameters does not fit. We consider a slightly smaller hyperparameter search set than the original one, as shown in Section 6.1, due to the computation and time budget for the grid search of all the combinations.

The above all can give rise to the difference between the results presented in our work and the original results.

[Q7] Suggestion on presenting BP and CCL convolutional kernels

We show the result in Figure (3) in the rebuttal document, where kernels learned by BP and CCL are visually different. Kernels from models trained with BP have more high frequency components, as manifested as neighboring white (e.g., weight with high values) and black pixels (e.g., weight with low values). In comparison, those with CCL have more low frequency components. We posit this might be because the error signal can contain more high-frequency information than the ideal target signal.

[Q8] Suggestion on similarity measurement

We thank the reviewer for pointing out this part. In Figure 6, we not only compare the similarities between pairwise features in the same layer (i.e., the diagonal entries) but also across different layers. We believe that using CKA can better capture relations across layers since it considers invertible linear transformation and orthogonal transformation.

[Q9] Does Figure 7 suggest feedforward-feedback learning rates should be symmetric?

Yes, it suggests that symmetric learning rates perform better.