Predicting Observation after Action in a Hierarchical Energy-based Model with Memory

We sincerely thank you for your thorough review and constructive suggestions. Your insights have helped us better articulate our contributions and refine our work. Below, we address your concerns in detail.

Weaknesses:

1. We would like to emphasize that the biological plausibility of our work is a key focus, aiming to explain how the brain performs representation learning. Utilizing biologically plausible networks to interpret brain mechanisms contributes significantly to the community, as demonstrated by works such as Tommaso Salvatori (2021) and Yuhang Song (2023). Moreover, our brain-inspired model achieves performance comparable to BP-based models on several benchmarks used in our experiments. Among non-BP methods, our approach represents one of the best-performing algorithms on similar benchmarks.
2. There are differences between our method and BP in terms of implementation. For detailed explanations, please refer to Question 5.
3. Predictive learning, as a form of representation learning, is already a mature upstream task. Our focus lies on predictive learning itself, and downstream tasks are beyond the scope of this paper.

Questions:

1. In the revised version, we will use a clearer citation format.

2-3. Predictive learning is inherently a type of representation learning, aimed at learning representations that facilitate prediction. This is already a critical area of interest, with many works focusing on predictive learning itself, such as Mufeng Tang (2023) (biologically plausible models), Aaron van den Oord (2019), Tung Nguyen (2021), and Mufeng Tang (2023) (BP-based models). As a general representation learning task, predictive learning supports a variety of downstream applications, including model-based RL and video generation. Our focus is on predictive learning as a representation learning task, and downstream tasks are beyond the scope of this paper.

4. One of the focal points of this work is understanding how the brain performs representation learning. By employing a biologically plausible model based on neural dynamics and local learning, we provide a framework to explain this mechanism, which constitutes a major contribution of this study. Additionally, our brain-inspired model achieves performance comparable to BP-based models on several benchmarks. While our algorithm does not yet show significant advantages in memory usage or latency over BP when implemented on von Neumann computers, it is well-suited for in-memory computing chips. These chips significantly reduce data transmission latency and bandwidth issues, offering a promising potential advantage for our algorithm over BP.

5. Thank you for thoroughly reviewing the details of our algorithm. While our method shares similarities with BP in terms of hierarchical information transfer, it differs substantially in its derivation and implementation. In our model, each layer contains not only the state neurons ($s$) but also additional error neurons ($e$). The joint operation of $e$and $s$ in each layer facilitates posterior sampling and gradient computation. BP, by contrast, relies solely on $s$ without involving error neurons. Furthermore, as each layer has error neurons, the synaptic updates in our model adhere to the Hebbian rule. The update rule for the first layer is presented in Eq. 15, while the subsequent layers follow the formula:
   $\tau_{\theta}\frac{\mathrm{d}\theta^l}{\mathrm{d}t} = -\nabla_{\theta^l}\mathcal{L}^{l}_t = \Lambda^l\hat{e}^l_t f(\hat{s}^{l+1}_t)^T.$ We will include this equation in the revised version.

6. Since our algorithm operates online, it cannot initialize the model with all past sequences at once; instead, it processes data sequentially.

7. We used MSE as the evaluation metric to facilitate comparison with prior work in this domain, such as tPCN, VPN, and ST-ResNet, all of which also adopted MSE.

8. For the CIFAR-10 dataset, we split each image into 16 patches to simulate the receptive field of real eye movements. In future work, we plan to use more realistic images to mimic eye-tracking experiments.

9. Thank you for the question. The performance of our model on the test dataset scales with the number of neurons, as shown in Table 1 and Figure 3.e. The observed degradation on seen images (training set) with increased neuron numbers is due to the increased difficulty of initialization. Energy-based models (EBMs) exhibit strong learning capabilities for both continuous and discontinuous functions in representation learning, whereas traditional forward models such as VAE face challenges with discontinuous functions. Please refer to Implicit Behavioral Cloning for more details.

10. Thank you for the suggestion.

We hope this explanation addresses your concerns and provides clarity regarding our work.

Thank you for taking the time to review our manuscript and for providing thoughtful feedback. Below, we address each of the concerns and questions you raised.

Weakness:

1. To clarify, our work assumes that the brain uses a predictive learning paradigm to perceive and interact with the world. Based on this assumption, we propose a biologically plausible network for predictive learning-based representation learning. The brain processes information through different pathways depending on the sensory-motor system involved. Our model aims to simulate these neural pathways. For example, in our saccadic eye movement experiments, hierarchical EBMs model the visual cortical pathway (V1-V4-IT neocortex), while CANNs simulate the collicular pathway for motor control. Similarly, in our movement experiments, EBMs model the visual cortical pathway, and CANNs represent grid cells in the entorhinal cortex. Since our work primarily focuses on the modeling aspects, we did not elaborate on specific cortical correspondences in the manuscript.

2. The specific task we address is predictive learning. Predictive learning is a form of representation learning aimed at acquiring representations optimized for prediction, making it an inherently valuable and widely researched topic. Many works have focused on predictive learning, such as Mufeng Tang (2023, biologically plausible model algorithms) and Aaron van den Oord (2019, BP-based algorithms). Predictive learning is a general and versatile task for representation learning. In our study, we compared our approach with five other models, presenting results across three benchmark datasets, as shown in Tables 1, 2, and 3.

3. Our work is based on the widely adopted Bayesian brain hypothesis, as discussed by Knill & Pouget (2004) and Friston & Price (2001), which posits that the brain understands the world through predictive learning. This hypothesis has inspired a significant body of work, including Gergo Orbán (2016) and Guillaume Hennequin (2014). Furthermore, it is supported by experimental evidence from studies such as Ulrik R. Beierholm (2009), Edward H. Nieh (2021), and Ralf M. Haefner (2016).

---

Questions:

1. Thank you for highlighting this point. To clarify, our task is not to predict the saccadic eye movement itself but rather to predict the post-saccadic image based on the known eye movement position.

2. In Figure 3a, "unseen" images refer to novel images that were not encountered during training. The models used in this row were trained on datasets that excluded the specific images shown. In Figure 4, "true" refers to the ground truth, i.e., the actual images.

3. Thank you for your suggestion. The shaded regions in Figure 3 represent the error bars.

We hope this explanation addresses your concerns and provides clarity regarding our work.

[10] Weilnhammer, V., et al. "A predictive coding account of bistable perception." PLoS Comput. Biol., 2017.

[11] Lotter, W., et al. "A neural network trained to predict future video frames mimics biological neuronal responses." arXiv, 2018.

[12] Watanabe, E., et al. "Illusory motion reproduced by deep neural networks." Front. Psychol., 2018.

[13] Samsonovich, A., and McNaughton, B. L. "Path integration and cognitive mapping in a continuous attractor neural network model." J. Neurosci., 1997.

[14] McNaughton, B. L., et al. "Path integration and the neural basis of the cognitive map." Nat. Rev. Neurosci., 2006.

[15] Burak, Y., and Fiete, I. R. "Accurate path integration in continuous attractor network models of grid cells." PLoS Comput. Biol., 2009.

[16] Seung, H. S. "How the brain keeps the eyes still." Proc. Natl. Acad. Sci. U.S.A., 1996.

[17] Zhang, K. "Representation of spatial orientation by the intrinsic dynamics of the head direction cell ensemble."

[18] Georgopoulos, A. P., et al. "Neuronal population coding of movement direction." Science, 1986.

[19] Ben-Yishai, R., et al. "Theory of orientation tuning in visual cortex." Proc. Natl. Acad. Sci. U.S.A., 1995.

Thank you for acknowledging our work on achieving biological plausibility，Innovative Prediction and Learning Sequence that bring greater stability and robustness over traditional methods, and validating the effectiveness of our memory mechanism. We hope the following responses can address your concerns:

W1, Q1, Q3: We appreciate your recognition of our model's good predictive performance. The error neurons employed in our work are supported by extensive experimental evidence. For example:

• [1] successfully explained end-stopping and other extra-classical receptive-field effects in the primary visual cortex by incorporating error neurons.
• [2–3] provided evidence for functionally distinct neuronal subpopulations, potentially corresponding to prediction and error neurons.
• [4] discussed multi-compartment neurons with a distinct apical dendrite compartment for storing prediction errors separately from the value encoded by the soma.
• [5] introduced a central microcircuit model of predictive processing using error neurons.
• [6] described predictive processing as a computational framework for cortical function, particularly in sensory processing.
• [7] extended formulations of the brain as an inference organ, using error neurons to orchestrate these processes.
• [8–12] explained a range of perceptual and neural phenomena, including repetition suppression, error responses, attentional modulations, bistable perception, and motion illusions.

We aim for our model to represent a general mechanism for the brain's predictive functions. For instance, in the context of vision, the hEBM framework in our model aligns with the hierarchical structure of the visual cortex. Similarly, CANN, as the memory module, has robust biological support, particularly in the hippocampus. Examples include place cells [13–15], saccadic movement position cells [16], head direction cells [17], movement direction cells [18], and orientation cells [19].

W2, Q2: In Table 2, we compare our model with the recent tPCN framework. To the best of our knowledge, our network is the first biologically plausible model capable of predicting different observations based on different actions. For instance, general PCN frameworks can only handle static tasks, and tPCN extends this to dynamic tasks but only with fixed input sequences. Our network can produce different sequences based on distinct actions.

W3 Figures 3c, 4b–e, and 6b–d in the paper demonstrate how model performance is influenced by network parameters and neuron counts. The question of how these parameters impact biological plausibility is intriguing. In future work, when applying the network to specific cortical regions, we plan to set these parameters based on real biological data.

W4 As mentioned in the discussion section, this is an interesting question. We intend to explore it in future research, although it is beyond the scope of the current paper.

ref: [1] Rao, R. and Ballard, D. "Predictive coding in the visual cortex." Nat. Neurosci., 1999.

[2] Bell, A. H., et al. "Encoding of stimulus probability in macaque inferior temporal cortex." Curr. Biol., 2016.

[3] Fiser, A., et al. "Experience-dependent spatial expectations in mouse visual cortex." Nat. Neurosci., 2016.

[4] Takahashi, N., et al. "Active dendritic currents gate descending cortical outputs in perception." Nat. Neurosci., 2020.

[5] Bastos, A. M., et al. "Canonical microcircuits for predictive coding." Neuron, 2012.

[6] Keller, G. B., and Mrsic-Flogel, T. D. "Predictive processing: A canonical cortical computation." Neuron, 2018.

[7] Kanai, R., et al. "Cerebral hierarchies: Predictive processing, precision, and the pulvinar." Phil. Trans. R. Soc. B, 2015.

[8] Auksztulewicz, R., and Friston, K. "Repetition suppression and its contextual determinants in predictive coding." Cortex, 2016.

[9] Feldman, H., and Friston, K. J. "Attention, uncertainty, and free-energy." Front. Hum. Neurosci., 2010.

Thank you for recognizing our work as posting a solution an interesting topic on the neural mechanism to perform prediction of future observations, as well as for acknowledging the value of our experimental results. We hope the following responses can address your concerns:

W1. The unseen images and seen images are chosen from the same dataset to specifically demonstrate the generalization ability of the model. For the experiments on unseen images, the network was trained on different images, not the ones shown in the demonstration. Below is the relationship between the MSE for unseen images and image numbers. Here’s how you can present the table and its description in English:

W2. The computational complexity of our network at each iteration before convergence is $O(n^2 \times L)$, where $L$ represents the number of network layers, and $n$ represents the number of neurons per layer. As a result, the scalability of our network on larger datasets is consistent with that of a typical feedforward network.

W3. The main contribution of our work lies in leveraging a fully biologically plausible network to achieve prediction tasks that the brain must perform. Specifically, hEBM is used for spatial compression, while CANN enables temporal compression. Removing certain components would render these processes biologically implausible within the neural system. For instance, removing the prediction error neurons would make the local learning rule infeasible. On the other hand, to better understand the model, we conducted several detailed experiments, particularly for the first task. Figures 3c, 4b–e, and 6b–d present some of these results.

Q1. hEBM demonstrates advantages over the forward model when learning discontinuous functions, as shown in ref [1]. Meanwhile, CANN provides an excellent representation for the action transfer task and enables efficient temporal compression.

Q2. In Table 1, we present a comparison with TransDreamer, and in Table 2, we compare with tPCN. These are recent works published in 2022 and 2023 respectively.

Q3. The impact of the number of layers on prediction accuracy is illustrated in Figures 3c, 4b–e, 6b–c, and 6f.

ref:

[1] Florence, Pete, et al. "Implicit behavioral cloning." Conference on Robot Learning. PMLR, 2022.