Learning Successor Representations with Distributed Hebbian Temporal Memory

Thank you for your valuable comments and questions. They shed light on some of the presentation weaknesses of the work, we have added corresponding clarifications that improved the overall paper’s quality.

W1 Segment Likelihood Approximation

the author does not explain why the 'independent' variable can be represented in the form of Eq. (11)

Our response: As we mentioned in the paper, the Eq. (11) is a linear approximation of log-likelihood of segment activation between two extreme cases. We don’t know if presynaptic variables are independent or not from the outset, that’s why we learn synaptic weights $w_{ul}$, which correspond to $p(s_l=1 | c^u_{t-1}=1)$ – probability that segment $l$ is active, given cell $u$ is active. We can estimate $w_{ul}$ empirically, just by counting coincidences of segment and presynaptic cell activities. Segment $l$ learns to respond to a cluster of active cells, that’s why if $w_{ul} \to 1$, that means cell $u$ is clustered, otherwise it is independent of the cluster that activates segment $l$.

W2 Sum-product Convergence

It is uncertain whether the sum–product message passing can converge within the given graphical structure.

Our response: There are no loops in our graphical structure, so it needs only one iteration of message passing (see Figure 1A).

W3 Modelling Actions

Actions also affect the transition matrix of HMM, but in Section 3.1, I did not see the author modeling actions.

Our response: Thank you for commenting on that, we have added corresponding clarifications to the text. We include actions in the model by forcing some of the variables $H^k_{t-1}$ to represent actions. That is, we assume that information about action is included in the hidden state of the model. If we have 4 actions, we set 4 states for one of the hidden variables and set its state from observation of the action.

W4 Biological Interpretation

The article does not clarify the specific biological correspondences of 'f' and 'm' within the context of sum–product message passing.

Our response: As we mentioned in the Section 3.1, Factor value $f$ corresponds to some learnable parameter of a dendritic segment. We speculate, that $f$ is segment’s efficacy, i.e. how strong the contribution of the segment to cell’s spiking is. It’s known that a segment may have several presynaptic cells and can be considered as a coincidence detector [1], from where factors with several variables come. Value $m$ is just normalized spike frequency of the corresponding cell. We agree, that it’s not clear from the text, so we added a separate section for clarifying biological interpretation of our model.

[1] Stuart, Greg J., and Nelson Spruston. 2015. ‘Dendritic Integration: 60 Years of Progress’. Nature Neuroscience 18 (12): 1713–21. https://doi.org/10.1038/nn.4157.

Thank you for your valuable feedback. We took into consideration all our comments and will make corresponding adjustments and extensions to the paper ASAP, and we will notify you about the changes. We hope that this will raise the significance of our inquiry and the quality of the presentation of our algorithm.

Q1 Policy Distribution

Section 2.1, shouldn't optimal policy be defined as $\text{argmax}_a$ not $\text{softmax}_a$?

Our response: Thank you noticing that! Our algorithm employs softmax strategy for enabling exploration, however, indeed, the optimal strategy is deterministic.

Q2 Including Actions Into the Model

The SR in RL is defined with respect to a policy; in DHTM, the target policy is ignored. Is this on purpose? If so, could authors comment on this and the difficulties of adapting their estimate with a changing policy?

Our response: Thank you for commenting on that, we will add corresponding clarifications to the manuscript soon. We include actions in the model by forcing some of the variables $H^k_{t-1}$ to represent actions. That is, we assume that information about action is included in the hidden state of the model. If we have 4 actions, we set 4 states for one of the hidden variables and set its state from observation of the action. We form on-policy SRs, i.e. relying on policy iteration theorem. DHTM can be used to form SRs for any policy, given it learned dynamics of the environment.

Q3 Successor Features instead of Successor Representations

Section 2.3, I’m quite confused about this section; although the SR could be used to model observations, you’ve discussed the SR up to this point as being a good representation of the hidden state. I’m not sure that the expected discounted observation occupancy, given a state, is the same object. It feels like successor features should be discussed here, as there’s an implicit assumption that the reward function lies in the span of the observation features. You might not be able to linearly predict the value function $V(h)$ given $M^\pi(\cdot | h) \in \mathbb{R}^{|\mathcal{O}|}$.

Our response: Thank you for noticing that! We have called it Successor Representation as a widespread term among neuroscientists (see [1] for example), and we aimed to bridge knowledge about place cells to our work. Nevertheless, we agree that it’s not common to define Successor Representation in observation space, and this may cause confusion for the reader familiar with the topic from the RL field. We will reformulate our work in terms of the Successor Features framework. In DHTM, variables $O$ are encoded observations, so they are, in effect, already in feature space, not raw observations from the environment. Variables $O$ are observations with respect to the memory itself, but not to the agent. To avoid confusion, further, we call these variables $\Phi$ to distinct them from raw observations.

[1] Samuel J. Gershman. 2018. ‘The Successor Representation: Its Computational Logic and Neural Substrates’. The Journal of Neuroscience 38 (33): 7193. https://doi.org/10.1523/JNEUROSCI.0151-18.2018.

Q4 Custom Metric: Pseudo-Surprise of SF

Section 4, can you describe how the pseudo-surprise is computed? I’m not sure I fully understand what’s going on here.

Our response: As we mentioned in the text, to measure pseudo-surprise of SF, we do the following:

1. Normalize SF by summing it over corresponding variables $\Phi$ and dividing SF by these sums. The result is SF-induced probability distribution $p(\varphi^k)$ of $\Phi^k$ variables.
2. Measure average surprise over future observed states $\varphi^k$, according to this distribution: $-\log p(\varphi^k=j)$, where $j$ is the observed state.

Normalized SF represents future observation (feature) profile for the current state. Pseudo-surprise shows whether SF is consistent with the observed feature states or not. For example, if SF doesn’t predict feature $j$ ($p(\varphi^k=j) = 0$), but we observe it, this’ll result in infinite surprise, which means that the SF is of a bad quality. Thank you for your comment, we will devote more space to this explanation in the text. For comparison, we will also put this metric calculated for uniform SFs.

W1 Scalability of the method

The method as presented wouldn’t scale to infinite horizon or large POMDPs. At the very least, I’d like to see a discussion on the challenges presented when scaling DHTM.

Our response: The main idea of our agent architecture is to combine planning and TD learning, which results in n-step TD learning. It doesn’t need infinite horizon predictions as it learns linear function mapping from hidden space to Successor Features space. We only need to predict the first several steps — to speedup TD learning and reduce its bias — and approximate the tail with the learned mapping (see Eq. (4)).

Scaling to POMDPs with large observation space is also possible due to usage of multiple feature variables, which constitute sparse distributed representation of observations. Sparsity [of observations and connections] is the important key to deal with high dimensionality in our approach. To show that, we also added experiments in AnimalAI environment, which gives more complicated visual input to the agent. We should have stated it clearer, as scalability is what we aimed for designing DHTM. Nevertheless, our method still may have scalability issues, so we will add corresponding discussion to the Conclusion section. Thank you for pointing this out.

W2 Architecture Description

The empirical methodology and agent architecture are poorly explained, which makes it hard to evaluate the method. More direct questions can be found below.

Our response: We agree that the agent’s architecture could be better explained. We took your comment into consideration and will extend the description of the agent and the experimental setup.

W3, W4 Lack of Empirical Results

The empirical results are of limited value; specifically, the authors use a single custom environment with non-standard metrics (see below for more questions about pseudo-surprise).

The paper could be better positioned in the literature regarding successor representations in POMDPs. For example, I expected to see a discussion with [1], which seeks to model an SR in POMDPs.

Our response: We agree that our custom environment experiments may be not very convincing, so we will add results of experiments in AnimalAI, which is a known benchmark for Reinforcement Learning architectures [2]. Besides, we are going to add standard n-step surprise metrics and qualitative examples of the resulting value functions. We have also tested our algorithm in a 2D maze experiment, similar to what is used in the paper you mentioned, and will add the results to the paper soon.

[1] Eszter Vertes and Maneesh Sahani. A neurally plausible model learns successor representations in partially observable environments. NeurIPS 2019.
[2] Crosby, Matthew, Benjamin Beyret, and Marta Halina. 2019. ‘The Animal-AI Olympics’. Nature Machine Intelligence 1 (5): 257–257. https://doi.org/10.1038/s42256-019-0050-3.

We have uploaded an updated version of the text. We have added baselines description to the Appendix and formulation of value function decomposition we use.

Thank you for your valuable commentary and advice! We agree with most of the observations and will do our best responding to them ASAP, and we will notify you about the paper updates. We hope that corresponding adjustments and extensions of our work will improve its quality and significance.

Q1 Spatiotemporal Property of Cortical Network Activity

In section 2.4, when you mention the "spatiotemporal property of cortical network activity," what specific aspect or characteristic are you referring to?

Our response: We are referring to such properties of cortical activity that only a small percentage of neurons are active at each moment of time and the semantics is spread (or distributed) across them.

Q2 Graph Model Superscripts

In the factor graph model, what do the superscripts $i$ and $k$ correspond to?

Our response: Superscripts $i$ and $k$ are used to enumerate different variables as in our model we have multiple hidden and feature variables at the same time, similarly to the Factorial HMM[1].

[1] Ghahramani, Z., and M.I. Jordan. 1997. ‘Factorial Hidden Markov Models’. Machine Learning 29 (2–3): 245–73. https://doi.org/10.1023/a:1007425814087.

Q3 Parameter Values for DHTM and Other Methods

What parameter values were used for DHTM for the results in Figures 3 and 4? What about for the other methods?

Our response: Thank you for pointing this out! We will add the most important parameters’ values of our model and baselines as an Appendix section. All parameters were chosen to optimize the performance of the methods, using grid search among the most plausible parameter values.

W1 Improving Readability of the Text

However, there remains three-quarters of a page that could be employed to break down and refine Section 3.1 on Distributed Hebbian Temporal Memory. Dividing this section into smaller subsections would aid in better organizing the material. Additionally, incorporating a notation glossary, either in the main paper or in an appendix, would be a valuable addition.

Our response: Thank you for your advice! We are going to divide Section 3.1 into several subsections and add a glossary at the end of the paper as soon as possible.

W2 Lack of Empirical Evidence

Although the model itself is quite compelling, the experiments fall short of doing it justice. Only a single experimental result is provided, where the performance is marginally inferior to one of the baseline models, the fchmm. There is potential for more comprehensive experiments that can showcase all the intriguing aspects of this model.

Our response: We agree that one experimental result may be not enough to justify the model. Therefore, we have added an easily interpretable 2D maze experiment to show qualitatively that DHTM value function estimate is adequate. Also, we have added examples of the learned Successor Features for our initial setup, which are easily interpretable too. Besides, we have added results of experiments in AnimalAI environment to show that our method is scalable and environment agnostic. We will upload an updated version of the paper ASAP.

Although our method is inferior to FCHMM according to some metrics, we should emphasize that, in contrast to FCHMM and other baselines, our method learns in fully online mode and doesn’t employ error backpropagation algorithm. FCHMM needs the whole sequence being available while updating learnable parameters. So, FCHMM provides a lower [surprise] bound that we seek to achieve, using only online Hebbian learning. However, hidden state representations, resulting from FCHMM are, in contrast to our method, dense, and that hinders linear mapping of hidden space to Successor Features. We will discuss this and other limitations of the baselines in the Conclusion section.

W3 Grammatical Errors

Minor issues include typos and grammatical errors scattered throughout the text.

Our response: Thank you for noticing that! We did another iteration of grammar checking.

We have uploaded an updated text with additional experiments in different environments.

We are grateful for your thorough commentaries on our paper. We think they have helped us to improve the paper significantly. An updated version of the paper will be uploaded as soon as possible, and we will inform you about the changes.

Q1, 2, 3, 6 Minor Text Improvements

Our response: Thank you for such a detailed commentary! We took into account your suggestions and observations, which have helped us to improve the overall paper's quality.

Q4, 5 Enhancing Clarity of Figure 1.

Our response: We are grateful for your suggestions! We agree that we should have included a more detailed description of the figure either into the main text or caption. Corresponding corrections are made, diagrams are enhanced with additional details to facilitate comprehension. We hope it will improve the overall presentation of the work.

Q7 Clarification of Algorithm 1.

In algorithm 1, link it with the equations being optimized during the learning process. At the moment, it is hard to tell how your proposed algorithm contributes to the standard reinforcement learning framework. 8.“The next step after computing p(h_{t}^{k}) distribution parameters is to incorporate information about .. How does this relate to algorithm 1?

Our response: Algorithm 1 scarcely portrays common training procedure for all methods used, not only for DHTM. Algorithm 1 is not valuable as a contribution, rather it is a clarification of memory testing procedure. There is no strict correspondence between steps described in the Section 3.1 and Algorithm 1, rather it’s a description of the agent's architecture workflow.

Q8 Learning Synaptic Weights of a Segment

How are the weights w_u_l being learned? Is it through equation 18? If it is, eq 18 should be moved to the main paper.

Our response: In Section 3.1 we mention that synaptic segment weights $w_{ul}$ are learning through local Hebbian-like plasticity. They are targeted to represent probability $p(s_l=1\ |\ c^u_{t-1}=1)$ that segment $s_l$ is active, given cell $u$ was active at the previous time-step. It can be learned just by counting activation coincidences and mismatches. But in our algorithm it is approximated as exponential moving average of segment’s $s_l$ frequency activation, given $c^u_{t-1}=1$: $\Delta w_{ul} = \alpha * \mathbb{I}[c^u_{t-1}=1] * (\mathbb{I}[s_l=1] - w_{ul})$, where $\alpha \in [0, 1)$ — learning rate. We will add this explanation to the main text to avoid reader confusion.

Q9 Agent Architecture Details

Details for the agent’s architecture are missing. Please provide them in the appendix. Ideally, given that you proposed a novel method, a computational graph of your architecture should also be included so that the readers can relate it to figure 1 and algorithm 1.

Our response: Thank you for your interest! We will add a more detailed description of the algorithm to an Appendix.

Q10, 11 Noisy Results

Figure 4 seems to have very high variance. How many seeds are being used to generate the results in this plot? Looks like more seeds are required.

Our response: As we mentioned in Section 4, we tested three seeds for every setup. We agree that, perhaps, we need to increase the number of trials to show stable results. However, we should note that high variance may be inevitable, due to usage of non-deterministic softmax strategy. We will update experiments and corresponding figures ASAP.

W1 2D Maze Experiments

The authors only consider one simple environment. It will be interesting to study their approach in 2D maze, such as minigrid environments which are commonly used to study successor representations. This environment will also allow more analysis and make better and more accurate comparisons with ground truth successor representations.

Our response: Thank you for your suggestion! We agree that experiments in 2D maze would be easier to interpret. Therefore, we are going to add them into an updated version of the manuscript.

W2 Learning Curves

At the moment, it is hard to understand why their proposed model is performing better than the baseline. The baseline models seem to have more complex architectures but no learning curves were presented to verify that the complexity reduces the learning efficiency of these models.

Our response: Thank you for pointing this out! We will add learning curves as well.

We have uploaded an updated manuscript with additional 2D maze experiments and other improvements you suggested.