We thank the reviewer for their comments. We hope our response clarifies most of the concerns.

proposed model will highly rely on reward.

While the current model proposes a reward dependent objective, we have also proposed a non-reward-dependent objective (Metric Representation) which recapitulates place field elongation against the trajectory, but not a high density at the target (Fig. 2C). Optimizing place fields either using the MR objective as a standalone or as an auxiliary improves the policy convergence rate (Fig. S15, red and brown) compared to fixed representations (blue), consistent with Fang & Stachenfeld 2024.

Tasks can be too simple.

We kindly refer the reviewer to our response to Reviewer 1 (Qrmv) that “the environments are also very simple”.

biologically implausible.

The question we were pursuing was whether there is a single, simple normative model that can recapitulate several learning-induced changes in place field representations. To do this, we felt some level of biological unrealism was permissible. Nevertheless, the reviewer is correct. We raised this biological implausibility as a limitation and have discussed avenues to make it plausible (e.g. random feedback with local learning rules). We are currently working on a biologically plausible representation learning model and preliminary results are promising.

1.remapping when reward has been changed?

Yes, when the target changes, some place cells that were coding for the initial reward location shift to the new reward location (Fig. S3), replicating the remapping phenomenon (Gautheir et al. 2018). Additionally, place fields that were not initially coding for reward but in the vicinity of the new reward location were recruited to code for the new target. Hence, the model does support partial remapping when the target changes. However, we do not see (1) the same proportion of reward place cells coding for the new target, and (2) reward coding place fields shift gradually to the new target location, whereas we could expect place fields to rapidly shift or jump to the new target location (Krishnan et al. 2022). The timescales for reward based remapping needs additional experimental data for verification.

2.comparison between your RM model and other RL models?

We will add comparison to other RL models to the discussion namely in 3 aspects:

Reward maximization (RM) algorithms (TD vs Q): Our RM model maximizes rewards by optimizing the place field representations for policy and value estimation. Other RL (e.g. Q-learning or SARSA) algorithms seek to maximize cumulative discounted rewards. Due to the reward dependency in these RL objectives, we expect other reward-maximizing RL algorithms to learn similar representations to our actor-critic based model.

Architecture (GBF vs MLP): Deep RL models that use MLPs for representation learning have also shown high density at reward location and backward shift of features (Fang & Stachenfeld 2024 ICLR). Adding Gaussian noise to deep network parameters also elicit a form of representational drift (Aitken et al. 2022). Although we constrained place field tuning curves to Gaussian distributions to analytically study representation learning, we believe insights from these analyses can be translatable to deep RL models.

Learning objectives (TD vs MR vs SR): The SR model learns transition probabilities based on the agent’s policy. Meaning, when the policy changes, the transition probabilities change, and the SR fields will subsequently change. Hence, SR fields do not influence policy learning, making this model inadequate to study representation learning for policy learning. If the agent has a reward maximizing policy, then the SR model recapitulates high field density at the reward location and field elongation against the trajectory (Fig. 2). Hence, the SR model seems to require 2 disparate components, making the SR model less parsimonious. Furthermore, the SR model is not a good candidate to study representational drift as SR fields have to be anchored to fixed representations (Eq. 72). Meaning, we should not expect SR field centroids to drift to a new location as observed in neural data (Fig. 3D Ziv et al. 2013; Fig. 5H Qin et al. 2023). Conversely, the MR agent is a reward independent objective (Foster et al. 2000) and learns to self-localize by path integration. Given its non-dependency on rewards, place fields do not reorganize to show a high field density at targets (Fig. 2C), though they cause fields to increase in size and shift backwards against the trajectory (Fig. 2A,B). We did not study representational drift using the MR objective in this paper, though this could be a future direction.

We hope these discussions would suffice on why the proposed model (Noisy GBF optimized by TD error) is a suitable candidate while being parsimonious and anatomically grounded to the biological neural circuits. We kindly request the reviewer to reconsider the score.

We thank the reviewer for their comments. We hope our response clarifies most of the concerns.

place fields in biology are organized similarly

We wanted to ask if a single, simple normative model can recapitulate several learning-induced changes in place field representations. We agree that there could be other mechanisms that elicit the same phenomena, which we briefly explored. Still, we think the proposed model is the most parsimonious in explaining the learning dynamics observed in place fields when animals are learning to navigate.

We do explore how other objectives also elicit similar representations in the subsequent section: i.e. path integration TD error (MR - metric representation) and state prediction (SR - successor representation) objectives. While the representations learned by these 3 different objectives could be similar (Fig. 2A,B), the dynamics of how place field representations change is different (Fig 2C,D,E), making this a prediction of our proposed model.

generality of the model

We would like to clarify that our single proposed model (noisy Gaussian basis function parameters modified by the TD error - Noise+TD) replicates all 3 phenomena without ad hoc modifications. Perhaps this was unclear, which we clarify below.

first model is formulated

While our partial model (TD) was explained first (Fig. 1), our full proposed model (Noise+TD) recapitulates high field density at targets (Fig. 3B, Fig. S3A,B).

successor representation agents and metric representation agents are introduced

We clarify that the successor representation (SR) and metric representation (MR) agents were introduced as alternative comparisons to our proposed agent (Noise+TD) where field parameters were optimized only by the TD error, not by the SR and MR objectives. Both partial (Fig. 2) and full Noise+TD (Fig. 3B) agents show field elongation against the trajectory when all the place field parameters are optimized.

noise is introduced

The stochasticity in the partial TD model was insufficient for representational drift (Fig. S7). Hence, our full model Noise+TD includes noisy parameter updates. To reiterate, adding noise to place field parameters did not detract the model from demonstrating (1) high field density at the reward location (Fig, 3B and Fig. S3), and (2) field elongation (Fig. 3B).

Hence, our full proposed model (Noise+TD) replicates all 3 neural phenomena, without needing ad-hoc modifications. We hope this clarification addresses the reviewer’s main concern about the model’s generality, and we will clarify this in the manuscript.

consideration of alternative plausible models

The SR and MR objectives are alternative plausible models to show that different objectives can recapitulate the field elongation while the representation dynamics are different (Fig. 2E), making this a prediction of our model.

design choices (successor representations, noise) are not evaluated

We would like to clarify that we did evaluate various design choices. SR does not influence policy learning, as the policy influences SR fields instead. Hence, the influence of SR fields on policy learning was not evaluated. Instead, we explored the influence of MR objective in policy learning. Fig. S15 shows that optimizing place field parameters using MR improved policy convergence (red) compared to using fixed place fields (blue). However, the rate of improvement was not as significant compared to optimizing place fields using the TD objective (purple). Using MR as an auxiliary objective (brown) to reward maximization (TD error) showed a slightly faster policy convergence, consistent with Fang & Stachenfeld (2024). We described this result in lines 372-375 right column.

We refer the reviewer to Fig. 4C, Fig. S11 and Fig. S12 which show the functional role of noise in policy learning when the target shifts or obstacle location changes. Specifically, when the target consistently shifts to a new location, partial TD agents without noise (blue) fail to learn the new target locations, suggesting they are trapped in a local minima. Instead, Noise+TD agents can continually learn the newly shifted targets. Hence, noisy place field representations increase the agent’s flexibility to learn new targets.

To conclude, our full proposed Noise+TD model replicates all 3 neural phenomena and demonstrates faster policy convergence when the task structure changes.

better informed by the literature

We refer the reviewer to the specific parts of the manuscript (Sec. 2 & 5) where we believe a discussion of the relevant literature has been included. It would be helpful if the reviewer could share other literature we missed to add as further discussion.

mapping of the actor-critic

Lines 68 to 84 left column

hippocampus and entorhinal cortex

Hippocampus: Lines 80 to 83 left column Entorhinal cortex: lines 84-87 right column, 393-394 right column

representation drift

Lines 66-82 right column, 434-403 left column

We thank the reviewer for their comments. We hope our response clarifies most of the concerns.

The writing of the paper is very dense

We will reduce the density to keep the description (e.g. parameter choices in methods, description of path integration based TD error in section 4.2, and figure captions) to a minimum and shift supplementary figures to the main paper. Additionally, we will get another page for the final draft.

figures are pixelated

We will use vector-based graphics e.g. .eps instead of .png for the revision!

is it surprising that the resulting representations are transformed Gaussians

Yes, we agree that the paper is more to advance the field of computational neuroscience and have made this edit “This paper presents work whose goal is to advance the field of Computational Neuroscience.” Transformed Gaussians are expected, but it is surprising that the TD objective is sufficient to capture the 3 different phenomena, that have been described by 3 disparate mechanisms i.e. (1) high field density is needed at salient stimulus (e.g. reward location) to maximize fisher information (Ganguli & Simoncelli 2014), (2) predictive coding of future location shows fields elongating and shifting backwards while enveloping obstacles (Stachenfeld et al. 2017) and (3) optimizing noisy parameters shows stable population code while individual place fields drift (Qin et al. 2023). Hence, it is not at all expected that the Gaussian basis function transformations seen in our model recapitulates all 3 phenomena including that shown by the SR algorithm.

phenomena…using pixel or other high-dimensional observations

Yes, this is a good question. Fang & Stachenfeld 2024 showed that optimizing deep representations using the metric representation-like objective recapitulates the backward shift of features. Nevertheless, whether representations in deep networks follow the dynamics observed in the current model is part of our future work. With respect to the current paper, we believe that this is a different question from the original goal, which was to analytically study how representation learning evolves in predefined place cell descriptions.

environments are also very simple

We used simple tasks to match the environmental setup used in experiments so as to study and predict place field representation dynamics. We explored an elevated level of task complexity by changing the target or obstacle location to serve as predictions of place field representations to test in experiments. Nevertheless, it could be possible to have different place field behavior in different or more complex environments. As a follow up, we are working on: multiple rewards (Lee et al. 2020 Cell), uncertainty in reward distribution (Tessarau et al. 2024 bioRxiv) and, sequential two alternative forced choice (Yaghoubi et al. 2024 bioRxiv). We appreciate other complexities to explore based on the reviewer’s recommendation.

1.Why SR fields decrease at the later stage?

Fig. 2C shows how the individual place fields change when optimizing the parameters using the SR algorithm. In the early phases of learning, place fields at the start location (purple) increase in size since the agent spends a higher amount of time at the start doing random walks (black). As the agent learns to spend a higher amount of time at the reward location, the fields at the start location decreases in size while those at the reward location increases. However, the rate of decrease in SR field sizes at the start location (purple) is more significant than the rate of increase in field sizes at the reward (green), resulting in a slight mean decrease in field size in Fig. 2A.

2.Why is the representation similarity matrix visualization different from the others?

This was an outlier example when the similarity matrix at T=75000 was slightly different from the other time points. But the population code becomes similar again as optimization continues. Fig. 3D shows the similarity matrix autocorrelation for the plots in Fig. 3B with $\sigma_{noise}=0.0001$ remaining largely stable (orange) with small deviations. Subtle differences in representational similarity matrices have also been observed in experimental data (Fig. 5F, Qin et al. 2023).

3.Elongation of fields by SR being subtle

Increasing the SR discount factor (Eq. 71) to 0.95, 0.99, 0.999, 0.9999 led to a faster increase in successor field magnitudes rather than the field width. Yet, the increase in SR field width was still significantly smaller than the field elongation observed when optimizing using the TD error. The SR fields ($\psi$) in our models are anchored to fixed place fields ($\phi$ Eq. 72) and the SR fields learn to represent the transition probability between each of these fixed place fields. Increasing the distance between the fixed place fields ($\phi$) or reducing their width could show a faster increase in the SR width compared to the increase in SR magnitudes.