We thank the reviewer for the valuable review; we especially appreciate the insights provided to expand on the experimental setup and validation context.

1 SNN and Biological Relevance

As pointed out by the reviewer, SNN is not relevant on this research, we apologise for the confusion. We've corrected the methodology to address this error. We use continuous-time recurrent neural networks (CTRNN) as the initial framework for the exogenous model.

We appreciate the reviewer's perspective on ring attractors, and would like to offer a complementary interpretation based on spatial encoding evidence. Kim et al. (Science, 2019). demonstrated that ring attractors represent spatial awareness through heading direction, a neural code that bridges internal representation and spatial behaviour. Their work showed how bump-like activity patterns maintain a persistent sense of orientation while smoothly updating with the fly's movements. This continuous integration of spatial state and movement suggests that ring attractors have evolved to handle representations that are simultaneously internal (maintaining spatial awareness) and behaviorally relevant (guiding navigation).

2 Baselines and 1-D circular variable action space

The reviewer raises a very valid point that was considered and implemented by the authors. In our experiments, we mapped the highway's benchmark navigation action space as a one-dimensional circular variable. We recognise this was not explicitly mentioned. We are working to include this as a new section in the appendix, alongside information about other benchmark action spaces, ring attractor experimental layouts (both single and double configurations), and baseline-RA experimental integration.

3 Role and impact of uncertainty quantification

The role of uncertainty quantification (UQ) in our ring attractor model is fundamental to both its theoretical foundation and practical performance for the exogenous ring attractor model. Our experimental results in Figure 2 clearly demonstrate that the uncertainty-aware version (BDQNRA-UA) consistently outperforms both the baseline (BDQN) and the simpler ring attractor model without uncertainty (BDQNRA), providing empirical justification for its inclusion.

The paper provides an explanation of UQ's role through mathematical formulations in Section 3.1.3 and explicit equations showing how uncertainty values (σₐ) directly influence the Gaussian functions driving ring attractor dynamics. This integration aligns with biological insights, as shown by recent work Kutschireiter et al., 2023, demonstrating the utility and performance of integrating uncertainty into ring attractors.

Rather than adding unnecessary complexity, UQ is tightly integrated with the ring attractor architecture through the Gaussian activation functions (σᵢ = σₐ in Equation 7), creating a natural mechanism for uncertainty-aware action selection.

Q1: Fixed, thank you.

Q2: Here, $m$ and $n$ represent the positions (indices) of neurons within the ring network, where these indices define each neuron's location and determine the distance between any two neurons via $|m-n|$.

Q3: A function approximation algorithm in this context is a method (typically a neural network) that learns to estimate Q-values (expected future rewards) instead of storing exact values for every possible state-action pair in reinforcement learning. It works by converting input states into feature vectors and using learned weights to approximate Q-values through the equation $Q(s,a) = θᵀx(s)$, making it practical for large or continuous state spaces.

Q4: Fixed, thank you.

Q5: While each neuron corresponds to an action-value pair, the ring attractor properties emerge from the circular connectivity pattern between neurons (defined by d(m,n) = min(|m - n|, N - |m - n|)) rather than from what the neurons represent.

Q6: Fixed, thank you.

Q7: Added brief explanation (line 410).

Q8: In this context, "mean computational overhead" refers to the extra processing time or computational cost added by the ring attractor implementation. Whenstated"297.3% overhead", it means the integrated CTRNN ring attractor model (BDQNRA) took 3 times more time to run than the baseline model (BDQN).

Q9: We are also to provide an appendix section that provides insights on integration for the different models and environments, including single and double ring implementations.

Q10: We agree with reviewer in that ablation studies are essential to demonstrate the ring attractor's impact on learning performance across experiments. We are exploring a potential reorganization of the manuscript, though we are still determining which sections could be moved to the appendix given significant space constraints.

Thank you for this excellent feedback, it has given us meaningful ways to strengthen our work:

1 Novelty

Problem Statement

The problem statement lies in conveying faster learning for decision-making algorithms, particularly within the RL framework. While ring attractors have been well-studied in neural models of decision-making, their theoretical integration into fundamental decision-making frameworks beyond neuroscience remains limited. We acknowledge that ring attractor models have been studied in theoretical work. However, our paper's novelty lies in several key contributions:

Novel Integration with RL/DL

We are the first to propose using ring attractors as a mechanism for action selection in RL, creating a bridge between neuroscience-inspired models and RL/DL architectures to provide spatial-aware decision-making that increases the learning rate of modern algorithms. To apply this concept to more mainstream approaches, we create a novel implementation of ring attractors using RNNs (Section 3.2), compatible with DL frameworks, while preserving their key properties the Section 3.2.

Our approach introduces explicit spatial encoding of actions, which is fundamentally different from existing methods. As demonstrated in Section 4.3, this leads to performance improvements, particularly in spatially-structured tasks (e.g., 110% improvement in Asterix, 105% in Boxing).

Theoretical Insights

Equations 5,6 describe the behaviour of our policy. We prove through ablation studies, Figure 4 and Figure 5 that the dynamics in the ring and correct layout of the action space as input signals is the only way that yields actual learning improvements.

2 Insight on why the model excels at action sampling

Ring attractors enhance BDQN's performance by encoding actions in a circular topology while incorporating uncertainty through Gaussian variance parameters σₐ, mathematically expressed through input signal equations $x_n(Q) = \sum_{a=1}^{A} {\frac{Q(s,a)}{\sqrt{2\pi \sigma_a}} \exp\left(-\frac{1}{2}\frac{(\alpha_n - \alpha_a)^2}{\sigma_a^2}\right)}$. This structure of the action space provides benefits beyond BDQN's Bayesian estimation by using explicit spatial encoding in the behaviuor policy plus uncertainty quantification expressed as variance of the action signals. As demonstrated by Sun et al. (2018), ring attractors provide a robust framework for combining cues according to their certainty, using this to our advantage during action selection.

3 Unclear performance boost between EffZeroRA and uncertainty estimation

Atari implementation uses our DL-based ring attractor model without uncertainty estimation. Uncertainty is only used in the CTRNN exogenous model with BDQN's BLR layer.

4 Compatibility of ring structure to arbitrary action spaces

As described in Section 3.1.2, actions are mapped to specific locations on the ring based on their spatial relationships, providing significant flexibility that enables generalisation. Our approach has demonstrated robust performance across diverse action spaces in the Atari 100k benchmark. Single-ring configurations excel in directional movement games (Asterix, Ms Pacman), while double-ring configurations effectively handle games combining movement with independent action dimensions (Seaquest, BattleZone). Additionally, ablation studies in the appendix indicate minimal performance degradation, compared to baseline, when actions are misplaced in a ring layout.

5 Implementation details

1. This is n clarified in section 3.2.1 (line 349)
2. Neural Ring Dynamics (Section 3.2.1): The learnable time constant τ controls information integration into the ring attractor. The forward pass uses fixed distance-dependent weights to maintain spatial relationships, while the hidden state dynamics U(v)m,n employs trainable weights to learn complex action dependencies. By adjusting τ, we can regulate input signals to the RNN layer, balancing spatial relationships with task-specific learning while preserving ring attractor dynamics. We are working on a new appendix section demonstrates the emergence of sustained ring patterns in our experiments.
3. It does not, all DL-based implementations are implemented as outlined in 3 Unclear performance boost between EffZeroRA and uncertainty estimation.
4. SNN does not play any role in this research, we apologise for the confusion. We've corrected the methodology to address this error. We use continuous-time recurrent neural networks (CTRNN) as the initial framework for the exogenous model.

Q1

This has been developed in the section above: 2 Insight on why the model excels at action sampling.

Q2

This has been developed in the section above: 4 Compatibility of ring structure to arbitrary action spaces.

Q3

This has been developed in the section above: 5 Implementation details.

We appreciate the reviewer's insightful comments and would like to provide further insights:

1. Role of the ring attractor in uncertainty quantification (UQ): The ring attractor primarily serves as a spatial encoding mechanism, while uncertainty quantification happens through the BDQN framework. The RA architecture provides a structured topology for organizing actions and their uncertainties, but does not directly participate in computing those uncertainties. The variance estimates from BDQN's Bayesian calculations feed into the Gaussian input signals to the ring attractor (via σₐ in eq. 1), allowing uncertainty to influence the activation patterns, and influencing the action selection process. The correct distribution and injection of uncertainty measurements into the ring improves action selection as presented in Figure 2 in Section 4.1 of the experiments and supported further by the ablation study in Appendix A.2.1. Ring attractors enhance BDQN's performance by encoding actions in a circular topology while incorporating uncertainty through Gaussian variance parameters σₐ, mathematically expressed through input signal equations $x_n(Q) = \sum_{a=1}^{A} {\frac{Q(s,a)}{\sqrt{2\pi \sigma_a}} \exp\left(-\frac{1}{2}\frac{(\alpha_n - \alpha_a)^2}{\sigma_a^2}\right)}$. This structure of the action space provides benefits beyond BDQN's Bayesian estimation by using explicit spatial encoding in the behavior policy plus uncertainty quantification expressed as variance of the action signals. As demonstrated by Sun et al. (2018), ring attractors provide a robust framework for combining cues according to their certainty, using this to our advantage during action selection.

2. What is the nature of the spatial relationship encoded by RA? The spatial organisation provided by ring attractors fundamentally enhances action selection by explicitly encoding the action space and enabling the distribution of spatial representations across the neural network. The correlation between actions is solved by a trainable hidden space as presented in the appendix, Sections A.2.3, A.4; and methodology, Section 3.2.1. As detailed in Section 4.2, games with primarily directional movement like Asterix utilise a single-ring configuration for eight directional movements, while games combining movement with independent actions like Seaquest employ a double-ring configuration; one for movement and another for secondary mechanics. The biological plausibility of this approach can be traced back to the studies presented in the appendix, Section A.1, where ring attractors provide spatial cue integration that we employ in the context of action selection.

3. I appreciate the newly added Appendices, though am still baffled at some implementation details. The learnable time constant τ in equation 13 controls the rate of information integration, since the weights are fixed to preserve the ring topology this learnable parameter enables that rate of transfer from previous layers in the DL agents can vary depending on the preference of the DL agent. As detailed in Section 3.2.1, this allows the network to balance spatial relationships with task-specific learning while preserving ring attractor dynamics through the fixed distance-dependent connection weights. The selection of σₐ = π/6 for BDQNRA was empirically determined to provide optimal balance between action discrimination and smooth transitions in the tested environments. As shown in Section 4.1, this value enables smooth action transitions while preventing interference with opposing actions. This middle step (BDQRA) in the integration between BDQN and a full integration with uncertainty with BDWNRA-UA was put in place to make easier the distinction on how each of the components spatial representation and uncertainty was actually providing an improvement compared to baseline. Equation 19 in Appendix A.4.1 is computed in a single forward pass during inference, despite Fig. 8 illustrating the temporal evolution of hidden states for visualization purposes. The complete forward pass combines input signals and hidden state information in one step through the matrix operations defined in the equation.

4. To help clarify some of the questions above, would the authors be open to share to their code? We acknowledge the importance of sharing our codebase for future research and are actively working on preparing it for release. While we were initially planning to share this at a later stage, we understand the value of making it available sooner. We will have the implementation details ready by the end of this week, including baselines for BDQN, DDQN, and Efficient Zero. Although we weren't able to prepare the codebase within one day of the request, it will be available at this repository.

We sincerely thank the reviewer for their insightful feedback, which has particularly strengthened our methodology:

1 Actions space assumptions

While our model does map actions to ring positions, as described in Section 3.1.2, it's not limited to purely directional actions. The paper demonstrates this through double ring configurations in Section 4.3, where games like Seaquest and BattleZone effectively use separate rings for movement and independent action dimensions. This architecture directly addresses cases where not all actions have inherent directional relationships. Regarding the potential for degeneracy in cases where actions lack directional mapping, Section 3.2 of our paper details how the DL implementation specifically handles this concern. The architecture employs a fixed forward path (V(s)) that maintains structural relationships through distance-dependent weights, while also incorporating trainable hidden state dynamics (U(v)) that can adapt to non-spatial action relationships. The empirical validation in Table 1 demonstrates that our approach works effectively across different action space configurations - from single ring games like Asterix (showing 110% improvement) to double ring games like Seaquest (32%). These results suggest that the ring attractor dynamics remain stable and effective after training, even when handling diverse action types. Additionally, ablation studies in the appendix indicate minimal performance degradation, compared to baseline, when actions are misplaced in a ring layout.

2 SNN and mathematical presentation

We apologise for the inconsistencies in the methodology, and we are working towards fixing all the issues that have been very well pointed out in this review. You are correct that Section 3.1 incorrectly describes a spiking neural network(SNN). We've corrected the methodology to address this error. We use continuous-time recurrent neural networks (CTRNN) as the initial framework for the exogenous model. Regarding the inconsistency between equations (4) and (5), we are working on reconciling all mathematical definitions and ensuring consistency across equations.

3 Experiment models presentation

We have added a brief clarification in line 380. Additionally, we will include integration specifications and implementation details for each model in the appendix. This revision should help readers better follow the relationship between our methodology and the specific model implementations discussed in the experimental results.

Q1: We acknowledged Zhang (1996), who first proposed ring attractors as a theoretical model for head direction cells, in the background placed in the appendix. As it is a key milestone we’ve moved the citation to line 30 in the text.

Q2: Apologies for the confusion with the nature of the ring attractor model. We've corrected the methodology to address this error.

Q3: Though these equations are well-known, their placement in the methodology is useful as they form the working components of our RL policy behaviour policy implementation. It may disconnect the theoretical foundation from our actual algorithmic steps, making the methodology harder to follow and reproduce.

Q4: Acknowledged and working on it.

Q5: Acknowledged and working on it.

Q6: Acknowledged and working on it.

Q7: Acknowledged and working on it.

Q8: Acknowledged and working on it.

Q9: Acknowledged and working on it.

Q10: Acknowledged and working on it.

Q11: Fixed, equations 5 and 6 wrongly displayed the current excitatory and inhibitory terms $v_n,u$ over the time integration constant $\tau$, which is incorrect. We have moved them outside of the equation parenthesis.

Q12: Fixed, thank you.

Q13: Acknowledged and working on it.

Q14: Acknowledged and working on it.

Q15: Fixed, added reference (line 268).

Q16: They represent the weights of the upstream Bayesian Linear Regression model (line 287, 304).

Q17: Fixed, developped further (lines 300-303).

Q18: Fixed (equation13).In the DL RNN implementation, no inhibitory neurons are present. Instead, standard DL neurons with tanh activation maintain the attractor state through their hidden state dynamics, rather than through biological-like lateral inhibition. We are working to provide visualisation of the RNN forward and hidden state dynamics in a appendix section.

Q19: The action space is split into two separate rings with weak connections between them, this implementation is not explicit in the original paper. We are also working to provide an appendix section that provides insights on integration for the different models and environments, including single anddouble ring implementations.

We sincerely thank the reviewer for their thorough examination of our work's fundamental principles:

1 Q-values and variance lower bound

We thank the reviewer for their careful reading of our work. However, we believe there may be a misunderstanding about how Q-values are used in our model. The Q-values are not used as connection weights within the ring attractor network. Instead, the connection weights between neurons are fixed and defined by distance-dependent functions (w_(E_m→E_n) = e^(-d^2(m,n))), which define the fundamental topology and dynamics of the ring attractor. Q-values are used only as input signals (scaling factors K_i) to the ring attractor network, as shown in equation 7. This formulation allows learned action values to influence the ring attractor's activity pattern while maintaining its core structural properties. Regarding potential numerical instability from variance terms, it only has relevance at the initialisation point, as the uncertainty estimates from our Bayesian approach maintain reasonable lower bounds due to the inherent uncertainty in value estimation.

2 DL RNN-based implementation

The relationship between RNNs and ring attractors is fundamentally grounded in their shared ability to model continuous-time neural dynamics, as established by Beer (1995) in their work on CTRNNs. As demonstrated in our exogenous model based on Touretzky's ring attractor, these networks inherently rely on recurrent connections to maintain stable attractor states, providing a natural motivation for using RNNs in our Deep Learning implementation. Additionally, we are working on an appendix section to visualise the emergence of sustained ring patterns in our experiments.

3 SNN and biological plausibility

As highlighted by the reviewer, SNN is not relevant to this approach. We apologise for the misunderstanding and have corrected the methodology. Furthermore, we acknowledge ring attractors primarily serve heading direction representation, as shown by Kim et al. (2017). Our innovation lies in adapting spatial awareness mechanisms for decision-making: just as biological ring attractors encode spatial representations for navigation, our model uses similar principles to organise actions in a circular topology.

Q1: The symbol $\alpha$ represents the preferred orientation for a given action $a$, while $\mu_a$ represents the value associated with that action. For clarity, we have revised this notation since it was inconsistent with Equation 7, using now the same variable in both equations with the distinction that this one represents an average over a sum of samples $\bar{Q}(s,a)$.

Q2: The symbol $\alpha$ represents the preferred orientation for a given action $a$, while $\mu_a$ represents the value associated with that action. For clarity, we have revised this notation since it was inconsistent with Equation 7, using now the same variable in both equations with the distinction that this one represents an average over a sum of samples $\bar{Q}(s,a)$.

Q3: While the reviewer raises important points about verification of ring attractor dynamics, our model demonstrates key ring attractor properties inherent in its design and implementation: The spatial encoding of actions in our circular topology is not arbitrary, we enforce distance-dependent weighting through equations (5-6) that explicitly model local excitation and global inhibition, a remark of ring attractor dynamics. The performance improvements we observe in spatially-structured tasks (Table 1) suggest the model successfully maintains stable representations of spatial relationships between actions, consistent with ring attractor behaviour. Our ablation study (Section A.1.2) shows that randomly disrupting the ring structure significantly degrades performance, indicating the ring topology actively contributes to information processing rather than being merely architectural. We acknowledge and are working on additional visualisations of neural activity patterns that would strengthen our claims.

Several neural architectures in machine learning have drawn inspiration from biological principles. One common example is Convolutional Neural Networks, which were influenced by the hierarchical organisation observed in the mammalian visual system Hubel and Wiesel, 1962. While our research addresses a different domain, we similarly attempt to incorporate insights from neural mechanisms, though we acknowledge the substantial abstraction involved in translating biological principles to computational frameworks.

We agree with the reviewer and moved Zhang (1996) key proposal from the appendix to line 30 of the main text, reflecting its importance as first milestone.

The authors have partially addressed my questions and improved the presentation of the results in the revised manuscript. I would like to increase my score from 3 to 5.

However, my question, "Is there a ring attractor at all in the model?" was not adequately addressed. I believe this is an important point that requires further clarification.