Spatial-Aware Decision-Making with Ring Attractors in Reinforcement Learning Systems

We would like to deeply thank the reviewer for their detailed feedback on our work, especially on the methodology front.

Regarding the Strengths of the Article

Briefly, we would like to point out that we have also conducted experiments with a 1D continuous action space, Fig. 2. The intent is to showcase the ability of the ring attractor to handle both continuous action spaces and discrete ones, the latter ones being used in well-established benchmarks such as Atari.

Regarding the Extension of Existing Models in the Article

We believe Section 3.2.1 provides guidance for practitioners. Algorithm 1 offers step-by-step dynamics loop for the CTRNN implementation with Eq. 5, 6, 7, 8, and 11 being the key equations to map fundamentals, while Section 3.2.1 presents the mathematical formulations for the DL architecture. Additionally, Appendix A.6 goes through the integration of multi-ring architectures with the DRL approach when in need to cover higher dimensional spaces, and Appendix A.7 walks the reader through the particular implementation for each environment evaluated in the article.

For further insights, "plug and play" implementation and a follow-up on how this is being implemented into a fully functioning algorithm, we refer to our available codebase in the link in the abstract where we show multiple implementations for the CTRNN implementation and DL-based implementation of the ring attractor.

Action: However, we acknowledge that this information might benefit from being more prominently featured in the article. We will work on adding distilled pseudo-code from the implementation in the appendix.

Concerning the Demonstration of Ring Attractor Dynamics

We believe our paper provides substantial evidence for the importance of attractor dynamics through a set of sections in the article. Section 3.1 gives the differential equations for the excitatory and inhibitory populations Eq. 6, 7 that generate the bump-of-activity characteristic of a ring attractor. Algorithm 1 shows that on every decision (action selection) step, we iterate these equations to reach attractor state, ensuring the network actually settles before an action is read out.

Most importantly, we believe several ablation studies deliberately disrupt the dynamics while keeping all other code paths unchanged as data points to support our statements empircally:

• CTRNN: BDQNRA-RM randomly permutes actions around the ring (scrambling distance-dependent interactions). This results in Fig. 6, showing a glorified forward pass that doesn't alter baseline model performance.
• DL-RNN: DL-no-ring replaces the exponential decay distance weight matrix with a standard initialization matrix (eliminating spatially-structured recurrence), causing again performance in line with baseline Fig. 6. Because the only difference from the full model is the absence of proper ring interactions, we believe these results constitute strong empirical support that the settling dynamics are present and causally useful.

Appendix A.4.3 tracks the learned weight matrices, showing that forward (input→ring) weights keep their spatial profile throughout training, while recurrent weights adapt task-specific asymmetries, Fig. 7, when leaving the DL-based ring attractor distance-based weights trainable. The persistence of this profile indicates that the network consistently reaches and learns to exploit the intended attractor regime.

On the Characterization and Ablation of an RNN Layer

While our DL implementation does utilize RNN architecture, it fundamentally differs from standard recurrent layers through its structured single/multi-ring circular connectivity and distance-dependent weights (Eq. 12). It modulates the input and hidden state through two parameters: the learnable time constant τ and the distance decay parameter λ. The key innovation lies not in adding computational layers, as shown in the ablation studies in Appendix A.4, but in encoding spatial relationships through biologically-inspired ring topology.

This structured approach preserves action relationships and enables the spatial encoding capabilities that drive our performance improvements. As mentioned above, Fig. 6 shows how the RL agent performs when distance-dependent weights are removed and the weights are set back to trainable. Effectively turning the ring attractor layer back to a standard RNN, when doing so, the DRL agent shows similar or slightly lower performance compared to baseline.

Addressing Formatting and Notation Issues

1. Differential Equation Notation

In an effort to improve readability, we will remove italics from the differential equations to set them apart from the distance-based variable.

Action: Use \mathrm{} to remove italics from differential notation in equations across the article.

2. Distance Function Clarification

Regarding lines 159-160 and Eq. 12, we respectfully clarify that d(m,n) refers to the same concept throughout: The discrete distance based on the number of neurons between two interacting neurons. The different formulations (|m-n| vs. min(|m-n|, N-|m-n|)) account for linear vs. circular distance calculations within the ring topology, both representing neuron-to-neuron distance.

3. Mathematical Operators

We agree with the reviewer and thank them for the feedback. Argmax as operator should not be in italics.

Action: Convert Argmax operation to roman format.

4. Differential Equations vs. Argmax

The differential equations (dv/dt) describe how neuron activations evolve until the ring attractor reaches a stable state, while argmax(V) operates on the final settled activation values V = [v1, v2, ..., vN] to select the action corresponding to the most active neuron, Eq. 8. The dynamics (dv/dt) drive the system to convergence, and argmax(V) reads out the final decision.

Action: In order to simplify understanding throughout the article, we will assume that Fig. 1 is at the final attractor (i.e., settled state). This will enable to swap argmax(dv/dt) for argmax(V) and point to Eq. 8, so it's easier for the reader to follow.

5. Parameter Notation Consistency

θ consistently denotes the learnable parameters of the DRL agent across the different methodologies and the appendix.

Action: We will reserve θ for the full parameter set (e.g., the neural network in Section 3.1.3) and use subscripts for specific subsets within the DRL agent.

Specific Questions

Question 1: In line 172, why Ki = Q(s,a)?

In the original ring attractor formulation (Eq. 1), Ki represents the magnitude or strength of a given input signal i. By setting Ki = Q(s,a), we establish that the magnitude of a given signal is given by a state-action pair for a given action. Effectively this means we map input signals (i) to actions in action space (a), integrating the ring attractor as policy from Eq. 1 in Eq. 5. We will add a remark to highlight the substitution of input signal (i) for actions (a) and we want to thank the reviewer for the point.

Action: We will signal this substitution at the beginning of Section 3.1.2 with the following sentence:

"To integrate the ring attractor with RL, we reformulate the input signals by mapping Ki to the Q-value Q(s,a), effectively substituting input signal index i with actions a∈A."

Question 2: In Eq. (5), why does parameter s exist on the right-hand but not on the left-hand?

We originally used xn(Q) for readability since the summation ∑(a=1 to A) Q(s,a) makes the dependence on specific Q-values explicit.

Action: We will revise to xn(Q(s,a)) for notational consistency throughout Eq. 5.

Question 3: How do you sample in line 257?

The sampling in line 257 refers to Thompson Sampling within an established Bayesian Linear Regression approach from the referred work by Kamyar Azizzadenesheli, Emma Brunskill, and Animashree Anandkumar, line 227. "Efficient exploration through bayesian deep q-networks." There is also a practical implementation of this method in the codebase linked in the abstract and we would be happy to explain in detail how this method works with our approach if there are any further questions.

Question 4: The time constant τ seems very quick. What is the final τ?

In our CTRNN implementation, we set τ = Δt where Δt represents the discrete time step size. For our experiments, we used τ = 1e-3 s, with a simulation time of 0.05 s. This gives a total of 50 iterations to reach attractor state. These values were chosen empirically to optimize the performance of the CTRNN model, and were preserved across experiments (environments, seeds and ablation studies).

Action: Add simulation time integration parameters to pseudocode in Algorithm 1.

Question 5: How does τ participate in Eq. 12?

In Eq. (12), τ appears in the input signal computation V(s)m,n = (1/τ)Φθ(s)T wI→H{m,n}. Here, τ acts as a scaling factor that controls the contribution of the input features to the hidden state. A smaller τ makes the network more responsive to immediate inputs, while a larger τ creates more integration over time. This is a learnable parameter allowing the network to discover the optimal balance between responsiveness and temporal integration for each specific task.

Action: We will expand the existing sentence in line 297 to:

"Here, τ acts as a learnable scaling factor that controls the balance between immediate input responsiveness and temporal integration within the ring attractor dynamics, with smaller values increasing sensitivity to current inputs and enabling faster adaptation, while larger values promote stability and smoother transitions between attractor states by integrating information over longer time horizons."

We hope these clarifications address the technical concerns raised and demonstrate our commitment to improve the quality of our article.

We thank the reviewer for acknowledging the additions and validating our approach.

Q1. Yes, that is correct. The experiement has been designed with two cues having identical properties, except for their amplitudes to demonstrate robustness under varying cue/input sizes. Below the same experiment using two identical input cues at Step 1 and Step 7.

Q2. Regarding the evaluation and deployment of the trained weights, training and extracting the Ring Attractor layer weights and learned parameters from the DRL agent checkpoint, this requires additional computational time. Given the time constraints of the revisited rebuttal deadline, we consider prudent to take this as an action to add this to the manuscript alongside the experimental tables presented above. We will incorporate this post-training weight analysis in Appendix A.4.

Meanwhile, we kindly direct the reviewer's attention to the current Appendix A.4.3, which outlines the frozen and trainable layer insights. The key point from our current analysis is that in the tables presented in the conversation above, we are examining the collective impact on the ring dynamics of both hidden-to-hidden and input-to-hidden weight contributions. The analysis presented in Appendix A.4.2, A.4.3 suggests that input-to-hidden weights make significant independent contributions to performance. Notably, these weights preserve their spatial structure throughout training even when left trainable, and when we remove this spatial representation from the input-to-hidden connections, we observe substantial performance degradation. This suggests that the spatial encoding is key during training and inference time.

Action: Add hidden and input state contribution tables presented above and generate one additional post-training table from Highway Benchmark in Appendix A.4.

We appreciate the reviewer’s acknowledgment of our efforts to provide detailed data points aimed at fostering constructive discussion.

Q1. As shown in the weight matrices presented above the table, the evolution of hidden-to-hidden weights reflects nuanced in-game relationships between actions. The relationship between two actions may be encoded as a negative weight if, in the context of the task, suppression of one action when another is active is beneficial. The attractor state of a particular hidden neuron is determined not by the sign or magnitude of any single diagonal element, but by the aggregation of all incoming connections to that neuron through the hidden-to-hidden weights. As the reviewer mentions, one of the key aspects is that the hidden state is not imposed directly on the output, but undergoes transformation through the activation function and neuron bias as per standard RNN equations: y_t = f(w_hh · h_t + b_y), where the hidden state h_t is processed through hidden to hidden weights w_hh and bias b_y before activation f. We agree with the reviewer that this transformation means that negative or not the highest hidden state values can still produce the desired output behavior after passing through the tanh activation and scaling parameter β, as seen in previous tables.

For example, in row 5 of the weight matrix (corresponding to neuron 5), the net input results from the combined influence of all other neurons is shaped by both the structured input-to-hidden topology and the learned recurrent weights. This aggregation, rather than any requirement for strictly positive self-connections, is what determines the stability of the attractor state, that we can validate through the response of synthetic cues and transitions between action above and through ablations studies in Appendix A.4. The hidden-to-hidden weight values are the product of the structured outputs from the upper layer and adjustments to the recurrent weights during training, driven by the specific environmental inputs and task demands. In this experiment, the weight distribution encodes task-specific utility, for instance, neuron 6, corresponding to the “move backwards” action in the Highway environment, has persistently lower activation because this action carries a penalty and is rarely used. We consider the presence of negative recurrent weights while still producing a stable attractor state as evidence of secondary behavior emergence within the ring-attractor layer, incorporated on top of the explict action space layout and attractor state.

We did not employ specific loss functions to enforce ring attractor properties post-training; rather, the architectural design itself constrains and preserves the spatial encoding necessary for ring-attractor-like behavior through using hyperbolic tangent activation. As detailed in Appendix A.4.2, the fundamental element is the explicit layout of the action space over the ring topology, which biases learning toward spatially coherent action sequences regardless of specific weight signs.

Q2. We consider the inertia observed in this highway environment represents an accentuated attractor behavior facilitated by the ring structure that acts as structured exploration bias, guiding the agent and learning faster, see Section 4.2, towards effective driving strategies while preventing chaotic, unrealistic action sequences that tend to occur at the begining of the training/exploration stage. This inertia characteristic has developed beyond the smooth transitions observed in our initial layout, seeing in the tables provided above, indicating task-specific adaptation. However, we emphasize that such pronounced inertia is not necessarily required or positive for all action combinations in both navigation or non-navigation tasks, hence its absence in previous experiments. The key contribution lies in the emergence of attractor states with the parameter τ modulating input contributions to hidden states; and crucially, the preservation of structural layout through explicit mapping of action spaces, extensively analysed in the manuscript in Sections 3.2, 4.2, Appendix A.4.1, A.4.2, A.7.

Thank you for your thorough review and suggestions, we are grateful you appreciated the work!

Related Work Critical Contrasting

Based on the reviewer point, we would like to go a step further and provide deeper insights into what sets apart our current approach from existing literature.

Action: We will extend the critical contrasting points throughout the literature review in Section 2 as follows:

Section 2.1 Spatial Awareness in RL

Line 62: "Although these approaches demonstrate the importance of spatial awareness in RL, they often lack the biological plausibility found in neural circuits."

Updated Line 62: "Although these approaches demonstrate the importance of spatial awareness in RL, they rely on emergent learning to develop spatial understanding through comprehensive architectural additions, requiring extensive training to discover action relationships. In contrast, our ring attractor approach explicitly encodes spatial structure in the action space, providing spatial inductive bias that accelerates learning rather than relying on the RL agent to autonomously discover all critical action space relationships, which may result in suboptimal or incomplete spatial understanding."

Section 2.2 Biologically Inspired Machine Intelligence

Line 78: "Their approach demonstrated rapid learning and adaptation to new tasks, similar to the flexibility observed in biological learning systems."

Updated Line 78: "Their approach demonstrated rapid learning and adaptation to new tasks, similar to the flexibility observed in biological learning systems. However, these approaches primarily focus on mimicking neural representations or learning dynamics, whereas ring attractors specifically encode spatial relationships in the action space itself, providing both biological plausibility and direct spatial awareness for action selection."

Section 2.3 Uncertainty Quantification

Line 98: "...incorporates efficient Thompson sampling and Bayesian linear regression at the output layer to factor uncertainty estimation in the action-value estimates."

Updated Line 98: "...incorporates efficient Thompson sampling and Bayesian linear regression at the output layer to factor uncertainty estimation in the action-value estimates. While these methods treat uncertainty estimation as a separate module applied to existing architectures, our ring attractor approach integrates uncertainty through the variance parameters of Gaussian input signals, making uncertainty awareness an intrinsic part of the spatial action representation."

Section 2 Summary (Lines 99-103)

Line 101: "However, there remains a gap in integrating these elements into a cohesive framework."

Updated Line 101: "However, there remains a gap in integrating these elements into a cohesive framework that simultaneously provides biological plausibility, explicit spatial encoding, and natural uncertainty handling within a single neural architecture."

These additions would transform the related work from a simple literature review into a more critical analysis that clearly positions the ring attractor approach relative to existing methods.

Experimental Scope and Applicability

While we acknowledge testing primarily on Atari and navigation environments, we believe our experimental scope demonstrates broader applicability than suggested:

• Diverse Action Spaces: Our experiments span discrete (8-action Mario Bros), continuous 1D (Highway environment), and complex discrete spaces (18-action Atari games) with multiple ring attractors as detailed in Table 2.

• Two Implementation Paradigms: We provide both CTRNN-based (Section 3.1) and deep learning implementations (Section 3.2), with the DL version specifically designed for "integration and comparison with existing DRL frameworks" enabling broader adoption.

• Scalable Architecture: Section 3.2.1 and Appendix A.6 walks the reader through the extensibility to N-ring configurations for higher-dimensional control, with our double-ring implementation Eq. 26, 30; showing how the approach scales to multi-dimensional action spaces.

• Performance Consistency: Table 1 shows consistent improvements across 19 different Atari environments with varying action complexities, relationships and different types of decision-making tasks, suggesting robust generalization.

However, we agree with the reviewer on the continuous control limitation. There is always room to grow further this piece of work, which we are aiming to cover in a subsequent publication with deployment in continuous 3D real-world robotics tasks. This limitation has been outlined in our future work, Section 5, where we explicitly identify "continuous control tasks" as a research direction.

Specific Questions

Question 1: How does your ring-attractor encoding compare, both conceptually and empirically, to existing spatial representations such as grid-cell encodings or attention-based spatial RL?

Conceptual Differences

Grid Cells vs. Ring Attractors: Grid cells [2] encode spatial positions in the environment potentially key for navigation algorithms composed of visual place recognition, path planning and other localization modules. While our ring attractors encode spatial relationships between signals, or in this case, actions. As noted in Section 2.2, grid cells "emerged naturally in agents trained on robotics tasks," whereas our approach explicitly structures the action space itself upfront.

Attention Mechanisms vs. Ring Attractors: Attention-based approaches [44] reason about spatial relations between entities, that are learnt over time in vast datasets or simulation environments to tune the self-attention layers. Increasing computational overhead, size and complexity of the DRL agent. On the other hand ring attractors maintain stable representations of action relationships through attractor dynamics, Eq. 1, 2, 3, 4; enabling the agent to distill information with little to no additional architectural and training overhead.

Empirical comparison: We haven't found any major research line that applied grid cells to well-known DRL benchmarks. Additionally, these two approaches are viewed as operating at fundamentally different levels of the decision-making hierarchy, with grid cells providing environmental state representation and ring attractors providing action space organization. Grid cells have been tied to navigation tasks so far, while ring attractors could be applied to spatial decision-making problems. The comparison would be like comparing a GPS system (grid cells mapping spatial location) with a 2D action palette (ring attractors organizing spatially-related choices into a circular selection interface), both spatial, but serving different purposes.

Architecture Differences

Topology & dimensionality:

• Ring attractor: 1D continuous manifold with periodic boundary; ideal for relationships between signals/actions, Eq. 5, 6, 7, 8.
• Grid cells: 2D hexagonal tiling of state space; emergent in RNNs for localization [2,10]. Our focus is on action selection and policy building, not spatial localization per se.
• Attention RL: learns soft relational representations over entities [44]; no hard prior on spatial continuity, relies on data to infer adjacency.
• Uncertainty handling: Our design injects σₐ from BLR directly into the bump width (σₐ in Eq. 5), giving an interpretable link between epistemic uncertainty and action diffusion; existing spatial encodings rarely bind uncertainty to the topology this tightly.

Action: We acknowledge that grid cells are another biologically plausible mechanism applied to RL navigation tasks, we will add the paragraph below in Section 2.2 covering the fundamental differences:

"In RL, grid cells and ring attractors serve distinct roles. Grid cells encode spatial state information through hexagonal firing patterns, functioning as fixed basis functions for value functions and spatial localisation (Banino et al., 2018; Cueva & Wei, 2018). Ring attractors encode persistent states (directional and spatial bias) through recurrent dynamics that actively maintain information over time (Khona & Fiete, 2022). While grid cells provide emergent spatial input features, ring attractors are dynamical components that provide intrinsic spatial encoding for memory states (Kilpatrick & Ermentrout, 2013), which we translate here to the encoding of the action space during the action selection process."

We would like to thank the reviewer for their constructive feedback and for the appreciation of our work.

We would like to respectfully challenge the characterization of our empirical validation as limited, given that our evaluation includes the widely-adopted Atari 100k benchmark with multiple diverse environments, comparisons against established state-of-the-art methods, testing across multiple domains (discrete/continuous control, navigation, game scenarios).

This has enabled us to demonstrate the applicability of our research across different environment configurations through various ring architectures and properties, as shown in Table 2. This enables our study to make claims beyond typical approaches in navigation, expanding to decision-making games. While real-world deployment remains an area we would like to assess our research in (Section 6), we believe the widespread adoption of the benchmarks used in this study and the provided code implementation will enable peers to validate and build upon our current approach.

We thank the reviewer for their thorough and constructive feedback and for the appreciation of our work. Below, we address each point raised by the reviewer:

Clarity and Connection Between Insights and Method

Let us clarify the connection between the deep insights and the proposed method, starting from the beginning:

1. Biological Plausibility, Appendix A.1: Ring attractors represent a computational principle where neurons are arranged in a circular topology to maintain spatial representations. Originally proposed by Zhang [45] for encoding heading direction and empirically discovered by Kim et al. [16] in Drosophila, these networks exhibit circular organization where head direction cells encode different spatial directions [36]. For our methodology, biological ring attractors integrate multiple sensory inputs through distance-weighted connections [46, 12] and maintain representational stability through excitatory-inhibitory dynamics [35], properties we implement through our distance-based weight functions in Eq. 4 and neural dynamics in Eqs. 2, 3.

2. Technical Translation of Neuroscience-based Principles, Section 3.1: Our method translates these biological principles by mapping actions to circular positions Eq. 5 and implementing synaptic connectivity through exponential decay functions w(Em→En)=e^(-d²(m,n)), Eq. 4 that mirror biological connection strengths between spatially proximate neurons. The continuous-time dynamics in Eq. 2, 3 capture the biological property of attractor states while enabling transitions between related actions, implementing the spatial encoding observed in biological systems where similar spatial directions maintain stronger mutual influence through the circular network topology.

3. Implementation Approach and Motivation, Section 3.1, 3.2: We developed two implementations to validate different aspects of biological translation: the CTRNN exogenous model in Section 3.1 provides biological fidelity through explicit excitatory/inhibitory dynamics in Section 3.1.1 and external Q-value processing in Section 3.1.2, enabling validation of ring attractor principles with uncertainty quantification in Section 3.1.3. The DL integrated model, Section 3.2, embeds ring topology within RNN weight matrices Eq. 12, Section 3.2.1 for deployment in DRL frameworks, maintaining the biological principle of distance-weighted spatial relationships and attractor states while enabling end-to-end training and scalability within existing DRL frameworks.

4. Performance Connection to Biological Insights, Section 4, Appendix A.4: The spatial encoding translates biological advantages into performance gains through action similarity leverage, demonstrated by our 53% improvement on Atari 100k benchmark, Table 1, Section 4.3. Our ablation studies represented in Fig. 5, 6 show that randomized action arrangements degrade performance, confirming the bioinspired spatial structure is necessary rather than incidental. Following uncertainty principles [42], our Bayesian implementation, Section 3.1.3, incorporates confidence-based decision making, showing that the foundation provides a framework for encoding spatial relationships in action spaces.

Action: We will include an introductory paragraph in Section 3 at line 105 as follows:

"Ring attractors represent a computational principle where neurons are arranged in a circular topology to maintain spatial representations. Originally proposed by Zhang [45] for encoding heading direction and empirically discovered by Kim et al. [16] in Drosophila, these networks exhibit circular organization where head direction cells encode different spatial directions [36]. For our methodology, ring attractors integrate multiple cues through distance-weighted connections [46, 12] and maintain representational stability through excitatory-inhibitory dynamics [35]. This bioinspired structure has been repurposed in this line of research to explicitly encode the action space during the decision-making process, as illustrated in this section."

Uncertainty Quantification Benefits

The ring attractor provides distinct advantages beyond standard Bayesian formulations:

Spatial Uncertainty Propagation: Unlike standard BLR which treats actions independently, our approach propagates uncertainty spatially across the ring Eq. 11, allowing uncertainty about one action to influence similar actions. BLR Eqs. 9, 11 yields per-action mean/variance; the ring uses σa as the width of each Gaussian in Eq. 5, so uncertainty directly modulates how broadly activity spreads across neighboring actions, not just the final argmax.

Action-Space Structure: The ring topology enables uncertainty to be distributed according to action similarity, not just individual action confidence. This is particularly beneficial for spatial tasks where action relationships matter.

Empirical Evidence: Fig. 2 shows BDQNRA-UA with uncertainty-aware ring attractors consistently outperforms both standard BDQN and BDQNRA without uncertainty, demonstrating the complementary benefits.

Key Performance Components

We believe our ablation studies in Section A.4 should identify the contributions of the key components to provide the boost in empirical performance:

Spatial Topology: Fig. 5 shows that randomizing action positions in the ring in the CTRNN-based model (BDQNRA-RM) eliminates most performance gains, indicating spatial structure is key in delivering faster and overall high learning performance.

**Circular Connectivity:**Fig. 6 demonstrates that removing distance-dependent weights in the DRL agent, effectively converting the ring attractor layer into a standard RNN, reduces performance significantly in spatial environments like Ms Pacman and Chopper Command.

Specific Questions

Question1: Scaling to Continuous and Higher-Dimensional Spaces

Continuous Spaces: Section 4.1 and Fig. 2 demonstrate successful application to OpenAI Highway environment with continuous 1D circular actions.

Higher Dimensions: Section A.6.2 and Table 1 show our double-ring architecture for games requiring multiple action dimensions (e.g., movement + combat). The approach scales to R rings via block matrix structures, Section A.6.3, maintaining O(R) complexity through selective coupling.

However, we agree with the reviewer that further work can be done to explore real-world complex scenarios. There is always room to grow further this piece of work, which we are aiming to cover in a subsequent publication with deployment in continuous 3D real-world robotics tasks. This limitation has been outlined in our future work, Section 5, where we explicitly identify "continuous control tasks" as a research direction.

Question 2: Performance Under Noise and Partial Observability

Future Work: We agree with the reviewer that we have yet to evaluate systematic noise robustness, which we aim to fulfill through 3D real-world robot navigation tasks.

Action: We will add in Section 5:

"Additionally, the extension to 3D spatial navigation tasks presents an organic expansion of this work, these scenarios will naturally incorporate realistic sensor noise to the system."

Regarding partial observability, we believe our benchmarks actually do test partial observability: Super Mario Bros uses limited scrolling windows preventing agents from seeing upcoming obstacles, while Atari games feature off-screen threats and limited sensor ranges.

Question 3: Ring Attractor Application Scope

For the CTRNN model, as these models are not trainable, it becomes an exogenous model that performs action selection as outlined in Section 3.1.2.

The ring attractor applies specifically to action selection with full integration through the DRL implementation. As described in Section 3.2.1, the ring attractor serves as the output layer of the DRL agent, processing action-value estimates while preserving their spatial relationships. The approach integrates fully within existing DRL agents and frameworks (DDQN, PPO, EfficientZero).

Question 4: Comparison to Related Spatial Architectures

Graph Neural Networks: While GNNs capture spatial relations through learned graph structures [3], they require learning spatial relationships during training, whereas ring attractors exploit an inductive bias that directly encodes the action space and drives the policy through explicit spatial embedding. Additionally, GNNs lack the temporal dynamics of ring attractors, missing the uncertainty encoding and attractor state stability that impact guide action selection.

Grid-Cell Inspired Architectures: Grid-cell architectures [2, 10] encode environmental spatial locations and focus on spatial memory and path integration, rather than directly embedding the action space into the policy. Grid-cell representations operate at the environmental level, whereas ring attractors provide direct action-level encoding that guides the action selection process.

Action: We acknowledge that grid cells is another biologically plausible mechanism applied in navigation, so because of several similarities we will add a paragraph in Section 2.2 covering the fundamental differences as follows:

"In RL, grid cells and ring attractors serve distinct computational roles. Grid cells encode spatial state information (where you are) through hexagonal firing patterns, functioning as fixed basis functions for value functions and spatial generalization (Banino et al., 2018; Cueva & Wei, 2018). Ring attractors encode persistent states (directional and spatial bias) through recurrent dynamics that actively maintain information over time (Khona & Fiete, 2022). While grid cells provide emergent spatial input features, ring attractors are dynamical components that provide intrinsic spatial encoding for memory states (Kilpatrick & Ermentrout, 2013), that here we translate to encoding of the action space."

We are grateful for the careful review and particularly appreciate the reviewer's recognition of our research.

Addressing Weaknesses

Clarifying Why Ring Attractors Help: The Role of Spatial Inductive Bias

Ring attractors maintain stable spatial information representations, preserving action relations lost in traditional flattened action spaces. The fundamental mechanism operates through distance-dependent connections that encode spatial relationships between actions, with the attractor state evolving according to Eq. 2,3 for the CTRNN implementation and Eq. 12 for the DL approach. The spatial inductive bias accelerates learning by explicitly encoding action relationships within the circular topology, enabling the agent to leverage similarities between neighboring actions in the spatial domain. This structured representation facilitates more efficient exploration and policy gradient estimation, as evidenced by our 53% performance improvement on the Atari 100k benchmark. Our analysis in Section A.4.3 demonstrates that when forward pass weights are made trainable, the network preserves the distance-dependent decay patterns throughout training, Fig. 7, suggesting that the spatial topology provides an inherently beneficial structure for action-value function approximation. This preservation suggests that the ring attractor has converged to an optimal spatial configuration that enhances learning efficiency.

Evidence from Ablation Studies:

Section A.4.1 presents the CTRNN ablation study where BDQNRA-RM randomly distributes the action space across the ring, disrupting the natural topology. Fig. 5 shows this manipulation eliminates performance gains, reducing results to baseline BDQN levels. From these ablation studies we may infer that the settling dynamics contribute to the observed improvements. The DL ablation study (Section A.4.2, Fig. 6) removes the circular weight distribution, eliminating spatially-structured recurrence and effectively converting the ring attractor layer into a standard RNN. Performance drops in spatial environments like Ms Pacman and Chopper Command. Section A.4.2 demonstrates that removing the circular weight distribution in our DL implementation eliminates the spatial structure, resulting in performance degradation in spatially-oriented environments like Ms Pacman and Chopper Command (Fig. 6), further confirming the importance of the ring topology. Because the only difference from the full model is the absence of proper ring interactions, these results suggest that the spatial structure may drive performance improvements.

Applicability Across Action Spaces:

The approach handles both continuous and discrete action spaces through unified spatial encoding. For continuous spaces, Fig. 2 demonstrates application to the Highway environment with 1D continuous circular action space. For discrete spaces, Table 1 shows improvements across the Atari benchmark.

Action: We will add at the beginning of Section 1 (line 32), following up from the mention in the abstract to the problem statement, this prior to our research:

"While many action spaces possess inherent topological structure (Todorov et al., 2012), traditional neural networks represent actions as orthogonal vectors that ignore these relationships. Ring attractors bridge this gap by providing a neural substrate that preserves spatial ordering and similarity between actions. This is particularly relevant given recent advances showing that spatial structure in representations improves sample efficiency [2, 44] and generalization [3, 13], yet these benefits have not been systematically applied to action selection mechanisms themselves."

Addressing the Questions

Question 1: Learning Deviation from Original Ring Attractor Dynamics

Standard Implementation Approach:

Our implementation maintains fixed, input-to-hidden connections as described in Section 3.2.1, where weighted RNN connections include fixed, non-trainable, input-to-hidden connections (wI→H) to maintain the ring's spatial structure, and learnable, trainable, hidden-to-hidden connections (wH→H) to capture emerging action relationships as mentioned in lines 283, 284.

Ablation Study Insights:

Section A.4.3 analyzes model dynamics with both forward pass and hidden-to-hidden weights made trainable, which should address this question. The results suggest two patterns:

Forward pass preservation: The forward pass connections preserve the ring structure over training time, with distance-dependent decay patterns maintained throughout the learning process, as demonstrated in Fig. 7.

Hidden layer adaptation: The hidden-to-hidden connections develop specialized patterns that enable the encoding of environment-specific relationships between neurons in the hidden space while building upon the structured spatial representation from the forward pass (Fig. 8).

These findings may indicate that our standard implementation approach where forward pass connections are fixed and only hidden-to-hidden weights remain trainable is reasonable, as the network appears to preserve spatial topology even when given freedom to deviate.The learnable time constant τ and scaling factor β from Eq. 12 control the temporal dynamics and magnitude of the attractor's hidden state contribution over time. While we have not provided specific data points for the evolution of these parameter values, our appendix analysis demonstrates the contribution and stability of both hidden and feedforward connections in the DL implementation. Fig. 7 and Fig. 8 show that these connections do not collapse during training and maintain reasonable, consistent scales over time. Further insights on the parametrisation of the DRL agent can be found in the code link attached in the abstract as well.

Question 2: Understanding the Strong Performance Improvements

Spatial Structure as a Contributing Factor:

The spatial encoding may translate into performance gains through action neighbouring. Our ablation studies suggest that randomized action arrangements degrade performance, which could imply that the spatial structure contributes to the observed improvements.

Empirical Evidence:

Section A.4.1, A.4.2 shows that BDQNRA-RM with scrambled distance-dependent interactions and EffZeroRNN with removed circular topology result in performance in line with baselines, while maintaining proper spatial relationships yields the 53% improvement observed on the Atari 100k benchmark. The controlled nature of this experiment suggests spatial structure as a contributing factor.

Complementary Uncertainty Benefits:

Additionally, the uncertainty-aware capabilities provide additional performance improvements beyond spatial encoding. Fig. 2 shows BDQNRA-UA with ring attractor and uncertainty awareness consistently outperforms both standard BDQN and BDQNRA without uncertainty.

Question 3: Limitations and When the Inductive Bias Might Not Help

Evidence of Performance:

Our experimental results suggest that ring attractors provide performance improvements across diverse RL tasks, with benefits extending beyond purely spatial tasks. Table 1 shows performance at or above baseline levels across all tested environments. For environments requiring multiple action dimensions, our double ring configuration (detailed in Appendix A.6.2) maintains spatial relationships within each action plane while allowing controlled interaction between dimensions through the coupling parameter κ, as demonstrated in games like Seaquest and BattleZone.

Potential Limitations:

The approach would likely face limitations in action spaces with completely uncorrelated actions or very small action spaces where the ring structure overhead provides no advantage, e.g. action space with two discrete actions.

Action: We will flag in Section 5 (line 382) additonal limitations:

"The ring attractor approach may exhibit reduced applicability in discrete action spaces with cardinality |A| < 3 where the circular topology overhead exceeds potential spatial encoding benefits, and in action spaces where the action correlation matrix approaches the identity matrix, indicating absence of spatial dependencies or sequential relationships between discrete actions."

Future Research Directions:

We agree with the reviewer that further work can be done to explore complex scenarios to put the inductive bias through further real-world tests. Which we are aiming to cover in a subsequent publication with deployment in continuous 3D real-world robotics tasks. This limitation has been outlined in our future work, Section 5.