Deep RL Needs Deep Behavior Analysis: Exploring Implicit Planning by Model-Free Agents in Open-Ended Environments

Thank you for your thoughtful comments! We will respond to specific concerns below:

1 (main purpose of the paper)

We will clarify the language of how we describe our contributions in the manuscript, thank you for pointing out that this could be unclear. Our four main contributions are as you have listed in the summary. The core goal of this work is to demonstrate how studying standard deep RL agents in complex naturalistic environments using analysis methods from neuroscience can reveal complex behaviors and neural patterns that are informative for computational neuroscience and ML research.

Further, we hope to encourage both computer scientists and computational neuroscientists to dig deeper into the joint behavior and neural data of agents in open-ended environments, as they provide controlled frameworks to test analysis methods intended for more complex real world biological data where controllability and statistical rigor may be lacking. Existing benchmarks do not easily provide the behavior, neural data, and naturalistic structure to be suitable for this, and thus we have provided both a new task and an analysis framework for it. Navigational circuits in large environments are especially poorly understood in both computer and biological sciences, as most well-studied tasks are small and/or fully observable arenas where complex planning with memory is not required. Thus, for novelty and impact we have focused on navigation in our analysis here, but many other aspects of an agent could be studied using similar techniques as well.

We will add further references to the intro and expand discussion to contextualize this in the paper, such as:

Abney, Drew H., et al. "Advancing a temporal science of behavior." Trends in Cognitive Sciences (2025).

2 (paper organization)

Given the space requirements, we tried to have a coherent and complete narrative in the main text while referencing extensive methodological details and less critical experiments in the appendix.

To address your specific points, we will fix the figure reference order and revise to better connect our findings in the results section with the analysis methods described in detail in the appendix that were used to obtain them. Thank you for helping us to improve the clarity of the paper!

For reproducibility, all analysis and task code are available via an anonymized github link present in the main text of the paper, along with basic documentation on how to reproduce our experiments. We are committed to maintaining the project github publicly post-review. If there are specific requests for improvement we would be happy to respond to them.

3 (task complexity)

We designed ForageWorld to find a balance between task complexity, transparency to analysis, and biological plausibility. It is certainly the case that more complex deep RL tasks exist, but many such tasks (for example, Minecraft) are much more difficult to study and in particular more difficult to instrument for behavior analysis. In addition, the increased computational cost of experiments in such environments is prohibitive to some statistical analysis, and many computational neuroscientists in particular do not have access to compute resources sufficient to run experiments in such complex environments.

Relative to common neuroscience tasks used to study foraging, e.g. n-armed bandit tasks which lack spatial or temporal components, ForageWorld pushes the complexity boundary on the neuro-side of neuroAI while still remaining tractable for neuroscience tools and neuroscientists to study. Very complex tasks would be difficult to analyze with the current toolboxes used in neuroscience, which were designed for tasks of low-to-intermediate complexity. Excluding such analyses and many potential users by working with a much more complex environment would decrease the impact of our work.

Further, ForageWorld as it exists is not trivial to learn- our agents require billions of timesteps of experience to train to competency on the task, and Craftax, the benchmark which we build upon to develop ForageWorld, has not been solved by any agent or architecture to date.

4 (model choice and hyperparameters)

Certainly, we are happy to provide clarity on our model design choices. We will revise the manuscript to make explicit why we use an RNN model as well. We use RNN models because such models can be studied readily with neuroscience analysis techniques and have lower complexity than transformer models while still remaining general-purpose learning agents with hundreds of thousands of free parameters, allowing us to strike a balance in terms of compute requirements, amenability to analysis, and model capability. Our baseline PPO-RNN model is largely identical to the best-performing model presented by Craftax, other than the addition of the path integration output head and changes to model hyperparameters. On the original Craftax task, this PPO-RNN model is only moderately worse than the best reported model on the Craftax benchmark, a much larger transformer-based model which would not be compatible with many neuroscience analysis techniques [*].

[*] Dedieu, Antoine, et al. "Improving transformer world models for data-efficient rl." arXiv:2502.01591 (2025).

While an LLM-based model might be able to learn faster in ForageWorld by using prior knowledge, our goal is not to maximize performance on this task, but to understand what an agent trained de-novo on the task learns, and to prove out analysis techniques for obtaining such understanding. Many of the neuroscience analysis techniques we use would not apply to such a model, and we would not be able to measure learned features such as forward/backward position encoding in the RNN (figure 5). In addition, as ForageWorld is not a language-based task translating between its symbolic observation space and an LLM would not be straightforward, and would introduce questions of prompting that have no connection to neuroscience or neural analysis but which would greatly impact performance. Further, using a pre-trained model such as an LLM makes connecting interesting behaviors to environment/reward function features (such as the emergence of planning in agent’s activations as a result of our open-ended reward function) impossible, as it is unclear what behavior is induced by the environment and what is derived from prior knowledge in the pre-trained model. Lastly, densely recording and analysing network activations across tens of thousands of timesteps per episode and tens or hundreds of episodes per training run (as we have done here) for an LLM would be intractable in terms of computational and storage requirements for the majority of academic labs.

Training and environment details can be found in Appendix A, and model and environment hyperparameters are listed and selected values motivated in Table 3 in the appendix.

5 (multi-agent support)

ForageWorld currently does not support multi-agent training, but we have ongoing work seeking to develop a multi-agent successor. Craftax also is not multi-agent.

6 (focus on deep RL training)

None of our analysis is specific to deep RL training in particular, and we hope that future researchers interested in our work try many other types of training algorithms. We use deep RL here because we are interested in what emergent capabilities and features an agent will learn from training on a relatively simple reward function (maintain high resource levels and avoid death), and because such RL algorithms combined with memory networks (like RNNs) are of particular interest to neuroscientists as the best known computational approximation of animal learning. However, given a pre-existing behavioral cloning dataset (which, as ForageWorld is a novel task, did not exist prior to developing our deep RL agent) it is entirely possible to apply similar analysis to an agent trained via BC, or to any other agent with an RNN-type model. We will note these potential extensions in our discussion of future work, thank you for the suggestion!

7 (necessity of world models for planning)

This view (that world models are necessary for explicit planning) is often stated in neuroscience and adjacent computational biology fields, although as written in the Bush et al (2025) paper [*] suggested by reviewer KWpH this topic is understudied considerably even in ML/AI (see the KpWH response about that contemporaneous work, which had a simpler task, shorter time horizons, full observability, and a reward structure that directly incentivizes planning). We will clarify this in the paper. On the neuro side, it was only recently appreciated that insects are individually quite intelligent, rather than just being swarm-governed or simple sensory-feedback automatons (see “Lars Chittka. The mind of a bee. Princeton University Press, 2022.” which we cited). The sort of planning a small insect brain can do is currently unknown, but their brains are only 1-2 orders of magnitude larger than our small DRL agent models, and the commonly-held historical view is that a brain of such size does not have the capacity to learn explicit planning without a world model (the existence of which has not been discovered by experimental neuroscience to date).

[*] Bush et al. Interpreting Emergent Planning in Model-Free Reinforcement Learning. ICLR 2025.

8 (Feed-forward comparison)

We did compare our RNN model to a feedforward agent, trained identically via PPO but with the RNN removed and replaced by an MLP of roughly similar parameter count. This ablation could not learn the task, as shown in Figure 2. This demonstrates that memory is required to learn our task to a non-trivial level of competency.

9 (Referring to two figures in section 4.2)

Yes this is intentional. Figure 2 refers to the performance of feedforward (memoryless) agents quantitatively, while Figure 10 shows an example of the suboptimal pathing of such an agent.

Thanks for these clarifications, I've gone through your rebuttal and the other author's critiques and I'm happy to raise my score primarily given the following points from your rebuttal:

• The ForageWorld environment contains sufficient complexity while remaining accessible to neuroscience analysis.
  • In particular, revisiting some of the analysis in the paper on structured exploration, adoption of multi-objective policies, and the emergence of behavioural competencies.
• Clarification of the claim involving World Modelling and the assumptions made in the neuroscience literature.
• Your commitment to address issues around paper clarity.

These points address many of the issues that I had with the quality, novelty, and overall clarity of the work, thanks for clearing this up. I still believe that it could be helpful to expand a bit on on your empirical findings regarding the contingency of world models in deep RL agents in naturalistic environments. You cover this a bit in the discussion section however it may be helpful to provide some insight as to: 1. whether there are conditions when a world model would be necessary for forage world type tasks, 2. the trade-offs to having a world model for the agent vs. the model-free approach taken, and 3. in what ways might a model-based agent reduce the clarity of the neural analysis.

Thank you for your helpful comments! To respond to specific questions:

1 (comparison to Bush et al, and causal interventions)

Thank you for bringing this work to our attention- we missed this recent paper during preparation, but it is definitely an important point of comparison, and we will discuss it specifically in the manuscript. It is also an interesting paper and was a thought-provoking read, the pointer is much appreciated.

However, we must point out the conference rules on contemporaneous work:

“For the purpose of the reviewing process, papers that appeared online after March 1st, 2025 will generally be considered "contemporaneous" in the sense that the submission will not be rejected on the basis of the comparison to contemporaneous work. Authors are still expected to cite and discuss contemporaneous work and perform empirical comparisons to the degree feasible. Any paper that influenced the submission is considered prior work and must be cited and discussed as such. Submissions that are very similar to contemporaneous work will undergo additional scrutiny to prevent cases of plagiarism and missing credit to prior work.”

Bush et al, with an Arxiv date of April 2nd, is considered contemporaneous work and thus a decision regarding our paper should not be based on comparison to this very recent paper.

That said, while Bush et al also show evidence of learned planning without a world model, ForageWorld differs from Sokoban in multiple important ways that make the planning evidence we see significant in excess of this prior work.

First, ForageWorld is partially observable and not deterministic- our agent can only see a small portion of the arena (9x7 grid within a 96x96 arena, versus the 8x8 grids of Sokoban), and predator and food spawns are random but geographically biased, meaning the agent must plan under uncertainty and unknown information, which add significant barriers to the emergence of planning within the RNN model.

Second, ForageWorld requires planning on a much longer timescale than Sokoban, with episodes lasting thousands of timesteps rather than dozens, meaning agents must plan and re-plan continuously within an episode rather than plan once and then execute.

Third, our reward function is open-ended (keep food/drink/health/energy levels high), and does not directly reward planning over short time horizons, whereas Sokoban provides short-term feedback if planning is not successful via failed levels. When trying to survive for longer periods of thousands of timesteps our agent seems to plan out its exploration and revisitation of food patches due to the dynamics of patch depletion, and potentially maps out potential routes to evade predators. These behaviors are emergent, and are only indirectly rewarded across long time horizons of hundreds or thousands of timesteps.

Lastly, our agent model and learning algorithm are very standard- we use a GRU with input/output fully-connected layers and PPO to train, as they are very commonly used in other research. This means that our finding of planning behavior is likely to generalize easily to other tasks/domains, and specialized models or algorithms are not required for planning to emerge. For example, the ability to perform multiple timesteps of computation per environment timestep is not essential for planning to emerge, per our results, which generalizes the findings of Bush et al.

In terms of doing a similar causal intervention as Bush et al, it’s conceptually possible, but our environment is larger and more complex than the small fully observable 8x8 grid world used in [1]. Their intervention worked because they were able to handcraft a simple environment to have two possible solutions, and could trigger the agent to pick one over the other. Our open-ended survival task does not have such a way to binarize two “solutions” to a given arena configuration, as there are many patches, paths, and movement strategies an agent could take. Scaling their approach to an environment of our complexity would be very interesting, but is not trivial. Methods to intervene in agents doing long-term planning in complex environments in order to modify or correct their plans would make an interesting topic for future research.

[1] Bush et al. Interpreting Emergent Planning in Model-Free Reinforcement Learning. ICLR 2025.

2 (additional analysis techniques)

We agree that there are a wider range of neuroscience tools that could be applied to ForageWorld in follow-up work. While there are many possible options, to expand the immediate toolkit in the current work, we have added analysis and code (to our anonymized github linked in the paper) for using generalized linear models (GLMs) to predict single GRU neural spiking activity from behavioral variables for each GRU neuron. This addition allows us to better understand the functional circuits that exist within our trained networks. While we are not allowed to provide experimental plots here due to the recent conference guideline emails, this analysis reveals single neuron-level behavioral variable encoding that complements the current decoder results.

The new neural-focused GLM experiments that we performed add evidence for an accumulation threshold circuit that pressures the agent to return to the origin the farther away it travels radially from the central start position. Specifically, we found the average response differential of position-selective neurons (i.e. the magnitude of the GLM regressor for a given position bin) increases with radial distance from the origin as the agent explores. This circuit could underlie the frequent origin revisitation we see our agents perform, with some sort of attractor driving the agent back to the origin in a ramping way with distance. Our GLM code is flexible, and can support single neural analyses using behavior time series that future researchers may come up with on our task or related ones, provided they use similar recurrent architectures and behavior logging. We have uploaded the code for this analysis to the anonymous github we linked in the manuscript and will add this analysis to the paper appendix.

Representational similarity analysis would be difficult because it is designed for a lower number of simpler input states and usually time averages neural activity (the neural GLM predicts the full activity time series in contrast). Our agent has 3 different time-varying physiological levels and highly varying pixel input each time step, thus the conditions one contrasts with RSA and how time series are segmented would have to be carefully considered. We will add a note about this to the paper.

Thank you for your thoughtful response!

1. I would like to point out that the Bush et al. is first published on 23 Jan 2025 on OpenReview as ICLR 2025 accepted paper. Therefore, unfortunately it does not fall within the definition of contemporaneous work. However, after considering the authors' rebuttal, I do agree with the authors that there is a considerable amount of difference between this paper and Bush et al. While this paper is not the first to demonstrate that RNN-based model-free DRL agents can exhibit planning-like behavior purely through emergent dynamics, it does consider this problem in a more difficult and general setting, thereby generalizing such finding of planning-like behavior.

2. The connection to neuroscience & ethology is an aspect that distinguishes this paper from Bush et al. and bridges DRL with neuroAI. The new neural-focused GLM experiment the authors introduced in the rebuttal further strengthens such connection.

Based on the above points, I am leaning more towards accept than borderline accept and will increase the score to 5.

Thank you for your comments. To respond to specific concerns:

1 (why not an LLM agent)

While an LLM-based model might be able to learn faster in ForageWorld by using prior knowledge, our goal is not to maximize performance on this task, but to understand what an agent trained de-novo on the task learns, and to prove out analysis techniques for obtaining such understanding. Many of the neuroscience analysis techniques we use would not apply to such a model, and we would not be able to measure learned features such as forward/backward position encoding in the RNN (figure 5). In addition, as ForageWorld is not a language-based task translating between its symbolic observation space and categorical action space and an LLM or VLM would not be straightforward, and would introduce questions of prompting that have no connection to neuroscience or neural analysis but which would greatly impact performance. Further, using a pre-trained model such as an LLM makes connecting interesting behaviors to environment/reward function features (such as the emergence of planning in agent’s activations as a result of our open-ended reward function) impossible, as it is unclear what behavior is induced by the environment and what is derived from prior knowledge in the pre-trained model. Lastly, densely recording and analysing network activations across tens of thousands of timesteps per episode and tens or hundreds of episodes per training run (as we have done here) for an LLM would be intractable in terms of computational and storage requirements for the majority of academic labs.

Using LLM architectures for computations underlying foraging, navigation, search & rescue, etc. tasks is an interesting future direction, but considerable work needs to be done on developing analysis techniques suitable for those architectures first. We will cite preliminary work in this direction for the future work appendix section, such as:

Qiao, Yanyuan, et al. "Open-nav: Exploring zero-shot vision-and-language navigation in continuous environment with open-source llms." arXiv preprint arXiv:2409.18794 (2024).

2 (why not use Minecraft)

While Minecraft represents a more challenging procedurally generated survival environment, and studying the behavior and neural activity of RL agents in it would be worthwhile research, there are a number of factors that would make the contributions of our work difficult or impossible to obtain using Minecraft.

First, instrumenting Minecraft to log the range of environment variables that we record would be impossible without assistance from Microsoft to modify the source code/API, which would make it impossible to perform many of our experiments (for example, relating patch revisitation rates to specific environmental factors, as shown in figure 3). This similarly limits the extendability of ForageWorld to modify factors such as arena generation or predator type/behavior.

Second, Minecraft’s game logic runs on CPUs, whereas ForageWorld is implemented in JAX and runs completely on GPU, allowing for massive parallelization in rollouts within a single compute node and training for billions of environment timesteps, which was necessary to observe the de-novo emergence of planning, for example. Performing a similar scale of experiment in Minecraft would require vastly more computational resources and software infrastructure to do massively distributed rollouts across a large compute cluster, and would greatly limit the number and duration of experiments that could be run. We also note that such compute resources are out of reach for many research groups, especially computational neuroscientists, which would limit the impact of our environment and analysis tools in follow-on work. This computational accessibility coupled with high task complexity was a major feature of Craftax as a framework to build upon, as it allows future work to easily build upon our work (as we have done with regard to Craftax).

Third, while doubtless many interesting phenomena could be observed in Minecraft, none of our main contributions would be significantly enhanced by using Minecraft- a more complex environment is not needed to observe foraging with patch revisitation in a long-time-horizon POMDP and the de-novo emergence of planning, nor for adapting neuroscience analysis tools to deep RL agents.

Lastly, we are interested in analyzing agent behavior and neural dynamics when trained from scratch on an open-ended foraging environment, and the complexity of Minecraft environments would make training such an RL agent to do sophisticated temporally-extended foraging involving predator evasion and patch revisitation a formidable challenge technically and scientifically, as to our knowledge prior work demonstrating complex behavior in Minecraft has relied heavily on pre-training on human data (though if there is any recent work training complex tasks from scratch in Minecraft we would be curious to learn about it and study the methods used).

3 (environment features that influence behavior)

This is a very interesting question, which we explore along several dimensions in the paper. We explore how factors such as arena size, agent field of view, and the presence/absence of the position prediction auxiliary loss term affect behavior (figure 2). We found that each of these factors produce changes in agent behavior and/or dynamics which are not obvious from comparison of reward curves. In future work there are of course more factors that could be varied to explore their influence (for example, a multi-agent version of the environment, including learned predators), and we hope to explore such extensions to the core foraging task in future work.

4 (additional model(s) for comparison)

We focused on a single standard DRL architecture because our focus was on developing and demonstrating the environment and analysis techniques, and on demonstrating that planning can emerge without explicit world modelling, not to argue that such emergence is a common feature of many algorithms/models, as making such a claim would demand a much wider range of environments and models to compare. For further comparison, we implemented and tested an additional DRL-RNN model, PQN [*] with an LSTM (changing both RL algorithm and model architecture). We found PQN-RNN models can perform the task comparably well to our PPO models, with similar behavior and learning curves. However, interestingly, relative to PPO agents the PQN agents seem to have higher position prediction performance, prefer curvier paths on short time scales, and are worse at killing predators (prioritizing predator evasion rather than the killing predators with tools as PPO learns to prefer). We will summarize these updates and provide the code in the updated manuscript.

[*] Gallici, Matteo, et al. "Simplifying deep temporal difference learning." arXiv preprint arXiv:2407.04811 (2024).

Thank you for your insightful comments! To address the specific questions:

1 (variations in reward design)

This is an interesting question, thank you for asking it. We did experiment with several different reward function variants during development of ForageWorld (for example, instantaneous reward for eating when hungry), but did not explore them in detail because the resulting behavior was degenerate/not foraging-like in various ways (the instantaneous reward agent for example would try to be on the edge of hunger as much as possible to maximize reward from eating). As our goal was to develop and study agents performing simulated foraging behavior, we did not discuss these variations in the paper since they do not lead to naturalistic behavior and there are many subtle environment design details that can break naturalistic foraging (for example, the spawn rate of cows relative to the food value per cow must be set carefully to prevent either unavoidable starvation or the agent managing to survive indefinitely in one patch). We will add a brief description of some of these variants to the appendix.

2 (incomplete figure 15 learning history plot)

Thank you for catching that- we had actually standardized on the shorter run durations for most experiments (as the major behavior and dynamics features seem to be established within that duration), but in hindsight that makes the directional vision experiment look out of place in that figure compared to the longer runs. We performed longer runs to show the full time series for that experiment and have amended our manuscript (the trend and conclusions are consistent with the original results).

3 (related work on emergent cognitive behaviors and interpretability)

This is a good suggestion- we will enhance our literature review to include work on that topic, including:

Abney, Drew H., et al. "Advancing a temporal science of behavior." Trends in Cognitive Sciences (2025).

Verma, Abhinav, et al. "Programmatically interpretable reinforcement learning." International conference on machine learning. PMLR, 2018.

Cheng, Zelei, Jiahao Yu, and Xinyu Xing. "A survey on explainable deep reinforcement learning." arXiv preprint arXiv:2502.06869 (2025).

4 (expanded definitions for metrics used in figure 4)

We apologize if we were unclear about the calculation of metrics that were plotted across the agent’s training period. We will provide more detailed definitions of each metric in the appendix and refer to it. The numbers given are the average per episode value of the metric at a given training time point. For example, predator kills is the mean number of predators killed in one episode (i.e. in one arena configuration). Food score is the mean value of how satiated the agent is in a given episode, with a score of 9 as zero hunger and 0 as max hunger. Predator “kill” and “defeat” are indeed the same phenomenon (reduced the predator HP to zero with repeated attack actions), we will relabel these captions to be consistent. Thank you for your help in clarifying the manuscript!

5 (paths used in analysis)

We thank the reviewer for pointing out this was unclear. Indeed, the paths here belong to the same episode, and are directly consecutive from 1 through 4 in the behavioral time series logs for the episode. Path 1 starts at t=0 of the episode. This is also true for the related appendix figures which show additional representative examples. The path differentiation is done manually for this figure with the purpose of illustrating how a representative agent frequently opens new episodes by doing azimuthally rotating loops as an exploration strategy. We will add this explanation to enhance clarity.

6 (survival time across episodes)

Survival time is closely correlated to total return, and thus we chose to only list returns. This follows from the reward function in section 3.1, which means an agent that survives for longer will very likely be spending more time above the 50% mark on its resource levels and thus receiving positive reward.

7 (relationship between survival time and other metrics)

As can be seen by comparing the Fig. 2 performance plot and the Fig. 4 learning history plot, most behavioral metrics change with improvements in survival time at first, but plateau 30-50% of the way through training. Increase in exploration area correlates with early survival time improvements, but late training improvements seem to be driven by more efficient predator defense plus possibly additional factors that we could not identify, thus leaving an open question for future work. We will expand the current discussion to address this point.

8 (unclear Fig. 11 caption)

We apologize that the caption was not written clearly. The top left plot is missing because there is no position predicting auxiliary objective in this experiment, thus no error in position prediction to report. The quoted sentence about the “absence of structured spatial uncertainty” was supposed to refer to Figures 2 and 17 which show further information about this experiment, where the worse decoder performance and worse survival performance in the absence of the position prediction suggests that internal spatial representations may be worse in the agents which do not predict their own position. We will improve the caption and reference those figures.

9 (clarifying the definition of late/early in figure 4 legends)

Thank you for the comment and we apologize for the confusion. “Early in Episode” corresponds to the first 1500 time steps of a new episode and “Late in Episode” is after the first 1500 time steps of a new episode. We will clarify this in the paper.