Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluation

Thank you for highlighting the strengths of our work. We respond to the comments below.

W1. Simple environments

On one hand, the environments may seem relatively simple; however, they enable rapid and targeted exploration of various aspects of memory and hypothesis testing. In addition, the T-Maze environment is a common standard memory test in RL [1, 2, 3].

The goal of these environments is to facilitate testing specific hypotheses related to memory without requiring the RL agent to solve additional tasks that involve learning unrelated skills or abilities.These examples are enough to demonstrate the core ideas of the paper.

W2, Q1. Procedural memory

Definitely, testing procedural memory is an important and valuable task; however, it lies beyond the scope of our work. We provide a definition of procedural memory along with examples to distinguish it from declarative memory. In this study, we focus primarily on declarative memory and propose a method for testing it.

W3. Visual aids or examples

We have aimed to support all key concepts introduced in the article with examples and visualizations. For instance, the illustrations in Fig. 1, Fig. 2, and Fig. 3 visualize and help clarify the definitions of declarative and procedural memory, long-term and short-term memory, and the general classification of memory types.

We are confident that, in addition to formal definitions, these visualizations will assist readers in better understanding the concepts and terms introduced in the paper, making the work more accessible to a wider audience.

Q2: Declarative vs. procedural memory

Declarative memory and procedural memory are indeed defined in the article as conceptually distinct concepts. Declarative memory is described as the use of knowledge within a single episode and a single environment, whereas procedural memory encompasses skill transfer across multiple episodes or environments. We agree that the current wording may lead to misinterpretation, and therefore, we have adjusted the definition to make it more precise:

• Declarative Memory $\Leftrightarrow (n_{envs}\times n_{eps}=1)$

• Procedural Memory $\Leftrightarrow (n_{envs}\times n_{eps}>1)$

Q3, Q4

Advantages of cognitive science definitions

Definitions from cognitive science, such as short-term and long-term memory, as well as declarative and procedural memory, are already well-established in the RL community, but do not have common meanings and are interpreted in different ways. We strictly formalize these definitions to avoid possible confusion that may arise when introducing new concepts.

Memory classification

The motivation for introducing a classification of different types of memory with respect to temporal dependencies and types of memorized information is motivated by practical goals. In the course of our research on memory in RL and a review of existing work in this area, we concluded that modern challenges and tasks in RL require such a classification to ensure proper memory testing in RL agents.

For instance, some interpret memory as employing transformers with extensive context windows, others as utilizing recurrent networks, and still others as a model’s ability to transfer skills across tasks. However, these approaches often differ fundamentally in design, making direct comparisons under identical conditions potentially invalid, or testing conditions suitable for one agent may not align with another’s memory mechanism.

How does this framework compare to existing memory evaluation approaches in RL?

Currently, in RL memory mechanisms are tested by running agents in memory-intensive environments and evaluating metrics without considering environment or agent temporal configurations. Section 6.1 shows the issues with this approach.

This experiment shows how naive validation of an agent with memory in a memory-intensive environment can lead to incorrect conclusions about its memory type. We derive three configurations of context length K and correlation horizons $\xi$ from Theorem 1 to evaluate: 1) long-term memory, 2) both long- and short-term memory, and 3) long-term memory only.

Our results demonstrate that the same agent trained in the same environment produces different results based on the K and $\xi$ configuration. When using our proposed Algorithm 1, the agent learns 0.53 ± 0.04, indicating its inability to solve the long-term problem. However, with the default configuration, it achieves 0.95 ± 0.02, which might suggest long-term memory, but since both long- and short-term memory were tested, we cannot definitively claim it has long-term memory.

[1] Esslinger, K. et al.. Deep transformer q-networks for partially observable reinforcement learning. arXiv:2206.01078.

[2] Grigsby, J. et al. Amago: Scalable in-context reinforcement learning for adaptive agents.ICLR 2024

[3] Pramanik, S. et al. AGaLiTe: Approximate Gated Linear Transformers for Online Reinforcement Learning.TMLR 2024

Thank you for addressing the typo in the paper. Related to my remaining concerns about novelty primarily related to the scope of the work. While procedural memory is intentionally left outside the experimental scope, its absence from evaluation feels like a missed opportunity especially that the flow of the paper initially introduces Memory DM and Meta-RL, but the testing methodology focuses solely on Memory DM, leaving the discussion incomplete.

We appreciate the reviewer's comments and are glad to hear that you share our vision of the need to formalize concepts related to memory in RL. We respond to the reviewer’s comments below.

Broad overview of use of the memory term

Our work is not a review paper. Our main goal is to formalize the basic concepts of memory in RL so that new memory-enhanced agents in RL can be compared and validated correctly. We take the definitions of memory from neuroscience as the basis of our formalism, since they have been used in RL for a long time, albeit in different senses, as we write about in Section 3 - Related Works. In turn, in Appendix B - Memory Mechanisms, we give an extensive description of what is basically meant by memory in RL.

Abuse of the term “memory”

We cannot talk about the abuse of the term “memory” precisely because everyone understands memory differently. For example, someone understands memory as the use of a transformer with a very large context window, someone understands it as the use of recurrent networks, and someone understands it in general as the ability of a model to transfer skills from one task to another. At the same time, these algorithms may have fundamental differences in their design and comparing them under the same conditions may not be correct, or the conditions for testing memory mechanisms for one agent may not be appropriate for another.

That is why we offer our definitions of different types of memory that allow us to fully describe the basic concepts in RL that are commonly understood by memory.

Purely review article

Writing a purely review article would be interesting to us as well, and we'll get into that next. However, for the moment, our goal is to propose practical ways to separate agent memory types in RL, as well as their validation in the Memory DM framework.

POMDPs

The definitions we propose are based on the POMDPs formalism in both the Memory DM and Meta-RL contexts, which is why we have placed this section at the beginning.

Related Works

In the Related Works section, since this is not a review paper but a practical one, we show that RL actively uses memory types from neuroscience, but with different meanings. Rather than providing a comprehensive review of all works on memory mechanisms in RL, we focus on our proposed contributions. Consequently, we do not conduct a detailed classification of existing memory mechanisms; instead, we provide an overview of them in the Appendix.

Cognitive science and RL

We base our definitions of memory on neuroscience, as concepts from neuroscience have long been used in Memory RL, as we discuss in the Related Works Section. A deeper look into cognitive science aspects of memory is out-of-scope for our work, as we focus specifically on the practical application of our taxonomy and memory validation algorithm.

Thank you for your interest in our work and your valuable feedback. We appreciate your proposal to serve as a reviewer, which encourages us to deepen our research. We look forward to discussing your ideas and suggestions to improve the paper and hope for further collaboration to enhance its quality.

[1] Mnih, V. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602.

[2] Schaul, T. (2015). Prioritized Experience Replay. arXiv preprint arXiv:1511.05952.

[3] Deverett B. (2019). Interval timing in deep reinforcement learning agents //NeurIPS 2019

We appreciate that the reviewer feedback and. Here are our responses.

W1. Focusing on RL

Our work focuses exclusively on RL, using neuroscience concepts as reference points to define memory types. We use the neuroscience framework because the terms “long-term/short-term, working, episodic memory, etc.” are already used in RL, but without a unified meaning. Therefore, we redefine them with clear, quantitative meanings to specify the type of agent memory, since the performance of many algorithms depends on their type of memory. We do not claim to have enumerated all types of human memory, since our work is focused exclusively on RL.

W2. RL is capturing declarative memory

Declarative memory of RL agents involves recalling and reusing arbitrary “facts,” which can be any environmental representation (e.g., a door’s color) learned during training and do not necessarily have to be verbalized. Our work focuses on Memory DM tasks, where agents use historical data for decisions. While LLMs excel at language tasks, declarative memory in RL often involves signal processing, making its study a distinct challenge.

W3. Declarative and procedural memory in RL

In RL, “memory” is used similarly in Meta-RL and Memory DM tasks, despite requiring different memory types, leading to comparison and validation issues (of two algorithms with a stated long-term memory one may not solve the same simple problems as the other one). To address this, we define procedural and declarative memory to clarify their roles in specific tasks.

W4. Practical application

Our definitions of declarative and procedural memory in RL are practical, such that they use two numerical metrics: the number of environments n_{envs} and episodes n_{eps}, enabling clear identification of the memory type required for a task. An environment refers to where the agent interacts and receives feedback, while an episode is the sequence from start to terminal state. These concepts are rooted in RL and widely used in existing benchmarks and baselines.

Q1. Memory meant to be a problem or a solution?

Memory can be both a problem and a solution, ​​which is why we titled our paper this way. In POMDPs, it helps agents overcome the challenge of incomplete environmental information by storing and retrieving past interactions. However, implementing an effective memory mechanism is a complex challenge.

Q2. Marr-Poggio levels

Yes, we consider that the methodology we propose for testing long/short-term declarative memory can be described using Marr-Poggio levels of analysis. However, this requires disclosure of par. 4 “Analyze the results,” in Algorithm 1, to formalize the system’s outputs. In the current version, we leave this point to the independent interpretation of researchers.

• Computational

Goal: Test long/short-term declarative memory in Memory DM tasks Input: Memory model, Oracle agent acting random at memory event recall, memory-intensive environment (Theorem 1). Language: Standard POMDP/MDP formalism, along with memory definitions from our paper. Output: 1 if memory model has tested type of memory, and 0 otherwise

• Algorithmic

Use Algorithm 1 to test long/short-term memory.

• Implementation

Focus on implementing memory-intensive environments and memory mechanisms, though the methodology is not tied to specific implementations.

Q3. Replay buffer

In Appendix we give examples of different memory mechanisms as they are understood in various works in the RL domain: “'In RL, memory has several meanings, each of which is related to a specific class of different tasks”.

We added replay buffers to the text due to the fact that well-known works [1, 2] treat it as a memory mechanism.

Q4. “Experiment did not follow methodology”.

This experiment shows how naive validation of an agent with memory in a memory-intensive environment can lead to incorrect conclusions about its memory type. We derive three configurations of context length K and correlation horizons $\xi$ from Theorem 1 to evaluate: 1) long-term memory, 2) both long- and short-term memory, and 3) long-term memory only.

Our results demonstrate that the same agent trained in the same environment produces different results based on the K and $\xi$ configuration. When using our proposed Algorithm 1, the agent learns 0.53 ± 0.04, indicating its inability to solve the long-term problem. However, with the default configuration, it achieves 0.95 ± 0.02, which might suggest long-term memory, but since both long- and short-term memory were tested, we cannot definitively claim it has long-term memory.

Q5. Time interval RL

In accordance with the definitions we introduced, in article [3], a feedforward agent employs a long-term declarative memory based on autostigmergy [3] to solve the task of reproducing time intervals.