Efficient Episodic Memory Utilization of Cooperative Multi-Agent Reinforcement Learning

Q1. I have concerns about the desirable trajectory. In paper, the author set $R_{thr}=R_{max}$. Since the desirable trajectories are the states that can achieve maximum returns, they must be the optimal states. What if the agents are impossible to achieve $R_{max}$. How to determine $R_{thr}$ in other environments?

A1. The reason we argue our contribution majorly lies on MARL community, especially cooperative MARL, is that in cooperative MARL settings, there is an explicit common goal that can determine the "desirability" of a trajectory. In other tasks where such an explicit goal is not presented, we need to determine $R_{thr}$ based on domain knowledge or a preference for the level of return to be deemed successful.

Considering the case where desirability is not specifically determined, We additionally conduct a parametric study on a single-RL task to see the effect of $R_{thr}$ value. For further details, please refer to Appendix D.9.

Q2. The dimension of $x$ is very small (i.e. 4 according to table 4 in the appendix). It's doubtful that it can reconstruct the global state.

A2. A major objective of our embedding function is to construct semantically similar memories clustered close. This is quite different from the objective of a generative model whose main purpose is to reconstruct inputs. As presented in the manuscript, considering reconstruction loss shows its merits in semantic embedding with $\textrm{dim}(x)=4$.

In addition, we use $\textrm{dim}(x)=4$ is for fair comparison with EMC, which uses $\textrm{dim}(x)=4$ in their episodic memory construction.

Q3. The approach adopts a state embedding instead of random projection. Does this make the approach more hard to converge?

A3. As the reviewer mentioned, semantic embedding requires additional training. However, adopting semantic embedding instead of random projection can improve the convergence of "policy" that we want to optimize because EMU guarantees the statistically coherent incentive toward desirable states.

Also, we can check the results from Figures 15-16 and EMU (XXX-No-SE) in ablation studies presented in Figure 9 and Figure 22. For small $\delta$, adopting semantic embedding or random projection shows not much difference in terms of convergence, indirectly viewed from the performance variance. However, their convergence speed is distinctively different. In other words, semantic embedding shows a much faster convergence to optimal policy. For large $\delta$, the semantic embedding (dCAE) shows a better convergence performance than a random projection.

Q1. EMU needs to set a return threshold to determine the desirability of a trajectory. This may require some domain knowledge to properly determine it, even when using $R_{max}$. This knowledge may partially explain its outperformance.

A1. First, thank you for the valuable comments. The reason we argue our major contribution lies on cooperative MARL is that in such tasks, the common goal is often explicitly specified such as scoring a goal in GRF or defeating all enemies in SMAC. By utilizing that signal, we can determine whether a trajectory is desirable or not and use that signal to prevent local convergence. However, if there is no such explicit-common goal, one may resort to a domain knowledge or a preference on the level of return to be deemed successful.

Q2. When the key encoder is updated, the proposed method needs to update all keys in the memory, which seems quite computationally intensive.

A2. We present the computational burden to update all key values in the episodic memory and to train $f_{\phi}$ and $f_{\phi}$ in Appendix C.3. Sorry for that if it was hard to find. In corridor task which has high-dimensional state space, the training and update only took less than 2 seconds at most, which is certainly negligible compared to MARL training.

Q3. It may be interesting to compare the proposed method with MAPPO.

A3. Since MAPPO relies on on-policy and other baseline algorithms including EMU are off-policy, it is difficult to set hyperparameters such as batch size, training iterations for a fair comparison. Thus, we work on the training framework and the corresponding hyperparameters, and if the time is available, we will present the result by the end of the rebuttal period.

(updated) We present the performance comparison of EMU with MAPPO in Appendix D.12. Please check the revised manuscript.

Q4. Can the author explain how to set the return threshold for desirability in experiments? Is this threshold dynamic or fixed?

A4. In our experiment, it is fixed. In cooperative MARL task, "desirability" is often determined explicitly as there is a common goal such as scoring a goal or defeating all enemies. Thus, in the task of SMAC and GRF we just use "desirability" signal generated by the environment, i.e., "battle_won" signal from the environment to determine whether an episode has achieved a goal or not. This signal can be identically generated if one set $R_{thr}=R_{max}$. The reason we present the desirability as $R \geq R_{thr}$ is for general definition considering the case where the explicit goal is not presented as in single-agent case. For these cases, we need to define $R_{thr}$ value, where the domain knowledge or preference on the level of reward to achieve should be introduced. However, again, in general cooperative MARL setting where the common goal is explicitly stated, we do not have to determine $R_{thr}$ value. We additionally conduct a parametric study on a single-RL task to see the effect of $R_{thr}$ value, considering the case where desirability is not specifically determined. For further details, please refer to Appendix D.9.

Q5. Is the incentive reward only effective when a trajectory is desirable? Does this mean episodic memory is useful only when a very good trajectory is explored, which can be hard?

A5. The incentive is only given to states considered desirable. This mechanism prevents local convergence toward states that have a good return at the early training phase but are not optimal. As the reviewer mentioned, the episodic incentive works after finding desirable trajectories which can be done by standard exploration with $\epsilon$-greedy or some additional incentive such as $r^c$. However, the problem is that even after we find desirable trajectories, the previous work could not fully utilize those good memories. On the contrary, in EMU, thanks to the semantic embedding, we can utilize semantically similar memories and thus encourage some exploration toward states that currently have not achieved goals but have the potential to do so. In this way, EMU can encourage exploration toward desirable states, deemed undesirable without semantic embedding.

Q1. Regarding scalability of EMU: It is unclear whether this method will scale to vision-based environments due to a) the memory requirements of storing many images, b) the optimization difficulty in reconstructing image-based states and c) the effectiveness of the introduced reward structure in high-dimensional state spaces. However, I acknowledge that many existing works in this area utilize feature-based observations. Regardless, it would greatly improve the strength of the results of this paper if some gains could be shown in vision-based environments.

A1. Thanks for the comment. First, we want to mention that our contribution lies majorly on applying episodic control method to cooperative MARL setting in a more efficient and robust way, by introducing a trainable semantic embedding and episodic incentive. As we evaluate EMU on multi-agent systems where high-dimensional state space, such as 322-dimension in MMM2 and 282-dimension in corridor, is already considered during centralized training, we deem our method also scalable to tasks with high-dimensional state spaces. In addition, episodic memory is saved in CPU rather than GPU, where the size RAM can be easily augmented.

If still saving all states is impossible in a certain task or a semantic embedding from original states becomes overhead, one may resort to pre-trained feature extraction model such as ResNet model provided by torch-vision in a certain amount for dimension reduction only, before passing through the proposed semantic embedding.

As an example, we implement EMU on the top of DQN model and compare it with original DQN on Atari task. For the EMU (DQN), we adopt some part of pre-trained ResNet18 presented by torch-vision for dimensionality reduction, before passing an input image to semantic embedding. We found a performance gain by adopting EMU on high-dimensional image-based tasks. Please refer to Appendix D.9 for experimental details and the result.

Q2. Regarding the evaluation of EMU: It appears that random projection performs almost equivalently to EmbNet/dCAE when compared using test win rates. The introduction of a new metric (overall win-rate) highlights that EmbNet/dCAE enable faster/more sample-efficient learning. However, improvement on this new metric is not as significant as the original win rate (which is the standard benchmark in the community). I would be curious to see the curve for EMU with random projection added to Sections 4.1 and 4.2 to better understand the significance of EmbNet/dCAE.

A2. We partially agree with the reviewer's statement since a long convergence time is one of predicaments in MARL training. Thus, we want to introduce a metric that measures both learning speed and quality. Although their win-rates seem comparable in relatively easy tasks such as 3s_vs_5z and 5m_vs_6m, adopting random projection instead of semantic embedding degrades the performance as presented in Figure 9 and Figure 22 in the manuscript. Please see EMU (XXX-No-SE) case compared to EMU (XXX). We also conduct additional experiments on EMU (XXX-No-SE) in other tasks. For details, please refer to Appendix D.11 of the revised manuscript.

Q3. It would be helpful to provide details about the state space, action space, environment reward and episode lengths for both SMAC and GRF in the Appendix.

A3. Thanks for the comments, we present the details about the dimension of state space, action space, reward settings, and episode lengths for both SMAC and GRF in Appendix D.10. Please refer to it.

Q1. Regarding the contribution of EMU

A1. First, thank you for the comments. From a certain perspective, there are some commonalities between the listed references and EMU, such as episodic memory usage and embedding for representation learning, but their main objective of using episodic memory is a bit different.

References presented in the manuscript aim to generate a better TD target by utilizing the values of state stored in episodic memory. The value in episodic memory can be deemed as the best return experienced throughout training. Aligned with previous works, our paper mainly aims to generate a correct TD target that could converge on a true TD target.

On the other hand, the listed references by the reviewer majorly deal with exploration, as specifically mentioned by the reviewer. We contrast our paper with the listed references by the reviewer.

[1] extends the count-based episodic bonuses to continuous spaces by introducing elliptical bonuses and encourages exploration with them. For feature embedding, [1] adopted an inverse dynamic model presented in [R1], to discard unnecessary state information for predicting the agent's action. [2] presents a noise-robust intrinsic reward, called surprise novelty, for exploration. [3] utilizes "familiarity" buffer to predict rare states and adopts contrastive momentum loss to prioritize long-tail states.

However, we design the dCAE structure to extract the features that are critical in determining the value of a given state, i.e., semantic embedding. This semantic embedding allows us to recall similar memories in a wider range of episodic memory space, yielding "Efficient memory utilization." Implementation of the listed references does not give us such functionality. Moreover, EMU deals with the problems that might be caused by implementing the conventional episodic control in cooperative MARL settings, by introducing "desirability." With this desirability, EMU presents an episodic incentive that generates a correct TD target while preventing local convergence toward "undesirable states". None of the listed references [1-3] aim to correctly model the true TD target as we do.

Considering the comments above, we hope the reviewer to see the differences between the listed references and EMU, and the contribution points raised by EMU.

[R1] Pathak, Deepak, et al. "Curiosity-driven exploration by self-supervised prediction." International conference on machine learning. PMLR, 2017.

Q2. Moreover, I believe that the comparison with related work is imperative to understand the position of the paper and its contributions and should not be relegated to the appendix.

A2. We want to point out that adopting an algorithm from a single-agent to MARL is often non-trivial and thus the community recognizes it as a contribution point of EMC. Moreover, a naive adoptation can adversely influence the overall performance, as EMC did in the complex MARL tasks. From this view, this paper argues that our contribution majorly lies on MARL community where the desirability is well determined by whether the common goal is achieved or not, leaving the possibility of adopting EMU to single-agent tasks, as additional merit of EMU. Thus, we present that application for readers in the Appendix rather than the main manuscript. Further differences between single-RL and MARL tasks are presented in Appendix A.3.

Q3. As a minor issue, writing also should be reviewed, but clarity in general is good. Beyond my recommendations above regarding writing, there is a common abuse of "the" through the text, e.g. "In spite of the required exploration in MARL with CTDE, the recent works on episodic control emphasize the exploitation of episodic memory to expedite reinforcement learning. Episodic control memorizes the explored..."

A3. Thank you for your comment. We have revised the manuscript to address any awkward usages of the.

First, thank you for the comment; it became clearer to see the benefits of EMU compared to methods possibly adoptable from a single-agent domain.

To respond the reviewer's request, we conduct additional experiments to check whether an exploratory incentive proposed in single-agent domain, specifically E3B in [1], works in MARL domain. Please refer to Appendix D.13 of the updated manuscript for experiment results. We found that replacing an episodic incentive of EMU with E3B-style exploratory incentive is not beneficial even after some hyperparameter tuning. With its original scale factor, the performance severely degrades.

It seems that adding a surprise-based incentive can be a disturbance in MARL tasks since finding a new state itself does not guarantee a better coordination among agents. In MARL, agents need to find the proper combination of joint actions in similar observations when finding optimal policy. On the other hand, in high-dimensional pixel-based single-agent tasks such as Habitat [Ref1], finding a new state itself can be a beneficial in policy optimization. Without modification of algorithm, one may choose a different level of a scale factor, but a hyperparameter search according to different tasks can be a hurdle of adopting such algorithm to various tasks. On the other hand, the proposed episodic incentive does not require such hyperparameter scaling to adjust the magnitude of incentive as it automatically adjust the magnitude according to its convergence status. This is one of merits of the proposed episodic incentive.

In addition, unfortunately, feature embedding structure presented in [1] is not directly applicable in MARL in following reasons:

1. There is joint action $a_{t}^{jt}$ in MARL while an inverse dynamics model $g$ used for training $\phi$ predicts a single action $a_t$ based on $\phi(s_t)$ and $\phi(s_{t+1})$ as $g(\phi(s_t),\phi(s_{t+1}))$.

2. In MARL, each agent's action $a_t^i$ is taken based on an agent-wise partial observation $o_t^i$. Thus, given global $s_t$ or $\phi(s_t)$ cannot properly predict $a_{t}^{jt}$. One could use partial observation as an input to $\phi$, as actionable representation learning adopted in [Ref2], but we want to learn $\phi(s_t)$ not $\phi(o_t^i)$.

For these reasons, incorporating the feature embedding structure from the single-RL domain and its learning framework into the multi-agent domain may necessitate extensive modification, resulting in a distinct line of research.

Considering the points mentioned above, we hope the reviewer understands the limitation of adopting algorithm from the listed references to MARL domain and sees the contribution points introduced by EMU.

[1] Henaff, M., et al., "Exploration via elliptical episodic bonuses.", Advances in Neural Information Processing Systems, 35, 37631-37646, 2022.

[Ref1] Ramakrishnan, Santhosh K., et al. "Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai." arXiv preprint arXiv:2109.08238 (2021).

[Ref2] Jeon, Jeewon, et al. "Maser: Multi-agent reinforcement learning with subgoals generated from experience replay buffer." International Conference on Machine Learning. PMLR, 2022.

Dear Reviewers,

Again, we would like to express our gratitude for the constructive comments you provided on our work. We genuinely appreciate the time and effort you have dedicated to this process. We uploaded the updated manuscript with our responses to address your comments. The revised or added parts of the manuscript are colored with magenta. If you require additional information or have any further inquiries about our paper, please let us know. We remain open to discussions and are ready to provide any necessary clarifications.

Sincerely, Authors.