Learning Progress Driven Multi-Agent Curriculum

Response to Reviewer ffyd

We thank the reviewer for the valuable comments. We note that the main concern comes from the comprehensiveness of the benchmarks and inclarity of the method description. In response, we have conducted additional experiments on new benchmarks, and we hope that the enhanced clarity in our revisions will effectively address these concerns.

1. More SMACv2 tasks and BenchMARL:

We have conducted additional experiments on four new SMACv2 tasks: Terran 5v5, Terran 6v6, Zerg 5v5, and Zerg 6v6. The results, available at this link, demonstrate that our algorithm, SPMARL, consistently outperforms the baselines and generates stable, interpretable curricula across all tasks.

For the BenchMARL tasks, Balance and Wheel, our method achieves performance comparable to the baselines, as shown in this link. This may be due to the fact that the exploration challenge in these tasks is not particularly severe, reducing the necessity of curriculum learning. However, we note that SPMARL is the only one generating curriculum that converges to the target task distribution.

2. Clarity on Section 4, Algorithm 1:

We apologize for the lack of clarity in explaining our method. In Section 4, we primarily apply the existing single-agent curriculum learning method, SPRL, to control the number of agents, which we refer to as SPRLM. Our main contribution is the introduction of the TD-error-based learning progress, a novel approach to curriculum learning, as noted by reviewer ZwRM.

Intuitively, our method first identifies tasks, defined by varying numbers of agents, that exhibit the highest learning progress. After training on these high-progress tasks and achieving the performance threshold $V_{LB}$, we adjust the task distribution towards the target by minimizing the $KL$ divergence between current task distribution and the target distribution. This process is constrained by the $KL$ divergence between the old and new task distributions to prevent rapid changes in the task distribution.

We appreciate the reviewer's valuable suggestions on the organization of the paper. We agree that moving the diagram and comparison table from the appendix to the main text will improve readability. We will revise the paper accordingly based on your helpful recommendations.

3. Related literature

We appreciate the reviewer's suggestion regarding the literature on Unsupervised Environment Design (UED), which falls within the domain of single-agent curriculum learning. UED approaches, such as [1], primarily focus on general environment design using state-action coverage as the objective. In contrast, our work addresses the increased credit assignment difficulty in multi-agent reinforcement learning (MARL) when applying reward-based methods. We acknowledge the close relationship between these fields and will update our paper to include a discussion on this topic.

Compared to single-agent RL, research on curriculum learning in MARL has seen significantly less progress. This is partly because many single-agent curriculum learning methods, such as SPRL and VACL, can be adapted to MARL to control the number of agents. However, our work specifically investigates the challenges that arise in MARL and proposes a TD-error-driven curriculum tailored for MARL.

[1] Teoh, Jayden et al. "Improving Environment Novelty Quantification for Effective Unsupervised Environment Design." Advances in Neural Information Processing Systems 37 (2024).

Response to Reviewer p7uW

We thank the reviewer for the appreciation of the novlty of our method. We hope our clarification and new experiments help to address your concerns.

1. Clarity on the variance comparison:

However, the settings of these two figures can be further explained, including details such as how many samples the variance is computed over. Given that this is a core claim of the paper, analysis on more environments should be shown.

In Figures 6 and 8, we simply compute the standard deviation of the collected episode returns and TD errors in each iteration, which are collected with the same context samples and used to estimate the objectives of the curriculum learning methods. The number of samples equals to the number of episodes done in the iteration which is at least $25$ episodes in SMACv2 since some episodes may terminate earlier.

Mathematically, assume we have collected a set of contexts $\\{c_1, c_2, \dots, c_n\\}, n \approx 25$, the corresponding episode returns $\\{R_{1}, R_{2}, \dots, R_{n}\\}$, and the TD-errors $\\{TD_{1}, TD_{2}, \cdots, TD_{n}\\}$ where $TD_{i} = LP(c_i) = \\frac{1}{2} E_{s, a \sim \pi(a \vert s, c_i)} \left[ \lVert R(s, \mathbf{a}) - V(s) \rVert^2 \right]$, the standard deviation is computed as $\sigma = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (x_i - \bar{x})^2}$, where $x_i$ can be $R_{i}$ or $\text{TD}_i$ and $\bar{x}$ represents the mean. Since TD-error on each context has been averaged over the samples of the whole episode, it shows lower estimation variance.

We ran additional experiments and performed similar analysis on two tasks from BenchMARL, i.e. Balance and Wheel. The results in link show that our method SPMARL demonstrates much lower standard deviation than the return estimation used in SPRL. In these experiments, the samples are also at least $25$ for each computation of the standard deviation, sometimes it can be more than $25$ when some episodes terminate earlier.

The variance reduction of using TD error compared to episode returns is also thoroughly analyzed in [1].

[1] Schulman, John, et al. "High-Dimensional Continuous Control Using Generalized Advantage Estimation." Proceedings of the International Conference on Learning Representations (ICLR), 2016.

2. Clarity on choosing $V_{LB}$:

The key hyperparameter, $V_{LB}$, is not ablated. How to choose hyperperparameter for new environments?

Thanks for raising this issue. We are sorry for not explaining it clearly. $V_{LB}$ is an important hyperparameter in our method. We empirically choose $V_{LB}$ to be $80\%$ of the final performance. For example, in Protoss 5v5, the converged win rate is around 0.8, we set $V_{LB}$ with 0.6. However, our ablation in link shows that $V_{LB}$ can be chosen over a larger range.

3. Number of random seeds:

How many seeds are the experiments in Fig. 3-4 evaluated on?

We appreciate the reviewer's feedback in highlighting this ambiguity. All our experiments were conducted using five random seeds.

4. Discussion on curiosity-driven learning:

Thank you for your insightful comment. Curiosity-driven exploration is a well-studied topic in RL, primarily focusing on developing intrinsic reward signals through techniques such as self-prediction, rather than modifying environments as in curriculum learning. However, we believe that general curiosity-driven exploration can be effectively integrated with curriculum design to further enhance exploration.

5. Clarity on state, action space

We thank the reviewer for raising this issue. In the appendix, in the next paper version, we include a detailed introduction of all the tasks used in our experiments.

Response to Reviewer ZwRM

We thank the reviewer for the appreciation of the simplicity and generality of our method. We hope new experiments and clarifications help to address your concerns.

1. Clarity on observability setting:

If I'm understanding the paper correctly, the setting is fully observable (albeit decentralized). Full observability feels like a potential hitch. In a rebuttal, would love to hear the authors' thoughts on that.

We agree with the reviewer that observability is important in MARL. In our experiments, we use partial observability for all the tasks except the XOR matrix game, since the XOR game is a stateless task. For example, in Simple-Spread, the agents can only observe $4$ nearest agents and landmarks, and in SMACv2 tasks agents can only observe enemies and agents in a certain range. Partial observability also helps in transferring policies across tasks with different numbers of agents since the observation space is fixed instead of growing with the number of agents. We use RNN policies for all the methods.

2. More benchmarks:

The simplicity of the method is very valuable. ... The framework also feels like it could be super general and applied across single-agent RL and MARL. That said, the experimental analysis seems to leave off some recent tools for benchmarking like JaxMARL and others.

We appreciate the reviewer for kindly acknowledging the contribution of our simple yet effective method and thank you for suggesting new benchmarks. We found that our current benchmarks such as MPE simple-spread and SMACv2 are also included in JaxMARL. Therefore, we mainly performed new experiments on more SMACv2 tasks and two tasks from BenchMARL [1] suggested by reviewer ffyd.

The results in link show the results on four new SMACv2 tasks, Terran 5v5, Terran 6v6, Zerg 5v5, and Zerg 6v6. We can see that our algorithm SPMARL consistently outperfoms the baselines and generates stable and intepretable curricula across all the tasks.

For the BenchMARL tasks, Balance and Wheel, our method achieves performance comparable to the baselines, as shown in this link. This may be due to the fact that the exploration challenge in these tasks is not particularly severe, reducing the necessity of curriculum learning. However, SPMARL is the only one generating curriculum that converges to the target task distribution.

[1] Bettini, Matteo, Amanda Prorok, and Vincent Moens. "Benchmarl: Benchmarking multi-agent reinforcement learning." Journal of Machine Learning Research 25.217 (2024): 1-10.