Learning Generalizable Skills from Offline Multi-Task Data for Multi-Agent Cooperation

W4: Since all tasks share the same space for the local skills $z^{1:K}_t$, why $z$ can embed task-specific knowledge?

A: Referring to Eqs. (10) and (11), $z$ is regularized by a contrastive learning objective. This objective ensures $z$ inferred in different tasks possess different distributions (i.e., skills in the same tasks will become closer, and in different tasks will be pushed away). Therefore, $z$ represents the task-specific knowledge.

W5: Multi-agent skills should provide abstractions at both temporal-level and subgroup-level. That is, each multi-agent skill should embed a (short) control sequence of a subgroup of agents. How is this aspect shown in the algorithm design?

A: Although we do not explicitly divide the agents into several subgroups, their common skills learn team-wise dynamics by predicting the common next global state and improve multi-agent cooperation by maximizing the team's accumulated rewards, achieving the temporal-level and team-level cooperative policy learning. Therefore, learning the common and task-specific skills is effective for offline multi-task MARL.

W6: Detailed instructions on how to reproduce the results in this paper using the released code.

A: Thanks for your suggestion, we have revised the instructions. Our code in the SMAC benchmark is based on ODIS's official code. Thanks to the clear instructions in the ODIS source, we make small changes in instructions to fit our method, experiments and can be reproduced based on current instructions. We will acknowledge ODIS in our official repository. In MAMuJoCo experiments, we produce our method based on OMIGA's [1] code rather than pymarl framework, which is different from the SMAC's experiments. We will release it in our official repository.

Reference

[1] 'Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization'

Thank you for your time and feedback on HiSSD. We address each of your questions below.

Weaknesses:

W1: The framework is quite convoluted compared to previous methods, which could potentially bring training difficulty.

A: In this paper, the training pipeline and hyperparameters tuning follow previous methods. To better illustrate the model sensitivity of our method, we conduct additional ablation studies for sensitive analysis and present the results below. The empirical results indicate that our method can be trained under different hyperparameter settings and no obvious training difficulty is observed.

Sensitivity Analysis on $\epsilon$ Source Tasks

Target Tasks

Sensitivity Analysis on $\alpha$ Source Tasks

Target Tasks

W2: The writing needs to be significantly improved.

A: Thanks for your reminder, we will further improve our writing.

W3: It's not clear why the common skills can integrate cooperative temporal knowledge and enable an offline exploration.

A: Firstly, cooperative temporal knowledge in this paper represents information for future prediction and value maximization from the global/team perspective. We train the learned skills to predict the future global state and maximize the team reward from the current local observations. Predicting the next global state from local observation requires the policy to understand the team-wise relationships, enabling the policy to extract more knowledge corresponding to the global information. Maximizing the team's accumulated value enables the policy to select the proper common skills that achieve the best cooperation performance, improving the policy's cooperation capability. Secondly, we utilize the IQL paradigm to train the common skills. This learning process greatly balances reward maximization and trajectory imitation from a global perspective, enabling the offline exploration of the cooperative multi-agent policy.

Thank you for your time and feedback on HiSSD. We address each of your questions below.

Weaknesses:

W1: This paper didn't adequately explain the clear causal relationship or mechanism between integrating cooperative temporal knowledge and learning common skills.

A: Theoretically, we treat the offline policy learning as probabilistic inference, formulating the trajectory distributions induced by the optimal policy and our trained policy as $p(\tau)$ in Eq. (2) and $\hat{p}(\tau)$ in Eq. (3), respectively. Our goal is to minimize the distance between these two distributions and the formal objective is given by Eq. (4), where the multi-agent policy needs to maximize the accumulated rewards and imitate the trajectory sampled from the offline dataset based on selected common skills. The first term in Eq. (4) is to learn common skills that maximize team rewards. This term benefits the multi-agent policy to achieve cooperation by selecting proper common skills. The second term in Eq. (4) is to predict the next global state based on common skills. This term helps to integrate both temporal-wise and team-wise knowledge into common skills. From human intuition, a superior cooperation multi-agent policy must achieve the same goal and understand the dynamic transitions from the team's perspective. Therefore we designed the proposed learning objective.

---

Questions:

W2 & Q1: Explain how performing global state prediction and value estimation directly leads to learning cooperative knowledge.

A: In this paper, cooperative temporal knowledge represents information for future state prediction and value maximization from the global/team perspective. In RL, temporal knowledge is highly related to the transition $(s_t, a_t, r_t, s_{t+1})$, which has been widely discussed in prior works [1, 2]. In MARL, predicting the next global state from local observation requires the policy to understand the team-wise relationships, enabling the policy to extract more knowledge corresponding to the global information. Maximizing the team's accumulated value enables the policy to select the proper common skills that achieve the best cooperation performance, improving the policy's cooperation capability.

Q2: How did the authors verify how distinct these two types of skills are? Was there any overlap or interference between the two skills?

A: The definition and learning process of these two skills are significantly different. The common skill is concatenated with the current observation to infer the observation embedding, while the specific skill is fed to the action output layer to guide the action execution. To clearly show the distinction between these two types of skills we visualize their distribution using t-SNE and present them in Fig. (5) in the revised paper. The empirical results indicate that the learned policy can distinguish different kinds of skills at each stage of the episode, and no obvious overlap between these skills is observed.

Reference

[1] 'Bridging State and History Representations: Understanding Self-Predictive RL'

[2] 'Joint-Predictive Representations for Multi-Agent Reinforcement Learning'

Q4: The task-specific skills learn task-specific knowledge of source tasks, why do these skills play an important role in unseen tasks?

A: Common skills represent the general knowledge that can be reused in different tasks. Meanwhile learning task-specific skills aims to discover knowledge related to the current task, achieving complementarity with common skills. Moreover, we conduct a variant of HiSSD named 'w/o Specific' which only learns common skills. We evaluate it on Marine-hard task set and present the results below. The empirical results indicate that the task-specific skills further improve the policy's generalization capability.

Components Analysis on Task-Specific Skills: 'w/o Specific' indicates only integrating common skills in HiSSD. Source Tasks

Target Tasks

W5: One concern for the paper is to evaluate HiSSD in maps that are more difficult (e.g., Stalker-Zealot task set in ODIS), to validate its effectiveness.

A: To further validate HiSSD's effectiveness, we conduct experiments on 'Stalker-Zealot' task set proposed by ODIS and present the averaged results below. Detailed results are presented in Appendix (F.2) in the revised paper. It should be noted that the stalker-zealot dataset proposed by ODIS is not well developed, which may hinder the training and generalization of the policy. Based on the results, we find no method is consistently best. HiSSD outperforms all baselines in half of the target tasks while still being competitive in source tasks, indicating the better generalization capability of hiSSD.

Average test win rates in the Stalker-Zealot task set with different data qualities. Source Tasks

Target Tasks

---

Questions:

Q1: Why present BC-best's results in Table 3.

A: Because our method is built on the BC-based method, we use BC-best rather than ODIS as a baseline in the ablation study to show the effectiveness of different components in HiSSD.

Q2: In Fig. (3), Large-scale tasks (i.e., 10m and 12m) overlap with each other but small-scale tasks don't. The paper said this is because 10m and 12m are similar. However, 3m, 4m, and 5m are also similar.

A: Our training objective ensures that the policy is able to distinguish task-specific skills when the target tasks are close to the distribution of source tasks. For the small-scale tasks that are similar to the source tasks, the policy can be easily generalized to the small-scale target tasks, and the discrepancy between these tasks can be learned more efficiently. As for the large-scale tasks (i.e., 10m and 12m) which are farther from the distribution of the source tasks, it is harder for the policy to capture significant discrepancies and it can only infer some knowledge related to the source tasks, therefore causing the clearer overlap.

Q3: Why the paper didn't conduct a senstive analysis for $\alpha$ in Eq. (6).

A: We set $\alpha$ and other hyperparameters following previous works. To better analyze the model sensitivity of our method, we conduct an additional ablation study under different $\alpha$ and present the results below. The results indicate that our method is robust to $\alpha$.

Sensitivity Analysis on $\alpha$ in Marine-hard Task Set Source Tasks

Target Tasks

Thank you for your time and feedback on HiSSD. We address each of your questions below.

Weaknesses:

W1: Although HiSSD is superior to previous skill-based methods, it fails to outperform ODIS in some tasks.

A: For the performance in marine task sets in SMAC, HiSSD outperforms ODIS on most tasks. HiSSD obtains the best scores in 68 tasks while ODIS obtains the best in 17. HiSSD fails to outperform ODIS mainly in a few large-scale tasks (e.g., 10m12m and 13m15m). This is due to 1) the distribution distance between large-scale tasks and source training tasks is farther than other target tasks. It is much more difficult for the policy to generalize skills to the large-scale tasks. 2) These large-scale tasks are more complex and the number of units is more asymmetric. All of the current methods are unable to obtain convincing performance (average win rates are under 15%).

W2: In MAMuJoCo, the paper didn't use ODIS as a baseline.

A: It's worth noting that the official code released by ODIS didn't conduct experiments on MAMuJoCo. Therefore, we didn't use ODIS as a baseline. Here we reproduce ODIS on MAMuJoCo environment and present the results below. For a fair comparison, we use the same architecture and hyperparameters as HiSSD. The empirical results indicate that HiSSD is competitive in source tasks and outperforms in target tasks compared to ODIS, which demonstrates the better generalization of HiSSD.

Reproduced ODIS on MAMuJoCo Source Tasks

Target Tasks

W3: The process of learning skills is similar to imitation learning, so it failed to perform well in unseen tasks (e.g., 10m12m and 13m15m), indicating low-level generalization ability.

A: Generalizing the multi-task offline policy to the unseen large-scale tasks (e.g., 10m12m and 13m15m) is more difficult: 1) the number of units in these tasks are more asymmetric than other tasks and 2) the task distributions are farther from source tasks than other unseen tasks. Therefore, it is difficult for the policy to extract useful skills and behavior patterns from the proposed offline multi-task dataset, leading to low absolute win rates. We consider improving policy's generalization capability in large-scale tasks as our future work.

W4: In the passage, some verbs sound a bit odd.

A: Thanks for your reminder, we will check and improve our writing.

W4: The visualization experiments do not show clear evidence about the latent skills. For example, the task-specific skills in the same task seem closer in Fig. (3).

A: We visualize the learned common and task-specific skills in Figs. (2) and (3) respectively. Fig. (2) indicates that the learned common skills will be clustered based on cooperative patterns rather than tasks. Fig. (3) shows the distribution of the task-specific skills in different time steps. Task-specific skills are trained to represent the task-relevant knowledge, serving as the common skills' complementary and enhancing the adaptive action execution. Therefore, task-specific skills in the same tasks should be clustered together. Moreover, as steps go on, different tasks in the same task set will degrade to various sub-tasks that are more similar. The distance between specific skills' distribution from the middle & end stages of each episode will be closer compared to the distribution in the early stage.

---

Questions:

Q1: From Eq. (3), does $q(s_{t+1} \vert c_t^{1:K})$ has any meaning in real? How does HiSSD model the sequence $c_t^{1:K}$?

A: In Eq. (3), $s_{t+1}$ represent the ground-truth global next state and $q(s^\prime_{t+1} \vert c_t^{1:K})$ indicates that we use a forward predictor $q$ to predict the next global state using all agents' common skills. For example, in the SMAC environment, $q(s^\prime_{t+1} \vert \cdot)$ indicates using common skills selected by each unit to estimate the whole battlefield in the next time step. This process can be formulated as $s^\prime_{t+1} = q(c_t^{1:K})$, where $c_t^{1:K} = \pi_h (o_t^{1:K})$ is each agent's common skill. These skills are inferred by a transformer-based policy $\pi_h$ based on their local observation $o_t^{1:K}$ and ${1:K}$ indicates agents' index. The shape of $c_t^{1:K}$ in SMAC and MAMuJoCo environments are $(n_\mathrm{agent}, n_\mathrm{entity}, n_\mathrm{hidden})$ and $(n_\mathrm{agent}, n_\mathrm{hidden})$, respectively.

Q2: How can we ensure a correct skill under unseen task distribution as the negative samples are drawn from the limited seen task.

A: We train the task-specific skills to extract potential complementary knowledge for common skills and identify various behavior patterns for potential sub-tasks from the multi-task offline data. Although the target tasks are unknown, the policy is able to extract certain behavior patterns learned from multi-tasks that are more related to the current task. Our prime goal is to demonstrate that compared to the methods in literature, integrating task-specific skills into offline multi-task learning can further improve the policy's generalization capability. Moreover, we conduct a variant of HiSSD without task-specific skills (named w/o specific) additional experiments as the ablation study and represent the results below in answer to Q4. The results indicate the effectiveness of task-specific skills when deploying the skill-based policy in unseen tasks.

Q3: Why are the two baselines, ITD3-CQL and ITD3-BC, not compared in the SMAC experiments?

A: It's worth noting that ITD3-CQL and ITD3-BC are built on TD3 algorithm, which is typically designed for continuous control tasks. Therefore, these baselines can not be directly applied to the SMAC environment with discrete action space.

Q4: How is the performance if we only adopt a common skill without using the task-specific skills?

A: We conduct a variant of HiSSD named 'w/o specific' which does not learn task-specific skills and report the experimental results below. We find that this variant obtained significant improvement compared to the behavior cloning baseline (BC), indicating the benefits of our common skills learning pipeline. Moreover, the original HiSSD outperforms the w/o Specific variant, demonstrating that learning task-specific skills from a multi-task dataset can further improve the policy's generalization capability.

Components Analysis on Task-Specific Skills: 'w/o Specific' indicates only integrating common skills in HiSSD. Source Tasks

Target Tasks

Thank you for your time and feedback on HiSSD. We address each of your questions below.

Weaknesses:

W1: HiSSD should reveal more about training details as the framework design may lead to instability.

A: For a fair comparison, most of the hyperparameters in our method are consistent with previous methods. To further analyze the sensitivity of our method, we add additional ablation studies and present the results below. We also added more training details in Appendix (D) in the revision. The empirical results indicate that our method is robust under different hyperparameter settings. This makes it easier for our method to tune hyperparameters. We keep the same settings as previous methods for a simple reproduction.

Sensitivity Analysis on $\epsilon$ in Marine-hard Task Set Source Tasks

Target Tasks

Sensitivity Analysis on $\alpha$ in Marine-hard Task Set Source Tasks

Target Tasks

W2: It can be better to present an algorithm of the whole training process and the execution process for understanding.

A: Thanks for this suggestion. We have presented the pseudocode in Appendix (B), which shows the brief training and execution process of our method.

W3: In the ablation studies, the HiSSD variants (e.g., w/o Planning and Half-Negative) often show strong performance compared to HiSSD with minor differences or even surpassing HiSSD.

A: Firstly, the Half-Negative variant in the ablation studies shares the same architecture and objectives with the original HiSSD while reducing the number of negative samples. Comparing HiSSD with this variant demonstrates that more negative samples will lead to better performance. Secondly, although the original HiSSD shows minor differences in some source tasks compared to other variants, it significantly performs best in target tasks, indicating the improvement of generalization capability.

Questions:

Q1: The difference between HiSSD and a very recent work 'Variational Offline Multi-agent Skill Discovery'.

A: VO-MASD proposes an efficient method to discover cooperative patterns between agents from a subgroup view. They utilize offline datasets to jointly learn individual and subgroup skills to enhance the online MARL. Compared to VO-MASD, our method aims to train a generalizable multi-agent policy in a fully offline learning pipeline to reduce interaction costs. Our method does not learn hierarchy-like common skills, we utilize task-specific skills to complement the common skills to enhance the policy's generalization capability. Our method embeds the common skills for a weighted next-state prediction, enabling an offline policy improvement. We have added the citations of HyGen and VO-MASD in the revision.

Thank you for your time and feedback on HiSSD. We address each of your questions below.

Weaknesses:

W1: Ablation study can be further expanded to have a comprehensive analysis of different components.

A: We have already conducted several ablation studies to analyze the effectiveness of different components in our paper. To further show the benefits of task-specific skills, we conduct a variant of HiSSD named 'w/o Specific' which only learns common skills. We evaluate it on SMAC's Marine-hard task set and present the results below. The empirical results indicate the effectiveness of task-specific skills in offline multi-task MARL.

Components Analysis on Task-Specific Skills: 'w/o Specific' indicates only integrating common skills in HiSSD. Source Tasks

Target Tasks

W2: Comparison to some recent related works like HyGen is missing.

A: It's worth noting that the problem setting of HyGen is different from HiSSD. HyGen attempts to integrate offline pre-training for skill discovery to enhance the online multi-task learning, while HiSSD aims to improve the policy's generalization capability by offline multi-task learning. HyGen first pre-trains the skill space with an action decoder using the global information, then conducts additional online exploration to maintain a hybrid replay buffer using offline and online data, improving the high-level policy and refining the action decoder simultaneously. This helps HyGen to correct the mistakes in value estimation when learning from poor offline data. Compared to HyGen, our method requires no online exploration and pretraining steps. HyGen only conducts common skills learning while our method leverages task-specific skills to complement the common skills. We compare our method to HyGen in the SMAC's Marine hard task set and present the experimental results below. When the offline data quality is good (i.e., Expert), HiSSD outperforms HyGen on most tasks, showing its good generalization of learning from offline multi-task data. When the data quality is medium, HyGen benefits from additional online learning and further improves the policy.

Expert data quality Source tasks

Target tasks

Medium data quality Source tasks

Target tasks