We appreciate Reviewer eJKo for recognizing the significance, quality, and completeness of our work. Below, we respond to the weaknesses.

(Weakness 1) Writing improvements

We appreciate Reviewer eJKo’s feedback on the writing quality. In the revised version, we will expand the captions to include more information and explanation.

(1) Figure 1: Training MOSDT on the MOSDB dataset. The arrows represent the sequence of operations and data flow among the dataset, the student networks, and the teacher network under the proposed policy self-distillation (PSD) framework.

(2) Figure 2: The network structure of the proposed MOSDT algorithm. The purple areas indicate our proposed innovative modules. The green areas represent data, while the blue areas represent networks. We enlarge one of the DSSL modules and display it in detail in the upper left corner. We only illustrate the situation at time t.

(3) Figure 3: Number of parameters. “Using KD”: Using conventional KD instead of PSD. “No PS”: No parameter sharing. MOSDT only requires 65% of the execution parameter count of CDT. PSD reduces the number of training parameters by 47% and the number of execution parameters by 58%.

(4) Figure 4: Visualization of the training on the “3x1Hopper” task. “No PS”: No parameter sharing. The purple curve in Subgraph (a) is enlarged and displayed in detail in Subgraph (b) for better viewing. Shaded areas represent sample standard deviations across multiple runs.

(5) Figure 5: The data distribution over the reward-cost plane of the proposed MOSDB dataset. Each point represents a return-cumulative cost tuple of a trajectory. The cost limit is set to 25.

Regarding the notation clarity, we confirm that the “L” in line 147 denotes the training sequence length instead of loss. We acknowledge that “L” could indeed be misunderstood as loss, so we will replace it with another notation, such as “M”, in the revised version. We will further check the clarity of notations to eliminate any possible misunderstandings.

(Weakness 2) Limitations of the full parameter sharing design

For agents with heterogeneous action spaces, full parameter sharing can effectively stabilize training and improve performance at the scale of the MOSDB dataset (up to 756 dimensions of observation spaces and 52 dimensions of action spaces). Full parameter sharing improves returns on 11 out of 18 tasks compared with no parameter sharing (cf. Table 2) and 12 to 14 out of 18 tasks compared with partial parameter sharing (cf. Table 8 in Appendix B.4).

For all tasks in the MOSDB dataset, agents have heterogeneous action spaces.

(1) MOS Velocity. Agents need to control distinct robot segments to move as quickly as possible while adhering to velocity constraints. The agents are heterogeneous because they play different roles in movements (e.g., the front and rear legs of “HalfCheetah” need to have different behavior patterns to move forward).

(2) MOS Goal. Multiple agents are required to navigate to targets matching their designated colors, while evading collisions and hazardous terrain. The agents are heterogeneous because they have different objectives (navigating to targets in different colors) and do not share rewards and costs (cf. Table 13 in Appendix C), which makes them not interchangeable.

(3) MOS Isaac Gym. It focuses on collaborative tasks between robotic agents, such as dual manipulators performing coordinated object manipulation like ball-handovers. Agents are heterogeneous because they play different roles in collaborations (e.g., dual manipulators are responsible for throwing and catching the ball, respectively).

To verify the effectiveness of full parameter sharing in more challenging scenarios with agents having heterogeneous action spaces, we will introduce a series of higher-dimensional power grid scheduling tasks to future versions of the MOSDB dataset.

(Weakness 3) Impact of CBE on training dynamics

We appreciate Reviewer eJKo for the suggestion to analyze the effect of CBE on training dynamics. According to the uploaded learning curves (cf. the link in the abstract), CBE helps improve training stability on almost all tasks by eliminating the noise in cost information. For training dynamics, CBE provides better training initializations and safety controls on most tasks.

The following statements need to be read in conjunction with the uploaded learning curves (cf. the link in our abstract). We observe three main advantages of CBE on training dynamics.

(1) Stabilizing training. On almost all tasks (e.g., “2x3HalfCheetah”, “2x3Walker2d”, and “3x1Hopper”), CBE results in smoother learning curves, compared with MOSDT without CBE. CBE mitigates volatility and reduces sharp declines. The cost binarization in CBE eliminates the noise in the cost information, thereby stabilizing the training.

(2) Better training initialization. In most tasks (e.g., “2x4Ant”, “4x2Ant”, and “Multi-Point1”), CBE yields higher returns with safe cumulative costs at the beginning of training. CBE directly transmits the prior information about the reward-cost correlation to MOSDT by embedding safety signals into returns, enabling agents to learn more efficiently. The cost binarization in CBE eliminates the noise of cost data, improving data efficiency, thereby offering a better training initialization.

(3) Better safety control. In most tasks (e.g., “2x1Swimmer”, “2x4Ant”, and “OverFingerMA”), CBE yields cumulative cost learning curves with faster and more stable convergence. For 11 out of 18 tasks, CBE reduces the cumulative costs (cf. the ablation study results in Table 2). CBE provides better safety control by improving the processing of cost information.

We will add the above analysis and related learning curves to the revised version.

We appreciate Reviewer mXFE for recognizing the significance, reasonability, and completeness of our work. Below, we respond to the weakness and the question.

(Weakness 1) Effectiveness of policy self-distillation (PSD) in offline MARL

We appreciate the Reviewer mXFE for the suggestion to analyze PSD without safety settings. To verify the effectiveness of PSD in offline MARL without safety constraints, we removed the safety constraints in the environments of the MOSDB benchmark and let agents run 1,000 steps during evaluation. In this setting, we train MOSDT with the proposed PSD and with the conventional knowledge distillation (used in MADTKD [1]), respectively. The results are shown in the tables below (with the same setting as Table 2).

Table A: Results of training MOSDT with PSD on the MOSDB benchmark (without safety constraints).

Table B: Results of training MOSDT with the conventional knowledge distillation on the MOSDB benchmark (without safety constraints).

According to these two tables, PSD improves returns on 12 out of 18 tasks (without safety constraints) compared to the conventional knowledge distillation. PSD improves most offline MARL tasks (with or without safety settings).

The primary focus of our paper is to tackle the novel and underexplored problem of multi-agent offline safe RL (MOSRL). Offline MARL methods, such as MADTKD [1], cannot address safety constraints, making them unsuitable for direct comparison within our safety-focused experimental framework. Our experiments thus prioritize the safety setting to demonstrate the risk control capabilities of MOSDT. We will conduct more in-depth research into the effectiveness of PSD in offline MARL without safety settings in subsequent work.

(Question 1) Clarification on the full parameter sharing design

We thank Reviewer mXFE’s interest in our full parameter sharing design. We note that full parameter sharing is not merely initializing all student networks with the same parameters before training. We share all parameters across all student networks throughout the training process, as described in Eq. (6). This design keeps the training stable during the entire process and significantly reduces model parameters. In practice, we only create one instance of the Student class for all agents. In the entire MOSDT network, only the teacher action heads and the teacher feature projectors are distinct for different agents. According to the ablation study, compared with no/partial parameter sharing, full parameter sharing yields the best performance (cf. Table 2 in Section 4.2 and Table 8 in Appendix B.4) and slashes parameter count to the greatest extent.

In the revised version, we will enhance the description of full parameter sharing. We appreciate Reviewer mXFE for highlighting this point, which allows us to improve the clarity of the paper.

References

[1] Tseng W C, Wang T H J, Lin Y C, et al. Offline multi-agent reinforcement learning with knowledge distillation[J]. Advances in Neural Information Processing Systems, 2022, 35: 226-237.

We appreciate Reviewer vSoS for recognizing the significance and quality of our work. Below, we respond to the weakness and the question.

(Weakness 1) Limited baseline comparison

We appreciate the Reviewer vSoS’s suggestion to broaden our evaluation by incorporating advanced offline MARL methods. We train MADiff [1] (a diffusion-based offline MARL method) and MOMA-PPO [2] (a model-based offline MARL method) on the proposed MOSDB dataset with their default settings for the tasks with the highest state dimensions. To make these two methods applicable to safety constraints, we input cumulative costs into them along with returns. The results are shown in the tables below (with the same setting as Table 1).

Table A: Results of training MADiff [1] on the MOSDB dataset.

Table B: Results of training MOMA-PPO [2] on the MOSDB dataset.

Compared with our proposed MOSDT, MADiff [1] yields lower returns on 15 out of 18 tasks and leads to unsafe policy on 1 task (Task No. 2). MOMA-PPO [2] yields lower returns on 14 out of 18 tasks compared with MOSDT, while maintaining safety. We will supplement these new results in the revised version and upload their training curves.

Although advanced offline MARL methods like MADiff [1] and MOMA-PPO [2] perform well in standard offline MARL settings, they lack safety-oriented designs, limiting their applicability to safety-critical MOSRL tasks.

(Weakness 2) Limitations of CBE

We acknowledge that the cost binarization in CBE may limit its adaptability in settings with soft, dynamic, or probabilistic constraints, as stated in Appendix D (lines 520-522). However, in most cases, the cost limits are fixed values, such as the temperature limits of cables in power systems [3] and the distance limit between vehicles and human in autonomous driving [4]. Many widely used safe RL benchmarks, such as Gymnasium [5] and DSRL [6], also adopt fixed values as their cost limits. In most cases where the cost limits are fixed, CBE offers a better alternative to specific cumulative costs, given its powerful effect (improving returns on 14 out of 18 tasks in Table 2), plug-and-play feature, and intuitive design.

In the future work, we will exploration several directions to improve our approach to accommodate soft, dynamic, or probabilistic constraints, enhancing its adaptability to more complex and realistic settings.

(1) Soft safety constraints: Replacing hard binary signals with a continuous cost representation. This would allow the model to reason about the severity of safety violations rather than treating them as all-or-nothing events, improving performance in environments with graded safety settings.

(2) Dynamic safety constraints: Introducing mechanisms to adjust thresholds based on contextual factors or trajectory history. This could enhance flexibility in settings where safety requirements evolve.

(3) Probabilistic safety constraints: Integrating uncertainty-aware models to handle probabilistic safety constraints, which could improve performance in applications where safety requirements are not deterministic.

(Weakness 3 & Question 1) Analysis of failure cases

We appreciate Reviewer vSoS’s suggestion on analyzing the failure cases without safety constraints violated. There are three major causes for the violation of safety constraints.

(1) High risk in the early stage of training. As shown in the uploaded training curves (cf. the link in the abstract), most failure cases show excessive cumulative costs at the initial stages of training. For example, on the “2x3HalfCheetah” task (the cost limit is 25), when 10% of the training (10k steps) is completed, the complete MOSDT only yields a cumulative cost of 13.1, while the MOSDT without PSD yields a cumulative cost of 18.8, and CDT [7] yields a cumulative cost of 62, resulting the violation of safety constraints of the latter two methods (cf. Tables 1 and 2).

(2) Instability of the cumulative costs in training. In most failure cases, the training curves of cumulative costs show severe fluctuations (e.g., the training curves of BCQ-Lag [8] on the “Multi-Point2” task). This instability indicates that the model often enters unsafe states and cannot maintain safety.

(3) Lack of communication among agents. Maintaining global safety requires cooperation among agents. Lack of communication among agents could cause them to violate safety constraints. For example, in the "2x3HalfCheetah" task, rollout visualizations of the MOSDT without PSD (i.e., lacking centralized training) revealed that the agent's hind leg persistently accelerated, while its front leg remained stationary to decelerate. This uncoordinated action led to over-speeding and eventual task failure (cf. Table 2).

We will supplement the above analysis in the revised version.

References

[1] Zhu Z, Liu M, Mao L, et al. Madiff: Offline multi-agent learning with diffusion models[J]. Advances in Neural Information Processing Systems, 2024, 37: 4177-4206.

[2] Barde P, Foerster J, Nowrouzezahrai D, et al. A model-based solution to the offline multi-agent reinforcement learning coordination problem[J]. arXiv preprint arXiv:2305.17198, 2023.

[3] Chen X, Qu G, Tang Y, et al. Reinforcement learning for selective key applications in power systems: Recent advances and future challenges[J]. IEEE Transactions on Smart Grid, 2022, 13(4): 2935-2958.

[4] Chen Y, Ji C, Cai Y, et al. Deep reinforcement learning in autonomous car path planning and control: A survey[J]. arXiv preprint arXiv:2404.00340, 2024.

[5] Ji J, Zhang B, Zhou J, et al. Safety gymnasium: A unified safe reinforcement learning benchmark[J]. Advances in Neural Information Processing Systems, 2023, 36: 18964-18993.

[6] Liu Z, Guo Z, Lin H, et al. Datasets and benchmarks for offline safe reinforcement learning[J]. arXiv preprint arXiv:2306.09303, 2023.

[7] Liu Z, Guo Z, Yao Y, et al. Constrained decision transformer for offline safe reinforcement learning[C]//International conference on machine learning. PMLR, 2023: 21611-21630.

[8] Xu H, Zhan X, Zhu X. Constraints penalized q-learning for safe offline reinforcement learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2022, 36(8): 8753-8760.

We appreciate Reviewer Twb8 for recognizing the completeness and usability of our work. Below, we respond to the weaknesses.

(W1) Motivation for multi-agent offline safe RL (MOSRL)

We further emphasize that MOSRL is not merely an incremental extension but a critical advancement addressing challenges in many important application scenarios.

(1) Autonomous vehicle coordination [1]. Driving policies for autonomous vehicles should be safe and cooperative to avoid any collisions with each other. Online exploration is impractical due to safety risk.

(2) Power grid scheduling [2]. A power grid involves multiple agents (e.g., generators). Safety constraints, such as preventing overload, are crucial. Online trial-and-error is infeasible due to potential catastrophic consequences.

(3) Robot collaboration [3]. In industrial settings such as manufacturing and logistics, multiple robots need to coordinate under strict safety constraints (e.g., collision avoidance during collaborative machining and transportation). Online learning may cause unacceptable asset losses.

We mentioned scenarios (1) and (2) in Section 1 (lines 29-30). We will emphasize more on the motivation of MOSRL in the revised version. The complex requirements in these scenarios reveal the limitations of existing RL methods. The developed MOSRL framework, algorithm, and benchmarks have the potential to meet the requirement of learning safe, cooperative policies from offline data.

Indeed, the MOSDB dataset has incorporated 18 practical tasks, including bionic robot racing, robot navigation and obstacle avoidance, and throwing/catching objects by bionic robotic hands. These tasks have significant implications in various real-world engineering domains, including autonomous vehicles and industrial automation.

We recognize the value in expanding the scope for greater practical significance. As stated in Appendix D (lines 522-523), future efforts will focus on expanding the MOSDB dataset, by introducing a series of power system operation tasks with real-world data.

(W2) Design motivation of MOSDT

Instead of a simple combination of CDT and MADTKD, MOSDT is a new algorithm tailored for the MOSRL problem, with several key innovations that significantly enhance its performance in multi-agent environments with safety constraints.

We propose two innovative designs to address key challenges in multi-agent environments, including policy self-distillation (PSD) and full parameter sharing. Inspired by the spirit of MADTKD that introduces policy distillation to offline MARL, we propose PSD to extend CDT to multi-agent settings. Different from MADTKD, PSD embeds student networks into the teacher and performs distillation synchronously with supervised learning, reducing training parameter count by 35% and training time by 24%. As shown in the following table (with the same setting as Table 2), compared with the PSD in our proposed MOSDT, policy distillation (in MADTKD) causes return losses on 11 out of 18 tasks and leads to unsafe policy on 1 task (Task No. 2).

Table A

The full parameter sharing among agents reduces training parameter count by 47% and execution parameter count by 58% (cf. Fig. 3).

We propose a plug-and-play module, cost binary embedding (CBE), to address key challenges in safety-critical settings. For MOSDT, CBE enhances the cost information processing capabilities of the CDTs by binarizing cumulative costs as safety signals and embedding the signals into returns. CBE improves returns on 14 out of 18 tasks (cf. Table 2).

(W3) Quality of the MOSDB benchmark

Our benchmark MOSDB can provide an informative comparison on most of the tasks in a widely adopted environment, Safety Gymnasium [4]. Indeed, MOSDT achieves SOTA returns on 14 out of 18 tasks (cf. Table 1). On 9 out of these 14 tasks, MOSDT achieves a lead of more than 5% compared to the suboptimal method. On some tasks, MOSDT achieves a return several times higher than the suboptimal method (9.2 times on “3x1Hopper”, 4.2 times on “6x1HalfCheetah”, and 2.3 times on “4x2Ant”). On 13 out of these 14 tasks, MOSDT achieves a lead of more than 10% compared to the average performance of the baseline models.

A few tasks in the MOSDB benchmark demonstrate small performance gaps among methods—we note that this is not unusual in other well-known benchmarks. For instance, in Safety Gymnasium [4], the best method achieves a lead of more than 5% compared to the suboptimal method on only 2 out of 6 Safety Velocity tasks. We agree with Reviewer Twb8 that for MOSDB, the statistical significance of performance comparison requires more attention.

The offline data in the MOSDB dataset is collected strictly following the procedure of a widely used single-agent offline safe dataset, DSRL [5]. Following existing well-known RL benchmarks, such as Safety Gymnasium [4], DSRL [5], and D4RL [6], we evaluate the quality of algorithms based on achieving SOTA average returns on how many tasks.

In the original submission, we separated mean values and standard deviations due to space constraints. We will merge them into one table in the revised version.

(W4) Necessity of MOSRL

As mentioned in Section 1 (lines 29-30), MOSRL is promising in various application scenarios, such as autonomous vehicle coordination and robot collaboration (cf. our reply to W1 for more details), where agents need to learn safe, cooperative policies using offline data.

As stated in Section 1 (lines 21-26), existing offline RL methods focus solely on either multi-agent or safety settings, leaving a gap in algorithms specifically designed for MOSRL. Existing offline MARL methods (e.g., MADT, MADiff, and PTDE) lack built-in safety mechanisms, rendering them ineffective for safety-critical applications. Current offline safe RL methods (e.g., CDT, SaFomer, and SDT) are tailored for single-agent settings, lacking the necessary coordination mechanisms for multi-agent systems.

To lay the foundation for MOSRL, we provide its first algorithm (MOSDT) and first dataset (MOSDB). We train MOSDT and other baseline methods to set up the MOSDB benchmark and verify the superiority of MOSDT in addressing MOSRL challenges.

(W5) Novelty Compared to MADTKD

Compared to MADTKD, the proposed MOSDT incorporates three key innovations.

(1) Policy self-distillation (PSD). As a two-stage method, it first trains an independent teacher network and then distills the policy to students. We introduce policy self-distillation (PSD) to MARL. PSD synchronously performs policy distillation alongside supervised policy learning, integrating student networks into the teacher network to streamline training and enhance parameter efficiency.

(2) Full parameter sharing. Existing offline MARL methods (including MADTKD) assign distinct network parameters to different agents. We propose full parameter sharing across all agents to ensure training stability and make MOSDT more lightweight.

(3) Safety property with CBE. As an offline MARL algorithm, MADTKD was not designed to address safety constraints. To enhance the safety performance of MOSDT, we propose a new plug-and-play module, CBE, which i) binarizes cumulative costs as safety signals, and ii) embeds the signals into returns.

(W6) Details of the MOSDB Dataset

The offline data in the MOSDB dataset is collected from Safety Gymnasium [4]. Please refer to [4] for detailed introduction on the nature of the tasks. We briefly describe these tasks below and will supplement this information in the revised version.

(1) MOS Velocity. Robots are required to move as quickly as possible while adhering to velocity constraints. Multiple agents need to control distinct body segments cooperatively.

(2) MOS Goal. Each agent is required to reach its color-designated target while avoiding collisions and hazardous terrain.

(3) MOS Isaac Gym. It focuses on collaborative robotic tasks, such as coordinated ball-handovers between dual manipulators, with enforced safety constraints on joint movements.

Regarding trajectory collection, as stated in Section 1 (line 77), we train two multi-agent safe RL algorithms on Safety Gymnasium [4], logging agent-environment interactions as offline data. As a commonly used concept in offline RL, “trajectory” is a fixed sequence of historical state-action-reward (cost) transitions.

As mentioned in Section 2 (line 114), the lack of MOSRL datasets is our key motivation for building the MOSDB dataset. The evaluation methods are described in Section 4.1 (lines 241-246). The baseline methods trained on the MOSDB dataset are introduced in Section 4.1 (lines 232-239).

We will integrate some of the above details of MOSDB into Section 3.4 in the revised version.

References

[1] Chen Y, et al. Deep reinforcement learning in autonomous car path planning and control: A survey. arXiv (2024).

[2] Chen X, et al. Reinforcement learning for selective key applications in power systems: Recent advances and future challenges. IEEE Transactions on Smart Grid (2022).

[3] Singh B, et al. Reinforcement learning in robotic applications: a comprehensive survey. Artificial Intelligence Review (2022).

[4] Ji J, et al. Safety gymnasium: A unified safe reinforcement learning benchmark. NeurIPS (2023).

[5] Liu Z, et al. Datasets and benchmarks for offline safe reinforcement learning. arXiv (2023).

[6] Fu J, et al. D4rl: Datasets for deep data-driven reinforcement learning. arXiv (2020).

Dear Reviewer Twb8,

We hope that our rebuttal has effectively addressed all your concerns. As the deadline for the discussion period draws near, we would greatly appreciate it if you could re-evaluate our manuscript and provide any feedback. Thank you for your time and consideration.

Best regards,

Authors of Paper Submission #4775

Dear Reviewer Twb8,

We hope that our rebuttal has effectively addressed all your concerns. As the deadline for the discussion period draws near (less than 24 hours), we would greatly appreciate it if you could re-evaluate our manuscript and provide any feedback. Thank you for your time and consideration.

Best regards,

Authors of Paper Submission #4775

Dear Reviewer Twb8,

We hope that our rebuttal has effectively addressed all your concerns. As the deadline for the discussion period draws very near (less than only 1 hour), we would greatly appreciate it if you could re-evaluate our manuscript and provide any feedback. Thank you for your time and consideration.

Best regards,

Authors of Paper Submission #4775