Efficient Multi-agent Offline Coordination via Diffusion-based Trajectory Stitching

Thank you very much for carefully reviewing our paper and providing constructive comments and suggestions. In response, we have added experiments and comparisons with different methods, which are presented as follows:

Q1: Reasons of generating the imbalanced dataset using random perturbations on expert trajectories.

A1: Our primary goal in generating imbalanced datasets is to create scenarios more challenging than those in mixed-quality datasets by explicitly introducing temporal and spatial imbalances. This approach pushes the boundaries of data augmentation techniques in Offline MARL. To achieve this, we apply random perturbations to expert trajectories, simulating both types of imbalances. This method constructs datasets with more pronounced imbalances compared to simple mixing strategies, allowing for a more rigorous evaluation of our method under challenging task settings.

Building on a general framework for addressing data imbalance in data augmentation [1], our approach extends beyond spatial imbalance to also tackle temporal imbalance. This expansion broadens the scope of our method and introduces additional challenges, enabling exploration of more diverse and realistic multi-agent settings. These innovations highlight the unique aspects of our approach.

Regarding to the performance of our method on the mixed-quality datasets mentioned by the reviewer, as shown in the simplified table below, MADiTS still significantly enhances dataset quality in mixed-quality md-replay datasets (with lines bolded). In tested environments, its performance approaches or even matches that of medium-quality datasets, demonstrating its ability to effectively leverage limited high-quality data to improve suboptimal data segments. For more detailed results, please refer to Table 5 in the revised manuscript.

Q2: The impact of MADiTS on the diversity of the trajectories.

A2: The dynamics constraints and behavior correction mechanism not only preserve the diversity of trajectories but also significantly improve data quality. To illustrate this, we provide visualizations of the synthesized trajectories under a fixed, non-stochastic environment's initial state, defined by the agents' starting positions and landmark locations. Comparisons between MADiTS and its variant without dynamics constraints or behavior correction reveal that our approach generates data with broader coverage and higher diversity. Further details and analysis can be found in Figure 7 and Appendix H.6.

Q3: Clarity regarding the computational cost of MADiTS and vague descriptors.

A3: Our method achieves computational complexity comparable to other baseline approaches. To demonstrate this, we provide a summary of the computational costs for MADiTS and baseline methods on the Cooperative Navigation task. The experiments were conducted on servers equipped with GeForce RTX 2080 Ti GPUs. As shown in the table below, MADiTS maintains computational complexity similar to other methods, staying within an acceptable range while ensuring competitive performance.

As for the vague descriptors, we have changed the sentence from “This iterative procedure continues until we obtain a satisfactory augmented episode trajectory” to “This iterative procedure continues until a satisfactory augmented episode trajectory is generated within the predefined limit or is discarded otherwise”. We greatly appreciate your suggestion.

Dear Reviewer bitr:

We thank you again for your comments and hope our responses could address your questions. As the response system will end in four days, please let us know if we missed anything. More questions on our paper are always welcomed.

Sincerely yours,

Authors of Paper10913

Thanks to the authors for their detailed response and for addressing some of my initial questions. I have two follow-up queries for further clarification:

1. What was the computational cost of the 12m experiments across the different methods (MADiTS, MADiff, MA-MBTS)? This would help better understand how the approach scales computationally with respect to the number of agents.
2. Do you have quantifiable measures of agent diversity? While examining trajectories is informative, it may not adequately capture or quantify diverse behaviours in a rigorous way.

Thank you very much for carefully reviewing our paper and providing constructive comments and suggestions. In response, we have conducted additional experiments and clarified the points of confusion. The detailed response is presented as follows:

Q1: The effectiveness of MADiTS on datasets with a larger number of agents.

A1: MADiTS demonstrates enhanced sample efficiency across various methods, even as the number of agents increases. To validate its effectiveness, we conducted experiments on the 12m map from SMAC, with 12 marine agents and 12 marine enemies. The results, shown in the table below, indicate that MADiTS outperforms the baseline method MA-MBTS and achieves competitive performance compared to MADiff, highlighting its broad applicability. Addressing scalability in large-scale MARL settings with hundreds or thousands of agents[1], possibly through techniques like agent grouping, is a promising direction for future work.

Q2: Results on SMACv2

A2: Our method enhances sample efficiency across various offline MARL approaches and benchmarks. In addition to the commonly used SMAC and MPE, we conducted experiments on the terran_5_vs_5 and zerg_5_vs_5 maps from SMACv2. As shown in the following table, MADiTS consistently improves the sample efficiency of algorithms such as BC, OMIGA, and CFCQL under different data imbalance settings, demonstrating its general applicability.

Q3: Confusion of the term “stitching”.

A3: In the context of our work, stitching refers to the head-to-tail concatenation of two trajectory segments to create a complete trajectory. This definition aligns with certain interpretations in the literature, where stitching is used metaphorically to describe various operations on trajectories. Existing works in the Offline RL domain employing stitching primarily focus on three main approaches:

1. Head-to-tail concatenation of two trajectory segments (e.g., SSD[2]): In this approach, each trajectory segment is generated to ensure high quality, and subsequent segments are initiated from the endpoints of the preceding ones, achieving head-to-tail concatenation.
2. Creating new transitions or sub-trajectories as bridges between existing trajectories (e.g., DiffStitch[3], MBTS[4]): This involves generating a "bridge" to connect trajectory A and trajectory B, resulting in a new optimal trajectory C, structured as A + bridge + B.
3. Describing the ability to learn optimal policies from suboptimal data (e.g., QDT[5], EDT[6], WT[7]): Unlike the previous two, this interpretation uses stitching as a metaphor for the ability to extract optimal policies from suboptimal trajectories, without explicitly concatenating or modifying trajectories.

Thank you for bringing this point to our attention. We have further clarified the definition and application of stitching in the revised manuscript.

Thank you for your inspiring and thoughtful reviews. We have prepared the following experimental results and comments for your proposed weakness and questions, and we hope they can relieve your concern:

Q1: Experimental analysis on methods with different hyperparameters and how they are determined.

A1: We use grid search to select these hyperparameters, which is popular in hyperparameters selection[1]. Take $\delta_{recon}$ as an example, which determines the strictness of the dynamics constraint. We first specify a list of candidate values, such as [0.003, 0.005, 0.01, 0.02, 0.05], and then evaluate the filtering strictness on a small-scale dataset. If $\delta_{recon}$ is too small, stitching a complete trajectory will take a long time. Conversely, if $\delta_{recon}$ is too large, no generated trajectories will be discarded. Details of experimental results for other key hyperparameters are included in Appendix H.3 of the revised paper.

Q2: Regarding to computational complexity.

A2: Our method achieves computational complexity comparable to other baseline approaches. To demonstrate this, we provide a summary of the computational costs for MADiTS and baseline methods on the Cooperative Navigation task. The experiments were conducted on servers equipped with GeForce RTX 2080 Ti GPUs. As shown in the table below, MADiTS maintains computational complexity similar to other methods, staying within an acceptable range while ensuring competitive performance.

Q3: How does the proposed method address the imbalance of the dataset in terms of time and space?

A3: We leverage a diffusion model to seamlessly stitch trajectory segments across different timesteps, effectively addressing temporal imbalances. To tackle spatial imbalances, we employ Integrated Gradients to identify these imbalances and optimize them by regenerating data through the diffusion model. We apologize for the earlier unclear explanation and have revised the introduction for clarity in the updated manuscript.

Reference:

[1] Liashchynskyi P., et al. Grid search, random search, genetic algorithm: a big comparison for NAS. arXiv preprint arXiv:1912.06059, 2019.

Dear Reviewer onPK:

We thank you again for your comments and hope our responses could address your questions. As the response system will end in four days, please let us know if we missed anything. More questions on our paper are always welcomed.

Sincerely yours,

Authors of Paper10913

Thank you very much for carefully reviewing our paper and providing constructive comments and suggestions, which have helped improve the work a lot. Our response is presented as follows:

Q1: a straightforward extension from diffusion-based single-agent RL? with single-agent states and actions replaced by global observations and actions.

A1: Our method is the first to integrate a diffusion model for data generation with Trajectory Stitching, enhancing coordination in multi-agent reinforcement learning (MARL) and addressing challenges beyond single-agent RL[1]. While both approaches aim to improve sample efficiency, MARL presents unique difficulties due to complex agent interactions. Specifically, we leverage the diffusion model's generative capabilities to handle two critical types of trajectory data imbalances: temporal, concerning relationships across timesteps, and spatial, focusing on interactions among agents. These imbalances complicate data representation and utilization, hindering MARL progress.

To address these issues, we propose two innovations. A bidirectional dynamics constraint ensures generated trajectories align with environmental dynamics. An offline credit assignment technique identifies and optimizes underperforming agents within trajectory segments. Experimental results show that MADiTS achieves superior sample efficiency across diverse MARL tasks, effectively tackling data imbalance and coordination challenges.

Q2: How does this approach compare to MADiff.

A2: Our method MADiTS differs from MADiff in addressing sample efficiency challenges in MARL. MADiTS focuses on data augmentation by combining a diffusion model with dynamic detection and credit assignment techniques for offline MARL policy training. In contrast, MADiff primarily uses a diffusion model for trajectory prediction in action planning. Specifically, MADiTS specifically targets data imbalance issues in MARL, while MADiff aims to enhance learning efficiency in standard offline MARL settings. To emphasize these distinctions, we compare MADiTS with MADiff across various environments. The results, summarized in the table below, show that MADiTS outperforms MADiff and other baselines, demonstrating its superior effectiveness in handling data imbalance challenges. We also include performance of other MARL algorithms on the imbalanced dataset, with detailed results presented in the Original column.

Dear Reviewer Af9P:

We thank you again for your comments and hope our responses could address your questions. As the response system will end in four days, please let us know if we missed anything. More questions on our paper are always welcomed.

Sincerely yours,

Authors of Paper10913

Thank you to the authors for making a serious effort to address my concerns. I believe the additional experiments provide better clarity on the contributions. I am now more positive about the paper and will increase my rating after seeing the authors updating the paper with the additional experiments and discussions.