Ad Hoc Teamwork via Offline Goal-Based Decision Transformers

Q1. Questions in Claims And Evidence.

A1. We apologize for any confusion. Firstly, we perform implicit modeling of teammates to capture their changes and update our TA-goal, so there is no explicit metric to measure the accuracy of teammate modeling. As for the generalization of the policy, we demonstrate this in our experiments. Specifically, we test the policy with teammates that have never been seen during training. The fact that our method achieves higher scores when cooperating with these unknown teammates, compared to other baseline methods, serves as evidence of the generalization ability of our proposed approach.

Q2. Questions in Methods And Evaluation Criteria.

A2. Regarding your concerns about data augmentation, more details can be found in our response to Reviewer TMEP, Response A1. Regarding the accuracy of TA-Goal, during training, we use the real state after $k$ steps as the ground truth, which is a common practice in many works. During testing, since there are no trajectories for reference, we cannot directly measure the accuracy of TA-goal's prediction. Our goal during testing is not to achieve the absolute accuracy of the TA-Goal but to assess its ability to generalize to unknown teammates in AHT.

Q3. Questions in Experimental Designs Or Analyses.

A3. The evaluation of teammate modeling accuracy and policy generalization is answered in A1. Since TA-RTG is a predicted value rather than a fixed input, it cannot be ablated in the traditional sense. Besides, we agree that further testing in more complex environments, such as SMAC, is necessary, and we plan to conduct such tests.

Q4. Questions in Essential References Not Discussed.

A4. Thank you for pointing out these relevant works. We acknowledge the importance of the two and will include a discussion of them in the revised manuscript to better situate our contributions within the existing literature.

Q5. Concern about innovation.

A5. Thank you for the comment. Our method is not designed to simply combine existing techniques, but is driven by a novel problem setting - offline AHT. This new setting raises unique challenges, which motivate our methodological choices. Rather than applying existing tools directly, we analyze why previous approaches like DT are insufficient and propose tailored solutions.

Our trajectory mirroring strategy is a data preprocessing method specifically designed for the AHT setting. It improves data efficiency and teammate diversity without introducing extra computational cost. After preprocessing, we learn policies from offline data. While we adopt the Decision Transformer (DT) architecture, we find that standard DT is not well-suited for our problem, as RTG cannot capture dynamic teammate behavior. To address this, we propose TA-RTG, a novel formulation that incorporates teammate-awareness. Furthermore, given the dynamic nature of the environment, we propose the novel concept of TA-Goal, a sub-goal representation predicted from TA-RTG, to guide the transformer’s decision-making in a hierarchical manner.

Q6. Concern about the method’s robustness when applied to low-diversity datasets.

A6. Thank you for raising this important point. While our method benefits from diverse offline data, we also aim to improve robustness through model design. Specifically, the proposed TA-RTG and TA-Goal mechanisms help the model infer and adapt to teammate behavior even under limited diversity. In our ablation studies, the variant without the trajectory mirroring strategy can be seen as a setting with reduced data diversity. We observe that the method still maintains reasonable performance under this condition, demonstrating a degree of robustness to limited diversity. Nonetheless, we acknowledge this as a valuable direction and plan to further explore robustness under low-diversity settings in future work.

Q7. Difference between TA-RTG and RTG.

A7. The main difference between the two is that TA-RTG considers teammate behavior in a team-shared reward setting. TA-RTG incorporates how teammates' actions influence the collective reward, adapting the agent's strategy based on the team context. TA-RTG shifts from relying solely on the agent’s observation to considering the team context, allowing it to adapt to changes in teammate behavior (TA-RTG and RTG ablation: https://anonymous.4open.science/api/repo/materials-E410/file/ablation_new.pdf).

Q8. Concern about poor generalization.

A8. Thank you for highlighting this important point. Our method addresses the generalization issue through both data-level and model-level designs. On the data side, the trajectory mirroring strategy increases teammate diversity. On the model side, the proposed TA-RTG and TA-Goal explicitly encode teammate behavior and adapt decision-making accordingly. These components together improve the model’s ability to generalize to unseen teammates. We will clarify this contribution further in the revised manuscript.

Thank you for your suggestions.

Q1. Concerns on the difficulty of collecting offline data and the applications of methods

A1.We apologize for the misunderstanding. Our claim that offline data is "relatively easy to collect" refers to its cost and safety advantages over online training, not to ignore practical challenges. We use simulator-collected data to avoid exploration risks (e.g., collisions in autonomous driving) and improve data efficiency with trajectory mirroring. For real-world data, we address challenges like long-horizon rewards using multi-sensor fusion [1] and hierarchical reward decomposition [2]. While data collection is challenging, offline RL remains valuable for providing a safe, low-cost baseline and enabling future online refinement.

Q2. Concerns on empirical validation and intuitive explanation of TA-Goal's advantages over RTG

A2.Towards empirical validation, our experiments (Fig. 3) show that TAGET (TA-Goal-based) outperforms DT (RTG-based). Intuitively, RTG is a long-term scalar target that doesn't adapt to changing teammate behaviors (e.g., obstacle avoidance). TA-Goal, on the other hand, dynamically adjusts based on teammate intent, enabling adaptive actions (e.g., flanking maneuvers) as shown in Fig. 4. https://anonymous.4open.science/api/repo/materials-E410/file/ablation_new.pdf

Q3. Why is the team context modeled as a multivariate random variable and selected Gaussian distribution

A3. The team context is modeled as a multivariate random variable to capture uncertainties in teammates' policies, enhancing decision robustness. The Gaussian distribution is chosen due to: (1) mathematical and computational advantages; (2) uncertainty modeling capability (3) research alignment—Gaussians are standard in deep probabilistic models (e.g., VAEs [3][4] ). Many existing works [5] adopt this approach, and we will clarify this in our paper more clearly.

Q4. The evidence to support this modeling strategy.

A4. In partially observable settings, the ego agent infers the global team context $h_i$ from local observations $o_i$. KL-divergence regularization (Eq. 6) aligns $z_i$ with $h_i$, enabling latent team dynamics capture without global info. This method has been validated in previous works [5][6] for improved generalization. If additional evidence is needed, we will include it in the revised manuscript.

Q5. Eq.(6). The first KL-divergence is a reverse KL-divergence.

A5. The first term in Eq. (6) is a reverse KL-divergence, following the beta-VAE approach [4], which encourages the latent representation to match a standard normal prior, preventing overfitting and improving generalization. The second term uses forward KL to align the proxy and team context representations.

Q6. Eq.(8) explanation about why the loss is formulated as binary cross entropy

A6. Thanks for the useful feedback. The groundtruth goal $\hat{G}$ is constructed by concatenating discretized observations (e.g., grid-world positions, task states) and one-hot encoding the global state $s_{t+k}$, where each dimension represents a binary indicator. We use BCE loss per dimension to handle binary classification, which aligns with the discrete nature of our environments. Using MSE for continuous regression would be suboptimal, as the goal representation is categorical and requires probabilistic modeling of binary outcomes. We will also fix the inconsistency between $\hat{G}$ and $G$.

Q7. Concerns about the applicability of methods in Methods And Evaluation Criteria.

A7. While collecting offline datasets is challenging, it is manageable in AHT. The core challenge is enabling the ego agent to handle unknown teammates, which previous works have addressed in two ways: achieving generalization through algorithms and constructing diverse sets of teammates. Even with algorithmic generalization, a diverse offline dataset is still needed for better performance, which doesn’t mean it’s not worth exploring. The offline method is not meant to replace online interactions but is motivated by safety and cost considerations, and can be further optimized when combined with online methods.

Q8. The dataset generation in Experimental Designs Or Analyses.

A8. Thank you for your feedback. For more details, please refer to our response to Reviewer cnPV, A1.

• [1] Li, et al. "Dynamic Semantic SLAM Based on Panoramic Camera and LiDAR Fusion for Autonomous Driving." IEEE TAITS 2025.
• [2] Matsuda, et al. "Hierarchical Reward Model of Deep Reinforcement Learning for Enhancing Cooperative Behavior in Automated Driving." JACII 2024.
• [3] Kingma, et al. "Auto-encoding variational bayes." 20 Dec. 2013.
• [4] Higgins, et al. "beta-vae: Learning basic visual concepts with a constrained variational framework." ICLR. 2017.
• [5] Gu, et al. "Online ad hoc teamwork under partial observability." ICLR. 2021.
• [6] Xie, et al. "Future-conditioned unsupervised pretraining for decision transformer." ICML, 2023.

Thank you for your kind suggestions and helpful feedback!

Q1: Concerns about teammate diversity.

A1. We apologize for the confusion about the teammate generation. First and foremost, Soft Value Diversity (SVD) is not a MARL algorithm but a framework specifically designed to generate diverse policies. SVD maximizes the differences in value estimates across observation-action pairs between different teams [1]. This design ensures diversity between training and testing policy sets. As a result, we did not conduct separate experiments to demonstrate the diversity of teammate sets, since this diversity is inherent to the SVD framework. To assuage your concerns, we've added the cross-play experiments between different populations, demonstrating significantly lower cooperation efficiency between populations compared to within-population cooperation, empirically validating that the policies represent truly distinct strategies. In cross-population cooperation experiments, each population sequentially designates one agent as the ego agent to collaborate with members from other populations. The matrix notation (Row1, Col3) specifically denotes an experimental configuration where Population 3's designated ego agent interacts with teammates from Population 1, establishing a systematic evaluation framework for inter-population coordination capabilities. As shown in the cross-play matrix(https://anonymous.4open.science/r/materials-E410/cross-play-matrix.md), in each row of the matrix, the diagonal positions behave best, which means that strategies between populations don't work well in coordinating, validating the diversity of our test teammate sets.

Q2. Concerns about the baseline and oracle upper bounds.

A2. Thanks for your raising the issue about baseline selection. As there are no existing methods specifically designed for offline AHT, our paper addresses this gap by proposing the first dedicated solution. To provide a reasonable comparison, we adapted methods from offline RL and online AHT. We’ve now added MADT [2] as a baseline, as you suggested. Our results show that TAGET outperforms MADT(https://anonymous.4open.science/api/repo/materials-E410/file/comparison.pdf). Due to time constraints, we only tested on the teammate test set of size 4, and we will fill in the experiments subsequently. It's important to note that standard MARL methods, including MADT, don’t fully align with the offline AHT setting, as they typically assume teammates of the same type. AHT, however, involves an ego agent adapting to diverse teammates. Our new experiments with MADT confirm this mismatch, highlighting the key strength of TAGET in handling teammate uncertainty.

Oracle Upper Bounds: We have included oracle upper bounds represented by the average self-play scores of test teammates, which are shown in the cross-play matrix. TAGET achieves 86.0%, 77.2%, and 95.1% of the oracle upper bounds in PP, LBF, and Overcooked environments respectively, which demonstrates strong performance considering the challenging nature of AHT with unknown teammates.

Q3. Questions about experimental analysis.

A3.Our offline interaction dataset includes ​2,300,000 transitions in PP, 1,430,000 transitions in LBF, and 5,500,000 transitions in Overcooked. While we have not yet conducted a systematic ablation on dataset size variation, our experiments with ​data augmentation ablations have revealed that our method still maintains reasonable performance under reduced data diversity. We will add related ablation experiments in the future. It's worth mentioning that our trajectory mirroring strategy mitigates the negative effect when the diversity of offline datasets is low (shown in Fig. 5).

• [1] Ding, Hao, et al. "Coordination Scheme Probing for Generalizable Multi-Agent Reinforcement Learning." (2023).
• [2] Meng, Linghui, et al. "Offline pre-trained multi-agent decision transformer." Machine Intelligence Research 20.2 (2023): 233-248.

We thank the reviewer for your constructive feedback.

Q1. Concerns about the trajectory mirroring strategy.

A1. Our trajectory mirroring strategy is a data preprocessing method specifically designed for the AHT, not a data augmentation method. It works by sequentially designating different agents as the ego agent in a given trajectory. As a result, the combinations of the remaining teammates change, increasing the diversity of teammate sets. This approach not only enhances data efficiency but also provides a wider variety of teammate combinations for training. Through ablation experiments, we have demonstrated that this preprocessing method improves algorithm performance without incurring additional computational overhead. If you believe that other augmentation methods could provide meaningful comparisons, we are open to considering further experiments, but our current focus is to showcase the effectiveness of this method in AHT.

Q2. Concerns about the teammate's diversity and teammate heterogeneity.

A2. Thank you for your insightful feedback. To clarify, the robustness of TAGET is focused on its generalization to unseen teammates, especially in terms of adapting to diverse and unknown behaviors. In our experiments (Fig. 3), we evaluate performance using diverse sets of previously unseen teammates to test this generalization. We have added the cross-play experiments between different populations, the cross-play matrix can be seen in https://anonymous.4open.science/r/materials-E410/cross-play-matrix.md.

Q3. Concerns about baseline

A3. Thanks for this insightful feedback! We agree that including classic offline RL methods like CQL [1] and QT [2] would provide a more comprehensive benchmark suite. We’ve now added CQL as a baseline. TAGET outperfoms CQL across all environments, which can be found in https://anonymous.4open.science/api/repo/materials-E410/file/comparison.pdf.

Q4. Concerns about evaluation metrics.

A4. There is currently no specific metric for measuring teammate adaptability. However, achieving a higher return generally indicates better cooperation with teammates in a game. To address this concern, we could consider comparing our agent’s adaptability with the Oracle Upper Bounds suggested by Reviewer cnPV, which would provide a meaningful comparison of our approach in terms of teammate adaptation.

Q5. Concerns about the novelty.

A5. Thank you for your comment. While we build on existing research such as Decision Transformer and return-to-go estimation, previous methods are not directly applicable to the novel problem of offline AHT. To address this, we introduce TA-RTG and TA-Goal to handle the unique challenges of offline AHT, where agents must infer and adapt to the behaviors of unseen teammates. This teammate-aware approach enables better generalization in AHT settings, differentiating our method from prior works. We believe this extension of offline RL into the AHT domain is a key innovation.

Q6. Concerns about the direct ablation for TA-RTG.

A6. Thank you for your suggestion. We have added direct ablation experiments for TA-RTG, and the results can be found in https://anonymous.4open.science/api/repo/materials-E410/file/ablation_new.pdf. The experimental results show that TA-RTG outperforms traditional RTG, as it is able to capture changes in teammates' strategies and adapt to unknown teammates.

Q7. Confusing figures and lack of detailed descriptions

A7. Thank you for your valuable feedback. We acknowledge that some figures, such as Figure 3, could benefit from clearer explanations. We will make sure to correct these issues in future revisions.

Q8. Concerns about the supplementary material.

A8. Thanks for your useful suggestion. We commit to including these additional experiment details, such as detailed hyperparameter settings, neural network architecture, the method for computing standard deviation and the computational time of each algorithm, in the Supplementary Material of the revised version. Furthermore, we will also open-source our code.

• [1] Kumar, A., Zhou, A., Tucker, G., & Levine, S. (2020). Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33, 1179-1191.
• [2] Kalashnikov, D., Irpan, A., Pastor, P., Ibarz, J., Herzog, A., Jang, E., ... & Levine, S. (2018, October). Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on robot learning (pp. 651-673). PMLR.