Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARL

We thank the reviewer for their time taken to review our paper. With the following response we hope to address the reviewers' concern regarding the novelty of our work as well as the question about theoretical scaling guarantees.

On the novelty and significance of Oryx

We understand that at a high level Oryx may seem like a simple combination of existing approaches; however, we would argue that this overlooks the many practical challenges we had to overcome in order to implement this algorithm successfully. Moreover, we believe that many significant and impactful contributions to science over the years have come from novel combinations of existing ideas (potentially from different subfields). Of course, if the results for Oryx were incremental at best, then one could justifiably lean towards rejecting it. But since we showed extensively that the combination of ideas in Oryx results in a substantial improvement over the state-of-the-art across a broad range of tasks, we believe that it's an important finding that should be shared with the research community. Particularly because scalable Offline MARL has huge practical potential for a range of problem areas.

Furthermore, we think that overly optimising for novelty at scientific conferences has actually resulted in some bad incentives that have led to some dubious research. Take, for example, the recent Offline MARL benchmarking paper [1], which showed that many of the novel ideas introduced to the field in recent years as conference papers turned out not to perform as well as relatively simple algorithms that “simply” combined existing ideas from the literature. Thus, we think that penalising us for not appearing to be novel enough is unfair, particularly in this case where we have provided substantial empirical evidence that the method works far better than anything else in the field.

More details on Oryx contributions Furthermore, would like to draw attention to some of the key aspects of our contribution, which we believe make it significant:

• Novel Sequential Autoregressive ICQ Loss: We developed and formally derived a novel sequential autoregressive version of the ICQ loss, which we believe is a significant theoretical contribution.
• Offline Training of Online Algorithms: A core challenge we addressed was modifying online (on-policy) algorithms like Sable and MAT to enable their effective offline training from off-policy samples from a replay buffer (dataset). This adaptation presented non-trivial technical hurdles.
• Network Architecture Modifications: We carefully considered and implemented modifications to the original Sable (and MAT) network architecture to incorporate the dual Q-value and Policy distribution heads required for the ICQ loss.
• Temporal Memory Handling: Appropriately handling the resetting of temporal memory in the retention-based memory of the Sable backbone, particularly when training from offline sequences, required careful attention to detail.
• Empirical Performance: Our work demonstrates substantially superior performance, not just marginal, across multiple benchmarks (SMAC, RWARE, Multi-Agent MuJoCo) and new challenging datasets featuring up to 50 agents. Having addressed the issue raised by the reviewer regarding comparison to MAT, we are confident our robust empirical investigation differentiates our work within the existing literature.
• Open-Sourcing and Reproducibility: To ensure full transparency and foster continued research, we have already made our implementation and the novel datasets openly available, complete with detailed instructions for experiment replication. We are confident this initiative will greatly facilitate further advancements stemming from Oryx-based models, and we eagerly anticipate the community's contributions building upon this foundation.

On Oryx’s theoretical guarantees Sable, on which Oryx is built, has theoretical guarantees when using the PPO training objective. This stems from the work by Kuba et al. (2022) [2]. In particular, HAPPO (multi-agent PPO with autoregressive updates) was shown to be an instance of their multi-agent mirror learning framework and to be theoretically sound. To obtain an instance of mirror learning requires only defining a valid drift functional, neighbourhood operator and sampling distribution. In Sable, these design choices are exactly as they are for HAPPO, and therefore, Sable inherits the same theoretical monotonic improvement and convergence guarantees as HAPPO. However, although Oryx shares a network architecture that is similar in spirit as Sable, it is fundamentally different as an offline algorithm and uses a different autoregressive objective. This makes it non-trivial to place Oryx within the same mirror learning framework and thus claim similar theoretical guarantees. We consider this an interesting avenue for future work.

While Oryx's theoretical guarantees remain an exciting avenue for future research, the significant empirical results and novel algorithmic contributions presented in this work already establish a strong foundation, demonstrating the considerable practical value and advancement this paper brings to the community.

In conclusion, we trust we have addressed the main concerns raised by the reviewer. Should the reviewer have any follow-up questions, please do not hesitate to ask. We would be happy to engage during the author-reviewer discussion period.

[1] Formanek, Claude, et al. “Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation.” NeurIPS ‘24

[2] Kuba, Jakub Grudzien, et al. "Heterogeneous-agent mirror learning: A continuum of solutions to cooperative marl." arXiv preprint arXiv:2208.01682 (2022).

We thank the reviewer for the time taken to review our paper. We appreciate the reviewer highlighting that our “set of benchmarks and baselines considered is comprehensive.”

Comparing Oryx to MAT

We acknowledge that it may not be immediately clear that our algorithm is not a CTDE algorithm and thank the reviewer for pointing this out; by no means did we intend to obscure this fact. We thought it was clear that by building on the MAT and Sable line of work, the reader would be aware that Oryx is a centralised approach. In hindsight, this assumption was misguided.

Our first iteration of the Oryx algorithm was built by extending the MAT network architecture (rather than the Sable one), but ultimately, we settled on the variant with the Sable network architecture because of its superior performance and scaling properties. But since Sable had already been shown to be better than MAT (at least in the online case), we thought it was not worth repeating here. We therefore only reported our performance compared to algorithms from the existing offline literature, and to date, all but MADIFF are CTDE algorithms.

Having now considered your review, we see that this was a mistake and can easily lead to confusion. To address this, we will add in our experiments comparing Oryx to the variant which used the MAT network architecture, and be clear about their difference to CTDE algorithms. Below are the results across all 43 SMAC datasets and all 6 RWARE datasets. On each dataset, we ran each algorithm over 10 random seeds.

However, converting MAT, an online and on-policy algorithm, to an offline algorithm was itself a non-trivial task, which involved

• Replacing the learning from a queue of online samples with learning from sampled batches of experience from a replay buffer (dataset).
• Replacing the PPO loss (which only works with online samples) with a loss function that works on off-policy samples.
• Modifying the MAT network architecture to accommodate the new loss.

In particular, we replaced the PPO-style loss in MAT with our novel Autoregressive version of the ICQ loss that we derived (and also use in Oryx). The MAT network modifications required for this were similar to the ones outlined in the Oryx paper.

Aggregated Results: Oryx vs. MAT

To compare the MAT variant and Oryx in a robust way, we applied the proposed statistical methodology from MARL-eval [1]. Namely, we provide 4 different aggregated metrics: the mean, median, interquartile mean, and optimality gap, with stratified bootstrap confidence intervals [in square braces]. To facilitate aggregation, the results for each dataset were normalised against the highest performance on that dataset before computing the aggregated performance metrics. From the tables below, it is clear to see that Oryx substantially outperforms the offline MAT+ICQ on both SMAC and RWARE datasets, with better scores across all metrics (for optimality gap, a lower score is better). We will add these results, along with a clarification of the difference between these methods and the CTDE methods, to our revised paper.

SMAC

RWARE

Raw Tabulated Results

We will include the raw tabulated results for these two baselines alongside the others in the appendix of the camera-ready version of the paper. We could not include them here because the tables used too many characters, given the character limit. From the results, one can see that MAT+ICQ is also a very strong algorithm when compared to the CTDE algorithms from the literature. Which, we agree, is to be expected given their Autoregressive action selection at evaluation time. To us, this result highlights the benefit of using this class of algorithms whenever possible, i.e., when fully decentralised execution is not required (more on this towards the end of our rebuttal). Our work is also the first to demonstrate the applicability of autoregressive action selection in the offline MARL setting.

On the Significance of our Contribution

We believe that developing Oryx involved several intricate steps and novel insights that extend beyond a simple integration of existing ideas. We would like to highlight the following key aspects:

• Novel Sequential Autoregressive ICQ Loss: We developed and formally derived a novel sequential autoregressive version of the ICQ loss, which we believe is a significant theoretical contribution.
• Offline Training of Online Algorithms: A core challenge we addressed was modifying online (on-policy) algorithms like Sable and MAT to enable their effective offline training from off-policy samples from a replay buffer (dataset). This adaptation presented non-trivial technical hurdles.
• Network Architecture Modifications: We carefully considered and implemented modifications to the original Sable (and MAT) network architecture to incorporate the dual Q-value and Policy distribution heads required for the ICQ loss.
• Temporal Memory Handling: Appropriately handling the resetting of temporal memory in the retention-based memory of the Sable backbone, particularly when training from offline sequences, required careful attention to detail.
• Empirical Performance: Our work demonstrates substantially superior performance, not just marginal, across multiple benchmarks (SMAC, RWARE, Multi-Agent MuJoCo) and new challenging datasets featuring up to 50 agents. Having addressed the issue raised by the reviewer regarding comparison to MAT, we are confident our robust empirical investigation differentiates our work within the existing literature.
• Open-Sourcing and Reproducibility: To further contribute to the field and facilitate future research, we have already open-sourced both our implementation and the novel datasets, along with clear instructions for reproducing our experiments. We are confident that this will pave the way for further innovation in models derived from Oryx, and we are excited about the potential for others to build upon this direction.

Moreover, we believe that classifying our contribution as merely "combining existing ideas" might overlook a common aspect of scientific advancement. Many significant breakthroughs often emerge from novel syntheses of existing concepts, and Oryx exemplifies this by integrating diverse ideas from the MARL field to achieve a result that is greater than the sum of its individual components. While an incremental improvement on a narrow set of benchmarks might warrant such a conclusion, our work demonstrates a significant performance gain over existing methods, supported by extensive experimentation. Furthermore, we have expanded the scope of research by developing and sharing a challenging set of benchmark datasets with up to 50 agents. Having addressed the experimental oversight highlighted by the reviewer, we are confident that our method represents a novel and significant contribution.

Final Note on CTDE vs Oryx (and MAT/Sable)

Indeed, our algorithm is not a CTDE algorithm; it belongs to the same class of algorithms as MAT and Sable. Thus, it is not directly applicable in settings where decentralised execution is required (although it's always possible to apply a decentralised policy distillation step at the end of training, if required [2,3]). Having said that, algorithms like MAT, Sable, and Oryx are nonetheless still practically very valuable as they lie somewhere in between the two extremes of fully independent actors (which scales well but may converge on a sub-optimal joint policy) and fully centralised learners (which scale poorly but have better guarantees to converge on an optimal policy in the limit) [4]. Thus, in problems with a large joint action space, which can naturally be factorised, Oryx (as well as MAT and Sable) provides a solution that trades off scalability on the one hand and quality of the joint policy on the other. For this reason, Oryx remains a highly relevant algorithm for many practical industrial applications.

In conclusion, we once again thank the reviewer for their valuable input on our paper. Having addressed the main concern we feel the work is significantly strengthened. Should the reviewer have any follow-up questions, please do not hesitate to ask. We would be happy to engage during the author-reviewer discussion period.

[1] Gorsane, Rihab, et al. “Towards a Standardised Performance Evaluation Protocol for Cooperative MARL.” NeurIPS ‘22

[2] Zhu, Zhengbang, et al. “MADiff: Offline Multi-agent Learning with Diffusion Models,” NeurIPS ‘24

[3] Li, Yueheng, et al. “Multi-Agent Guided Policy Optimization.” Arxiv 2025

[4] de Kock, Ruan, et al. “Is an Exponentially Growing Action Space Really That Bad? Validating a Core Assumption for using Multi-Agent RL.” AAMAS ‘25

Dear reviewer,

We believe that you have misinterpreted the RWARE results. Our results show that Oryx outperforms MAT+ICQ across all 4 metrics with no overlap in confidence interval [1]. Therefore, we can conclude with statistical significance that Oryx is superior to MAT+ICQ on RWARE.

With the additional experimental evidence showing that Oryx is superior to MAT+ICQ, which are both offline CTCE methods, we have addressed your only stated concern with the methodology of our paper. Therefore, we feel strongly that your rating of 2 is no longer justified. For instance, a rating of 2 implies that our paper has “technical flaws, weak evaluation, inadequate reproducibility, and incompletely addressed ethical considerations.”

• Technical flaws: None have been raised.
• Weak evaluation: We have resolved the only weakness mentioned by your review. Furthermore, the breadth of benchmarks and baselines is far greater than the widely accepted standard in the field [2]. We tested on over 88 datasets, from 5 different environments. We covered all the recent SOTA offline algorithms as well as demonstrated superior performance to a robust implementation of a novel baseline MAT+ICQ.
• Inadequate reproducibility: We publicly release all of our code and datasets with clear instructions on how to reproduce our results. This is once again a substantial improvement over the widely accepted state of the field [2].
• Incompletely addressed ethical considerations: clearly does not apply.

Finally, with regards to your statement that “(1) there is a fundamental limitation of using ICQ due to the IGM assumption used for value decomposition”, we are not entirely sure what you mean, but if you are referring to the QMIX component in the original MAICQ algorithm, then we will answer as follows. The variant of the ICQ loss that we derive in this paper builds on the single-agent version of the ICQ loss that is introduced in the first half of the paper [3], not the MAICQ loss, which includes QMIX (value decomposition). Please make sure to see our proof in the appendix of the paper for all the details on the derivation. Thus, our autoregressive version of the ICQ loss should be seen as another way to extend the original ICQ loss to the multi-agent setting, entirely separate to the value decomposition variant (MAICQ), which relies on the IGM principle.

If, however, we misunderstood what you are trying to say, please do elaborate so that we can accurately provide a response.

[1] Agarwal, Rishabh, et al. “Deep Reinforcement Learning at the Edge of the Statistical Precipice”, NeruIPS ‘21

[2] Claude Formanek, et al. “Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation”, NeurIPS ‘24

[3] Yang, Yiqin, et al. “Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning”, NeurIPS ‘21

We thank the reviewer for their constructive comments on our paper. We appreciate the reviewer highlighting the novelty of our algorithm and acknowledging that the new benchmark datasets we provide are valuable contributions. Concerning the author's questions, we now address each in detail:

• How does inference memory scale with agent count (e.g., in 50-agents)?
  • The Sable [1] paper showed the scaling properties of the network for agents up to 1000. The memory scaling properties are significantly better than MAT (which is quadratic in the number of agents) and competitive with algorithms that use independent actors. This suggests that Oryx will scale to even more than 50 agents. The main bottleneck we encountered with scaling beyond 50 agents was handling the very large datasets. Generating datasets for systems with so many agents became quite an engineering challenge to handle. We stopped at 50 for this paper because we felt it struck a good balance between pushing the frontier of research while still being manageable for other future researchers to use without too much engineering overhead. In the future, building a better data pipeline should allow us to test Oryx at even greater scale.
• Retention blocks and sequential decoding likely increase training costs vs. non-sequential methods (not quantified).
  • Yes, it is true that sequential decoding increases computational complexity in the number of agents. In MAT, the complexity scales quadratically in the number of agents, which can be prohibitive for large numbers of agents. Sable, on the other hand, has far superior scaling properties thanks to the retentive network architecture. While not as good as non-sequential (independent) action selection methods, it is very competitive up to 1000 agents. Sable (and by extension Oryx) therefore strikes a good balance between superior performance and competitive scaling. It is for this reason that we chose to use the Sable network as a starting point for Oryx rather than the MAT network.
• Can sequential updates handle highly heterogeneous agents?
  • Yes, it can! Although how to achieve this depends on what exactly you mean by heterogeneous agents. If you mean the agents assume different roles in the environment, but their observation and action spaces are the same, then Oryx handles this gracefully right out of the box. If, on the other hand, agents have different-sized observation spaces and action spaces, you can still apply Oryx, but first need to pad the spaces to the same size. This is relatively easy to do and has worked well when we have tried it. Moreover, we are confident about Oryx’s ability to handle heterogeneous agents because it builds on the foundations set by the Heterogeneous Agents RL [3] line of work, which first introduced the sequential updating scheme that was then extended to Sequence Modelling by MAT [2], and finally Sable.

Once again, we thank the reviewer for their comments. Should the reviewer have any follow-up questions, please do not hesitate to ask. We would be happy to engage during the author-reviewer discussion period.

[1] Mahjoub, Omayma, et al. “Sable: a Performant, Efficient and Scalable Sequence Model for MARL.” ICML ‘25

[2] Wen, Muning, et al. “Multi-Agent Reinforcement Learning is a Sequence Modeling Problem.” NeurIPS ‘22

[3] Zhong,Yifan, et al. “Heterogeneous-agent reinforcement learning.” Journal of Machine Learning Research, January ‘24

We thank the reviewer for their positive review of our work and their constructive questions. By addressing them, we feel we can substantially improve the clarity of our paper's message. Here we address each of your questions in turn:

• While sequential updates may help coordination, do they scale well in environments with hundreds of agents or continuous control tasks?
  • Similar to our response to the reviewer vYPS, since Oryx uses a modified version of the Sable network architecture, it inherits many of its scaling properties. In particular, the Sable paper showed superior scaling trends to MAT [1] and competitive trends compared to fully independent actors for up to 1000 agents. This suggests that Oryx will scale to even more than 50 agents. The main bottleneck we encountered with scaling beyond 50 agents was handling the very large datasets. Generating datasets for systems with so many agents became quite an engineering challenge to handle. We stopped at 50 for this paper because we felt it struck a good balance between pushing the frontier of research while still being manageable for other future researchers to use without too much engineering overhead. In the future, building a better data pipeline should allow us to test Oryx at even greater scale.
  • Oryx works very well in continuous control settings. All of the Multi-Agent MuJoCo tasks are continuous control problems, and Oryx significantly outperformed other algorithms on most of those datasets.
• In the sequential update scheme, how sensitive is the overall policy quality to the order in which agents are updated?
  • In Oryx, we randomly shuffle the order of agent updates at each learning step, similar to MAT [1] and HAPPO [2]. We agree that if you had some informed prior on which order of agent updates would be best, this may help speed up learning. However, we did not experiment with this because we wanted to keep our method as general as possible. And developing a method to automatically learn the best order of updates is out of the scope of this work, but it is an interesting direction for future work.
• In sequential updates, each agent optimizes assuming the policies of the other agents are fixed. Does this introduce error accumulation across the agent chain?
  • Indeed, the accumulation of error as agent policies are updated one after another is a central challenge in offline MARL. However, since Oryx uses an autoregressive policy scheme (like MAT [1]) where agents condition on the actions of agents that came before them, the error will be less than fully independent policies. This fact gets at the central benefit of using a model like Oryx.
• In your implementation of Implicit Constraint Q-learning for each agent, how is the constraint enforced during policy updates?
  • The ICQ constraint is enforced in two ways. First, during the critical learning step (Q-learning), one strictly only uses the actions observed in the dataset for computing the target value for the next state. This avoids extrapolation error due to out-of-distribution actions in the value function. Then, at the policy learning step, the policy distribution is updated with respect to in-distribution actions only (i.e. actions in the dataset), which implicitly avoids out-of-distribution actions. Together, these constraints mitigate the accumulation of extrapolation error. More details on the ICQ loss can be found in the original paper [3]. Central to our contribution is the sequential and autoregressive variant of the ICQ loss.

On the Ablation Study

Finally, with regard to the reviewer's comment about the lack of ablation studies, we specifically designed the TMAZE experiments to isolate which components of Oryx were important. Our main takeaway from the TMAZE ablation study was that removing any one component from Oryx (autoregressive actions, memory, or ICQ) resulted in failure on the task. Highlighting that only by combining all three ingredients together did the algorithm succeed. We conducted the ablation in our custom-designed TMAZE environment because we felt its simplicity made the results more interpretable. If, however, the reviewer feels the ablation study would be strengthened by conducting it over a more complex, but less interpretable, environment (e.g. RWARE), we would happily consider adding that in for the camera-ready version.

In conclusion, thank you once again for your very constructive questions. They have helped us sharpen our messaging in the paper. In particular, we will add a discussion on scaling to the revised version, along with more detail on how exactly the autoregressive policy updates work in practice. Should the reviewer have any follow-up questions, please do not hesitate to ask. We would be happy to engage during the author-reviewer discussion period.

[1] Wen, Muning, et al. “Multi-Agent Reinforcement Learning is a Sequence Modeling Problem.” NeurIPS ‘22

[2] Zhong,Yifan, et al. “Heterogeneous-agent reinforcement learning.” Journal of Machine Learning Research, January ‘24

[3] Yang, Yiqin, et al. “Believe What You See: Implicit Constraint Approach for Offline Multi-Agent Reinforcement Learning” NeurIPS ‘21