Agent-Centric Actor-Critic for Asynchronous Multi-Agent Reinforcement Learning

We appreciate your insightful comments and would like to clarify our contributions and address concerns as follows:

[Q1] Parameter Comparison

We agree that comparing parameters is important, so we compared the total number of parameters between ACAC and Mac-IAICC. The results are as follows:

• Overcooked/Rand: ACAC (320k) > Mac-IAICC (241k)
• Overcooked-Large/Rand: ACAC (874k) < Mac-IAICC (963k)

Crucially, ACAC significantly outperforms Mac-IAICC in Overcooked-Large despite having fewer parameters. This strongly indicates ACAC's performance gains stem from its architectural design (more effective processing of asynchronous information), not just model capacity.

 

[Q2] Use of RNN in baselines

Yes, all macro baselines utilize RNNs as ACAC.

 

[W1] Practicality of Pre-defined Macro-Actions

Our framework doesn't assume uninterrupted macro-actions. In the environments used (e.g., Overcooked), actions can be interrupted by other agents (e.g., blocking). When this occurs, the action terminates early, allowing the agent to immediately choose a new macro-action based on the situation. This provides reactive adaptation to dynamic interactions. While the environment handles these interruptions, our paper focuses on ACAC's ability to handle the resulting asynchrony in decision points for effective learning in the CTDE framework.

 

[W2] Padding Issue & Critic Novelty

We appreciate the perspective on the padding problem. While padding is simple, our paper argues and results show its impact is significant in asynchronous MARL, especially complex tasks. It can cause misleading temporal information(padding obscures the actual time elapsed between an agent's valid observations, hiding crucial duration information), inaccurate history abstraction(reusing past observations via padding makes it hard to distinguish new vs repeated stale information, leading to inaccurate joint history representations), and hinder credit assignment. Our experiments (Figs 7, 8) consistently show ACAC (no padding) outperforms padding-based methods, especially in complex settings (Overcooked-Large), suggesting addressing padding is crucial, not minor. This motivated ACAC's design to handle asynchronous inputs directly.

 

[W3] Component Contribution Verification

We agree ablation studies are crucial, so we have performed ablations for time embedding ("No TE") and self-attention ("No SA", using MLP) across Overcooked and Overcooked-Rand environments. Since the complex Overcooked-Rand-B map showed the clearest and most significant performance differences between configurations, clearly highlighting component impacts, we focus on these results here for brevity. Full results will be included in the revised manuscript. We report the average final performances and standard errors over five random seeds.

Table 3. Ablation study on time embedding (TE) and self-attention (SA).

These results show that removing either the TE or SA component severely degrades both stability and the final performance, confirming that each component is vital to ACAC's effectiveness.

Regarding the PPO/GAE ablation, directly applying PPO to Mac-IAICC (N critics) is non-trivial. However, our ACAC vs. ACAC-Vanilla comparison (Sec 5.2, Fig 7) isolates the PPO objective's impact. They share the same architecture; only the optimization differs (PPO vs. simpler policy gradient). ACAC's superior performance demonstrates the benefit of our PPO-based objective.

 

[Experimental Designs 3] Validity of Ablation Comparison

This experiment aimed to isolate the negative impact of processing padded/duplicated information, which is difficult within standard padding methods. ACAC-Duplicate uses the ACAC architecture, including time embedding and per-agent history encoders, but adopts the padding method's history update rule (update on any agent's new observation). This creates a valid intermediate comparison, effectively showing the performance degradation caused by processing duplicated information within our agent-centric framework.

 

[Experimental Designs 4] Lack of Implementation Details

Key architectural details are in Sec 3/Fig 4, with hyperparameters in Appendix F/Table 1. Please feel free to ask if further hyperparameter details are needed. We will release the full source code publicly for reproducibility.

We are grateful for your thoughtful remarks and would like to provide clarification on our contributions and address the raised concerns.

 

[Q1,W1] Analysis of Modified GAE

Similar to the original GAE, our proposed method does not guarantee convergence in general. However, just as the original GAE converges when $\lambda$ equals 0 or 1, our proposed GAE also inherently converges in these specific cases. The rationale behind using macro-action-based $\lambda$-discounting in GAE within ACAC is detailed in our response to reviewer Cf7Q (Analysis of Modified GAE); please refer to our comments there for further clarification.

 

[Q2] Information Loss Comparison

Given an episode obtained using macro-actions, both ACAC and padding-based methods theoretically acquire the same amount of information. However, the padding-based approach continuously uses information from observations that are not actually collected, making it difficult to distinguish between cases where no new information is obtained and those where identical information from a previous observation is repeated. This can result in information distortion. To eliminate such distortion without losing information, ACAC employs an agent-centric encoder that utilizes only each agent's available information for history encoding. This structure allows ACAC to more accurately estimate joint histories compared to padding-based methods, thereby achieving superior performance.

 

[Q3,W3] Evaluations on the SMAC

We agree with your point that evaluating our approach in environments like SMAC could enhance the persuasiveness of our results. Indeed, several studies have explored hierarchical approaches in SMAC; however, most of them have focused on synchronous scenarios where macro-actions have a fixed, identical duration for all agents. Our work, in contrast, specifically addresses scenarios involving asynchronous macro-actions, where each macro-action has varying durations. Consequently, conducting performance comparisons in synchronous SMAC environments would not meaningfully reflect the contributions of our method.

To compare ACAC and the baseline methods effectively in SMAC under asynchronous settings, we would need to explicitly define macro-actions with varying durations. Unfortunately, it is not feasible to develop and test such an asynchronous version of SMAC within the limited time frame available for this rebuttal. Nevertheless, we are actively working on developing an asynchronous variant of SMAC. We strongly anticipate that ACAC will exhibit superior performance over existing baselines, such as Mac-IAICC and Mac-NIACC, once evaluated in this asynchronous SMAC environment.

 

[Q4] Influence by Opponent Policies

Thank you very much for suggesting an evaluation in environments like SMAC-Hard, where multiple opponent policies are mixed. Similar to our earlier explanation regarding SMAC, we believe conducting evaluations in such environments would be meaningful only within an asynchronous version of SMAC. This, however, requires developing a dedicated asynchronous SMAC environment. Additionally, effectively responding to various opponent strategies, as seen in SMAC-Hard, would necessitate developing a module explicitly designed for opponent strategy inference. While this is beyond the current scope of our research, which primarily targets asynchronous MARL scenarios, we anticipate that ACAC's per-agent history encoder could be extended or enhanced to perform opponent strategy inference. This capability would potentially enable the model to adapt effectively to different opponent strategies, representing an exciting and promising direction for future research.

 

[W2] Sensitivity Analysis on Hyperparameter

We agree your suggestion, so we have performed sensitivity analysis on hyperparameter, clipping ratio and GAE $\lambda$(micro-level vs macro-level $\lambda$-discounting), across Overcooked and Overcooked-Rand environments. Since the complex Overcooked-Rand-B map showed the clearest and most significant performance differences between configurations, clearly highlighting component impacts, we focus on these results here for brevity. Full results will be included in the revised manuscript. We report the average final performances and standard errors over five random seeds.

Table 1. Sensitivity analysis on hyperparameter: clipping ratio (ε).

Table 2. GAE $\lambda$ comparison.

The results shows that performance is robust to clipping ratio variations; ACAC's macro-action based GAE discounting outperforms the original timestep-based discounting.

We are grateful for your feedback and would like to offer a detailed explanation of our contributions while addressing the concerns raised.

[W1] Analysis of Modified GAE

It is known that the original GAE does not guarantee convergence in general. However, specific boundary cases are clearly defined: when $\lambda$=0, GAE reduces to the TD-error; when $\lambda$=1, it simplifies precisely to the empirical return. Similarly, the proposed GAE in our paper maintains this important property—when $\lambda$=1, it equals the empirical return, and when $\lambda$=0, it becomes the multi-step TD-error between consecutive decision points ($\delta_{l(k)}$ in our formulation, where $l(k)$ is the $k$-th timestep a new observation becomes available for any agent).

In MacDec-POMDP settings featuring temporal abstraction via macro-actions, there can be two choices for designing the advantage function based on how $\lambda$-discounting is applied:

• 1. Applying $\lambda$-discounting at the micro-timestep level: This approach discounts future TD errors based on the number of primitive timesteps elapsed.
• 2. Applying $\lambda$-discounting at the macro-timestep level: This approach discounts future TD errors based on the number of macro-action decision steps taken.
• Let's illustrate using an example where macro-observations (and value estimates $V_t$) are obtained at timesteps 0, 2, 5 and 6:
  • Rewards: $r_0, r_1, r_2, r_3, r_4, r_5, r_6, \dots$
  • Values: $V_0, -, V_2, -, -, V_5, V_6, \dots$
  • Multi-step TD errors between decision points:
    • $\delta_0 = r_0 + \gamma r_1 + \gamma^2 V_2 - V_0$
    • $\delta_2 = r_2 + \gamma r_3 + \gamma^2 r_4 + \gamma^3 V_5 - V_2$
    • $\delta_5 = r_5 + \gamma V_6 - V_5$
  • Below are the resulting advantage calculations at t=0 for each approach:
    • (1) Micro-level $\lambda$-discounting : $A^{\lambda, \text{micro}}_0 := \delta_0 + (\lambda^{2} \gamma^{2}) \delta_2 + (\lambda^{5} \gamma^{5}) \delta_5 + \dots$
    • (2) Macro-level $\lambda$-discounting : $A^{\lambda, \text{macro}}_0 := \delta_0 + (\lambda \gamma^{2}) \delta_2 + (\lambda^{2} \gamma^{5}) \delta_5 + \dots$ We adopted the second approach (macro-level $\lambda$-discounting) for ACAC, as it emphasize both the significance of future rewards and the critical decision points associated with macro-actions. As in Table 2 of response for RGMo (Sensitivity Analysis on Hyperparameter), this approach effectively handles the variable intervals in our asynchronous setting and contributes to the strong performance observed.

 

[W2] Real-world Applicability

Thank you for raising this important point regarding the evaluation environments. We agree that verifying applicability beyond grid-world tasks is crucial. While settings requiring asynchronous coordination are common in the real world, the field of Asynchronous MARL, especially involving macro-actions, is still emerging. Consequently, there is currently a limited set of established benchmarks available, most of which are grid-based environments like BoxPushing and Overcooked. We aimed for a comprehensive evaluation within the current scope of the field and thus tested our ACAC method on these known and publicly available benchmarks specifically designed for macro-action-based asynchronous MARL. We recognize the importance of demonstrating applicability in more realistic scenarios. Extending our work to real-world problems is a key direction for our future research.

Your valuable comments are much appreciated. In response, we aim to clarify our contributions and address the points of concern.

[Q1] Qualitative Analysis Request

Thank you for the insightful suggestion to correlate attention scores with behavior. This is an excellent direction for future work to gain deeper understanding.

 

[Q2] Hyperparameter Sensitivity

We conducted experiments to analyze hyperparameter sensitivity. Due to space limitations in this response, we have detailed these results in our separate response provided to reviewer RGMo (Sensitivity Analysis on Hyperparameter). We kindly request your understanding and refer you to that specific comment for the detailed results.

 

[Q3,W3] Continuous Action Space

We believe ACAC is structurally suitable for continuous actions with standard actor-critic modifications. The main challenge is the current lack of asynchronous MARL benchmarks with continuous action spaces and defined macro-actions. Defining these requires careful effort, preventing results within the rebuttal period, but it remains important future work.

 

[Q4,W1] Analysis of Modified GAE

Similar to the original GAE, our proposed method does not guarantee convergence in general. However, just as the original GAE converges when $\lambda$ equals 0 or 1, our proposed GAE also inherently converges in these specific cases. The rationale behind using macro-action-based $\lambda$ discounting in GAE within ACAC is detailed in our response to reviewer Cf7Q (Analysis of Modified GAE); please refer to our comments there for further clarification.

 

[W2] Computational Complexity of ACAC Structure

Defining N=#agents, K=#observations per agent, Z=obs dim, H=hidden dim:

• Actor: Complexity is nearly identical, aside from ACAC's minor time embedding overhead.
• Critic:
  • History Encoding
    • Mac-IAICC: Encodes the concatenated joint history (size NZ) whenever any agent gets a new observation (up to NK times total). This leads to a worst-case complexity of O(N²KZ).
    • ACAC: Processes each agent's observation (size Z) individually through its dedicated encoder. Across all agents and observations (NK total), the complexity is O(NKZ). This scales better than Mac-IAICC's encoding by a factor of N.
    • Thus, ACAC's history encoding scales linearly with N (O(NKZ)), offering better scalability than the quadratic O(N²KZ) of joint encoding, especially for larger N.
  • Value Estimation
    • Mac-IAICC: Typically uses an MLP processing aggregated hidden features (size NH). Computed up to NK times, giving O(N²KH) total complexity.
    • ACAC: Employs an attention mechanism over N agent representations (size H), requiring O(N²H) computation per step where an observation arrives. In the worst case (NK steps), this totals O(N³KH) complexity.
    • While a simpler MLP aggregation (as in Mac-IAICC) offers one potential alternative for scalability, potentially trading off some performance, another promising direction for future work is to explore the integration of efficient attention mechanisms [1] that reduce complexity (e.g., to linear O(NH) per step) while aiming to retain the expressive capacity for modeling inter-agent dependencies.
• Practical Runtime: Despite theoretical differences, the observed runtimes for ACAC and Mac-IAICC were similar in our experiments (N=3, 6). This suggests ACAC's more efficient history encoding helps offset the computational cost of the attention module in practice for these agent populations.

[1] Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020). Linformer: Self-Attention with Linear Complexity. arXiv preprint arXiv:2006.04768.