In-Context Fully Decentralized Cooperative Multi-Agent Reinforcement Learning

1. Response to Weakness 1 and Question 1.

(1) Intuition. Our goal is to construct a representation of the agents' joint policy. Intuitively, joint policies that yield similar expected returns tend to have similar value functions, and their representations should be located close to one another in the representation space. Contexts based on episodic returns naturally satisfy this requirement, and the context-based value function produces similar outputs for similar joint policies. Specifically, such episodic return-based contexts cluster the agents' joint policies according to their returns (or associated value functions), resulting in a one-to-many mapping from contexts to joint policies.

(2) Empirical Evidence. As shown in Tab. 1 of the main text, for the matrix game, we define 4 episodic return intervals $[-12, 0), [0, 6), [6, 8), [8, +\infty)$ based on prior knowledge of the task. For agent $1$, the contexts are associated with distinct return ranges corresponding to different policies of another agent $2$. The context-based value function $Q^{1}$ accurately approximates the rewards of all joint actions (i.e., all deterministic joint policies learned by value-based methods). The above analysis also applies to agent $2$. These results empirically demonstrate the effectiveness of episodic return-based contexts in representing the agents' joint policies.

(3) Theoretical Consideration. The theoretical foundations for the correspondence between episodic return-based contexts and the agents' joint policies have not been explored in this work. We leave this theoretical analysis for future investigation.

2. Response to Weakness 2 and Question 2.

Case (1) considers the scenario where the context changes between time steps, i.e., other agents update/change their joint policy at every (or every few) time steps. In this case, the contexts based on episodic returns fail to uniquely represent the agents' changing joint policy, leading RAC to struggle with the resulting severe non-stationarity. The underlying reason is as follows: when other agents change their joint policy within episodes, the generation probability of current agent $i$'s local trajectory $\tau^{i}=(s_{0},a_{0}^{i},r_{0},s_{1},\ldots,s_{H-1},a_{H-1}^{i},r_{H-1},s_{H})$ is determined by multiple joint policies of other agents. Consequently, the corresponding episodic return $R(\tau^{i})=\sum_{t=0}^{H-1}r_{t}$ is also determined by these multiple joint policies of other agents, and thus fails to uniquely correspond to any single one. We will add this discuss in the revision to enhance clarity.

3. Response to Weakness 3 and Question 3.

Thanks for your valuable suggestions. In terms of the KL supervision weight $\beta$, a straightforward approach to ensure comprehensive coverage of the return space is to equip RAC with efficient exploration techniques. As for the number of context bins $m$, as discussed in the Response to Question 2 of Reviewer KFLW, adaptive context modeling may be achieved by dynamically clustering the received returns. We plan to explore both directions in the next stage of our work.

We will add this discussion in the revision to encourage further research on in-context fully decentralized cooperative MARL.

4. Response to Weakness 4 and Question 4.

Thanks for your valuable advice. At this stage, we primarily focus on evaluating the impact of the KL term. We consider two specific cases: (1) When the KL term is fully disabled, the separately learned $Q_{\operatorname{S}}^{i}$ degenerates into standard IQL, resulting in performance equivalent to that of IQL. (2) When the learning of $Q_{\operatorname{S}}^{i}$ is removed, RAC is constrained to directly use $\phi^{i}$ for action selection, and we assess its performance in the matrix game, predator and prey, and the SMAC map 5m_vs_6m.

As shown in the table above, RAC_w_$\phi^{i}$ achieves superior performance in the matrix game and predator and prey tasks. This highlights the effectiveness of the individual optimistic marginal value $\phi^{i}$ in RAC for mitigating relative overgeneralization. However, RAC_w_$\phi^{i}$ fails to learn effective policies on 5m_vs_6m. We attribute this failure to insufficient coverage of the return space, hindering the construction of meaningful contexts based on episodic returns. A detailed explanation is provided in the Response to Question 3 of Reviewer KFLW. For complex tasks such as SMAC, integrating RAC with efficient exploration techniques remains a promising direction for future work. We will also make every effort to complete additional ablation studies on alternative action selection strategies and soft context clustering, and incorporate them into the revised manuscript.

5. Response to Weakness 5 and Question 5.

Thanks for your valuable advice. We will adopt abbreviations in the revision to enhance readability.

Dear Reviewer 7ZBz,

During the rebuttal period, we have provided detailed responses to your concerns regarding:

(1) Clarifications on context separability in terms of the discriminability among joint policies;

(2) Discussion on the limitations of our method in Case (1), where contexts change between time steps;

(3) Further extensions involving adaptive context modeling and simplified hyper-parameter tuning;

(4) Enhancements to the ablation studies;

(5) The use of abbreviations to improve readability.

We sincerely hope that the current responses can resolve your concerns. Looking forward to your response if you have any further concerns about this work.

Best regards,

The Authors

Dear Reviewer 7ZBz,

We sincerely appreciate the time and effort you have dedicated to providing us with your valuable feedback. We believe that we have addressed all of your concerns. Should you have any further questions or suggestions, please do not hesitate to share them, and we would be glad to address them promptly.

Best regards,

The authors

1. Response to Question 1.

Our goal is to construct a representation of the agents' joint policy. For joint policies that yield similar expected returns, their corresponding value functions are also similar, and thus their learned representations should lie close to each other in the representation space. Contexts based on episodic returns naturally satisfy this requirement, and the context-based value function produces similar outputs for similar joint policies. Specifically, such contexts cluster the agents' joint policies according to their returns (or associated value functions), resulting in a one-to-many mapping from contexts to joint policies, rather than a one-to-one correspondence.

As noted by the reviewer, a potential drawback of representing the agents' joint policy using episodic returns is the high variance caused by the stochastic policy and the stochastic environment. A promising approach to this problem is to perform multiple Monte Carlo rollouts to approximate the expected return of a policy more reliably. We will highlight this limitation in the revised manuscript. Further extensions about reducing the high variance and exploring alternative representations are left for future work.

2. Response to Question 2.

Thank you for the careful reading. We will correct this typo in the revision.

3. Response to Question 3.

The computational complexity of our approach contains two components:

(a) In terms of the context-based value function $Q^{i}(s,a^{i},c)$, its inputs include both states and contexts, and the outputs correspond to the value estimates of all possible local actions of agent $i$. Its input space scales linearly with the dimension (number) of contexts, and deep neural networks can generalize well across such space. The number of its output is only $|A^{i}|$, where $A^{i}$ is agent $i$'s local action space.

(b) In terms of the individual optimistic marginal value $\phi^{i}(s,a^{i})=\max_{c\in\mathcal{C}}Q^{i}(s,a^{i},c)$, when we set the number of contexts $m$ too large, the enumeration of all possible contexts typically leads to high computational complexity. A promising approach is to use sampling-based derivation-free heuristic search method, such as CEM [1], to find approximate maxima. In detail, we iteratively draw a batch of $N$ random context samples from a candidate distribution $D_{k}$, e.g., a Gaussian, at each iteration $k$. The best $M < N$ samples (with the highest context-based value estimates) are then used to fit a new Gaussian distribution $D_{k+1}$, and this process repeats $K$ times. Such a sampling-based approximation can efficiently reduce the complexity caused by exhaustive search over a enormous context space, making the approach more tractable when $m$ is too large.

4. Response to Question 4.

In this work, the learning processes of $Q_{\operatorname{S}}^{i}$ and $Q^{i}$ form a positive feedback loop: $Q_{\operatorname{S}}^{i}$ generates informative transitions to ensure effective updates of $Q^{i}$, while $Q^{i}$, in turn, guides $Q_{\operatorname{S}}^{i}$ in selecting per-agent local cooperative actions to produce more informative transitions. This mutual reinforcement facilitates efficient, fully decentralized cooperative policy learning. However, such mutual reinforcement heavily relies on appropriately weighting the supervision loss term. As acknowledged in Sec 6 (Conclusion: Limitation and Future work), RAC necessitates a thorough coverage of the entire return space to efficiently update the context-based value function, while separately learning $Q_{\operatorname{S}}^{i}$ and tuning the supervision loss term may perform poorly in more complex tasks. To address this limitation, integrating RAC with efficient exploration techniques is a promising direction, which we leave for future work.

Regarding the context-based value function, it is trained based on the transitions generated by the separately learned $Q_{\operatorname{S}}^{i}$. More explanations about the learning complexity of the context-based value function are detailed in the Response to Question 3.

5. Response to Question 5.

Thanks for your valuable suggestions. We evaluate QMIX and VDN on the matrix game, predator and prey, and five SMAC maps, reporting the median asymptotic performance along with the standard error across five random seeds. The results, presented in the table above, show that RAC significant outperforms both QMIX and VDN on the matrix game and predator and prey, where QMIX and VDN suffer from sub-optimal policies due to their representational limitations regarding the factored global action value function. On the five SMAC maps, RAC exhibits superior performance over other fully decentralized baselines but underperforms relative to QMIX and VDN. We attribute this to RAC's limited ability to handle partial observability and to adequately cover the entire return space. Future work will focus on enhancing RAC by incorporating belief state representations and integrating more efficient exploration techniques. We will add the comparison results in the revision for a better presentation.

6. Response to Question 6.

For a cooperative game, relative overgeneralization refers to the scenario where sub-optimal equilibrium is preferred over the optimal ones. In the SMAC benchmark, due to challenges such as partial observability and algorithmic limitations (e.g., representational limitations in value decomposition methods or over-estimation in Hysteretic Q-learning), baseline methods often converge to sub-optimal policies on certain maps, as evidenced by their poor performance. In this context, relative overgeneralization steams from both environmental and algorithmic factors, rather than solely from the reward design. We will add this discussion into the revised manuscript to improve clarity.

Reference

[1] De Boer P T, Kroese D P, Mannor S, et al. A tutorial on the cross-entropy method[J]. Annals of operations research, 2005, 134(1): 19-67.

1. Response to Question 1.

The non-stationarity arises from the evolving policies $\pi^{-i}$ of other agents $-i$. From the perspective of each agent $i$, its local task dynamics is determined by $\pi^{-i}$, as stated in Eq. (1) of the main text. In this work, we associate each context $c$ with a unique $\pi^{-i}$, and accordingly formalize the task, as locally perceived by each agent $i$, as a contextual Markov Decision Process (CMDP). A CMDP comprises a family of MDPs that share the same state and local action spaces but differ in their state transition and reward functions, which are determined by the associated contexts. Accordingly, the non-stationarity induced by changes in other agents $-i$' policies $\pi^{-i}$ can be attributed to context switching.

Furthermore, the influence of other agents $-i$' policies $\pi^{-i}$ on a given agent $i$ is directly reflected in the rewards it receives. In this work, we focus on case (2), where all agents keep their policies fixed within each episode and change them only between episodes. Under this setting, for a fixed joint policy $\boldsymbol{\pi}=(\pi^{i}, \pi^{-i})$, the training episodic return can be regarded as a representation of $\boldsymbol{\pi}$ (and, by extension, of $\pi^{-i}$ when agent $i$'s local action or policy is known). This perspective is also adopted in related works [1]. In such case, the training episodic returns are naturally associated with specific $\pi^{-i}$, and thus we refer to them as contexts.

2. Response to Question 2.

As shown in Fig. 2 of the main text, RAC significantly outperforms I2Q and Hysteretic Q-learning on multiple tasks, including predator and prey, 5m_vs_6m, 10m_vs_11m, 3s_vs_4z, 3s_vs_5z, and 2s_vs_1sc. We will extend our evaluation to more benchmarks to better confirm the superior performance of RAC.

3. Response to Question 3.

Thanks for your valuable suggestions. We evaluate QMIX and VDN on the matrix game, predator and prey, and five SMAC maps, reporting the median asymptotic performance along with the standard error across five random seeds. The results, presented in the table above, show that RAC significant outperforms both QMIX and VDN on the matrix game and predator and prey, where QMIX and VDN suffer from sub-optimal policies due to their representational limitations regarding the factored global action value function. On the five SMAC maps, RAC exhibits superior performance over other fully decentralized baselines but underperforms relative to QMIX and VDN. We attribute this to RAC's limited ability to handle partial observability and to adequately cover the entire return space. Future work will focus on enhancing RAC by incorporating belief state representations and integrating more efficient exploration techniques. We will add the comparison results in the revision for a better presentation.

4. Response to Question 4.

We begin by providing a theoretical insight on the relationship between non-stationarity and relative overgeneralization from a context modeling perspective. Non-stationarity arises due to the lack of access to other agents' actions or policies. If a perfect context (i.e., an accurate representation of other agents' joint policies) can be obtained, the context-based value function effectively conditions the current agent’s policy on the policies of the others. In this scenario, the agent is essentially learning a best-response policy, leading to convergence toward a Nash Equilibrium. However, in cooperative games, multiple Nash equilibria may exist, and not all are desirable. Without convergence to a Pareto-optimal solution, agents may fall into suboptimal equilibria, manifesting as the relative overgeneralization problem.

We attribute the inability of existing methods to simultaneously address non-stationarity and relative overgeneralization to two main factors.

(1) For methods under the centralized training with decentralized execution (CTDE) paradigm, this training framework inherently alleviates the non-stationarity but retains relative overgeneralization. This issue is evident in, for example, the representational limitations of factored global action value functions in value decomposition methods [2], or the centralized-decentralized mismatch encountered in multi-agent policy gradient methods [3].

(2) For methods under the fully decentralized learning paradigm, they typically lack the capacity to model other agents' policies (contexts) due to the absence of centralized information. As a result, they primarily focus on addressing relative overgeneralization in isolation, as seen in Distributed Q-learning, Hysteretic Q-learning, lenient learners, and related methods.

5. Response to Question 5.

To the best of our knowledge, this work is the first to formalize the task, as locally perceived by each agent, as a CMDP, thereby attributing the non-stationarity of local task dynamics to switches between unobserved contexts. A theoretical foundation for this formulation could be established by explicitly associating the context with the joint policy of other agents. We leave this theoretical extension for future work.

6. Response to Question 6.

Thanks for your valuable feedback. We will incorporate the context modeling process into a figure in the revision for more clarity.

7. Response to Question 7.

The lazy agent issue refers to the scenario where partial agents select non-cooperative actions. In RAC, the individual optimistic marginal value enables agents to accurately identify and select their cooperative actions. As a result, each agent tends to select its local cooperative actions, effectively addressing the lazy agent issue.

8. Response to Question 8.

Thanks for your careful reading. We will address this issue in the revision.

Reference:

[1] Qiu W, Ma X, An B, et al. Rpm: Generalizable multi-agent policies for multi-agent reinforcement learning[C]//The Eleventh International Conference on Learning Representations. 2023.

[2] Rashid T, Farquhar G, Peng B, et al. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning[J]. Advances in neural information processing systems, 2020, 33: 10199-10210.

[3] Wang Y, Han B, Wang T, et al. Dop: Off-policy multi-agent decomposed policy gradients[C]//International conference on learning representations. 2020.

Dear Reviewer GYBm,

During the rebuttal period, we have provided detailed responses to your concerns regarding:

(1) the rationale behind representing contexts using episodic returns and attributing non-stationarity to context switches;

(2) the performance against existing baselines (Hysteretic Q-learning, I2Q) and additional CTDE methods (i.e., add comparison results against QMIX and VDN);

(3) the discussion on why existing methods struggle to simultaneously address non-stationarity and relative overgeneralization;

(4) the clarifications on our contribution (i.e., the first work to attribute non-stationarity to context switching) and details about context modeling (i.e., include an illustrative figure);

(5) the advantage of our method in addressing the lazy agent issue.

We sincerely hope that the current responses can resolve your concerns. Looking forward to your response if you have any further concerns about this work.

Best regards,

The Authors

We begin by revisiting two key insights underlying our work. (1) We aim to mitigate non-stationarity during training (i.e., in value function updates) by shaping contexts based on episodic returns, under the assumption that other agents update / change their policies only between episodes (that is, the case 2 setting, where context changes between episodes). This assumption allows us to focus exclusively on episode-wise non-stationarity. (2) During execution (i.e., in value function estimation), we address the issue of relative overgeneralization, thereby enabling agents to select their cooperative actions. Together, these two insights form the foundation of our approach.

1. Response to Weakness 1 and Question 1.

We develop RAC under the assumption that contexts change only between episodes, where agents keep their behavior (exploration) policies fixed within each episode and update their evaluation policies only between episodes. To follow this assumption, for agents with $\epsilon$-greedy policy, we keep $\epsilon$ fixed during each episode and decrease it between episodes, as described in Lines 244-247 of the main text.

We begin with this simplified case to creatively develop an in-context fully decentralized cooperative multi-agent reinforcement learning (MARL) framework. For the more general case where contexts (other agents' policies) change between time steps, we acknowledge that RAC, in its current form, is not equipped to handle such dynamics. Extending RAC to address this more challenging scenario is the subject of our ongoing work.

2. Response to Weakness 2.

As stated at the outset, during execution, we focus on addressing relative overgeneralization using the per-agent individual optimistic marginal value $\phi^{i}$. To support this, we learn a separate $Q_{\operatorname{S}}^{i}$ to achieve a coarse coverage of the return space, thereby enhancing the learning of the context-based value function $Q^{i}$, which facilitates the computation of $\phi^{i}(s,a^{i})=\max_{c\in\mathcal{C}}Q^{i}(s,a^{i},c)$. Based on the assumption that other agents update policies only between episodes, we address episode-wise non-stationarity during training by learning the context-based value function. In contrast, we do not need to consider real-time, step-wise non-stationarity during execution; instead, our primary objective at execution time is to guide agents toward cooperative action selection using $\phi^{i}$ or its imitator, $Q_{\operatorname{S}}^{i}$.

3. Response to Weakness 3.

We agree with your argument. For cooperative tasks of moderate complexity, RAC directly using $\phi^{i}$ to select actions achieves superior performance, as demonstrated by RAC_w_$\phi^{i}$ in Sec 5.2. For more complex tasks, such as SMAC, we propose learning $Q_{\operatorname{S}}^{i}$ as a coarse means to cover the return space. A more straightforward alternative would be to enhance RAC with efficient exploration techniques, for example, by separately learning an exploratory joint policy for all agents to generate informative transitions, a strategy commonly adopted in the single‑agent RL literature. We leave this extension for future work.

4. Response to Weakness 4.

RAC is specifically designed to simultaneously address both non-stationarity and relative overgeneralization. When these two issues are relatively mild, RAC is expected to perform comparably to baselines such as IQL, as evidenced by their similar performance on certain maps. However, when these challenges become more pronounced, the performance gap between RAC and other baselines widens, as demonstrated by RAC's superior results on tasks such as the matrix game, predator and prey, 5m_vs_6m, 10m_vs_11m, 3s_vs_4z, and 3s_vs_5z.

Regarding scenarios where I2Q outperforms RAC, it is important to note that I2Q and RAC address both non-stationarity and relative overgeneralization from different perspectives. Each method exhibits distinct adaptability to different tasks, and it is not guaranteed that RAC will be optimal across all tasks.

5. Response to Weakness 5.

As noted in our Response to Weakness 3, a more straightforward approach to covering the return space would be to equip RAC with efficient exploration techniques, such as separately learning an exploratory joint policy for all agents. This approach could eliminate the need to tune the additional hyper-parameter $\beta$, and we plan to investigate it in the next stage of our work. For the remaining hyper-parameter $m$, we recommend determining an approximate value through a grid‑search procedure.

6. Response to Question 2.

In this work, we accurately determine $R_{\operatorname{max}}$ and $R_{\operatorname{min}}$ based on prior knowledge of the task. We have also considered incremental context modeling; that is, as the return space covered by RAC expands, the context space should be expanded accordingly. As the reviewer suggested, when employing online clustering of returns to construct contexts, it is important to ensure that the same return interval consistently maps to the same context across multiple clustering processes. We plan to further explore approaches for adaptively defining contexts in the next stage of our work.

7. Response to Question 3.

$Q^{i}(s,a^{i},c)$ is conditioned on the state $s$, agent $i$'s local action $a^{i}$, and the context $c$ instantiated as the episodic return interval. During the early training process, coordinated behaviors among agents are limited, and the achieved returns tend to fall within a low-value region (corresponding to small values of $c$). Consequently, when computing the individual optimistic marginal value $\phi^{i}(s,a^{i})=\max_{c\in\mathcal{C}}Q^{i}(s,a^{i},c)$, $\phi^{i}(s,a^{i})$ tends to approximate $Q^{i}(s,a^{i},c)$ associated with small $c$, which in turn corresponds to sub-optimal actions yielding low episodic returns. In summary, accurately learning $Q^{i}$ necessitates comprehensive coverage of the return space. We plan to achieve this by equipping RAC with efficient exploration techniques in the next stage of our work.

In this work, the learning processes of $Q_{\operatorname{S}}^{i}$ and $Q^{i}$ form a positive feedback loop: $Q_{\operatorname{S}}^{i}$ generates informative transitions to ensure effective updates of $Q^{i}$, while $Q^{i}$ guides $Q_{\operatorname{S}}^{i}$ in selecting per-agent local cooperative actions to produce more informative transitions. This mutual reinforcement facilitates efficient, fully decentralized cooperative policy learning. However, if $\beta$ is poorly set, such mutual reinforcement may be disrupted. As depicted in Fig.~3 (e), on the 5m_vs_6m, setting $\beta=10.0$ causes $Q_{\operatorname{S}}^{i}$ to overly imitate the poorly trained $Q^{i}$ during the early training process, and in turn $Q_{\operatorname{S}}^{i}$ fails to generate informative transitions. Consequently, RAC fails to learn an efficient policy and suffers from poor performance.

8. Response to Question 4.

Thanks for your valuable suggestions. We evaluate QMIX and VDN on the matrix game, predator and prey, and five SMAC maps, reporting the median asymptotic performance along with the standard error across five random seeds. The results, presented in the table above, show that RAC significant outperforms both QMIX and VDN on the matrix game and predator and prey, where QMIX and VDN suffer from sub-optimal policies due to their representational limitations regarding the factored global action value function. On the five SMAC maps, RAC exhibits superior performance over other fully decentralized baselines but underperforms relative to QMIX and VDN. We attribute this to RAC's limited ability to handle partial observability and to adequately cover the entire return space. Future work will focus on enhancing RAC by incorporating belief state representations and integrating more efficient exploration techniques. We will add the comparison results in the revision for a better presentation.

9. Response to Limitation.

Thanks for your valuable advice. We will place emphasis on the following directions in our future work: (1) developing adaptive context modeling, (2) ensuring comprehensive coverage of the return space through efficient exploration, and (3) extending to the more general setting in which contexts (other agents' policies) change between time steps.

Dear Reviewer KFLW,

During the rebuttal period, we have provided detailed responses to your concerns regarding:

(1) the assumption that contexts change between episodes;

(2) the role of episodic return-based contexts in addressing episode-wise non-stationarity;

(3) the rationale behind separately learning $Q_{\operatorname{S}}^{i}$;

(4) the performance against other baselines and additional CTDE methods (i.e., add comparison results with QMIX and VDN);

(5) the further extensions involving adaptive context modeling and simplified hyper-parameter tuning.

We sincerely hope that the current responses can resolve your concerns. Looking forward to your response if you have any further concerns about this work.

Best regards,

The Authors

Response to Q2.

We appreciate the reviewer's insightful question and acknowledge that our previous explanation may have been too high-level, leading to potential misunderstandings. As discussed in our Response to Question 3, $Q^{i}(s,a^{i},c)$ is conditioned on the state $s$, agent $i$'s local action $a^{i}$, and the context $c$ instantiated as the episodic return interval. During the early training process, coordinated behaviors among agents are limited, and the achieved returns tend to fall within a low-value region (corresponding to small values of $c$). Consequently, when computing the individual optimistic marginal value $\phi^{i}(s,a^{i})=\max_{c\in\mathcal{C}}Q^{i}(s,a^{i},c)$, $\phi^{i}(s,a^{i})$ tends to approximate $Q^{i}(s,a^{i},c)$ associated with small $c$, which in turn corresponds to sub-optimal actions yielding low episodic returns. In summary, accurately learning $Q^{i}$ necessitates comprehensive coverage of the return space.

For tasks with moderate complexity, directly selecting actions using $\phi^{i}(s,a^{i})$ for each agent yields superior performance, as stated in our Response to Weakness 3. For more complex tasks such as SMAC maps, we empirically find that separately learning $Q_{\operatorname{S}}^{i}(s,a^{i})$ and optimizing it with the standard IQL value loss and the supervision loss lead to satisfactory performance.

Importantly, we clarify that the inclusion of $Q_{\operatorname{S}}^{i}$ is not a unique choice, and other alternatives are viable. In the context of SMAC maps, the ability of $Q_{\operatorname{S}}^{i}$ to aid in return-space coverage can be attributed to the following two main factors:

(1) The competitive performance of vanilla IQL on SMAC maps, which makes IQL a reasonable candidate for generating informative transitions.

(2) The incorporation of additional supervision signals, such as the proposed supervision loss, through which $Q^{i}$ helps guide $Q_{\operatorname{S}}^{i}$ in identifying and selecting agents' local cooperative actions, improving its effectiveness in supporting exploration and learning.

From perspective (1), $Q_{\operatorname{S}}^{i}$ could be replaced by other RL algorithms that demonstrate pretty performance. From perspective (2), its exploration capability could be further enhanced using intrinsic objectives (e.g., curiosity-based rewards) or alternative supervision signals derived from $Q^{i}(s,a^{i},c)$ to encourage broader or more targeted exploration.

We sincerely thank the reviewer for prompting a deeper discussion on this topic. We will incorporate this elaboration into the revision to better communicate the flexibility and extensibility of our framework.

Thanks for your timely feedback. Our responses are detailed as follows.

Response to Q1.

We would like to emphasize that each work is typically featured by a critical innovation. The primary contribution of this work lies in the introduction of in-context fully decentralized cooperative MARL, a unified framework for addressing both non-stationarity and relative overgeneralization. The current focus on Case (2), where contexts (i.e., agents' policies) change only between episodes, serves as a temporary simplification. This treatment also underscores the potential of Case (1) as a promising direction for future research.

To further advance this line of research, we are actively working on extending the framework to Case (1), where contexts change between time steps. Our ongoing work effectively addresses this scenario through dynamics-aware context modeling. Specifically, from the local perspective of each agent $i$, we denote its perceived local task dynamics by $P^{i}(s_{t+1}|s_{t},a_{t}^{i},c_{t}^{i})$ and $R^{i}(s_{t},a_{t}^{i}, c_{t}^{i})$, where the context $c_{t}^{i}$ is associated with the real-time joint policy $\pi_{t}^{-i}$ of other agents $-i$ at time step $t$. As policies evolve over time, both $c_{t}^{i}$ and the local task dynamics $P^{i}, R^{i}$ also change. Accordingly, we propose to represent the contexts by real-time task dynamics distribution.

Under Case (1), the step-wise changing local task dynamics distribution parallels the setting of concept drift, where the underlying data distribution evolves over time. To address such shifts, maintaining a sliding window to hold the latest data within the data stream has proven effective in capturing the real-time data distribution, ensuring model adaptability and accuracy in dynamic settings [1]. Accordingly, we propose to maintain a sliding window alongside per-agent local trajectory, and model the task dynamics distribution using latent variables based on the transitions within this sliding window. This allows us to integrate dynamics-aware context modeling within the same unified framework proposed in this work, enabling the learning of context-based value functions that address step-wise non-stationarity.

This ongoing work directly corresponds to the future work outlined in Lines 336-338 of the main text (Sec.~6, Conclusion: Limitation and Future Work):

Second, RAC currently focuses only on the scenario where context changes between episodes (case (2)). To address case (1), where context changes between time steps, we plan to construct contexts by modeling the local task dynamics distribution over different time intervals. We leave them as our future works

We also present preliminary empirical results to validate the proposed method on the matrix game and 5m_vs_6m. As shown in the table below, the proposed method succeeds in selecting the optimal joint action with $+8$ reward on the matrix game, and outperforms baselines on 5m_vs_6m, demonstrating its potential effectiveness.

In summary, we sincerely appreciate the reviewer's insightful comments, which inspired us to broaden the scope of our work to address both Case (1) and Case (2). We believe these two works form a strong foundation for the proposed in-context fully decentralized cooperative MARL framework. Further investigations into addressing step-wise non-stationarity remain a promising and worthwhile direction for future research.

Reference

[1] Lu J, Liu A, Dong F, et al. Learning under concept drift: A review[J]. IEEE transactions on knowledge and data engineering, 2018, 31(12): 2346-2363.

Response to Q3.

We apologize for the confusion caused by our earlier statement. In this work, we accurately determine $R_{\operatorname{max}}$ and $R_{\operatorname{min}}$ based on prior knowledge of the task. As such, there is no scenario where the actual returns go far outside these initial bounds.

The notion of incremental context modeling is part of our ongoing exploration into extending the proposed framework and is not implemented in the current version of the work. Nevertheless, we appreciate the reviewer's interest and would like to share some preliminary ideas related to this direction.

We identify two key challenges:

(1) How to adaptively identify the bounds $R_{\operatorname{max}}$ and $R_{\operatorname{min}}$?

(2) How to ensure consistent one-hot context representations when using an adaptive discretization of the return space across time? That is, we should ensure the same one-hot context representation consistently maps to the same return value interval across multiple adaptive modeling processes of the return bounds.

To address (1), we consider a simple adaptive approach where $R_{\operatorname{min}}, R_{\operatorname{max}}$ are maintained as the minimum and maximum returns observed so far. Based on these values, we dynamically partition the return space into $m$ intervals. However, this leads to challenge (2): as the return bounds evolve over time, the semantic meaning of each one-hot encoded context (i.e., return interval) may change, violating consistency and potentially confusing the learning process.

To address this issue, we propose a warm-up phase during which only $R_{\operatorname{min}}$ and $R_{\operatorname{max}}$ are collected over the first $T_{\operatorname{warm-up}}$ time steps, without updating $Q^{i}(s,a^{i},c)$ and calculating the supervision loss. After this initialization, the return space is discretized into $m$ fixed intervals, and any out-of-bound returns are assigned to the first or last interval. Only then do we begin training $Q^{i}$ and computing the supervision loss to guide $Q_{\operatorname{S}}^{i}$.

We refer to this approach as Adaptive_RAC, and evaluate its performance on the matrix game and 5m_vs_6m. In the matrix game, we find that setting $T_{\operatorname{warm-up}}=1000$ has yielded the best performance, where Adaptive_RAC succeeds in achieving the highest reward $+8$. On the 5m_vs_6m, we experiment with values of $T_{\operatorname{warm-up}}$ to 50000, 150000, 250000, 350000, 450000, and 500000. We report the corresponding test win rates as well as the recorded values of $R_{\operatorname{max}}$ and $R_{\operatorname{min}}$ under these configurations. The results, summarized in the table below, demonstrate that setting $T_{\operatorname{warm-up}}$ to $350000, 450000, 500000$ can accurately find the maximum episodic return bound $20$ and yield satisfactory performance. In contrast, smaller warm-up durations $T_{\operatorname{warm-up}}=50000, 150000, 250000$ struggle in sub-optimal policy and inaccurate return bounds.

We will include Adaptive_RAC as a baseline in the revision, and provide comparative results on the remaining tasks to further validate and enrich our contributions.

We note, however, that the effectiveness of this approach still depends on the return diversity observed during the warm-up phase. Inadequate coverage during this phase may lead to an incomplete context space, as also discussed in our Response to Q2. This limitation highlights the need for future work on both efficient return-space exploration and more flexible context representations, as advised by the reviewer.

Following the reviewer's valuable suggestion, we will place emphasis on the following directions in the revision to inspire further researches within the proposed in-context fully decentralized cooperative MARL framework: (1) developing adaptive context modeling, (2) ensuring comprehensive coverage of the return space through efficient exploration, and (3) extending to the more general setting in which contexts (other agents' policies) change between time steps.

We are grateful for the reviewer's constructive feedback, which motivates important future extensions of our work.