Fair Cooperation in Mixed-Motive Games via Conflict-Aware Gradient Adjustment

We appreciate the reviewer’s valuable comments and questions.

Below, we provide our responses to the reviewer’s concerns and questions.

[4-1] Regarding "discussion of solution concepts from game theory"

Regarding "What equilibrium is being targeted here?" We clarify that our work does not aim to explicitly compute or characterize a game-theoretic equilibrium (e.g., Nash or correlated equilibrium). Instead, our focus is on learning Pareto-optimal joint policies in mixed-motive multi-agent environments.

In these settings, pure equilibrium solutions (such as Nash equilibrium) often result in inefficient or unfair outcomes. Rather than relying on equilibrium concepts, we directly optimize the joint training dynamics such that all agents’ individual value functions converge, while maximizing the performance. This guarantees that the learned outcome lies on the Pareto front.

To evaluate the quality of these outcomes, we adopt the α-fairness framework, which allows us to capture different trade-offs between efficiency and fairness:

• $\alpha=0$-fairness (equivalent to average return) captures only task completion;
• $\alpha=1$-fairness (geometric mean) and $\alpha=\infty$-fairness (minimum individual return) incorporate both task performance and fairness, with increasing emphasis on fairness.

We will clarify in the final version that our solution concept is Pareto optimality.

[4-2] Regarding "the direction of motivation"

We represent the direction of motivation through gradient adjustment. In reinforcement learning, the policy is guided by the value function. FCGrad modifies this guidance by adjusting the gradient direction to promote fairness and cooperation.

[4-3] Regarding "Fairness and resolving conflict seem to be two separate issues"

FCGrad aims to achieve fairness through conflict-aware gradient adjustment. Fairness in our context refers to the actual returns received by each agent, not to taking the mean of gradient directions. FCGrad allows agents to prioritize the collective objective when they are in an unfair situation (i.e., receiving less return than others), and vice versa. This adaptive adjustment promotes fairness while still optimizing expected returns. As shown in our theoretical results, this procedure yields a monotonic non-decreasing improvement in the value functions.

Regarding optimality, we emphasize that it depends on the chosen metric. This is precisely why we adopt the $\alpha$-fairness framework, which jointly considers both task completion and fairness. For example, in terms of average return ($\alpha=0$), which is widely used in the literature and ignores fairness, FCGrad performs comparably to the Col baseline (which maximizes collective return) in the Cleanup and Harvest environments, and even slightly worse in Unfair Coin. However, in terms of $\alpha=1$ and $\alpha=\infty$, which incorporate fairness to increasing degrees, FCGrad consistently outperforms the baselines. This demonstrates that FCGrad achieves a favorable trade-off between fairness and performance.

More broadly, $\alpha$-fairness defines a family of Pareto-optimal solutions with varying emphasis on fairness. By adjusting gradients to navigate toward different regions of the Pareto front, FCGrad enables a flexible trade-off between fairness and efficiency, rather than sacrificing one for the other.

[4-4] Regarding "motivation"

The example provided by the reviewer—division of labor that maximizes average return but is unfair—is precisely the kind of solution FCGrad is designed to address (as also mentioned in the Introduction). As shown in Figure 4(a), the Col baseline, which optimizes collective return, performs poorly in terms of Min ($\alpha = \infty$), indicating that some agents receive disproportionately low rewards under "efficient but unfair" policies. In contrast, FCGrad achieves a minimum return close to 40, suggesting that even the least-rewarded agent is actively collecting apples.

While some prior works attempt to address fairness through reward shaping (as discussed in Section 2.4), these typically rely on gifting mechanisms—where agents transfer rewards to others. Fairness is then measured over the sum of task rewards and gifts. However, this can be problematic: for example, one agent may clean pollution alone while receiving only "intrinsic" rewards, which does not reflect actual participation in the task. In contrast, we define fairness solely based on task rewards to better capture each agent’s true contribution.

FCGrad addresses this imbalance through gradient adjustment. Specifically, when an agent’s individual return is lower than the collective return, FCGrad updates its policy in the direction of its individual value function (e.g., toward collecting apples). Conversely, agents whose individual return exceeds the collective return shift their gradient toward the collective objective (e.g., cleaning pollution). This dynamic leads to more balanced task participation and promotes fairness while maintaining high overall performance.

To capture the trade-off between fairness and optimality, we adopt $\alpha$-fairness as a unified metric. For $\alpha > 0$, it balances both aspects, with higher $\alpha$ placing more emphasis on fairness. As our results show, FCGrad outperforms baselines in terms of $\alpha = 1$ and $\alpha = \infty$, achieving strong fairness while maintaining high task performance. To the best of our knowledge, we are the first to report $\alpha$-fairness as an evaluation metric in mixed-motive multi-agent reinforcement learning.

4-5) Regarding "connection with [Duque et al.]"

We appreciate the reviewer pointing out the work by Duque et al., which indeed shares a high-level motivation with ours—namely, addressing the misalignment between individual and collective objectives, particularly in social dilemmas where selfish behavior leads to suboptimal outcomes.

While both works aim to promote cooperation, their approaches differ fundamentally. Duque et al. attempt to resolve this misalignment by aligning agents' advantages, shaping the behavior of other agents to promote robust cooperation. Their method seeks to mitigate the conflict, rather than leveraging it.

In contrast, our method explicitly detects and addresses such conflicts during training via conflict-aware gradient adjustments. Moreover, fairness is a core objective in our framework. Unlike Duque et al., which does not incorporate fairness metrics, FCGrad is designed to promote both cooperation and fairness by dynamically adjusting the learning direction based on disparities in agent returns.

In summary, while the high-level motivation is similar, our objectives and mechanisms differ: Duque et al. focus on advantage alignment to foster cooperation, whereas we explicitly exploit gradient-level conflicts to achieve fair and cooperative outcomes.

• We are glad that many misunderstandings have been addressed.

• We agree with the reviewer that reward distribution mechanisms can also address both metrics.

• We acknowledge our limitation: SSD problems are originally aimed at resolving the tension between short-term individual incentives and long-term collective benefits in a temporally extended setting—where the collective return is the ultimate objective. Our formulation is suboptimal in that regard. We will clarify this point in the final version of the paper.

Please let us know if there are any additional points that require clarification, so we can further address the reviewer’s concerns.

Best,

Authors

We apologize if our earlier statements came across as overly strong.

When we referred to the “ultimate goal,” our intention was to describe the evaluation goal in most SSD studies, which is then reported as collective return in the evaluation results(e.g., Figures 4, 5, and 6 in [1]). Prior work typically evaluates performance based on collective return, in settings where individual incentives and long-term collective benefits are not naturally aligned. This misalignment creates tension, and the objective is to resolve it so that the total number of apples collected is maximized. In such cases — as in Cleanup — the absence of direct rewards for cleaning creates the social dilemma, which in turn hampers collective return. As the reviewer correctly noted, this goal is pursued as “each agent optimizes its own individual reward in an agent-wise manner.”

As the reviewer also mentioned, existing approaches make use of reward redistribution (what we refer to as reward reconstruction). This assumes that redistributed rewards are real, tangible payoffs — akin to currency in real life — that can be used to address fairness issues, e.g., compensating an agent that only cleans. Under this assumption, division of labor naturally emerges. We find this approach compelling and elegant, and it has been the focus of much prior work.

In contrast, as the reviewer acknowledged, our work adopts a different assumption. Rather than assuming the availability of a currency-like signal that is independent of an agent’s direct actions or the environment, we focus on a more primitive notion of fairness — one based solely on rewards obtained from actual state–action outcomes. This leads us to ask: if one agent cleans entirely while others only collect apples, is such an arrangement still acceptable in the absence of currency? In such a setting, one could argue that it would be fairer — and potentially better from another perspective — for all agents to share the cleaning task, even if that changes the collective reward outcome.

We fully understand the reviewer’s concern that, under the traditional reward redistribution assumption, our method could be suboptimal in terms of collective reward. However, under our setting, there can be multiple optimal solutions depending on the trade-off between fairness and cooperation. For this reason, we evaluate using multiple α values in α-fairness to explore different points on the Pareto frontier.

We will make sure to clearly explain this alternative assumption and the scope of our study in the final version.

Thank you again for your thoughtful comments.

[1] Jiachen Yang et al.,"Learning to Incentivize Other Learning Agents," NeurIPS 2020

We sincerely thank the reviewer for actively engaging in the discussion and helping us refine our explanations.

First, our work does not aim to find an equilibrium; rather, it seeks Pareto-optimal outcomes. Both equilibrium concepts (e.g., Nash, correlated equilibrium) and Pareto optimality are well-studied in game theory, but they are distinct. While an equilibrium focuses on stability under strategic deviations, Pareto optimality focuses on efficiency—solutions where no agent’s outcome can be improved without reducing another’s.

Second, α-fairness in our paper is not a training framework but an evaluation metric. Our method does not directly use α-fairness during learning; we only use it to evaluate outcomes. As noted in the paper, we also report the α = 0 case (collective return), which corresponds to pure efficiency without fairness considerations. From the behavioral perspective, we understand that the reviewer may be referring to sharing the cleaning task in Cleanup. In that sense, policies learned with FCGrad may achieve slightly lower collective return than “selfish agents spontaneously performing reward redistribution.” In our view, this latter behavior is effectively equivalent to all agents directly optimizing collective reward, which is exactly what we report as the “Col” baseline in our experiments. Indeed, in Cleanup and Harvest, “Col” sometimes achieves a higher maximum collective return than FCGrad, and this is even more evident in Unfair Coin (Figure 3). However, for other α values, FCGrad outperforms “Col.” In addition, we think the reviewer's suggestion that directly incorporating α-fairness into the learning process could be an interesting direction. To the best of our knowledge, such approaches have been explored in bandit settings, but applying them in complex RL environments remains challenging. We will leave this as an avenue for future work.

Finally, as mentioned above, game theory is not solely about equilibrium concepts. Our paper is about Pareto-optimal solutions, and α-fairness is used as a metric to evaluate them. While FCGrad may not achieve the highest performance for α = 0 (collective return), it outperforms baselines for α = 1 and α = ∞, which incorporate both task completion and fairness. We will make sure to clarify this distinction, as well as the role of α-fairness, more explicitly in the final version.

Once again, we truly appreciate the reviewer’s active participation in this discussion, which has helped us better articulate the scope and positioning of our work.

We appreciate the reviewer’s valuable comments and questions.

Below, we provide our responses to the reviewer’s concerns and questions.

[3-1] Regarding "individual-collective conflicts"

We believe that individual–collective conflict implicitly includes individual–individual conflicts in the form of "one agent versus the others." For example, consider a three-agent case where the collective reward is defined as the average of individual rewards. If agent 1’s reward is lower than the average, i.e., (R1+R2+R3)/3 > R1, then it must be the case that the average of R2 and R3 is greater than R1, implying at least one agent is outperforming agent 1.

[3-2] Regarding "lack measurement of 'task completion' itself"

We respectfully clarify that the main results in Section 4 are based on $\alpha$-fairness, which simultaneously captures both "task completion" and "fairness." As discussed in the paper:

• $\alpha=0$-fairness (equivalent to average return) captures only task completion;
• $\alpha=1$-fairness (geometric mean) and $\alpha=\infty$-fairness (minimum individual return) incorporate both task performance and fairness, with increasing emphasis on fairness.

[3-3] Regarding "large-scale complex heterogeneous multi-agent systems"

Our current experiments focus on fairness and cooperation in environments that exhibit asymmetries, which we consider a form of heterogeneity. In addition, Cleanup and Harvest have been considered as fairly large-scale and complex environments. Scaling FCGrad to extremely large or structurally heterogeneous agent populations is an important challenge, and we agree it would be valuable future work.

[3-4] Regarding "the periodic occurrence of gradient conflicts"

We agree with the reviewer that the theoretical framework assumes the periodic occurrence of gradient conflicts. While this assumption may not always hold in real-world scenarios, our empirical results demonstrate that FCGrad performs well in the tested environments, where such conflicts naturally arise during learning. Investigating robustness under less frequent or non-periodic conflict patterns would be a promising direction for future research.

[3-5] Regarding "The accuracy of gradient estimation"

We agree with the reviewer that the quality of gradient estimation can impact learning. However, similar to general stochastic gradient methods, we mitigate this issue by using a sufficiently large number of samples when estimating gradients.

[3-6] Regarding "the specific differences between FCGrad and PCGrad"

First, PCGrad was originally proposed for multi-objective learning, whereas FCGrad is designed for achieving fair cooperation in multi-agent reinforcement learning. Second, PCGrad does not adaptively consider which gradient direction should be prioritized. It resolves conflicts by projecting each conflicting gradient onto the normal plane of the other and summing them (see Section 2.3 of our paper).

In contrast, FCGrad adaptively selects which gradient to project based on the expected returns: the gradient associated with the lower expected return is projected onto the normal plane of the other. This mechanism promotes fairness by favoring the underperforming objective in conflict situations.

[3-7] Regarding "reward functions"

In our setup, each agent has access to all individual rewards provided by the environment. The collective reward is computed as the average of these individual rewards. Thus, both individual and collective rewards originate from the environment, with the latter being a derived aggregate.

[3-8] Regarding "perform on the task itself"

As described in our main results (Section 4), we use $\alpha$-fairness as a unified metric that captures both task performance (explicit reward) and fairness. Specifically, $\alpha = 0$ corresponds to average return (pure task performance), while $\alpha = 1$ and $\alpha = \infty$ incorporate increasing emphasis on fairness. FCGrad achieves competitive performance under $\alpha = 0$ and outperforms other baselines under $\alpha = 1$ and $\alpha = \infty$, indicating it balances both objectives effectively.

[3-9] Regarding "How scalable is the algorithm to multi-agent systems"

Since FCGrad is a decentralized algorithm, it is easy to scale, as reviewer ZK8G mentioned as one of our strengths (“2. It is a decentralized training approach which gives it scalability.”) We believe scaling to very large populations is a potential future direction.

We appreciate the reviewer’s valuable comments and questions.

Below, we provide our responses to the reviewer’s concerns and questions.

[2-1] Regarding "Details of proof"

Due to space limitations, we include only the interpretations of the statements in the main paper. We will provide a proof sketch in the final version.

[2-2] Regarding "FCGrad uses the weighted sum of two gradients: g = ..."

That is correct. We will revise g to g_fcgrad in line 168.

[2-3] Regarding "Table 1: bold and significant tests"

We will include the standard deviation in the final paper. The following table shows the mean and standard deviation of Jain and Gini indices for each algorithm. (We have added two more seeds for Harvest and Cleanup.)

The following table shows the p-values from Student’s t-test comparing FCGrad to each baseline:

Note that these results reflect fairness only, without considering task performance. In the Unfair Coin Game, FCGrad achieves higher fairness metrics than all baselines, with strong statistical significance. For Cleanup, FCGrad outperforms all except wei, which achieves similar fairness. In Harvest, most algorithms except col show comparable fairness to FCGrad. Even though FCGrad does not always outperform others in raw fairness scores, it achieves statistically significant improvement in terms of α-fairness (please refer to our response to Reviewer ZK8G03 [1-2]).

[2-4] Regarding "a brief discussion about the dual purpose of this research: can someone use the same paradigm as FCGrad to intentionally decrease fairness?"

Thank you for the helpful and meaningful suggestion. We would like to clarify that FCGrad does not directly optimize for fairness as an explicit objective; rather, fairness emerges as a consequence of balancing individual and collective objectives. Therefore, we do not believe FCGrad can be trivially repurposed to intentionally decrease fairness. In fact, an extreme case of not considering fairness at all would be optimizing only for the collective objective, i.e., the Col baseline, which may lead to highly imbalanced outcomes despite maximizing the overall return.

That said, as the reviewer suggests, there are certainly ways an agent could attempt to manipulate fairness, such as by maximizing its own reward while actively minimizing others’. Detecting and preventing such adversarial behaviors presents an important and underexplored direction for future work in fairness-aware multi-agent systems.

We appreciate the reviewer’s valuable comments and questions.

Below, we provide our responses to the reviewer’s concerns and questions.

[1-1] Regarding "a diagram for the model"

We will include a diagram describing our model in the appendix of the final paper. Thank you for the helpful suggestion.

[1-2] Regarding "Increasing the number of runs"

We have added two more seeds (6 in total) for Cleanup and Harvest.

The following table shows the performance of FCGrad and the baselines over 6 seeds:

The following table shows the p-values from Student’s t-test:

As shown in both tables, FCGrad consistently outperforms the baselines in terms of α-fairness (α=1 and α=∞), with statistical evidence. The only exception is the "wei" baseline in Cleanup (Min performance), which can exhibit high variance as α increases in α-fairness. However, FCGrad (43.6) still achieves better performance than Wei (31.1).

[1-3] Regarding "Regarding the expected behavior of agents trained under different regimes"

We appreciate the reviewer’s thoughtful questions regarding the robustness of FCGrad agents when interacting with agents trained using different regimes or exhibiting off-policy behaviors.

We would first like to clarify that the theoretical guarantees of FCGrad (e.g., monotonic improvement and fairness) assume that all agents adopt the FCGrad update rule. Nevertheless, even in mixed training regimes, our design encourages coordination rather than exploitation due to the structure of the collective reward.

In particular, since the collective objective is designed to yield higher returns than purely individual optimization (i.e., col > ind), agents trained with other regimes are naturally incentivized to engage in coordination. If an agent trained with a different policy attempts to exploit FCGrad agents by behaving selfishly, it will often find that its own individual reward remains limited, potentially perceiving this as an unfair outcome. Over time, it would thus be driven toward more cooperative behavior to access the benefits of the collective reward.

Conversely, if an FCGrad agent perceives that others are not cooperative or are unable to adapt, it will naturally shift toward optimizing the individual objective. This dynamic ensures that coordination is pursued when beneficial, but exploitation is not blindly tolerated. In this way, FCGrad agents exhibit a degree of robustness when interacting with off-policy behaviors by adapting their gradient direction based on observed reward feedback.

We believe this structure makes FCGrad well-suited for mixed-agent scenarios, and we thank the reviewer for raising this important point.