Learning to Balance Altruism and Self-interest Based on Empathy in Mixed-Motive Games

Thank you for your valuable comments on our work!

The comparison to the original gifting methods and the individual contributions of the gifting mechanism.

Our primary contribution lies in designing an algorithm that can dynamically adjust the gifting strategy to different co-players, performing exceptionally well in multi-agent decision-making and adaptation. In contrast, most of the previous works predefine fixed gifting amounts, lacking adaptation ability. LIO and LToS [4] have learned gifting strategies, but underperform in mixed-motive scenarios.

As far as we know, [1] first used the gifting mechanism to promote cooperation in sequential social dilemmas. The implementation of the gifting in [1] is to equip the agent with the extra action of a "gifting beam", which gives other agents in the beam a reward of $g$, and incurs an immediate penalty of $−g$ to itself. We try to approximatively implement this method by adding an additional gifting action to the A2C agents. However, since this action incurs an immediate $−g$ penalty, even a small $g$ will quickly reduce the probability of choosing this action, and the result is almost the same as that of A2C without gifting in our baselines which underperforms LASE.

There are also some works that focus on how social dilemmas can be resolved by pre-defining the proportion of rewards to be distributed (gifted) [2] [3]. At one extreme, equal distribution makes everyone's goal the group's total benefit, the same as our GO baseline. For a clearer comparison, we conduct an additional ablation experiment using the converged gifting weight (as shown in Table 2) as a fixed gifting weight for each agent.

LASE outperforms because it can encourage cooperative behavior by rewarding some specific agents. In addition, LASE can dynamically adjust the proportion of rewards shared with different agents, which helps avoid being exploited by others.

To sum up, our contribution to the gifting mechanism is mainly reflected in the fact that LASE can adjust the gifting strategy dynamically by inferring the social relationships with others, and can effectively promote group cooperation and avoid exploitation under various sequential social dilemmas.

[1] Lupu et al., Gifting in multi-agent reinforcement learning, AAMAS 2020

[2] Wang, et al., Emergent prosociality in multi-agent games through gifting, IJCAI 2021

[3] Willis et al., Resolving social dilemmas through reward transfer commitments, ALA 2023

Ablation study, contribution and effectiveness about the counterfactual reasoning module.

The social relationships inferred through counterfactual reasoning guide our gifting strategy and form the core of our algorithm, making ablation experiments on this module challenging. A feasible method is to use an end-to-end trained neural network to determine gifting weights, as done by LToS [4]. So we add LToS as one baseline:

It shows that the counterfactual reasoning module helps LASE outperform LToS. Meanwhile, LIO as a method that does not use counterfactual reasoning but directly uses neural networks to output gifted rewards has also been compared as a baseline in the paper.

Counterfactual reasoning is an effective idea in multi-agent learning, however, most previous research has focused on cooperative tasks using the CTDE framework [5] or competitive tasks using counterfactual regret minimization (CFR) [6]. In contrast, LASE employs counterfactual reasoning to infer social relationships in a decentralized manner, achieving excellent performance in mixed-motive games. We believe this innovation has the potential to significantly enhance community development.

[4] Yi, et al., Learning to share in multi-agent reinforcement learning, NeurIPS 2022

[5] Foerster et al., Counterfactual Multi-Agent Policy Gradients, AAAI 2018

[6] Zinkevich, et al., Regret minimization in games with incomplete information, NeurIPS 2007

Compare the Equality with other baseline methods.

As an evaluation metric, fairness should be evaluated alongside reward to measure algorithm performance effectively. Some algorithms may fail to address decision-making issues in mixed-motive games where each agent receives a small reward, but the reward disparity between agents is minimal, resulting in high fairness. Clearly, these methods are not effective. An effective method should both maximize group reward and ensure intra-group equity. Thus, we compared only LASE and GO, which achieve the highest rewards. We have now included the fairness results of other baselines:

As you guess, LASE does not always outperform the baseline on fairness metrics, such as SI and A2C in SSH, LIO in SSG, IA in Coingame, and SI in Cleanup. However, LASE can significantly enhance group benefits while maintaining high fairness, which other baselines can’t.

Limitations and broader societal impacts.

We have mentioned the limitations of our work in Section 7, Conclusion, including the assumption that each agent's reward function will be modified by other agents' gifts, whereas in reality, people may refuse gifts. Additionally, LASE currently focuses on finite discrete action spaces, and extending it to continuous action spaces is our next goal.

We discussed the broader impact of our work in Appendix D. If you have any further questions, please feel free to ask! We are happy to discuss this with you!

The complexity of the framework and the scalability of the method.

Thank you very much for your suggestions on LASE’s scalability! To test LASE’s scalability, we have extended Cleanup and Snowdrift as follows:

Here are the experimental results:

As shown above, LASE still outperforms the baselines in extended environments. Thus, we say that LASE has a certain degree of scalability. We believe that learning two additional networks for opponent modeling and social relationship inference in a decentralized manner is not overly complicated, ensuring the method's scalability. However, it is undeniable that the current version of LASE requires inferring the social relationships of each agent individually, leading to increased computational complexity as the number of agents increases greatly. The scalability of LASE will be a main focus of our future study. A possible approach is updating the relationship between agents less frequently.

The explanation of the observational conversion network and the SR policy network. An ablation study about mapping $i$’s observation to $j$’s policy directly.

Thank you very much for your suggestion to replace the PT module with a single network for the ablation study! We trained the network $p(\mathbf{\hat{a_t}} | o^i_t, \mathbf{a_{t-1}})$ with supervised learning by minimizing the MSE loss of the predicted joint action $\mathbf{\hat{a_t}}$ and the real action $\mathbf{a_t}$, and get the following results:

The results show that the two methods perform comparably in SSH, SSG, and Cleanup, while LASE performs better in Coingame. However, since both approaches predict others' actions based on the same local observations, they essentially serve as opponent modeling tools and thus exhibit similar performance.

We use the PT module in LASE for two main reasons:

First, since the SR value network $\phi$ is required to calculate the egocentric $Q$-value when carrying out counterfactual reasoning, and the SR policy network $\mu$ predicts others' actions with the egocentric policy model, both networks share the CNN and some FC layers to extract the observation features. And they are optimized with the same reward signal under the actor-critic framework (Eq. 5). This sharing is common in actor-critic framework and helps improve training efficiency. Therefore, although the PT module in Figure 1 consists of two networks, the additional parameters and computational overhead mostly stem from the observation conversion network, and it introduces essentially the same amount of parameters as using a single perspective taking network.

Second, based on the psychological theory that perspective-taking is a crucial component of cognitive empathy [1], we employ the PT module instead of an end-to-end trained neural network to predict opponent actions, which helps computationally and comprehensively model empathy, the essential mechanism in human society. Since this approach does not result in significant performance loss or increased memory and computational demands, we believe our work contributes new insights into opponent modeling and encourages the community to incorporate human cognitive processes into AI agent design.

[1] Davis, Mark H. "Measuring individual differences in empathy: Evidence for a multidimensional approach." Journal of personality and social psychology 44.1 (1983): 113.

Removing the cleaning beam in Cleanup.

In our implementation, the cleaning beam is replaced by a cleaning action which takes effect only at the waste location, thereby maintaining the dimensionality of the action space. Since the river where the waste accumulates and the apple orchard are located on opposite sides of the map, removing the cleaning beam prevents the agent from directly cleaning waste near the apple orchard. Meanwhile, the time cost of traveling between the river and the apple orchard requires agents to collectively learn a more explicit division of labor strategy in the dilemma. This is why we claim that removal will make the environment harder.

The parameters of the observation conversion network and SR policy network.

Sorry for the confusion! The observation conversion network $\eta$ and the SR policy network $\mu$ do employ separate sets of parameters. In Equation 4, even though the loss is computed forward through $\mu$, is not used to update $\mu$, but only to update $\eta$. The update method of $\mu$ is in Eq.5.

Epsilon greedy

We use epsilon greedy mainly to enhance exploration and avoid falling into local optima, which is more likely to occur in mixed-motive games. The introduction of epsilon greedy allows for a controlled level of exploration. The utilization of epsilon greedy is not an anomaly. The classic on-policy algorithm SARSA [2] uses epsilon greedy. A low epsilon value does not cause drastic policy changes, so on-policy methods remain effective. This strategic balance between exploitation and exploration has proven to be empirically sound, fostering a robust approach to algorithmic performance. A similar example is PPO, which uses trajectories sampled from a slightly offset policy for update, but it is still classified as an on-policy algorithm, and it has demonstrated strong empirical performance.

[2] Sutton, Richard S. "Generalization in reinforcement learning: Successful examples using sparse coarse coding." NeurIPS 1995

Estimate the uncertainty of their social relationship.

We select the $w^{ij}$ data from the last $10^6$ timesteps of training to calculate their mean value $\overline{w}$ and standard deviation $s$, which estimates the uncertainty of social relationships. The calculation method is as follows:

We conduct a comparative experiment to replace the input of SR policy network $\hat{o}^j$ with $j$’s real observation $o^j$ . Here is the results:

The results show that the mean value of LASE's inferred social relationships closely matches that of scenarios with actual observations, although partial observability significantly increases uncertainty. Considering that the social relationships between people in real life tend to be relatively stable and do not change drastically, we think that a possible solution to handle the uncertainty of social relationships is to introduce some smoothing techniques for $w^{ij}$ to reduce the variance of social relationships over time. This approach will be explored in our future work.

Meanwhile, It is important to note that Figure 8 and the corresponding analysis show that LASE is able to correctly infer the relationships with different co-players and respond properly. Specifically, in the experiments conducted in Section 6.3, one LASE interacts with three rule-based co-players: cooperator, defector and random. The results show the $w^{ij}$ given to the cooperative co-player is the largest, significantly higher than that given to the other two co-players. The $w^{ij}$ given to the random co-player comes second, and the smallest is given to the defector. These results demonstrate the consistency between LASE’s estimates of the relations and the ground truth.

Thanks again for your valuable insights! We strongly agree that this issue has an important impact on trust in real-world scenarios, and we will add relevant results and analysis in a revised version of the paper. At the same time, we would also like to thank you for providing the literature about partial observability, which is very helpful to our follow-up work!

The usage of the term “empathy”

Empathy includes both emotional empathy and cognitive empathy [1]. The former refers to the ability to feel and share others’ emotions. The latter involves imagining and understanding others’ thoughts, feelings and perspectives. In particular, [2] shows that human response is empathically modulated by the learned relations with others. Although there are similarities between Theory of Mind (ToM) and empathy, ToM mainly focuses on attributing mental states to others [1]. Based on [2], we design LASE, incentivizing the cooperative behavior of others through gifts, which is closely related to the emergence of cooperation in real world [3].

[1] De Waal, Frans BM, and Stephanie D. Preston. "Mammalian empathy: behavioral manifestations and neural basis." Nature Reviews Neuroscience 18.8 (2017): 498-509.

[2] Singer, Tania, et al. "Empathic neural responses are modulated by the perceived fairness of others." Nature 439.7075 (2006): 466-469.

[3] Yalcin, Ӧzge Nilay, and Steve DiPaola. "A computational model of empathy for interactive agents." Biologically inspired cognitive architectures 26 (2018): 20-25.

The real-world application

Our work employs decentralized learning to infer the influence of other agents on oneself in mixed-motive games, enabling broad applicability in complex real-world multi-agent interactions. A potential application scenario is multi-agent automated negotiation. With the rapid advancement and increasing application of machine learning and LLMs, we believe that in the future, agents may assist or even replace humans in various fields such as E-commerce for automated negotiations and autonomous driving.

For example, bargaining in e-commerce is a classic mixed-motive game scenario. In bargaining, both the buyer and seller share the common goal of reaching an agreement. However, each party also aims to maximize their own profit, leading to competition over the transaction price. Consequently, during this process, each side must infer the other's willingness to cooperate and adjust their negotiation strategy accordingly. If the other party shows a high willingness to cooperate, such as when a seller notices that the buyer is very eager to get the good, the seller can propose a higher price. Conversely, if the other side displays a very unfriendly attitude, the agent should consider making some concessions.

Another example is autonomous driving. When two vehicles traveling towards each other meet on a narrow road, they infer the other's manner based on its behavior. If the other one shows courtesy, the focal vehicle proceeds first; if the other one is brash or in a hurry, the focal agent yields to avoid conflict.

Clarify issues

Thanks for your valuable suggestions! The overall framework and flow of LASE as well as the relationship between each module can be seen in Figure 1 in the paper and Algorithm 1 in the appendix. For clarity, here we present the pseudocode for both Social Relationship Inference (SRI) and Gifting modules in detail:

# Trajectories collection
... ...

# SRI
for i in Agents:
	for j in Agents_without_i:
		predicted_obs = obs_conversion(i_obs)
		predicted_action_prob = SR_policy(predicted_obs)
		for j_virtual_action in all_possible_actions:
			SR_q_values.append(SR_value(i_obs, j_virtual_action + others_real_actions))
		counterfactual_baseline = predicted_action_prob @ SR_q_values
		w_ij = (SR_value(i_obs, real_joint_actions) - counterfactual_baseline) / M # Eq (3)
	w_ii = 1 - sum([w_ij for j in Agents_without_i])

# Gifting
for i in Agents:
	i_weighted_reward = 0
	for j in Agents:
		i_weighted_reward += w_ji * j_reward

# Training with weighted rewards
... ...

These two modules are executed after all agents have completed an entire episode of interaction with the environment, adjusting the rewards to influence subsequent RL training. Specifically, in SRI, each agent $i$ predicts another agent $j$’s action probability using Perspective Taking and assesses the impact of all possible actions of $𝑗$ on its $Q$-value using the SR value network. Then $i$ infers the social relationship $w^{ij}$ with $j$ using Equation 3.

After all agents have completed SRI, the Gifting module is responsible for distributing rewards. Specifically, the weighted reward for each agent $i$ is the sum of the gifted rewards $w^{ij}\times r^j$ from other agents $j$, including $i$’s own reward left to itself $w^{ii}\times r^i$.

Insight into counterfactual baseline design choice

Thank you very much for your suggestions on designing the counterfactual baseline! We apologize for the confusion we may have encountered while attempting to implement the three different baselines you mentioned. We are not entirely sure if we fully understood your points and would greatly appreciate the opportunity to discuss this matter in more detail to ensure we correctly implement your suggestions!

• Fixed neutral baseline: If you are referring to using a fixed hyperparameter as a neutral baseline, we believe that this approach may not be appropriate due to the continuously evolving $Q$-values throughout the RL learning process. If you are referring to the neutral value of $i$'s $Q$-values conditioned on the various possible actions of $j$, we think that our ablation study LASE w/o, where $\sum_{a_t^{j'}}\frac{1}{|\mathcal{A}^j|}Q^i(o^i_t,(a_t^{-j},a_t^{j'}))$ is used as a counterfactual baseline, has already addressed this.

• Learned cooperative behavior baselines: Are you referring to "cooperative behavior" as $\text{max}_{a^{j'}_t} Q^i(o^i_t, (a_t^{-j},a_t^{j'}))$? If so, based on Equation 3, this would lead to $w^{ij} \leq 0$, resulting in no gifting occurring. This would cause the algorithm to fail and revert to the A2C baseline.

• Worst-case action baselines: We interpret this baseline as $\text{min}_{a^{j'}_t}Q^i(o^i_t, (a_t^{-j},a_t^{j'}))$. We replace LASE's counterfactual baseline with the Worst-case Action Baseline (WAB), and the self-play results are as follows:

  Self-play	SSH	SSG	Coingame	Cleanup
  WAB	17.624	118.731	0.009	43.741
  LASE	18.948	117.784	38.736	33.467

  WAB outperforms LASE on SSG and Cleanup, which is because using $\text{min}_{a^{j'}_t}Q^i(o^i_t, (a_t^{-j},a_t^{j'}))$ as the baseline results in $w^{ij}\geq 0$. This overly optimistic estimation of social relationships promotes group cooperation in self-play settings, but it also raises the risk of exploitation when interacting with unknown agents. To demonstrate this, an adaptive experiment on WAB is conducted in the same manner as described in Section 6.3 of the paper. A WAB is trained with three rule-based agents and three A2C agents, respectively. We record the mean gifting weights assigned by the WAB to other agents and the rewards of the WAB after gifting:

  Gifting weight	Cooperator	Random	Defector	A2C_1	A2C_2	A2C_3
  WAB	0.321	0.254	0.184	0.266	0.221	0.222
  LASE	0.268	0.091	0.027	0.0391	0.1339	0.2194

  Reward after gifting	Rule based agents	A2C agents
  WAB	5.778	4.467
  LASE	15.129	8.752

  Due to WAB giving excessive gifts to other agents and not receiving reciprocation as it would in self-play, the actual reward obtained by WAB is significantly lower than that of LASE. This demonstrates the weakness of WAB being easily exploited.

  Additionally, we observe that in Coingame, the social relationship converged to $w^{12}\approx w^{21}\approx 1$, indicating that the agents' optimization objective shifted towards maximizing the opponent's rewards. In Coingame, only collecting coins of the opponent's color impacts its reward. Finally, the two agents converged to a behavior of not collecting any coins, demonstrating that excessive gifting is detrimental to learning in this environment.

Limited domain coverage

Thank you very much for your suggestions! As a standardized testing platform that integrates various environments including cooperation, competition, and mixed motives, MeltingPot can further assist in testing the scalability and robustness of algorithms. We will consider selecting some of these environments as experimental settings in our future work to enhance the persuasiveness of our paper.

Thank you again for your insightful comments! We will make corresponding changes in the revised version of the paper based on your and other reviewers’ suggestions, including but not limited to improving and clarifying the description of the algorithm, adding discussions and experiments on the counterfactual baseline, and expanding the experimental environments.