Contracting with a Learning Agent

Thank you for your feedback! We address the points raised in the review below.

Win-win dynamics: In general, win-win situations arise in “general-sum” games, where the players are not complete adversaries, but rather can increase and share the overall welfare. In the constructions we used for demonstrating win-win dynamics, the structure was that different agent actions resulted in different levels of welfare. Thus, both players were better off if the principal could initially invest to create a strong motivation to play high-welfare actions. More broadly, one takeaway is that win-win scenarios, at least from our example, can be those where such an investment has an increasing return in terms of welfare.

Multiple agents: The question of how to optimally incentivize multiple learning agents is an interesting one to consider in future work. A main challenge is that if agents jointly produce an output—meaning the payoff for the principal is a function of the agents’ joint action profile—it may become complicated to disentangle their contributions and incentivize them. It seems plausible that if agents’ contributions to the payoff are additive and contracts are linear (one alpha per agent), some multi-agent generalization of free-fall contracts might still be optimal. However, we have not analyzed this scenario. Other issues that may arise in interaction between multiple agents that one would need to consider in the model are multiplicity of equilibria and free-riding problems between the agents.

Different learning algorithms: We agree that studying different types of learning approaches is interesting. Which type of learning is a good question. Some approaches may eliminate the possibility of free-fall contracts, but could have other limitations. We address below the two potential directions mentioned in the review. We also note that no-swap-regret is known to be the right notion for non-manipulability and results in the optimal static contract remaining optimal.

Bounded memory: If we think about agents whose response at time $t+1$ is only a function of the history of steps $[t-k, t]$, with some constant $k$, other issues may arise, which are not necessarily in the benefit of the agent. This is related to the fact that such agents are not regret-minimizing. An extreme example is when $k=1$. Here, the principal can alternate between incentivizing some good action and paying zero in the following step. The principal ends up not paying anything , but the agent is always responding with a lag of one step and so he plays the good action half of the time. This kind of example is, in fact, not pathological, but could be extended to larger memory size.

Agent who is aware of the principals strategy: In this case we need to think through how the agent reasons about their strategies in the repeated game, and are we still thinking about some notion of learning, or about general strategies in the repeated game. The latter case has been studied in prior literature, as we discuss in the introduction, and this was not our focus. A potentially related notion from online learning that one could think about in the context of the first case is contracting with an agent who is minimizing policy regret. This is an interesting direction to look into.

Pareto-optimal contracts: The current solution, which maximizes the principal’s utility, is already Pareto-optimal (PO). If we consider PO contracts more generally, the analysis would include contracts that balance utilities between the principal and agent. We suspect that for the classic contracts setting, the entire PO curve can be recovered using similar linear programs to those used to compute the principal-utility-maximizing contract. When moving to the mean-based agent setting, our results already show that the PO curve is improved (we find a point with higher principal utility). However, there will be part of the PO curve that favors the agent and is bound by the total welfare, which cannot be improved. This is an interesting question to more carefully consider.

Real-world interpretation: Our results could be interpreted in two main directions: from the perspective of the agent or the principal. On the agent’s side, a potential message is that in repeated contract scenarios, one should be careful in choosing which learning algorithms to implement. Using simple off-the-shelf options like Multiplicative Weights or Follow the Perturbed Leader may make the agent susceptible to exploitation by a sophisticated principal. On the principal’s side, the results show that one can do better by considering dynamic contracts, especially if there is reason to believe that the agent is using simple learning strategies or has some delay in response.

Conditions related to win-win scenarios: We currently do not have a full characterization of the conditions for win-win scenarios. We can think of some such conditions. For example, “all actions have the same welfare” makes it impossible, because principal utility and agent utility sum to welfare and without increasing the welfare pie, principal gains come at the cost of agent losses. On the other hand, our analysis shows that the agent’s utility from a freefall contract is as if the final (stopping) action was played the entire time, so we know sufficient conditions need to imply that this stopping action has higher agent utility than the static contract’s induced action. This is an interesting question for future research.

Learning in contracts with MDP models: While MDPs have been studied in contract settings. In the work we mention in the paper, the MDPs are either due to the principal having long-term constraints [reference 9 in the paper], or due to an underlying evolving state of the world [reference 53 in the paper], but not with learning agents. We have not considered the direction of learning agents with states. It would be an interesting direction for future research.

Thank you for your feedback! We will further extend our discussion of the learning models, and in particular, the comparison between mean-based regret minimization and no-swap regret. Please see also our responses to the other reviews regarding this point.

Thank you for your feedback! We address the main point raised in the review below.

Mean-based learning and free-fall contracts: We completely agree that—knowing now the exploitability of mean-based learning agents—studying different types of learning approaches, such as learning with recency bias or loss aversion, would be an important direction to explore. Such approaches can be realistic for human learners while preventing the principal from implementing free-fall contracts.

There is, however, significant interest in establishing clear results for mean-based learning in our setting. First, we note that the result that a principal can exploit mean-based regret-minimizing learners in a large class of contract games using a very simple family of contracts (free fall) is not trivial and certainly not obvious before doing an analysis. Furthermore (see also our response to review HskF), our results show that it is not always better to use smarter learning strategies (after accounting for how the principal might adjust their dynamic contract). While this analysis (Theorem 3.2) uses a particular construction, it demonstrates that whether it is beneficial to use learning and which type of learning to use is more subtle and depends on the details of the game.

Additionally, as also suggested in the review, with human agents, one could think of free-fall contracts as happening over at least limited time spans. There are lines of work in economics studying delayed or insufficient responses to new information (see [1,2,3] below), or even limited learning over extended time periods [4]. Essentially, a decision that was extremely good for a long period of time may have momentum and will not always be abandoned in the first (say) month of a bad outcome. Mean-based regret minimization is a form of learning with these properties.

We would like to emphasize, however, that we think there is value in our analysis regardless of whether mean-based regret minimization is the way human agents act. Mean-based regret minimization is a prominent algorithmic approach to learning in repeated games, which has been extensively studied in other contexts. In particular, algorithms of this family have been traditionally motivated in the theoretical economics literature as “simple,” “adaptive,” and “natural” theoretical models of learning by boundedly rational agents (see, e.g., [5] and references therein). It is thus important to understand the implications of such learning approaches for repeated contracts, especially in comparison to other types of learning and to static contract setting. In this sense, our analysis of mean-based and no-swap regret learners in this paper is a first step in exploring contracts with learning agents more broadly.

[1] Carroll, C.D., Crawley, E., Slacalek, J., Tokuoka, K. and White, M.N., 2020. Sticky expectations and consumption dynamics. American economic journal: macroeconomics, 12(3), pp.40-76.

[2] Bouchaud, J.P., Krueger, P., Landier, A. and Thesmar, D., 2019. Sticky expectations and the profitability anomaly. The Journal of Finance, 74(2), pp.639-674.

[3] Ba, C., Bohren, J.A. and Imas, A., 2022. Over-and underreaction to information. Available at SSRN 4274617.

[4] Benjamin, D.J., Rabin, M. and Raymond, C., 2012. A Model of Non-belief in the Law of Large Numbers. Available at SSRN 1945916.

[5] Hart, S. and Mas-Colell, A., 2013. Simple adaptive strategies: from regret-matching to uncoupled dynamics (Vol. 4). World Scientific.

Thank you for your feedback! We address the points raised in the review below.

Bandit feedback: One possibility for bandit feedback is when the agent does not observe the contract $p_t$ but only observes the payoff induced by this contract for the action that was played. A different scenario is when the agent may observe the contract but does not know the outcome distributions or what they imply for the actions that were not played, which again requires some additional exploration. Our analysis, however, holds for full feedback or any of these partial feedback scenarios.

Theorem 3.2: The theorem shows that there is a positive measure of contract games where learning by the agent and an optimal dynamic contract by the principal lead to a strict Pareto improvement (win-win) outcome. This is, however, not true for all contract games; there are games where learning reduces a player’s utility compared to a myopic best response (for instance, the example in the introduction). The construction in Theorem 3.2 shows that there are cases where using either a best-response strategy or, alternatively, smarter learning (no-swap-regret) leads to a loss compared to using simpler learning (mean-based learning; see our remark at the end of Section 3). So, whether it is beneficial to use learning, and which type of learning to use, depends on the game.

Theorem 4.3: Whether or not $(\epsilon,1)$ is feasible just depends on whether $(1 + \epsilon)$ is less than the ratio between the optimal (known-time horizon) dynamic contract and the optimal static contract; for any problem there is some sufficiently large $\epsilon$ to make $(\epsilon, 1)$ infeasible. The theorem statement is perhaps easier to conceptualize by considering its contrapositive: if there is an $\epsilon > 0$ such that $(\epsilon, 1)$ is feasible, then for any $\gamma > 1$ there exists an $\epsilon_\gamma$ such that $(\epsilon_\gamma, \gamma)$ is feasible as well. We will provide additional context for this theorem to better clarify the situation.