Thank you for taking the time to review our paper. Please see our responses below:

• Variance in iterates: as also noted in our response to reviewer c4Mz, MeVa does not rely on REINFORCE in principle. We did not use any REINFORCE gradients for the experiments because these games can be differentiated exactly. Thus the results in Tables 2 and 3 are not subject to REINFORCE variance. However, we did note that our $U$ formulation requires a way of estimating $\nabla f$, and REINFORCE is the obvious choice. REINFORCE is in no way required though, as one could also use a model $\hat{f}$ to estimate $f$ and approximate $\nabla f$ by $\nabla \hat{f}$ (this is what the $V$ formulation with $\gamma=0$ would provide).

• Ablations: unfortunately Eq 5 can not be used directly, as the rollouts $x_{>t}$ need to come from the meta-policy itself. Eq 5 (and 6 for that matter) are implicit, and can only be made explicit by fitting a model. We will however include a comparison (on the matrix games) to MetaMAPG, which can be seen as using Eq 5 but evaluating the expectation using naive learning as the meta-policy for the rollouts. For a fair comparison we will modify MetaMAPG to use exact gradients rather than policy gradients, which is the M-MAML variant discussed in the M-FOS paper.

• Exploration: our exploration is done on the meta level, where actions are continuous, and it is not obvious what form the "default" exploration should take. The typical $\epsilon$-greedy approach is undefined, as there is no uniform distribution on $\mathbb{R}^n$. We could use the uniform distribution on the sphere to get uniform random directions, but then it is unclear how to scale these. We could add $\mathcal{N}(0,\epsilon^2)$ noise to $\bar{\nabla}\hat{V}$, however this scheme seems problematic in general as different parameters can have wildly different sensitivities (think neural networks), in which case we do not expect perturbing them with the same variance will lead to meaningful diversity. Nevertheless, we will rerun the ablation with this additive gaussian noise as the control.

• We can reorganize related work, but please hear us out on the current structure:

  • We discuss LOLA front and center because it is the seminal work in this area.
  • In Section 1 we mention MetaMAPG and M-FOS because they are meta-level approaches like MeVa.
  • In Section 2.2 we go over the idea of looking ahead, the inconsistency in LOLA and how it may be resolved. Here HOLA and COLA are briefly relevant.
  • In Section 2.3 we discuss looking further ahead, which brings in the meta-game. Here Meta-(MA)PG and M-FOS become relevant.
  • In Section 3.1 we introduce MeVa, and relate it to MetaMAPG and M-FOS.

  We propose to make two improvements:

  • We will add discussion of LOLA and COLA in Section 3.1.
  • We will discuss COLA at the first mention of consistency ("It is self-consistent: ..."), which gives us an opportunity to clarify what is meant by (in)consistency.

Thank you.

Thank you for your review. Please see our responses below:

• Reliance on REINFORCE: MeVa does not rely on REINFORCE in principle. We did not use any REINFORCE gradients for the experiments because these games can be differentiated exactly. However, we did note that our $U$ formulation requires a way of estimating $\nabla f$, and REINFORCE is an obvious choice. REINFORCE is in no way required though, as one could also use a model $\hat{f}$ to estimate $f$ and approximate $\nabla f$ by $\nabla \hat{f}$ (this is what the $V$ formulation with $\gamma=0$ would provide).
• Do the improvements in the paper lead LOLA-based methods closer to scaling to a feasible solution? Yes and no. Our method requires significantly more data and computation upfront to learn the value function. However we solve two issues with LOLA that could very well keep it from scaling to neural networks:
  • LOLA's extrapolation rate $\alpha$ needs to be large. Large learning rates don't work with neural networks. MeVa's discount rate $\gamma$ provides a different way of looking further ahead than just enlarging the learning rate, which all else being equal should be helpful for neural network policies.
  • LOLA's inconsistency (the assumption of naivety) is a manifestation of the exact issue it sought to address in naive learning (inconsistency due to the assumption of stationarity). In a sense, we feel that MeVa is the logical conclusion of the idea of LOLA.
• The generalization of our method to more than two agents is straightforward: each agent learns their own meta-value approximation and follows its gradients. How the practical implementation will scale computationally and datawise with the number of agents is another matter, although we have no reason to believe it will scale particularly poorly.
• Yes, M-FOS and MeVa are equivalent in performance, and if one had to choose, one would choose M-FOS for its scalability. However, M-FOS' performance on the coin game is not stellar, so there is value in exploring a variety of solutions to the same problem. One interesting takeaway is that our restriction to local updates did not hurt performance -- MeVa is just as well able to dynamically exploit naive and LOLA learners as M-FOS is.
• "We argue that this throws out the baby with the bathwater" -- we will remove this line.

Thanks again for your time!

We would like to provide an important clarification regarding the final sentence of your summary:

Unfortunately, head-to-head results between meta-values and M-FOS are not presented due to computational restrictions.

The restrictions are not computational but methodological; essentially it is not clear how to perform the experiment fairly. It is easy to compare LOLA vs LOLA head-to-head because the meta-policy (LOLA) does not have any parameters to be learned upfront. M-FOS and MeVa do have parameters, and comparing M-FOS vs LOLA (or MeVa vs LOLA) involves training M-FOS (or MeVa) with optimization trajectories so they can adapt to LOLA's behavior.

Comparing M-FOS vs MeVa head-to-head requires training them jointly. M-FOS and MeVa learn meta-policies; let us call the process by which they learn these meta-policies ``meta-meta-policies'' (concretely, naive learning). The purpose of a head-to-head evaluation of meta-policies (e.g. LOLA vs LOLA) is to learn something about the interaction between these meta-policies. However, when trained jointly, these meta-policies are moving targets, and whatever we observe will be confounded by the interactions of the meta-meta-policies.

Moreover, we would wish to choose hyperparameters for each method, in order to give both a fair opportunity. Do we choose the ones that make MeVa look good, or the ones that make M-FOS look good? Or do we choose the ones that make both look equally good? At the risk of carrying the terminology too far, we might call the hyperparameter optimization scheme a meta-meta-meta-policy. It's two-player general-sum games all the way up.

Thank you for a very thorough review. We believe several of the issues raised are the result of misunderstandings, we hope to clear them up below.

• Section 1: (Clarity around proposed improvements to LOLA):

  The point is unclear rather than redundant. In the context of LOLA, "consistency" means that the optimization process in the simulated updates is the same as that in the true updates. In LOLA, the simulated update is a naive one, while the true update is a LOLA update. In naive learning, the simulated update is no update at all, while the true update is a naive update.

  LOLA exhibits two kinds of inconsistency:

  • The simulated update is naive when the true update is a LOLA update.
  • The simulated optimization process only optimizes the opponent and not the agent itself (equivalently: the agent itself is simulated to not learn at all).

  We swept this second kind under the rug in the name of simplicity: LOLA as described by Eqn (2) only exhibits the first kind of inconsistency, while LOLA proper (given in Appendix B) exhibits both. Reviewer rws5 also appeared to be confused by this, so we will make it explicit, ideally reworking the equations to be written from a single player's perspective like in MetaMAPG. Apologies for the confusion.

  We would moreover like to clarify that consistency in this sense is not related to convergence of the algorithm. It also does not mean invariance to the opponent's algorithm; a consistent surrogate explicitly needs to account for the opponent's algorithm. It may perhaps be construed as an equivariance.

• Section 1 (Clarity around Extrapolation): Extrapolation is used in the sense of forecasting the future (of the optimization process). We will make sure to clarify this in the paper as it is indeed a pretty overloaded term.

• Section 2.2: Good point, we will add more discussion for both the Logistic game and the matrix games.

• Section 2.3 (M-FOS Description): You say "learning an arbitrary meta-policy from this value function should be seen as "learning with learning awareness" rather than merely acting"; we think this mixes up levels of learning. Learning an arbitrary meta-policy would be "learning to act with learning awareness", whereas learning a local meta-policy is "learning to learn with learning awareness". We will clarify this in the paper, as it is a subtle point.

• Section 3 (In place of implicit gradients): it is used to update the policies $x$ in the inner learning process. We use $V$ as a surrogate for $f$; it is a game on which naive learning is naive-learning-aware.

• Section 4.1 (Grounding): the practical difference between the $U$ and $V$ formulations is that the $U$ formulation provides useful gradients from the start. Whereas $\nabla V$ relies entirely on the model, $\nabla f + \gamma \nabla U$ sort of "defaults" to the naive gradient $\nabla f$ and learns to modify it eventually. In practice it appears the gradients $\nabla V$ (and $\nabla U$) start out nearly zero, and falling back to the naive gradient allows the system to discover a (minimal) variety of policies. In the process of fitting these policies, the model starts producing more useful gradients.

  One failure mode that we have observed with $V$ is a tendency for policies to become overly cooperative. This is a consequence of the TD error $f(x) + \gamma V(x') - V(x)$ (where $x'=x+\alpha \nabla V(x)$) having minimal weight on the ground truth term $f(x)$ and comparatively large weight (effectively $\frac{\gamma}{1-\gamma}$) on the bootstrapped return estimate $V(x')$. With $V$, the $f(x)$ term in the TD error is the only way in which ground truth enters the system. With $U$, the ground truth additionally enters the system through the use of $\nabla f$ in the optimization process. This is the main sense in which we say $U$ is more strongly grounded in $f$.

  Evidently, we need to discuss this more explicitly in the paper.

• Section 4.3 (Exploration): There may be a misunderstanding -- we do not apply the noise to the inner policy ($x$) but to the meta-policy ($V$). A perturbation to the meta-policy will cause the meta-rollouts ($x$ optimization trajectories) to go in systematically different directions than they otherwise would. By flipping signs on the learned features in terms of which $V$ predicts the value, we hope to make those directions semantically meaningful. Adding gaussian noise to the parameters of $V$ would potentially do a similar thing, however it is unclear how to scale such noise especially when parameters have an extremely wide range of different sensitivities. Multiplicative noise seems more appropriate to us.

• Section 3 and 4 (what exactly is the meta-MDP): It is indeed not episodic. We will fully specify the meta-MDP in the paper. Most of the pieces are there in the prose in Sec 2.3, except the action space which we left vague because it differs between the prior work and ours (where it is local and depends on the state).

Thank you!

Thank you for your review. Please see our responses below.

• Theoretical Guarantees: we do briefly discuss convergence at the end of Section 3.2. We also expected to be able to inherit convergence results from Q-learning. Unfortunately, multi-agent Q-learning is not known to converge even in the discrete/tabular case. The Bellman operator is generally not a contractive map when multiple agents maximize different objectives.
• Experiments: we will include a comparison to MetaMAPG on the matrix games. This will take a few days to come together. The M-FOS authors (Sec 5.1) note that MetaMAPG does not scale beyond 7 updates, and use a variant based on exact gradients. We will follow them in this regard.
• Reproducibility: apologies, we should have submitted the code here on OpenReview. It is available at https://github.com/MetaValueLearning/MetaValueLearning .
• The targeted scenarios are somewhat restrictive:
  • We present the method in two-player form for simplicity, but the generalization to n players is straightforward: each player learns and optimizes their own meta-value.
  • We are not sure why you say the method is tailored to zero-sum meta-games; that does not appear to us to be the case. E.g. on IPD, some meta-policies lead to defection whereas others lead to cooperation, with different total payoffs.
  • It is true that we assume access to opponent parameters. We believe working on top of opponent models is entirely possible, but the science is better done in a controlled setting. Even there, the nonstationarity problem of MARL is far from solved.
• In two-player zero-sum games, might some baseline methods that compute equilibria also be included in the comparison experiments? The only zero-sum game we consider is Iterated Matching Pennies, and it only has a single Nash equilibrium at the uniform policy pair. Naive learning finds it without issue, so it's unclear what these other algorithms would add. Could you elaborate?

We hope to have addressed some of your concerns. Thank you!

Thank you for your review. We respond to the issues raised below.

Weaknesses:

1. It is true that the complexity of the problem increases with the number of agents, however it is not clear that this is a weakness of our method, as opposed to an inherent fact of life in MARL.
2. Another reviewer also raised a concern about the clarity of the notation in Section 2. We will revisit the notation and ensure the meta-MDP is fully defined.
3. The current algorithm is hard to follow because it was written to have full details regarding the application of the practical techniques. We will move this to an appendix and provide a simplified, readable presentation in the main text.

Questions:

1. Our words were confusing and we will rewrite them. To see the gap, consider two LOLA agents A and B training against each other. A assumes B uses naive learning, when in truth B uses LOLA as well. And similarly, B assumes A uses naive learning, when A actually uses LOLA. In a sense, both players think they are smarter than their opponent. This is the gap. The same holds in naive learning, where each agent assumes the other to not be learning at all. And the same holds in HOLA2 when A assumes B uses LOLA and B assumes A uses LOLA, when in fact they are both using HOLA2.
2. Two agents that follow the gradient of (3) make the correct assumption about how optimization would proceed, namely that both agents follow the gradient of (3). (Our notation in Section 2 is somewhat simplified and does not reflect LOLA's original proposal to only imagine the opponent's update; instead we consider the agents to be aware of their own learning as well. Ignoring the agent's own update would be a different, unrelated source of inconsistency.)

We hope that clears things up.