The (co)limit of metabeliefs

We thank the reviewer for their comments.

To respond to the questions: (1) Real-world implications of our theory, would be, for example, to choose the appropriate level of metabeliefs for specific problems, offer guarantees on approximation given a particular choice of level, and reason about compositions of problems.
(2) We are unaware of any comparable effort to formulate the problem of metabeliefs. If the reviewer has specific suggestions, those would be appreciated.
(3) Yes, we do plan to write subsequent papers on specific applications! As suggested in the introduction, we believe there are quite significant implications across learning, reinforcement learning, and social inference. In each case, engaging with the literate will require additional notion and technical machinery, and we therefore believe it best to tackle these applications in subsequent papers. If the reviewer raises their rating a bit, that would help us move faster toward that goal!

Thanks again!

We thank the reviewer for their comments. To respond to the reviewer's question, we are not aware of any comparable methods of modeling metabeliefs to which we may compare. If the reviewer has specific examples in mind, those would be appreciated.

We agree that understanding how the colimit might be useful in machine learning is important. We believe that applications are beyond the scope of the current paper, given the already significant technical content. To improve the clarity of the connection, we have elaborated the example in Section 4: Computing distance in the colimit: rock-paper-scissors. In particular, we give the solution to the $3\times 3$ optimization problem, and simplify the integration to remove the infima. We hope this helps make potential connections more clear.

We wish to thank the reviewer for their question. To respond directly: The reviewer is incorrect, Nash equilibria are different from colimit spaces induced by metabeliefs. Specifically, the question at the end of paragraph 4 is ``Can we consider beliefs at all levels simultaneously?''. Beliefs are different from strategy. Nash equilibria assume away the problem of beliefs and this is one reason that such equilibria are not effective models of behavior. Our contribution is to offer a precise mathematical formalization of reasoning about metabeliefs of all levels.

More technically:

• The reviewer mentions that ``the limit $P^{\infty}(\mathcal{X})$ is the fixed point which is the definition of a Nash equilibrum...." The space $P^{\infty}(\mathcal{X})$ here is a topological space consisting of every distribution in $\mathcal{P}^n(\mathcal{X})$ for every finite $n$, as well as points of $\mathcal{X}$, modulo identifying the embedding of each $\mathcal{P}^n(\mathcal{X})$ into $\mathcal{P}^{n+1}(\mathcal{X})$. To say that this space is a Nash equilbrium does not make sense: the space is a collection of equivalence classes with a topology imposed on them, a Nash equilibrium is a set of strategies for each player in a non-cooperative game such that each player has no preference for switching strategies.
• The setting of Nash equilibria applies to non-cooperative games in which each player knows the equilibria strategies of the other players. Our setting is applicable to non-cooperative games, cooperative games, and things that are not games, and is focused in the rock-paper-scissors example on Bob not knowing Alice's strategy, her equilibrium strategy, or even exactly what her space of strategies might be.
• In order to fix the confusion surrounding the rock-paper-scissors example, we have changed "In order for Bob to generate his own strategy, he must consider what he knows...." from the example on page 4 to the following: "In order for Bob to generate his own strategy, he may consider what he knows...."

We apologize for any confusion created by our wording, and thank the reviewer for their comments.

The point of that Nash equilibrium is that we don't need or want to do the infinite regress of meta-beliefs. It is not as though the people who have studied this over the decades did not explore the tree of meta-beliefs; it's just that it is too difficult and not needed, "every distribution... for every finite n" is the wrong thing to do. Every finite set of beliefs about beliefs can give the wrong answer!! (Rock-paper-scissors is too symmetric to explain the problem; I found a good example at https://artint.info/3e/html/ArtInt3e.Ch14.S4.html). In such examples, every finite sequence of meta beliefs gives an arbitrary answer. It's the fixed point / equilibrium, not the space of all finite parts, that forms the rational belief. An agent doesn't need to do the reasoning about meta-beliefs to arbitrary depths, which you just assume is a useful thing to do. "Nash equilibria assume away the problem of beliefs" - precisely because the infinite regress is the wrong thing to do (and finite subparts are even more useless). In the paper "In practice, metabeliefs are typically truncated to a finite sequence." isn't true; it is typical that the equilibrium is used (discounting means finite part is a good approximation to the equilibrium).

In a Nash equilibria an agent doesn't need to know the strategies of the other players (perhaps selecting which Nash equilibrium, but is there is only one then there is no reason to deviate). There are related equilibria (such as correlated equilibria) for other games (where "game" is a very general concept of multi-agent interaction). There is a reason they rejected the explicit (infinite) regress of meta-beliefs.

The motivation of the paper seemed to be mis-guided. Perhaps there is a venue for this paper, but ICLR isn't it. (Perhaps KR or AAMAS).

I don't disagree that agents need to reason about the probability of the other agents actions. The other agent has beliefs that result in a distribution over their actions. It is that distribution over actions that an agent needs to find the best response to. None of your examples seems to justify beliefs about beliefs to an arbitrary depth.

There is a reason why there is not a recent literature on meta-beliefs to arbitrary depths. It doesn't work and we have much better (and simpler!) solutions.

We wish to thank the reviewer for their enthusiastic support and helpful and detailed feedback. We have addressed the reviewer's two main questions as follows:

• On page 3 in Lemma 2, we have added the following clarification for $n=0$: In place of "...$\mu$ is a unique distribution in $\mathcal{P}^n(\mathcal{X})$ such that $\mu$ is not a single-atom Dirac distribution." We replace with: "...$\mu$ is a unique distribution in $\mathcal{P}^n(\mathcal{X})$ such that $\mu$ is not a single-atom Dirac distribution or $\mu$ is a unique point in $\mathcal{X}$ (recall that $\delta^0(x)=\delta_x$ is the Dirac distribution in $\mathcal{P}(\mathcal{X})$ with support equal to $\{x\}$)."
• $W_p^{\infty}$ is indeed positive definite. We have added this proof to the proof of Lemma 6 given in the appendix: ``Let $\mu\in\mathcal{P}_p^m(\mathcal{X})$ and $\nu\in\mathcal{P}_p^n(\mathcal{X})$ with $m\geq n$ such that $\mu$ and $\nu$ are not single-atom Dirac distributions (so they could be either different distributions or points in $\mathcal{X}$). By definition, $W_p^{\infty}([\mu],[\nu])=W_p^m(\mu,\nu_m)$, where $\nu_m$ is the unique element in the intersection $[\nu]\cap\mathcal{P}_p^m$. Then because $W_p^m$ is a metric, $W_p^m(\mu,\nu_m)\geq 0$ with $W_p^m(\mu,\nu_m)=0$ if and only if $\mu=\nu_m$. However, if $\mu=\nu_m$, then because we have chosen $m$ so that $\mu$ is not a single-atom Dirac distribution, $\nu_m$ is not a single-atom Dirac distribution. Therefore since $\nu_m\in [\nu]$ and $[\nu]$ is the unique non-single-atom Dirac distribution (or point in $\mathcal{X}$) in $[\nu]$, we have $\mu=\nu_m=\nu$."

We also fix all of the typos in the list provided, and wish to once again thank the reviewer for providing such a detailed list. Here are some more detailed responses to a few of the small remarks:

• The T1 claim may be found in Parthasarathy (2005) "Introduction to Probability and Measure", though the statement can also be amended to change T1 to, say, locally compact and metrizable, and the statement is easy to prove directly.
• Many of the applications (particularly to machine learning) implicitly use compact spaces; however in the statement of part 4 of Theorem 1, "compact" can be changed to "paracompact," and the statement is still true (albeit weaker). We have amended this.
• We have fixed the right-hand side of Equation 10 by removing the unnecessary and confusing portions of the notation: the beginning and ending curly braces and the $\nu\in\mathcal{P}_p^i(\mathcal{X})$.
• $\Delta^i$ commutes with $\bigcup$ since for any function $f$ and sets $A,B$, $f(A\cup B)=f(A)\cup f(B)$. To make the passage from equation 11 to equation 12 more clear, we have added a few lines to the proof (including why $\Delta^i$ commutes with $\bigcap$). This addition begins immediately following Equation 11, "Applying $\Delta^i$..." and continues until just before Equation 14, terminating with, "Now, Equation 13 becomes:".