Modeling Boundedly Rational Agents with Latent Inference Budgets

Thank you for your review!

Clarity and Presentation:

We will try to improve clarity of presentation in our revision, specifically as it relates to the plots. Are there specific aspects that you would like to see improved?

Lack of Thorough Literature Review:

We discuss related work in section 2 as well as several others in sections 4-6. Are there papers that you think are missing or warrant further discussion?

Quality of Experiments:

We have included experiments in 3 very different domains as well as relevant state-of-the-art approaches for each of the domains. Are there specific comparisons with other models of bounded rationality that you think are missing?

Modeling of $\beta$:

This comment is not quite right: our approach does allow beta to vary per-step. However, we currently model these choices as independent; we do think that explicitly modeling state-dependence of beta is an interesting future direction.

Ambiguity in Parameter Inference:

$\eta$ represents the parameters of the model that produces the distribution over $\beta$. In all 3 domains, this denotes the parameters of a simple feedforward network which are learnt using gradient descent. We will make this clear in our revision.

Robustness Concerns:

We already include ablations in each of the three sections: Imitation Learning, Imitation Learning + Fixed Inference, Imitation Learning + Boltzmann Rational Inference and finally, Imitation Learning + Runtime Budget Inference (which is our method). As evidenced by large differences in task success, there is a lot of variability in the real chess and language data, which in particular includes human players making weird or improbable moves in the lowest skill buckets. Nonetheless, L-IBMs are just as accurate in these buckets as the others.

Literature Review and future direction:

Discussed above.

Practical Implications:

We find this line of work to be an important contribution to the cognitive science and linguistics community as evidenced by the recent Nature paper [1]. Another key motivation for us was towards the development of better Human-AI systems. Like recent works like Diplomacy [2] points out, being able to model human behavior better would allow us to build AI systems (even, those that involve LLMs) that can better understand sub-optimal human behavior and therefore better cooperate with humans.

[1] van Opheusden, Bas, et al. "Expertise increases planning depth in human gameplay." Nature (2023): 1-6.

[2] Meta Fundamental AI Research Diplomacy Team (FAIR), et al. "Human-level play in the game of Diplomacy by combining language models with strategic reasoning." Science 378.6624 (2022): 1067-1074.

Computational Complexity:

Learning of $\eta$ is discussed above. We note that our method is more efficient than the alternatives as multiple values of $\beta$ can be evaluated as quickly as we can evaluate just one, because of the fact that we consider any-time algorithms. We also use stochastic gradient descent to learn $\eta$ and use inference algorithms like MCTS, RSA and BFS that are known to scale well in large-scale domains.

Thank you for your review!

Improving clarity:

Are there specific decisions that you would like us to elaborate on? We will work towards improving clarity of our presentation in our revision.

Comparison of other proposals:

Are there other alternative general proposals for these domains that you would like for us to compare to? In both Colors-in-context and Chess, we're comparing to what we believe to be the current state-of-the-art model. (In Chess, this is not just inferring a temperature parameter.)

Figure 1:

Fig. 1 was meant to be conceptual but we will improve the clarity of our assumptions in the next revision.

Typo:

Thank you for catching the typo. We have fixed this in our revision.

RSA:

In colors-in-context, we consider the case where runtime is truncated at a fixed # of iterations. You can do more RSA iterations if you want, but on this task as past work [1, 2] has shown it doesn't make a difference.

[1] McDowell, Bill, and Noah Goodman. "Learning from omission." Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.

[2] Monroe, Will, et al. "Colors in context: A pragmatic neural model for grounded language understanding." Transactions of the Association for Computational Linguistics 5 (2017): 325-338.

Figure 6:

We have improved Figure 6 in our current revision. Please let us know if it is better.

Thank you for your review!

Introducing anytime inference algorithms earlier:

Thank you for the suggestion! We will introduce anytime algorithms earlier in the paper to make it easier to follow as well as work towards improving the clarity of our presentation in our next revision.

Magnitude of improvement:

1. Chess being a sequential game, small improvements in per-step prediction accuracies can often be significant over the course of the game due to cascading errors.
2. In scientific applications, we often care about inferring skill parameters for their own sake, and not just to improve predictive accuracy. In this sense, we view it as a strength of L-IBM that it can infer these population-level parameter differences from only small differences in behavior. Indeed, in both language understanding & chess, we're not aware of past work that can automatically derive these differences from behavior alone; the ability to do so has direct applications in cognitive science [1] and computational linguistics [2].

[1] van Opheusden, Bas, et al. "Expertise increases planning depth in human gameplay." Nature (2023): 1-6.

[2] Franke, Michael, and Judith Degen. "Reasoning in reference games: Individual-vs. population-level probabilistic modeling." PloS one 11.5 (2016): e0154854.

Applications:

In addition to the relevance to the RL community that you pointed out, we also find this line of work to be important to the cognitive science and linguistics community as evidenced by a recent Nature paper [1] and others [2]. As it relates to the ML community, one of our key motivations was towards the development of better Human-AI systems. Like recent works like Diplomacy [3] points out, being able to model human behaviour better would allow us to build AI systems that can better understand sub-optimal human behaviour and therefore better cooperate with humans.

[1] van Opheusden, Bas, et al. "Expertise increases planning depth in human gameplay." Nature (2023): 1-6.

[2] Jeon, Hong Jun, Smitha Milli, and Anca Dragan. "Reward-rational (implicit) choice: A unifying formalism for reward learning." Advances in Neural Information Processing Systems 33 (2020): 4415-4426.

[3] Meta Fundamental AI Research Diplomacy Team (FAIR), et al. "Human-level play in the game of Diplomacy by combining language models with strategic reasoning." Science 378.6624 (2022): 1067-1074.

How hard is it in general to predict the player’s next move in chess?:

The task of predicting the player's next move in chess is still a challenging and open problem given the size of the action spaces as well as other components that influence them like player styles and partner styles. Note, that even state of the art models in chess [1, 2] are still far from perfect accuracy.

[1] McIlroy-Young, Reid, et al. "Aligning superhuman ai with human behavior: Chess as a model system." Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020.

[2] Jacob, Athul Paul, et al. "Modeling strong and human-like gameplay with KL-regularized search." International Conference on Machine Learning. PMLR, 2022.

Is there an extension of anytime inference that can handle “amortized computations” like dynamic programming for value functions or other pre-planning methods?

Yes, and we're doing it already in the first experiment. We model maze exploration using BFS; BFS is just a special case of Dijkstra's algorithm, which is a classic dynamic programming algorithm. We can think of the inference states $f_i$ as holding the state of a dynamic program. In fact, value iteration is an anytime algorithm as well, where the inference states $f_i$ holds the current values for all states.

What did you practically use to implement the inference procedures in this paper? I think the methods would be useful to comment on a bit.

Maze Domain:

We implemented truncated BFS using Pytorch and Numpy and we also used the mazelib library to generate the data.

Colors-in-context Domain:

We trained the models using Pytorch and Huggingface libraries. We specifically implemented RSA using Pytorch.

Chess Domain:

For chess, the base policy and value functions were trained using Ray library and Pytorch. MCTS was specifically implemented using Numpy. We also used the pettingzoo library for simulating moves.

We have added these details to our revision and also plan to release all of our code publicly.

Thanks for the review!

Accuracy differences:

1. Although the performance improvements are small in an absolute sense, in chess (which is sequential in nature), these improvements at a per-move level can quickly cascade at a trajectory level.
2. In scientific applications, we often care about inferring skill parameters for their own sake, and not just to improve predictive accuracy. In this sense, we view it as a strength of L-IBM that it can infer these population-level parameter differences from only small differences in behavior. Indeed, in both language understanding & chess, we're not aware of past work that can automatically derive these differences from behavior alone; the ability to do so has direct applications in cognitive science (e.g., [1]) and computational linguistics (e.g., [2]).

[1] van Opheusden, Bas, et al. "Expertise increases planning depth in human gameplay." Nature (2023): 1-6.

[2] Franke, Michael, and Judith Degen. "Reasoning in reference games: Individual-vs. population-level probabilistic modeling." PloS one 11.5 (2016): e0154854.

Modeling Assumption:

1. The key underlying assumption is that the inference procedure is an anytime algorithm. While not allowing arbitrary inference procedures, this still accommodates a wide variety of algorithms that are used in practice. Most importantly, even with these assumptions, our method successfully explains real behavioral data from humans.
2. The challenge in computing the normalizing constant in Sec. 5.3 is a property of all rational speech acts models, and not inherent to to latent budget modeling. As reference games admit an arbitrarily large natural language utterance set, it is indeed the case that we have to include some form of approximation. We use the same approximation that past work has shown to be effective. [1, 2]

[1] McDowell, Bill, and Noah Goodman. "Learning from omission." Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.

[2] Monroe, Will, et al. "Colors in context: A pragmatic neural model for grounded language understanding." Transactions of the Association for Computational Linguistics 5 (2017): 325-338.

Improving Clarity of Plots

In our uploaded revision, we have improved the clarity of the plots to make them easier to parse. Please let us know if this is better!

$\beta_\mathrm{temp}$ and $\beta_\mathrm{runtime}$ in RSA:

Intuitively, $\beta_\mathrm{temp}$ can be seen as a greediness parameter. A low $\beta_\mathrm{temp}$ indicates that the speaker is optimally "best-responding" to the listener and a high $\beta_\mathrm{temp}$ indicates that the speaker is choosing actions from a uniform distribution. Empirically, human language users are somewhere between the two extremes, and past work has found that tuning $\beta_\mathrm{temp}$ is necessary to get a good fit to human data [1]. Despite this, we find here that individual human differences in language production are not explained by differences in $\beta_\mathrm{temp}$, but they are explained by an L-IBM (inferring $\beta_\mathrm{runtime}$) that models individual participants as literal or pragmatic listeners.

[1] Frank, Michael C., and Noah D. Goodman. "Predicting pragmatic reasoning in language games." Science 336.6084 (2012): 998-998.

Relationship between $\beta_\mathrm{puct}$ and ELO rating:

$\beta_\mathrm{puct}$ (as used in popular strength-modeling approaches like [1]) doesn’t simply control exploration vs. exploitation. Instead, it biases exploration towards a specific policy prior. As $\beta_\mathrm{puct}$ tends to infinity, it is identical to playing the prior policy (effectively reducing to imitation learning). When $\beta_\mathrm{puct}$ is set to 0, it is equivalent to greedily picking the search Q-values. As it relates to ELO rating, what we are noticing here is that stronger players start deviating more from their base policy to instead depend more on their MCTS search Q-values. This is therefore indicating that stronger players rely more on search compared to weaker players.

We have added this discussions to our revision!

[1] Jacob, Athul Paul, et al. "Modeling strong and human-like gameplay with KL-regularized search." International Conference on Machine Learning. PMLR, 2022.