Utility-based Adaptive Teaching Strategies using Bayesian Theory of Mind

We are thankful to the reviewer for their constructive remarks and literature references, which have greatly enriched and strengthened the foundation of our work. We would like to particularly thank the reviewer for pointing to the work of Rafferty et al. that we missed, when responding in place of Reviewer C7a4.

The paper's formulation is ambiguous. The decision to frame the learning task as a POMDP lacks clear justification. While this relates to the proposed method, more concrete examples are needed for justification.

Response:

We acknowledged the reviewer’s concern and included a reference that models the learner’s task as a POMDP following a similar approach, page 3 (modification in red). In a teacher-learner setup, it's crucial for the learner to lack access to specific environmental information for the teacher to be beneficial. Thus, introducing the learner’s task within a POMDP framework is a straightforward and elegant method to illustrate the uncertainty the learner faces. This uncertainty is what the teacher attempts to reduce or alleviate through its demonstrations.

The task formulation for the ToM-teacher isn't clearly presented in the manuscript. From my perspective, the teaching task resembles a POMDP, akin to references [1, 2]. The teacher doesn't fully know the learner's internal state but tries to optimize the student's performance by providing demonstrations. Unlike earlier works, this paper also seeks to estimate the learner's transition function parameters (goals and observation function). Clarifying the task's formulation is crucial for understanding the technical value of the proposed approach, which remains unclear.

Response:

We concur with the insightful observation made by the reviewer regarding the potential enhancement of our paper through a more structured formalization of the teacher's task. Recognizing this valuable suggestion, we incorporated to the paper a formalization of the teaching task within the contextual MAB framework (which is a simpler particular case of POMDP) and added corresponding references to the paper (modification in blue).

In our adapted formalization, the teacher has partial observability over the environment’s state which is in our case the learner’s mental state. This strategic integration aims to clarify and strengthen the conceptual underpinning of the teaching task, facilitating a more comprehensive understanding of the teacher-learner dynamics within our proposed framework, see Section 3.3.2 (modification in blue).

The paper's objective is unclear. The authors claim, "The goal of this work is to study whether learner-specific teachers who model the learner’s internal state are more efficient than learner-agnostic ones." Given the vague problem definition, it's unclear if this captures the paper's essence. Is the primary takeaway that Bayesian ToM-teachers excel at understanding human internal state changes?

Response:

We acknowledge the reviewer's concern regarding the paper's unclear formulation and objectives. In response, we have clarified these aspects in the revised introduction (modification in red).

One of the main focuses of our work was to explore the utility of teachers utilizing a Theory of Mind (ToM) model compared to learner-agnostic baselines. While the superiority of ToM-teachers over learner-agnostic teaching strategies was expected, we were intrigued by the discoveries surrounding partial observation of the learner’s trajectory and the utilization of an approximate model of the learner’s policy. In fact, Section 5.2 results demonstrate that in scenarios with limited data where the teacher only has an approximate understanding of the learner's policy, employing a ToM model can be misleading. In this case, a learner-agnostic teaching strategy which aims to maximize the mean utility across all learners proves to be more useful.

We chose a Bayesian model because Bayesian inference is commonly used to model theory of mind mechanisms, as extensively discussed in our related work on cognitive science. However, since our work is confined to simulated environments and virtual agents, further investigation, particularly involving human users, would be necessary to extrapolate our findings and draw conclusions about humans. Yet, we consider such research beyond the scope of this paper.

We thank the reviewer for their positive evaluation of our work and for helpful comments, including spotting typos. In particular, we are glad that they found the paper easy to read and that the reviewer appreciated the notebooks.

Generalizability: The proposed ToM technique assumes access to an approximate policy of the student. Also all environments discussed in the paper were deterministic. In practice, while the set of student goals are limited, the policy they follow may be far from ideal and the presence of stochasticity may confuse the teacher further to reach a reasonable belief. Would be great to discuss these limitations in the paper.

Response:

The reviewer brings up an interesting point. We introduce the rational ToM-teacher, characterized by a stochastic approximate policy of the learner. This policy is further defined by a Boltzmann temperature parameter, lambda, representing the assumed degree of rationality of the learner. A higher temperature parameter implies a noisier rationality in the learner's behavior. Therefore, in theory, with adapted temperature parameters, the rational ToM-teacher can be effective for noisily rational learners

Although our experiments only involve learners following a deterministic shortest path policy, representing perfect rationality with the lowest temperature parameter, we could have employed stochastic, noisily rational learners. In such cases, learning the temperature parameter would have been necessary. We added a precision on that point in Appendix B.3 (modification in red). However, learning a good policy within a complex POMDP, such as large Minigrid environments with sparse rewards (for instance, requiring the key of a specific color to open a door), constitutes a distinct research challenge beyond the scope of this paper.

Limited novelty: the main idea of the paper is not novel and similar ideas have been explored to infer agent policies and goals as cited by the authors. The main difference is to expand the inference space to include an observation model.

Response:

We acknowledge the reviewer's concern regarding the novelty of our work. We recognize that the introduction of our objectives may have been misleading. In the revised paper, we have clarified our focus, emphasizing the results obtained in the regime of limited observation of the learner’s behavior and ToM-teachers with an approximate understanding of the learner's policy (modification in red). Additionally, we have incorporated references highlighted by the reviewers to better underscore the novelty of our work (modifications in green). We trust that these modifications will demonstrate the significance of our findings and the novelty of our approach to the reviewer.

"receptive field", does the agent see behind walls? I believe the answer is yes, but would be great to clarify in the main doc.

Response:

If the learner has a limited observation function, it does not see behind walls (we set the argument ‘see_through_walls=False’ to generate our environments with the MiniGrid library). We added a clarification of this point in the main document in Section 4, point on the learner (page 6) (modifications in red).

Why are demonstrations action sequences only? Why exclude observations? Is it due to a deterministic assumption?

Response:

The observation function differs between the teacher and the learners. The teacher has full observability while the learners can have limited and different observation functions. The idea is that the same demonstration as a sequence of actions will result in different sequences of observations for two learners with different observation functions (i.e. receptive field sizes). Hence while being exposed to the same demonstration, two learners with different observation functions will acquire different knowledge of the environment. The demonstration, presented as a sequence of actions, serves as a method for the teacher to guide the learner in the environment. However, the teacher must infer from prior observation what the learner will perceive while being guided. We added a clarification in Section 3.3.1 page 4 (modifications in red).

Before preparing our response to all reviewers, we would be glad if the reviewer could help us better understand the weaknesses of our work by pointing to some concrete instances of the relevant literature that we missed and that the reviewer seems to have in mind. The reviewer says that we do not mention “the extensive literature on formalizing teaching and cooperation”, they say that “This literature has grown large and quite mature in recent years, including extensive mathematical and computational theories from a variety of perspectives.” and even say that “several papers touch on topics that are quite close to the ideas here including sequential teaching under perturbations on belief and policy, teaching as a POMDP, proofs of robustness of standard ToM reasoning in cooperative settings, etc.”, but doesn’t give even a single instance of such papers that we missed.

Further in the review, the reviewer says “In recent years, there have been several related NeurIPS papers, searching for ‘teacher’ or ‘teaching’ or ‘cooperation’. There are older papers formalizing teaching as a POMDP problem. There are also newer papers on inferring beliefs of agents.”. Again, mentioning a few instances of such papers that the reviewer may have in mind would help a lot. Finally, the reviewer says “I would strongly recommend that the authors review the last few years of NeurIPS (also ICML) papers for topics such as ‘teacher’, ‘teaching’, ‘cooperation’ and ‘cooperative’. Not all papers with those words will be related, but one will find quite a lot. Specifically, there are relatively theoretical papers that outline a mathematical framework and imply the results here that appeared in NeurIPS and ICML. There are also papers in NeurIPS that tested human experiments, which necessarily have errors in beliefs. Similarly, please read the literature related to RSA (the Goodman and Frank paper cited) which have a number of interesting models and results. Please also search broadly for teaching and POMDPs as there is related work in that direction also. The current work will benefit from reconceptualization in light of these works.”

Again, the reviewer seems to have some concrete papers in mind, but does not mention them. So we would be extremely grateful to the reviewer if they could provide at least 2 or 3 of the most relevant papers they have in mind, so that we can enrich our perspective with these works if they are actually relevant, or eventually better delineate our scope if we think they are not.

In particular, as the reviewer mentions "cooperation" and "cooperative", we would like to clarify that our work is not much related to the abundant literature about cooperation in multiagent systems, where indeed one can find POMDP-based formalizations and Bayesian inference mechanisms, but teacher-learner interactions are not considered. Rather, our work is inspired by cognitive sciences, and more specifically by recent works on inferential social learning (ISL) (Gweon et al., 2021), which argues that humans learn from evidence generated by others and generate useful evidence to help others learn.

"we introduce a teacher equipped with a Theory of Mind (ToM) model that we refer to as ToM-teacher" Important to acknowledge most papers have a ToM component.

Response:

Our objective was to explicitly model the ToM employed by the teacher to instruct the learner. While there are numerous models demonstrating ToM capacities and incorporating ToM components, to our knowledge, in most of the literature, the ToM model is defined from the learner toward the teacher and is used to modulate how the learner processes the information provided by the teacher. In fact, none of them explicitly defines a model of ToM from the teacher to the learner and shows how this model can be leveraged to modulate teaching in itself. While the current literature focuses on how ToM is used to learn, Gweon research and our work focus on how ToM is used to help others learn.

Given the introduction focusing on cognitive science, it is surprising to see decision trees and A-star as part of the learner. Is there reason to believe that these are reasonable models of humans?

Response:

We do not claim that decision trees and A* algorithms are reasonable models of humans. However, we intentionally use a known policy for the learner in order to analyze the impact of inaccurate prior on the learner’s policy. We specifically use a decision tree, due to the complexity of learning in a large 2D navigation POMDP with sparse rewards—a challenge beyond this work's scope.

We could have used a trained agent. No matter if a trained or handcrafted policy is used, the ToM-teacher assumes the learner behaves rationally. We anticipate a properly trained agent to exhibit rational behavior, creating similar uncertainty for the ToM-teacher, whether the learner follows a trained policy or a decision tree. Thus, we believe that employing a decision tree learner does not impact the value of our contributions.

We thank the reviewer for their constructive comments on our work. We are glad that the reviewer found that we are tackling a relevant problem and that they found our approach suitable.

My main concerns are regarding the presentation and impact.

Regarding presentation, I find the formalization unclear and sometimes misaligned with the literature. For example, the environment is introduced as a POMDP but then formalized (seemingly?) as an MDP (there are no observations?).

Response:

The learners’ environment is framed as a POMDP, yet unlike the conventional formalism, the observation functions are considered internal to the learners rather than to the POMDP. As a result, within the same environment, two learners with varying observation functions will receive different observations from the same environment states. We added in the main paper a reference to a similar work using the same formalization (modification in green).

In the revised version of the paper, we've introduced a teaching contextual Multi-Arms Bandit (which is a simpler case of POMDP) that delineates the teacher’s task from that of the learners’ in Section 3.3.2. We hope that the reviewer will find this revision clarifies our method (modifications in blue).

Additionally, the Bayes-adaptive POMDP is mentioned but then it seems that the uncertainty is limited to the state (is the belief only over the state?), which is not a problem solved by the BA-POMDP at all.

Response:

We referred to the BA-POMDP framework to motivate the use of a belief over the state of the environment which was introduced as a ‘belief state’ in the paper [1] “The belief state specifies the probability of being in each state given the history of action and observation experienced so far, starting from an initial belief b0.” However, as the learner uncertainty relies solely on the state of the environment, it is true that referring to Bayesian approaches of POMDP is more accurate. We thank the reviewer for this useful remark and removed the BA-POMDP reference from the paper, page 4 (modifications in red).

[1] Stéphane Ross, Brahim Chaib-draa, and Joelle Pineau. Bayes-adaptive POMDPs, NeurIPS, 2007.

Various (important) notions are not defined, including the learner's initial belief, which the teacher is trying to affect, which makes it hard to understand what exactly the demonstrations are accomplishing.

Response:

The initial belief of the learners is said to be uniform (Section 3.2, page 4, “Unless mentioned otherwise, we assume that the learner’s initial belief bi,j_0 on the state of Mj is uniform over the set of possible states Sj_B .“).

In the revised version of the paper, the initial uniform belief of the learner, which has not yet interacted with the teacher, is considered to be an observation of the learner’s mental state (which is the hidden state of the teaching contextual MAB). This is defined in Section 3.3.2 page 5 (modification in blue) and we hope that the reviewer finds this formalization clarifies this point.

In the demonstration environment, the teacher offers guidance to familiarize the learner with the new environment. Starting from an initial uniform belief, the learner follows the teacher's demonstration –which is a sequence of actions– and updates its belief about the state of the new environment from the observations it has accumulated by following the teacher’s demonstration. Subsequently, during the evaluation phase, the learner behaves on its own but starts with the initial belief updated by the teacher’s demonstration rather than an uniform initial belief. This is detailed in the first two paragraphs of Section 3.3.1 pages 4 and 5.

Another example is the data observed by the teacher (tau^obs) which does not seem to be defined anywhere.

Response:

The data observed by the teacher was defined in Section 3.3.2 paragraph 4, page 5, “From a past trajectory τ^obs = {(s_k, a^{obs}_k )}^{K−1} _{k=0} of an unknown learner L on the first environment Mobs [...]”.

We thank the reviewer for highlighting this unclear point. In the new version, the teacher’s observation of the learner is more clearly defined as an observation function of the learner’s internal state, see the new Section 3.3.2 page 5 (modifications in blue).

We thank the reviewer for their time and effort in evaluating our work, and are glad that the reviewer found the soundness and presentation of our paper excellent.

1) It's not clear why the method requires two learning environments - a simple one for the teacher to interact with the learner and gain knowledge of the student's mental state; and another more complex environment where the teacher performs real teaching. Is it possible to unify these two and let the teacher teach on the fly as it interacts with the student?

Response:

Teaching 'on the fly' while interacting with the learner is possible but still requires sequences of observation, demonstration selection, and evaluation. We decided to split the framework into two environments to separate the Theory of Mind (ToM) modeling (teacher modeling the learner’s internal state) from the teaching (teacher leveraging its model of ToM to provide a demonstration).

2) If there are significant distinctions between simple and complex environments. How does it affect the teaching process? Is the knowledge learned by the teacher in the simple environment going to become less effective?

Response:

The first environment’s complexity can vary, and in additional experiments which we included in Appendix G of the revised version, we show that actually performing the observation stage on a simpler environment causes more ambiguity in the learner’s behavior, causing the teacher to have more difficulty to identify the learner’s goals and beliefs. Indeed, since the learner can have different widths of sensory capacity, being observed in a larger, more complex environment can actually help better identify its sensors. However, if the observation environment was overly complex for the learner to engage with objects, its behavior would not be indicative enough of its goal, making inferring internal states impossible. Hence, the need for a reasonably simple environment in which all the learners can interact with the objects without the help of the teacher.

On the other hand, the second environment, where the teacher conducts teaching, needs to be complex enough for some learners to require the teacher’s assistance to achieve the task. If learners can easily accomplish tasks without the teacher's help, providing an empty demonstration would be an optimal solution: no teaching would be needed. Hence, the need for a ‘complex’ environment. Finally, note that a learner’s internal state (goal and observation function) remains consistent across different environments and is independent from the complexity of those environments. Consequently, the insights gained by the teacher during the observation phase can be leveraged for teaching in any environment, regardless of its complexity. We added in Appendix G of the revised version of the paper a precision on this point discussing the influence of the observation environment size (modifications in orange).

3) The teacher requires full knowledge of the underlying POMDP. This is really unrealistic unless in very specific domains. This requirement limits the applicability of the proposed method.

Response:

Indeed, we assume the teacher's access to the POMDP. Specifically, our method requires the teacher to predict a learner's reward in the environment based on a given policy conditioned by a particular belief. In our method, this involves simulating trajectories within the environment, which relies on the teacher's knowledge of the underlying POMDP. However, for reward prediction, the teacher can use a pre-trained model. We could consider a model trained to predict the reward of a learner following a particular belief-conditioned policy (the teacher’s approximation of the learner’s policy), therefore taking a belief as input. Yet, this would require access to a large amount of trajectories of such learners with various beliefs following the teacher’s approximate policy.

We chose to focus on uncertainty on the learner’s mental state rather than on the POMDP from the teacher’s point of view. We acknowledge the interest of considering teachers with partial observability but we consider that this study is out of this work’s scope.