Beyond Single Stationary Policies: Meta-Task Players as Naturally Superior Collaborators

We are grateful for the time and effort that reviewer QDdz has invested in reviewing our paper, and we appreciate the recognition of the main advantages of our method. Additionally, thank you for your insightful comments regarding Theorem 1 and its proof.

Clarification on the Relationship Between Stationary and Non-Stationary Policies

Thank you for your thoughtful and perceptive question. Your question highlights an important aspect of our theorem formulation, and we acknowledge the need for clearer explanation in our original manuscript. Our approach is developed on the key insight that, despite the inherent diversity in human behaviors, the underlying meta-tasks within specific collaborative contexts are quite similar. This observation facilitates the transformation of a non-stationary cooperative problem into one involving a series of stationary policies addressing distinct meta-tasks.

Theorem 1 outlines this transformation by situating the collaborative process between humans and AI within a Non-Stationary MDP framework. To manage the inherent non-stationarity, a methodical decomposition of the broader non-stationary decision-making process into a series of temporally contiguous stationary segments, which represent the stationary process of meta-task accomplishing.

Initially, Theorem 1 defines trajectories collected from interactions with what we referred to as a 'stationary human policy.' To clarify, this term was intended to denote a stationary policy that humans use to accomplish a specific meta-task. It was not meant to imply that our theorem begins by establishing convergence for a stationary human policy and then extends this to a non-stationary human policy. We acknowledge that the term 'stationary human policy' may have led to some confusion. Thank you for your insightful query. In response, we have revised the terminology to 'stationary meta-task performing policy' in our manuscript to improve clarity and readability.

Clarification of Convergence in Stationary and Non-Stationary Contexts

Your question regarding convergence is astute. We believe our CBPR framework is innovative, offering a straightforward approach to achieving convergence in non-stationary human-AI cooperative settings.

To ensure convergence while interacting with a non-stationary human, a cooperative policy must: 1) recognize changes in human behavior policies, and 2) adapt a consistent response policy that aligns with these changes.

In the proof of Theorem 1, we first establish the convergence properties of the Bayesian update mechanism. This ensures that, with the convergence provided by Bayesian updates, the belief on the current meta-task will consistently converge. This convergence is crucial for the CBPR framework to effectively track changes in human behavior. Subsequently, we demonstrate that when utilizing Bayesian updates, CBPR algorithms consistently converge to a specific meta-task cooperative policy, provided that the meta-task being performed by the human remains unchanged until this fixed response policy is achieved.

A key advantage of our CBPR method is the use of Bayesian updates to dynamically update beliefs on human behavior. This approach is effective for both stationary and non-stationary policies. The belief about the meta-task will consistently converge to match the behavior that the human is exhibiting. The primary distinction between stationary and non-stationary human policies is that the meta-task in the former remains constant, while in the latter, it evolves over time.

We hope these clarifications address your concerns and enhance the readability and comprehension of our work. We thank you again for your valuable input, which has significantly contributed to improving our paper.

We really thank you for your valuable comments on improving our work. We sincerely hope the reviewer can raise the assessment score of our paper if the following responses have successfully addressed the concerns.

Q1: The uncertainty of σ?

The human-AI collaboration problem we address is inherently non-stationary. We utilized a rule-based agent with noise (i.e. τ ) a stochastic policy (π), thus the reward is uncertain. Signal model P (σ | τ, π), which is also known as observation model represents the distribution of observed signal σ when AI performs the meta-task τ with policy π in vanilla BPR. Note that in CBPR, we choose the episodic reward as the signal σ of signal model. During offline stage of CBPR, we construct each signal model (i.e. performance model) by fitting a Gaussian distribution given a stochastic AI policy π and a noisy rule-based agent τ (see l124-128 of the main text and l533-534 of the Appendix). To further clarify the definition of σ, we will add additional explanation of signal model in the revision.

Q2: The assumption of Equation 1? Why the belief updating reasonable?

Thanks for the question. We would like to clarify that only the AI decision makers are acting to maximize the expected utility, whereas humans may exhibit varying levels of initiative to cooperate in human-AI collaboration. The rationale of the updating the belief in Eq 1 adheres to the vanilla BPR, which reuses prior experience from the class of tasks to cope with unknown task variations. Note that Following works primarily focused on extending BPR in competitive tasks (lines 93-95 in original paper). In this work, we firstly extend BPR in human-AI collaboration scenario. Human-AI collaboration is essentially an incomplete information game since human players may exhibit varying levels of initiative to cooperate and AI players do not have complete knowledge of payoff functions of human players. On the contrary, AI players need to adapt to the non-stationarity of human players, and are acting to maximize expected utility.

Q3: ρ seems make no difference in mean reward, why is this the case?

We examine the setting of rho in the evaluation of our experiments. According to Eq 4, when ρ=0.1 the weight of the inter-episode belief decays to 0.01 after just 2 time steps, while when ρ=0.9, the inter-episode belief nearly decays to 0.01 within 40 time steps. Therefore, for a game with 600 time steps in our evaluation, only the first 40 steps of integrated belief will be influenced by the inter-episode belief, leading to the observation that ρ make no difference in mean reward. In other words, to demonstrate the effectiveness of ρ, the inter-episode belief should decay to below 0.01 after a greater number of time steps in an episode. To further evaluate this new setting, we set ρ = 0.99 and ρ = 0.95. Therefore, we have conducted more ablation study of ρ (Fig 3 in rebuttal PDF), the result indicates that in simple layout (i.e. Cramped Rm.) since the partner’s policy is simple, variations in ρ have little impact on the reward. However, in complex scenarios (i.e., Soup Coord.), adjusting rho can enhance cooperative performance to a certain extent.

Q4: Results of (Figure 4) and Figure 6.

We provide a more detailed analysis of our results in Fig 4 as follows and include them in our updated paper:

The inherent advantage of SP and FCP agents. Partners with low, medium and high skill levels are represented by checkpoints (essentially are SP agents) at initial, the middle and the end of training of FCP. There- fore, SP and FCP agents have an inherent advantage in the evaluation presented in Figure 4. Despite this, CBPR performs better when faced with partners of a lower skill level. When collaborating with real humans (Fig 6), FCP and SP no longer have such advantage. This leads to almost all FCP and some SP performing well against agents of various skill levels (Fig 4), but falling short when facing human players (Fig 6).

The cooperative advantage of CBPR in non-separated layouts. In separated layouts (i.e. Asymm. Adv. and Soup Coord.), agents can usually complete tasks independently without considering the hindrance of the other partner’s moves to themselves. However, players’ own position (e.g. stand still in front of the serving areas) can obstruct their partners from completing the task in the non-separated layouts. Therefore, non-separated layouts require more cooperation between players compared to separated layouts. As shown in Fig 4, CBPR’s better performance in Cramped Rm. and Coord. Ring suggests its advantage in collaborative tasks.

The double-edged sword of SP’s simple policy. In Asymm. Adv., SP agent exhibits outstanding perfor- mance when it cooperates with the agent of high skill level (Fig 4c). We replayed the game and found that the SP agent learned the simplest and most effective policy (i.e. in the right room, just pick an onion from onion dispenser and then place it in a pot within the shortest path) during training. However, other agents exhibit some superfluous actions due to their own complexity. However, when SP cooperates with the agent of low skill level, it performed poorly because the SP agent on the right only learned the simplest policy (putting onions in the pot), and when the agent with low skill level on the left does not deliver the cooked soup, SP will wait in place rather than deliver the cooked soup. Thus, cooperating with SP agents results in low performance (Fig 4d).

Q5: Why is it necessary to identify the task?

We would like to clarify the process of ”identify” meta-task in the online stage of CBPR. Identifying task is to calculate the probabilities to generate a queue Q of human behaviors given each meta-task τ . During the online collaborating, we are unable to ascertain human cooperative strategies and initiative; however, human behaviors reveal the meta-tasks they are currently undertaking. Therefore, it is necessary to identify the task.

We thank the time and effort reviewer 3rF3 has invested in reviewing our paper. For the Weakness1, Weakness2, Q1 and Q3 you mentioned, we have added experiments to support our insight. We apologize for the confusion caused by Figure 2. We have simplified it in the rebuttal PDF to make the framework of CBPR more clear. We provide detailed explanations for the other questions and hope this can address your concerns.

Weakness1 and Q1: The difficulty of predefined meta-tasks and the unpredictability of human behavior.

For many tasks, defining meta-tasks based on human knowledge is not so difficult. For instance, in the context of autonomous driving, proactive lane changing and forward driving can serve as meta-tasks. In robotic scenarios, navigation and robotic arm grasping can also be considered as meta-tasks. To address leftover tasks not included in predefined meta tasks, we introduce a meta-task category as ”others” (Fig 1, bottom-Left).

For the undefined and unpredictable tasks, they can, to some extent, be addressed by the “others” category by learning a general model for them in our current framework. To demonstrate this, we add an ablation study based on section 4.2 to examine the impact of the number of meta-tasks undefined for the Soup Coord. scenario, in which we defined 4 more meta-task (place_onion_and_deliver_soup, place_tomato_and_deliver_soup, pickup tomato and place mix, pickup_ingredient_and_place_mix). The results show that without "others" category, the performance deteriorates significantly, while the performances degrade relatively gracefully with less meta-tasks defined and more included in "others" category.

In some extremely complex tasks, it may be difficult to define what a meta-task is, which will be the direction of our future research. We will explore approaches to decompose tasks into finer-grained pieces so that definitive meta-tasks can appear and become definable. We will add these discussions to limitations and future work in our revised manuscript.

Weakness 2: Lacking experiments in other in real-world applications.

Following your recommendation, we conducted additional experiments using the CARLA autonomous driving simulator. To acquire driving behavior data, we employed human driving models from the SUMO traffic simulator, selecting random origins and destinations for each vehicle to collect driving data. This data was used to train a Behavioral Cloning (BC) agent (Agent 1) to autonomously control the vehicle. For the Control-Based Priority Routing (CBPR) agent, we designed two meta-tasks: the "line up" meta-task, developed by selecting vehicle queuing data and training a second BC agent (Agent 2), and the "others" meta-task, utilizing BC Agent 1.

We evaluated both BC and CBPR agents in a 'Queue for a Left Turn' scenario, employing a rule-based simulated human. This simulated human intervention ensured the vehicle remained in line whenever the AI attempted to change lanes. Initially, as vehicles began to queue, the AI chose to turn right and overtake. At this point, human would redirect the vehicle to maintain its position in the queue through a left turn. After a single instance of human intervention, the CBPR agent switched to the "line up" meta-task and continued to wait in the queue, whereas the BC agent repeatedly attempted to overtake by turning right, necessitating continuous human intervention. Detailed results are shown in Figure 4 of the rebuttal material.

Q2: The learning process is more deterministic?

In Appendix D, we provide the training curve of all agents (Fig 9-12) over five random seeds, which indicate the learning process is not that deterministic. The shaded area in Fig 3 is indicated by standard error, which is the standard deviation divided by the square root of the the number of seeds to represent the randomness of multiple experiments. To make the Fig 3 easier to understand, we scaled down the errors in original paper. We have replotted the Fig 3 using the standard deviation as shown in Fig 1 of rebuttal PDF file.

Q3: CBPR performance in the episode with longer horizon?

Thanks for this interesting question. Following your suggestion, we increased the original horizon setting from 600 to 3000, and then had the CBPR agents cooperate with the policy-switching agents, recording the cooperative rewards under five different frequencies of policy switching. The experimental results are shown in Fig 2 of rebuttal PDF, which implies that CBPR remains highly efficient in long-horizon tasks.

Q4: CBPR scales with LLMs?

Thank you for your question, which has inspired us to undertake some work combining large models with CBPR in the future. We consider CBPR scaling with LLMs or large models in following two aspects: Meta-task identification \ In Overcooked simulator, meta-tasks are easy to defined. However, in environments where meta-tasks are not clearly defined or are too complex to categorize, the context comprehension ability of large models can deconstruct multiple meta-tasks based on behavior data demonstrated by humans. Acceleration of belief $\beta_k(\tau)$ convergence \ In an environment where two meta-tasks do not have clear boundaries. LLMs or large models can accelerate the convergence of $\beta_k(\tau)$ by adding an additional factor $\lambda$ to the numerator of Eq 1 or Eq 3. $\lambda$ can be determined by self-reflection [1] of LLM.

[1]Shinn et al. Reflexion: Language Agents with Verbal Reinforcement Learning. 2023