Designs for Enabling Collaboration in Human-Machine Teaming via Interactive and Explainable Systems

We thank the reviewer for noting that they enjoyed reading our paper, that the paper's contribution to the field is significant, and that our paper is well-written and the ideas are clear. We have responded below to the weaknesses and questions noted by the reviewer.

Contextual Pruning Results and Novelty -- The two post-hoc pruning strategies we utilize are to prune a tree given the boundaries of each state variable and removing redundancies within the tree. This need for pruning arises naturally during the training of the IDCT model via reinforcement learning as different subspaces of the tree model may become more important (higher probability of reaching certain sub-trees) or completely inactivated (impossible to reach a certain sub-space due to changes in weights caused by gradient descent). It is important to note that these pruning techniques follow a similar ideology behind neural network pruning but do not modify the model's behavior in any way. Neural network pruning approaches, on the other hand, often remove weights with smaller magnitudes or activations, which result in model changes that can harm performance.

These specific pruning strategies have not been applied to differentiable decision tree models before to improve interpretability. In training the IDCT model, we conducted a hyperparameter search over different tree sizes and chose the best-performing model post-pruning with under 32 leaves. In the end, we found that a 256-leaf IDCT model was most amenable to training in each domain, and contextual pruning was able to reduce this large tree into a simple tree of three leaves and two leaves, respectively (a 64-128x reduction in model size). This pruning technique is essential as trees of arbitrarily large depths can be difficult to understand [1] and simulate [2], and that a sufficiently sparse DT is desirable and considered interpretable [3].

[1] Abhishek Ghose and Balaraman Ravindran. Interpretability with accurate small models. Frontiers452 in Artificial Intelligence, 3, 2020.

[2] Himabindu Lakkaraju, Stephen H. Bach, and Jure Leskovec. Interpretable decision sets: A454 joint framework for description and prediction. In Proceedings of the 22nd ACM SIGKDD455 International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 1675–1684,456 New York, NY, USA, 2016. Association for Computing Machinery.

[3] Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of458 interpretability is both important and slippery. Queue, 16(3):31–57, 2018.

Human Improvement from Iteration 1 to 4 -- In the rebuttal document, we have added in an additional display of the reward trajectories for each participant, with a single color assigned to each participant's behavior for each domain. These figures, while showcasing the trajectories of each participant well, can get cluttered and thus we chose to showcase our findings via a scatter plot. In IV2-D1: Forced Coordination, you can see that many participants do not linearly increase from Iterations 1 to 4. Often, mistakes in tree logic can lead to an agent that does not collaborate well and as this domain requires collaboration, leads to a sub-100 score.

The improvement in IV2-D2: Optional Collaboration between iterations one and four was found to be significant (p<0.01). This implies that that in this domain, users were applying a conscious strategy when modifying the agents policies. However, even in this domain, the improvement was not necessarily monotonically increasing for all participants.

Additional Limitation -- We thank the reviewer for noting this additional limitation and will add this into our section.

We thank the reviewer for noting that our study in Human-AI teaming would be beneficial to the community.

Unsupported Claims and Design Decisions -- As the field of Human-AI Teaming is still relatively new, some motivations are positional. For example, in considering lines 129-132, some may argue that AI agents and humans do not need consensus or that the human should always defer to the AI's strategy. In some domains, this may be effective. However, in domains where agents and humans need to collaborate closely, we believe the process of consensus will lead to better collaboration performance and understanding between teammates. This is validated in our human-subject study findings (see Figure 4b of the manuscript), where users that can interact with the AI's policy significantly improve over repeated gameplay. There are several works that display that social robots that can adapt online and develop social relationships achieve more successful and sustained interactions with users [1].

[1] Leite et al. Empathic robots for long-term interaction: evaluating social presence, engagement and perceived support in children.

The design guidelines are directly related to the results of our paper.

• Creation of white-box agents that achieve competitive initial performance to black-box agents: If model performance is equal, interpretable models provide increased accountability compared to black-box models. Importantly, these models with user interaction led to positive team development.
• Design of learning schemes to support the generation of collaborative behaviors: This is derived from the case study and analysis of Fictitious Co-Play. Designing objective functions so that agents collaborate well with humans and/or online adaptation schemes to lead to personalized, effective teammates is an important research direction.
• Creation of interfaces that enable mixed-ability users to improve team collaboration: This is based on system usability scores collected from users, emphasizing the need for creating modification interfaces that support a wider variety of users.
• Evaluation of teaming in a larger number of interactions: This was derived from our user study findings, where a higher # of interactions may have provided a better understanding of the team development process.

The design decision behind the IDCT is that this tree-based model affords transparency and supports training via RL. This allows us to directly compare with prior frameworks in Overcooked that leverage RL for training collaborative agents and provides us with an interpretable model that users can modify and visualize.

Case Study regarding the Teaming Gap with Current Models -- We agree that this case study could be expanded. This first study (Figure 1) is conducted with an actual human player following a scripted strategy while collaborating with an agent publicly available from Carroll et al. While a human could have conducted a larger number of trials with the agent, it was clear from the set of trials that the agent could not adapt to the human-preferred strategy.

The second study focuses on Ficticious Co-Play. A heuristic collaborative strategy and heuristic individual strategy are programmed, receiving scores of 408 and 306. Then, an FCP agent is trained, converging to a score of 295.06 \pm 1.86 over 50 teaming simulations. This FCP agent is also evaluated with humans in the study, achieving scores ranging from approximately 120 to 315 in the last teaming round (Figure 4b-right). These gameplay scores of FCP, both in teaming with synthetic agents and real humans, are far below a heuristic collaborative strategy, signifying the gap. A full assessment including other benchmarks and analyzing agent collaborativeness, both quantitatively and qualitatively, while interesting, would likely be a full paper in itself.

Technical Details regarding the Approaches -- The details regarding the IDCT approach are found in Appendix C alongside pictures of the agent models (Figure 6 and 7). In line 321, we note that the trained policies had two leaves in Forced Coordination and three leaves in Optional Collaboration. As mentioned in Appendix C.4, we train our IDCT models with 256 leaves. A reduction to a depth of one (2 leaves) or two (3 leaves) is a pruning reduction of 128x and 64x. We will update these numbers and shift these details into the main paper.

Details of the FCP baseline are also found in Appendix Section C.4. The AI-Led Policy Modification is an adaptation of Carroll et al.'s approach to an online setting with an IDCT. We provided high-level details about this approach in Line 279-284 and within the footnote on page 6. We will include further details about the online optimization procedure in the Appendix.

User Study Detail Clarifications -- Our study was conducted at a university with a diverse population majoring in different engineering disciplines, economics, and sciences. All users had some college education and were enrolled at an engineering-focused university as an undergraduate or graduate student. Users were asked demographics information, experience with games and decision trees, and conducted a personality survey. This information was included when determining significant trends.

In future studies, it would be beneficial to increase the # of participants and gameplay trials. As noted in the Procedure, the domains are randomly ordered. Our experiment is a 5 (teaming method; between) x 2 (no. of domains; within) x 4 (no. of repeated evaluations; within) mixed-factorial experiment.

Results Supporting Author Claims - The statement noted was comparing conditions IV1-C1 to IV1-C2, IV1-C3, and IV1-C4. This statement meant that given the same tree-based model, the ability to interact adds subjective benefits. The reviewer is correct in noting that participants did assess higher-performing agents more positively in their subjective ratings. We will clarify this statement.

We thank the reviewer for their valuable feedback. We have responded to the weaknesses and questions noted by the reviewer.

Instability of L1 -- We thank the reviewer for this comment. The L1 regularization is only applied to the action leaf nodes of the tree policy. This regularization serves to make the categorical distributions within each leaf node more sparse so that the agent acts more deterministically. The mechanism used in our architecture to train an interpretable tree model via reinforcement learning is called differentiable crispification and was formulated by prior work (see [1]). From our observation, the L1 regularization successfully improved how deterministic the resultant tree-based policies were.

[1] Paleja, R., Niu, Y., Silva, A., Ritchie, C., Choi, S., & Gombolay, M. (2022). Learning interpretable, high-performing policies for continuous control problems. arXiv preprint arXiv:2202.02352.

Population bias and how it might affect the generalizability of your findings? Also, maybe sex/age-based difference analysis is also worth exploring. -- Our study was conducted at a university with a diverse population majoring in a variety of different engineering disciplines (Electrical and Computer, Biomedical, Aerospace, Chemical and Biomolecular), economics, and variety of sciences (computer science, neuroscience, atmospheric). All users within our experiment had some college education and were enrolled at an engineering-focused university. As such, these findings may not generalize beyond this population. In our data collection, we collected demographic information including age and sex. In our analysis, these variables were used and we did not find any significant trends that displayed that age or sex led to differences in Human-AI collaboration performance.

In generalizing our results to a broader population, we believe are findings can be augmented by other literature in human-AI interaction to design better interfaces for specific populations or domain-specific collaborative agents.

If possible, could you explain more about the potential challenges in scaling the human-led policy modification approach to more complex and dynamic environments beyond the Overcooked-AI domain? -- Thank you for this excellent question. In scaling to more complex and dynamic environments, the tree size needed to represent a high-performing agent will likely increase. In these cases, users may require more time to interact with and understand an agent's policy. There are also several capabilities that can be added to human-led policy modification which may make the process quicker and easier. For example, model verification can be added into the tree modification interface to detect problems with logic (detect regions of the tree that cannot be reached). This would allow the human to receive other types of feedback prior to teaming with the AI. For complex games, agent policies can also operate over different levels of abstraction providing the human with a tradeoff with fine-grained control of the agent policy and tree size. There may also be other more accessible mediums, such as language, that the human can use to program a large tree policy.

Missing citations -- We thank the reviewer for noting this deficiency. Within our literature review, we will add the following recent papers that are closely related to our work.

[2] Hong, Joey, Sergey Levine, and Anca Dragan. "Learning to influence human behavior with offline reinforcement learning." Advances in Neural Information Processing Systems 36 (2024).

[3] Wang, Chenxu, et al. "On the Utility of External Agent Intention Predictor for Human-AI Coordination." arXiv preprint arXiv:2405.02229 (2024).

[4] Guan, Cong, et al. "One by One, Continual Coordinating with Humans via Hyper-Teammate Identification."

[5] Tulli, Silvia, Stylianos Loukas Vasileiou, and Sarath Sreedharan. "Human-Modeling in Sequential Decision-Making: An Analysis through the Lens of Human-Aware AI." arXiv preprint arXiv:2405.07773 (2024).

**Abbreviations and Reference Format ** -- We have updated our paper language per your comments and updated our references to be consistent.

Ethics Review Flag -- As mentioned in Line 342 of our paper, our experiment was approved by a university institutional review board (IRB). All participants in our experiment signed a consent form, received description of the risks involved in our study, and received compensation.

We thank the reviewer for noting that our paper is well-motivated, provides valuable insight into designing human-AI collaboration interfaces, and studies a relatively underexplored domain. We have responded to the weaknesses and questions noted by the reviewer.

Comparison to Explainability Approaches -- The reviewer is correct in that we do not compare against explainability approaches that utilize local explanations to explain the decision-making of autonomous agents. While these approaches should be tested to explain behavior in the tight human-AI collaboration settings we consider, local explanations can be misleading and may not accurately represent the agent's decision-making behavior. This would lead to another dimension that needs validation. As this area of research (transparency and adaptability of collaborative agents in repeated human-AI collaboration) is relatively new, we hope our research can spawn further studies of this kind.

How can Section 4 be better linked with Section 5? -- Section 4 presents an interpretable machine learning architecture to train collaborative AI teammates, Interpretable Discrete Control Trees (4.1), a training advancement to enhance interpretability, and a mechanism to allow humans to modify the tree in simple ways, including tree deepening, decision variable modification, and leaf node modification. The creation of this capability (interpretable tree architecture + human modification) is our main proposed condition, IV1-C1: Human-Led Policy Modification in Section 5.

The following conditions IV1-C2: AI-Led Policy Modification, IV1-C3: Static Policy - Interpretability, IV1-C4: Static Policy - Black-Box all utilize the architecture introduced in Section 4 but ablate the interaction and interpretability. We tried to depict this gradual feature reduction across our proposed conditions in Table 1. To improve the linkage between Section 4 and Section 5, we propose to create a new section between 4 and 5 that presents information regarding the training and results of the IDCT model. In creating this section to improve clarity, we would augment details from the author response alongside the paper lines 301-321.

How do you envision improving the transparency of the proposed methodologies? -- To improve the transparency of the proposed methodologies, we have created a flow diagram (uploaded as Figure 1 in the attached rebuttal pdf) of each condition that helps display visually the experiment flow and interaction being assumed within the proposed approach. We have also included a diagram of how IDCT agents are generated as part of Figure 2 (left). Both of these figures will be added into the main paper as part of the appendix and alongside the new aforementioned section will improve the paper's clarity.

What are the evaluation metrics for the chosen collaboration settings? -- The main objective evaluation metric used to evaluate the performance of the collaboration is the game score. Section B of the appendix describes the exact scoring function that makes up the game score. In short, high score bonuses are obtained for full dishes served, and minor score bonuses are given for smaller objectives like filling a pot or picking up a dish. In the first domain, Forced Coordination, successful dish serving is not possible without resources being handed between the AI and human. In the second domain: Optional Collaboration, agents can serve dishes without explicitly collaborating with the human, but this domain was intentionally designed such that collaboration with the human via timely resource handoffs would result in a higher score than without collaboration.

How is interpretability evaluated? -- The interpretability of the model isn't evaluated explicitly. In our hyperparameter search for training the tree-based agent policy, we choose the highest-performing model with under 32 leaves. Prior work [1] notes that there is a cognitive limit on how complex a model can be while still being human-understandable and thus, we prioritized selecting a high-performing IDCT model that was still relatively small. We note that as conditions IV:C1-C4 utilize the same model, utilizing interpretability metrics such as tree size do not provide additional information in understanding the tradeoff between these conditions.

[1] Lakkaraju et al. Interpretable decision sets: A joint framework for description and prediction.

Are there any other collaboration environments that can be considered for this line of research? How do you envision these findings being generalized to other collaboration tasks? -- We believe research in repeated teaming with collaborative, transparent agents should be studied in more complex domains such as Minecraft, Dota 2, and Starcraft. Our work utilizes the relatively low-dimensional Overcooked setting where techniques like online optimization and tree manipulation by a human end-user can be done within a short time period. This is vital in allowing for a feasible single-session user study to be conducted.

However, many of our takeaways generalize beyond our single setting of Overcooked to the field of human-AI collaboration and also have applicability to real-world applications in collaborative robotics. In many settings, teams of agents and humans will need to go through stages of team development. Our work shows that model transparency and the ability to interact with policies help in this regard. While there may be domains where white-box models are already competitive with the initial performance of black-box models, there is still research needed to improve interfaces so that users can interact successfully with agent models. Within our domain, we found through a usability survey that there was a large disparity between users finding the interface good (>75) and bad (<35). This finding also generalizes to many domains that have human collaborators coming from different backgrounds and expertise levels.

After considering all the feedback and responses, while I am not fully convinced of the paper's methodological strength, the idea and its execution are both original and solid. The paper offers insightful findings, which motivated me to raise my score.