Understanding Your Agent: Leveraging Large Language Models for Behavior Explanation

3) On the subject of ‘optimality’

We don’t assume optimal behavior, and will change the subtitle of Figure 5. The major confusion is likely raised by Figure 3, where the example includes the LLM response “The medic did not consider this information when making its decision, which would imply a non-optimal decision-making process on the part of the medic.” Note that this text is entirely generated by the LLM. Interestingly, GPT-4 assumes the exploitative policy as optimal and the explorative policy as suboptimal. Relevant discussions on this have been covered in section 4.1.

4) Regarding concerns about the user studies

The fact that humans didn’t identify explanations unfaithful to the agent doesn’t necessarily lead to the conclusion that our explanations didn’t convey sufficient information. Instead, it points out that users may overly pay attention to prose, so it’s particularly important to ensure the content provided to the users is faithful. To demonstrate this, we describe our user study in more depth. For each of the 20 cases, the user is given a figure of the current state, a figure of the future state (in which the agent accomplishes its next goal), and two explanations of random order. Then they are asked the question “which explanation best helps you understand the agent's behavior”. We provide the future state such that participants can better evaluate explanation helpfulness with respect to the agent’s actual behavior.

In the study introduction, we explicitly informed participants that explanations may not be correct, so they were aware of this. However, the fact that participant preference was not affected by hallucination indicates either that, a) participants were unable to identify hallucinations even given the current and future states, or b) participants did identify it but considered it inconsequential. Our study design does not allow us to disentangle these effects, but it does not emphasize the necessity of using explanations which minimizes inaccuracies and hallucinations, which we show that our approach excels at.

Due to our study design, we could not ask participants to predict the agent’s next action (because we provided them with the future state). In this case, we opted for giving more information to participants such that they can better judge explanation helpfulness by considering the agent’s future behavior, rather than withholding it and analyzing explanation comprehension by prompting participants to predict actions themselves.

5) Minor comments

As stated in the paper, Template refers to a textual representation of the decision path. Examples have been provided both in Figure 2 and in our appendix. Thanks for pointing out these issues, we’ll update the manuscript accordingly.

Regarding the high hallucination rate, the maximum rate for BR (Path) is 20% for the explorative short-term policy, with the average rate falling at 10.7% across all policies. This is significantly lower than 30%, and we note that a considerable amount of hallucinations are from reasoning instead of from facts. Examples of them are “If the engineer is in room (0, 0) and moves north, they will be in room (0, 0)” while the engineer cannot move north in the first place, “there is no rubble blocking the way” while rubble is not blocking the way in our settings. Wrong presentation of facts is extremely rare in our proposed method.

We would like to thank Reviewer ccmh for their time and detailed feedback. We appreciate that you find our motivation and methodology strong. Below is our response to your raised concerns and questions.

1) The approach seems somewhat limited in scope, requiring a considerable amount of work from a designer and a description of the world model

We argue that our approach actually requires considerably less effort than previous methods, which typically involve building a dataset of annotated natural language explanations for agent state-action pairs and fine-tuning a language model. Policy distillation is a fairly straightforward process (we use DAGGER in this work, see our response to #2 below) that only has to occur once. Similarly, a domain expert has to define a natural language description of the task MDP and a few in-context learning examples, but these are also only one-time efforts and are trivial in comparison to constructing an entire dataset for fine-tuning (as most prior methods require). We will clarify the writing to more clearly indicate that the description of the MDP is a requirement.

2) Being able to summarize behaviors into a decision tree

We use dataset aggregation (DAGGER)[1] to sample state-action pairs and train the decision tree (using CART). DAGGER proceeds by collecting a set of visited states at each iteration under the current distilled policy and the corresponding actions from the agent’s original policy, which is aggregated to the set of all previous state-action pairs and used to train the decision tree.

The state of our environment is an NxMx5 tensor where N is the number of rows (5 in our case) and M is the number of columns (4). The last dimension represents a binary feature vector for each room and consists of: the presence of the medic, the presence of the engineer, the presence of rubble, the presence of a victim (may be hidden if there is rubble), and whether the room has been explored.

[1] Ross, Stéphane, Geoffrey Gordon, and Drew Bagnell. "A reduction of imitation learning and structured prediction to no-regret online learning." AISTATS 2011.

We quantified the distilled policy accuracy by rolling out 5000 episodes with random initializations, and computing the action accuracy using the original policy’s action as the target. The accuracy values are given below where we can observe a range from ~83% to 100%. We will add these results to the appendix of our paper.

We would like to point out a few things with regard to the distilled policy.

1. Our environment was intentionally designed so as to exhibit ambiguous states, where different behaviors will produce the same action at time t, but different actions at time t+1. This ambiguity is the cause for some of the loss in performance (especially the explore policies).
2. We only produce explanations when the action of the distilled DT matches that of the original policy. So in the worst case here, we still explain ~83% of the agent’s actions, which we think is reasonable.

I thank the authors for their detailed rebuttal comments; they were indeed helpful for clarity. Unfortunately, I share some of the concerns with the other reviews and also agree that the rebuttal comments (while helpful for elucidating the approach the authors have taken) do not resolve the more fundamental issues with the approach that I (and others) have pointed out. The two central lingering concerns are:

1. The simplicity of the proposed domain and policies. While having a simple domain is great for showing that the method aligns with designer expectations, it is insufficient to show its potential more broadly and I am not convinced that this will work on domains more complex than the one shown. I am also still a bit on the fence about whether the effort put forth by the designer is an issue. My instinct still remains that the designer's work to design; however, the authors' comments about this step being a "one time effort" did reduce my concerns somewhat.
2. The limitations of the user study, both in its lack of ability to test user's ability to use explanations to demonstrate better understanding of the agents and also the uncertainty over the impact of hallucinations (whether factual or not) are not addressed.

I think there is potential for this work, and I am excited to see it move forwards. However, I do not think that some of the fundamental questions have been worked out yet. As such, my score remains unchanged.

3) How were the examples hand-annotated and how did we avoid over-fitting to to our choice of policies?

We first note that the agent policies themselves were exhibiting goal-directed behavior in our experiments by design (we explicitly crafted policies to exhibit exploitation and exploration type behaviors). Any reasonably accurate distilled policy will then mimic such goal-directed behavior as well. This actually ties into our broader point: we don’t necessarily agree that goal-directed metrics are inappropriate for explaining agent behavior in general. In some sense, any explanation which attempts to reason about high-level behavior is at risk of “anthropomorphic bias” unless a policy explicitly includes hierarchical behavior planning or computational theory of mind models. And yet, we can clearly observe goal-directed behavior in standard deep neural network policies that do not include such mechanisms [2] – to such a degree that it is useful for humans to reason about agent behavior in terms of them. So while there may be cases in which inferred goals do not explicitly represent the underlying policy, we feel that the idea and metrics in general still have merit, particularly from a comprehensibility standpoint for humans.

[2] Vinyals, Oriol, et al. "Grandmaster level in StarCraft II using multi-agent reinforcement learning." Nature 2019.

We include additional details regarding the hand-annotation below.

We designed policy-agnostic quantitative metrics to make sure our evaluation approach can be applied to other tasks and domains. These are Strategy, Category, Goal, Action, and Intent accuracy. These metrics are intended to work with any environment and agent that exhibits high-level strategies and low-level goals so as to be useful for future work. The downside is that such metrics must be manually annotated from natural language explanations, requiring considerable human effort. However, we find these provide far more detailed insights than automated metrics which may be erroneous (e.g. asking an LLM to evaluate itself) or only operate at the dataset-level (e.g. BLEU, BERTscore). We define our metrics in the manuscript in Sec. 4 but restate them here with additional details.

Suppose that the engineer is taking an exploitative strategy, and the decision path largely consists of decision rules operating over rubble locations (similar to the same path in Fig. 2).

• Strategy: Whether the overall strategy is identified. In this case, whether the explanation contains “The agent seems to prioritize the removal of rubble” for an exploitative engineer, as opposed to “The agent seems to be prioritizing exploration” for an explorative engineer.

• Category: Whether the agent’s goal category was identified in the explanation. For example, if the explanation is “The engineer is prioritizing the removal of rubble and moves south to room (0,1) as that brings it closer to room (0,4) which contains rubble”, then the explanation correctly identifies that the agent is seeking out rubble (which happens to be in room (0,4) in this case, although the specific location is not part of the category).

• Goal: Whether the agent’s specific goal was correctly identified in the explanation. In the explanation above, if the agent’s actual goal was to remove the rubble in room (0,4) (which we can observe by looking at future actions), then the goal is correct. However, if the agent instead moved south to room (0,1) in order to then move east and remove rubble in room (1,1), the goal would be incorrect even though the category would still be correct.

• Action: Whether the agent’s next action was successfully predicted.

• Intent: Whether the agent’s intent for taking the next action was successfully identified. If the explanation is “The engineer will move south to room (0, 3) to get closer to room (0, 4) which contains rubble” for an exploitative policy, Intent is correct. If the explanation is “The engineer will move south to room (0, 3) to get closer to room (1, 4) which is unexplored” for an exploitative policy (which should take rubble in room (0, 4) as priority), Intent is incorrect.

We would like to thank Reviewer Pb35 for their thorough and thoughtful review. We appreciate the positive comments about our method and that you think our idea is interesting. We respond to your questions and concerns below.

1) How model-agnostic is this approach given that it seems to rely on interpretable state representations?

You are correct that the distillation process used here requires inputs which can be directly consumed by a decision tree (and also interpreted by an LLM, but the DT is the bigger constraint in our experience). However, we also feel that our approach is applicable to environments which contain different state spaces, e.g. images or uninterpretable latent representations. We argue that such representations can be handled by first extracting a set of semantic features, e.g. visual concepts [1], which are then used to distill a structured behavior representation. Ultimately this is an issue that applies to all interpretable/explainable methods – how do we construct human-intelligible representations from inherently uninterpretable ones – and so we don’t feel that this is unique to our method, particularly as we are not directly addressing this problem.

[1] Koh, Pang Wei, et al. "Concept bottleneck models." ICML 2020.

2) Can this method generalize to a wider range of policies, or to a richer range of environments?

These are reasonable points, which we think can be reduced to: how descriptive can a decision tree/path be of an agent’s strategy, intent, and goal? We agree that it may be possible for a given decision path to be interpreted multiple ways, and that the generated explanation is influenced by the in-context learning examples which are given in the prompt. However, we feel that is also a strength of this approach: if an agent is capable of producing similar-looking decision paths that may have different underlying intents, then this can be represented through in-context learning examples which themselves express contradictory explanations or uncertainty. We noted this behavior ourselves in the paper, e.g. when providing the model with no behavior representation (no DT path), the LLM often exhibited uncertainty and refrained from generating specific explanations for agent behavior (even when using the “standard” set of in-context examples in this case). There is even a potential for prompting the LLM for a set of explanations and then measuring agreement above them to quantify certainty.

Despite this, we emphasize that our approach has several benefits over previous methods:

1. We leverage the encoded knowledge within the pre-trained LLM to avoid having to curate annotated datasets of states and explanations, which is impractical when attempting to explain arbitrary agent behavior.
2. The LLM itself is capable of performing reasoning over the given inputs (guided by the in-context examples), which often leads to more useful explanations as a human observer no longer has to perform this reasoning themselves (e.g. from textual translations of decision paths).

Lastly, we want to note that decision trees are rather expressive in terms of behavior. For example, with your example of an agent that is exploratory in the top-half of the grid and exploitative in the bottom-half, we would expect a decision tree to represent this with different decision path structures for rooms in the top half vs bottom half.

We would like to thank Reviewer 5qZT for their time and insightful comments. We are glad that you appreciate the model-agnostic nature of our approach, the interactivity it enables, and our qualitative/quantitative evaluation. Below is our response to your raised concerns and questions.

1) Few details are given of the agent, the decision tree, environment, and the language model

We have provided details regarding our experiments in Appendix A, but we restate them here with some additional detail for clarity:

Environment: The environment is a partially observable multi-agent urban search and rescue task, consisting of a grid of inter-connected rooms that may contain rubble and/or victims. There are two agents with heterogeneous skills: a medic which can save victims and an engineer which can remove rubble. The goal of the task is to save the victims, however, they may be trapped by rubble if both a victim and rubble are present in the same room. In this case, the agents must work together: the engineer must remove the rubble so the medic can save the victim.

Agent: We leverage two types of agent behaviors, an exploration type and an exploitation type. The exploration type will first explore all rooms to identify exposed victims and rubble (which may possibly hide trapped victims). The exploitation type acts as a greedy agent, which begins by exploring but acts to either save victims or remove rubble (depending on the agent type) as soon as one is found. All of our results produce explanations for both the medic and engineer agents.

Language model: We use GPT4 via OpenAI’s developer API. We include a concise description of the environment (similar to above), a description of the behavior representation (if there is one), and three in-context learning examples.

Decision tree: We use dataset aggregation (DAGGER)[1] to sample state-action pairs and train the decision tree (using CART). DAGGER proceeds by collecting a set of visited states at each iteration under the current distilled policy and the corresponding actions from the agent’s original policy, which is aggregated to the set of all previous state-action pairs and used to train the decision tree.

[1] Ross, Stéphane, Geoffrey Gordon, and Drew Bagnell. "A reduction of imitation learning and structured prediction to no-regret online learning." AISTATS 2011.

3) Preference is heavily involved in the user study regarding the evaluation of behavior explanations

The quality of the generated explanations are evaluated from a holistic perspective, with the quantitative metrics evaluating faithfulness and the human study evaluating preference. We believe that a good explanation needs to be both faithful and comprehensible from a human's perspective. It is true that users didn't prefer the non-hallucinated explanations over hallucinated ones (Sec. 5.1), but whether this is because they considered the hallucination inconsequential or because they didn’t detect it in the first place we cannot say. However, this supports our argument that we need to generate explanations which minimize hallucinations in the first place, and our quantitative results are intended to show that this is the case. If we use explanations with hallucinations and users blindly trust them, this can mislead the user and potentially lead to dangerous situations.

2) Concerns about generalization to other environments, language models, and continuous action spaces

Our method was evaluated on a single environment due to the time-consuming nature of hand-annotating responses for quantitative evaluation and running user studies. However, we argue that the depth of our analysis on this environment offers intriguing insights, and this doesn’t limit the generalization of our approach to other environments.

Environment and policy-agnostic metrics: We designed policy-agnostic quantitative metrics to make sure our evaluation approach can be applied to other tasks and domains. These are Strategy, Category, Goal, Action, and Intent accuracy. These metrics are intended to work with any environment and agent that exhibits high-level strategies and low-level goals so as to be useful for future work. The downside is that such metrics must be manually annotated from natural language explanations, requiring considerable human effort. However, we find these provide far more detailed insights than automated metrics which may be erroneous (e.g. asking an LLM to evaluate itself) or only operate at the dataset-level (e.g. BLEU, BERTscore). We define our metrics in the manuscript in Sec. 4 but restate them here with additional details.

Suppose that the engineer is taking an exploitative strategy, and the decision path largely consists of decision rules operating over rubble locations (similar to the path in Fig. 2).

• Strategy: Whether the overall strategy is identified. In this case, whether the explanation contains “The agent seems to prioritize the removal of rubble” for an exploitative engineer, as opposed to “The agent seems to be prioritizing exploration” for an explorative engineer.

• Category: Whether the agent’s goal category was identified in the explanation. For example, if the explanation is “The engineer is prioritizing the removal of rubble and moves south to room (0,1) as that brings it closer to room (0,4) which contains rubble”, then the explanation correctly identifies that the agent is seeking out rubble (which happens to be in room (0,4) in this case, although the specific location is not part of the category).

• Goal: Whether the agent’s specific goal was correctly identified in the explanation. In the explanation above, if the agent’s actual goal was to remove the rubble in room (0,4) (which we can observe by looking at future actions), then the goal is correct. However, if the agent instead moved south to room (0,1) in order to then move east and remove rubble in room (1,1), the goal would be incorrect even though the category would still be correct.

• Action: Whether the agent’s next action was successfully predicted.

• Intent: Whether the agent’s intent for taking the next action was successfully identified. If the explanation is “The engineer will move south to room (0, 3) to get closer to room (0, 4) which contains rubble.” for an exploitative policy, Intent is correct. If the explanation is “The engineer will move south to room (0, 3) to get closer to room (1, 4) which is unexplored.” for an exploitative policy (which should take rubble in room (0, 4) as priority), Intent is incorrect.

Environments with high-dimensional inputs, continuous action spaces, and higher complexity: Our approach readily supports continuous action spaces as-is; decision trees can be used for regression tasks (continuous actions) and most tree-construction algorithms natively support this, e.g. CART, ID3. High-dimensional inputs such as images can also be supported by first extracting a set of semantic features from the images, e.g. visual concepts [2].

As for complex environments which induce complex behaviors, this gets into the issue of how complex of a policy a decision tree can encode. While we do not directly investigate this question in our work, we can conjecture here about what we expect. Previous works [3] have shown that DTs can achieve surprisingly good performance purely through distillation, however, it is likely that this approach may not work for all complex policies. When that is the case, we expect that our method would also work with other model representations for the behavior. For example, instead of a decision tree maybe we utilize a gradient boosted tree or a differentiable decision tree (that is “crispified” [4]). While we have not investigated these specific approaches yet, LLMs have shown at least some ability to interpret even complex neural network representations [5], so we expect it would perform at least as well on decision tree variants.

[2] Koh, Pang Wei, et al. "Concept bottleneck models." ICML 2020.

[3] Bastani, Osbert, Yewen Pu, and Armando Solar-Lezama. "Verifiable reinforcement learning via policy extraction." NeurIPS 2018.

[4] Paleja, Rohan, et al. "Learning Interpretable, High-Performing Policies for Autonomous Driving." RSS 2022.

[5] Bills, Steven, et al. "Language models can explain neurons in language models." 2023.

3) How the performance of the tree compares with the surrogate policy, and how the performance gap influence the generated explanations

We quantified the distilled policy accuracy by rolling out 5000 episodes with random initializations, and computing the action accuracy using the original policy’s action as the target. The accuracy values are given below where we can observe a range from ~83% to 100%.

We would like to point out a few things with regard to the distilled policy.

1. Our environment was intentionally designed so as to exhibit ambiguous states, where different behaviors will produce the same action at time t, but different actions at time t+1. This ambiguity is the cause for some of the loss in performance (especially the explore policies).
2. We only produce explanations when the action of the distilled DT matches that of the original policy. So in the worst case here, we still explain ~83% of the agent’s actions, which we think is reasonable.

4) Generalization to environments with continuous/dense inputs

Our approach readily supports continuous observations and action spaces as-is; decision trees can split over continuous features and also be used for regression tasks (continuous actions), and most tree-construction algorithms natively support this, e.g. CART, ID3. High-dimensional inputs such as images can also be supported by first extracting a set of semantic features from the images, e.g. visual concepts [1].

As for complex environments which induce complex behaviors, this gets into the issue of how complex of a policy a decision tree can encode. While we do not directly investigate this question in our work, we can conjecture here about what we expect. Previous works [2] have shown that DTs can achieve surprisingly good performance purely through distillation, however, it is likely that this approach may not work for all complex policies. When that is the case, we expect that our method would also work with other model representations for the behavior. For example, instead of a decision tree maybe we utilize a gradient boosted tree or a differentiable decision tree (that is “crispified” [3]). While we have not investigated these specific approaches yet, LLMs have shown at least some ability to interpret even complex neural network representations [4], so we expect it would perform at least as well on decision tree variants.

[1] Koh, Pang Wei, et al. "Concept bottleneck models." ICML 2020. [2] Bastani, Osbert, Yewen Pu, and Armando Solar-Lezama. "Verifiable reinforcement learning via policy extraction." NeurIPS 2018. [3] Paleja, Rohan, et al. "Learning Interpretable, High-Performing Policies for Autonomous Driving." RSS 2022. [4] Bills, Steven, et al. "Language models can explain neurons in language models." 2023.

We would like to thank Reviewer NiyW for their time and helpful feedback. We appreciate that you consider the model-agnostic nature of our method a strength. Below is our response to your raised concerns and questions.

1) Additional information on metrics

We designed policy-agnostic quantitative metrics to make sure our evaluation approach can be applied to other tasks and domains. These are Strategy, Category, Goal, Action, and Intent accuracy. These metrics are intended to work with any environment and agent that exhibits high-level strategies and low-level goals so as to be useful for future work. The downside is that such metrics must be manually annotated from natural language explanations, requiring considerable human effort. However, we find these provide far more detailed insights than automated metrics which may be erroneous (e.g. asking an LLM to evaluate itself) or only operate at the dataset-level (e.g. BLEU, BERTscore). We define our metrics in the manuscript in Sec. 4 but restate them here with additional details.

Suppose that the engineer is taking an exploitative strategy, and the decision path largely consists of decision rules operating over rubble locations (similar to the same path in Fig. 2).

• Strategy: Whether the overall strategy is identified. In this case, whether the explanation contains “The agent seems to prioritize the removal of rubble” for an exploitative engineer, as opposed to “The agent seems to be prioritizing exploration” for an explorative engineer.

• Category: Whether the agent’s goal category was identified in the explanation. For example, if the explanation is “The engineer is prioritizing the removal of rubble and moves south to room (0,1) as that brings it closer to room (0,4) which contains rubble”, then the explanation correctly identifies that the agent is seeking out rubble (which happens to be in room (0,4) in this case, although the specific location is not part of the category).

• Goal: Whether the agent’s specific goal was correctly identified in the explanation. In the explanation above, if the agent’s actual goal was to remove the rubble in room (0,4) (which we can observe by looking at future actions), then the goal is correct. However, if the agent instead moved south to room (0,1) in order to then move east and remove rubble in room (1,1), the goal would be incorrect even though the category would still be correct.

• Action: Whether the agent’s next action was successfully predicted.

• Intent: Whether the agent’s intent for taking the next action was successfully identified. If the explanation is “The engineer will move south to room (0, 3) to get closer to room (0, 4) which contains rubble.” for an exploitative policy, Intent is correct. If the explanation is “The engineer will move south to room (0, 3) to get closer to room (1, 4) which is unexplored.” for an exploitative policy (which should take rubble in room (0, 4) as priority), Intent is incorrect.

2) More details on tree-based distillation

We use dataset aggregation (DAGGER)[1] to sample state-action pairs and train the decision tree (using CART). DAGGER proceeds by collecting a set of visited states at each iteration under the current distilled policy and the corresponding actions from the agent’s original policy, which is aggregated to the set of all previous state-action pairs and used to train the decision tree.

The state of our environment is an NxMx5 tensor where N is the number of rows (5 in our case) and M is the number of columns (4). The last dimension represents a binary feature vector for each room and consists of: the presence of the medic, the presence of the engineer, the presence of rubble, the presence of a victim (may be hidden if there is rubble), and whether the room has been explored.

[1] Ross, Stéphane, Geoffrey Gordon, and Drew Bagnell. "A reduction of imitation learning and structured prediction to no-regret online learning." AISTATS 2011.