REVEAL-IT: REinforcement learning with Visibility of Evolving Agent poLicy for InTerpretability

Thank you for your time and valuable suggestions! We have updated the paper following your suggestions; please refer to the highlighted part in the rebuttal revision. Now, we address your main concerns as below:

REVEAL-IT does not require for stronger assumptions than other interpretability methods.

However, this work assumes (unless I am wrong - see below) the environment provides a set of subtasks that can be trained on, which is a large assumption. Therefore the generality of this method is somewhat limited. To what extent do subtasks need to be provided/can be inferred?

• The The main reason that the traditional conterfacutal/causal methods can't work in complex environments/tasks is that constructing a true causal model/SCM relies on a perfect understanding of the data generation process, any correlation, intervention and even counterfactual inquiries. However, given the inherent complexity of the world, it is often impractical to access a fully specified SCM. This claim has been fully discussed in previous works and survey papers[1], [2]. [1] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. [2] Jean Kaddour, Aengus Lynch, Qi Liu, Matt J. Kusner, and Ricardo Silva. Causal machine learning: A survey and open problems.

What if the environment/task does not have apparent sub-task structure for training?

I am also under the impression that environment subtasks are needed for REVEAL-IT, but the authors perform experiments on OpenAI Gym MuJoCo environments, which don't have them?

• Regarding to the GYM environment, we agree that there are no obvious sub-tasks in GYM or similar environments. Instead of using a random sample from the replay buffer (PPO or other online RL methods), we store the trajectories that bring the agent higher learning progress and do the experience replay based on these trajectories (similar to the idea in HER[1]).

[1] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, Wojciech Zaremba, Hindsight Experience Replay, NIPS 2017.

Is REVEAL-IT an addition to existing algorithm?

Table 2 indicates that adding REVEAL-IT to existing methods improves their performance, but in Table 1 the authors present REVEAL-IT as its own algorithm, so once again it is unclear what is going on here. Is REVEAL-IT standalone or an addition to existing algorithms?

• In terms of the structure of our method, we can say REVEAL-IT is an addition to existing algorithms. However, REVEAL-IT does not have any constraints on the basic RL algorithm used for the control policy training. This is the reason we compare REVEAL-IT (PPO-based) with different baselines in ALFworld, and we compare with classical RL algorithms in GYM.

Curriculum learning is not the key contribution.

The methodology is a bit unclear ... and then halfway through the paper the authors introduce a GNN predictor and curriculum learning. Curriculum learning is not discussed at all in the Related Works section. And yet the authors show that their method significantly outperforms other methods in ALFWorld (Table 1), so it is clear that this is not just about interpretability.

• REVEAL-IT does not exclusively focus on interpretability but acts as a framework that improves the understanding of an RL agent's learning process and improves the agent's learning efficiency based on it. These two contributions are complementary rather than contradictory.
• We do not claim to innovate in curriculum learning itself but use it to demonstrate the utility of its explanation-driven approach. Curriculum learning is a tool within REVEAL-IT to operationalize the insights derived from the GNN explainer. We have clarified this in Line.163-166.
• Although curriculum learning is not our main contribution, we agree that including the discussion in related work can improve the paper. Thank you for the suggestion. Please refer to the new vision for the udpates.

First of all, I would like to thank the authors for their detailed response. I still have a few questions that I would appreciate further clarification on.

• For Q1, my question is whether training the same model (using essentially identical policy architectures) multiple times would yield consistent interpretability results. How do you ensure alignment of the neural representations in this case? Perhaps I am misunderstanding something, but further clarification would be greatly appreciated.

• Regarding the question I raised about the "alignment between the learned structural information and the actual policies or concepts," I did not find a specific answer to this. Could the authors provide some clarifications or pointers?

• Regarding this one:

REVEAL-IT is designed to intuitively visualize the learning process of the agent and help people understand what specific abilities the agent has learned in a subtask to enable it to complete the final task. Therefore, we choose to visualize the policy update process, but considering the complex structure of the policy itself and the excessive number of updates. Even with the visualization, the results are not easy to understand; thus, we design a GNN explainer to simplify the policy graph and highlight important updates.

I understand your point here, but what confuses me is whether there is any fundamental evidence demonstrating that this approach is inherently better than alternatives, such as using magnitude, partial derivatives, or conditional mutual information, which I believe can also be used for visualization.

• Side note: Thank you for considering and comparing additional encoders. Very impressive and helpful

Thank you for your time and valuable suggestions! We have updated the paper following your suggestions, please refer to the highlighted part in the rebuttal revision. Now, we address your main concerns as below:

REVEAL-IT is not trained across models.

The authors assert that variability in learning across models will not pose an issue since the learned explainer is model- or task-specific. However, for applications in multi-task learning, pre-training, or other forms of generalization, this variability is crucial and challenging.

• REVEAL-IT is purposefully designed to address specific tasks and environments rather than pre-trained setups. On the other hand, ALFworld is a benchmark that contains different tasks (refer to the task list in Appendix A). Therefore, the specificity is a feature, not a limitation, ensuring tailored explanations and efficient training for a given task or model.
• REVEAL-IT does not have any specific requirements for the basic RL algorithm for training the control policy, which is not equal to REVEAL-IT can work with different RL algorithms in different environments/tasks simultaneously.

REVEAL-IT can work with simple networks, but it does not target on this.

The impact of network size on the results should be investigated through ablation studies. If the network size is small, do the same phenomena observed in Figure 2 still occur?

• If the RL policy's structure is overly simplistic, i.e., every node can be completely active during the testing phase, the role of REVEAL-IT may be constrained. However, generally speaking, a simple policy typically correlates to a simple job and environment, which is not the aim that REVEAL-IT needs to solve.
• On the other hand, the structure of RL policy in GYM is simpler than it in ALFworld. The results in Table 2 prove that REVEAL-IT can still bring some improvement in some tasks.

The benefits of GNN explainer.

What are the primary benefits of using GNNs for contribution analysis of each node or weight? Why not directly use magnitude, partial derivatives, or conditional mutual information to assess the importance of each weight?

• REVEAL-IT is designed to intuitively visualize the learning process of the agent and help people understand what specific abilities the agent has learned in a subtask to enable it to complete the final task. Therefore, we choose to visualize the policy update process, but considering the complex structure of the policy itself and the excessive number of updates. Even with the visualization, the results are not easy to understand; thus, we design a GNN explainer to simplify the policy graph and highlight important updates.

The gap between the learned structural information and the policy has already been addressed.

To ensure the framework is generalizable and learns universal, principled representations, it would be beneficial to further explore the alignment between the learned structural information and the actual policies or concepts, either empirically or theoretically.

• The GNN explainer in REVEAL-IT is trained on the visualized graphs of the control policy updates. Meanwhile, the experiment results in ALFWorld demonstrate that the framework can identify and utilize key updates to optimize task sequencing, which indirectly validates the alignment between learned structures and policies.

Compare with other encoder methods.

Approaches could include using sparse autoencoders (potentially with larger models) [1] or examining the alignment between individual components and their corresponding concepts, modularities, interactions, and causal relations [2-5].

• In ALFworld, REVEAL-IT is trained and tested in the visual environment, which has no prior/extral knowledge from the text engine (text world in ALFworld). This is the same for the baseline methods (BLIP-2, LLaMA-Adapter, InstructBLIP and MiniGPT-4.)
• To compare with other encoder methods, we need to train REVEAL-IT in the text engine (without visual environment), and also we need to compare it with the baselines in the same conditions. Therefore, we have added new experiments for comparison with LLM-based baselines. We report the results in the table below, and we have also included it in the paper. Please note that the experiment setting is different with the conditions in Table 1.

ReAct: Shunyu, et al. React: Synergizing reasoning and acting in language models. In ICLR, 2023. Reflextion: Noah Shinn, et al. Reflexion: Language agents with verbal reinforcement learning. In NeurIPS, 2023. AutoGen: Qingyun Wu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversation framework.

Thank you for your time and valuable suggestions! We have updated the paper following your suggestions, please refer to the highlighted part in the rebuttal revision. Now, we address your main concerns as below:

The definition of activated nodes in RL policy.

Question 1&2: What is the GNN Explainer’s training objective? What is the definition of “active nodes” in Step 1 of section 4.2?

• We add a ReLU function on the first and third layers in the policy network to visually identify the activated nodes through the evaluation process. We introduced this in section 5.2 line 352-355.
• The GNN explainer learns to simplify the visualization of policy updates to help humans understand in which sub-tasks the agent has learned relevant capabilities to complete the task. Please refer to the weakness.4 in our reply to reviewer mUUC for more details of GNN explainer.
• The GNN explainer is trained to simplify the original graph (policy visualization) to a simpler sub-graph that the GNN predictor can still make the same prediction on it.

The sub-task sequence optimization in REVEAL-IT.

How exactly does the GNN explainer choose the distribution of subtasks to train on? and how does it help in GYM environment?

• GNN explainer does not directly change the distribution of subtasks for training. Alternatively, we rank the sub-tasks in terms of the learning progress predicted by the GNN predictor. We introduce this in section 4.2 Line 247-251. To avoid falling into the local optimum and the situation where the predictor is not accurate enough in the early stage of training, we introduced $\epsilon-$greedy in task selection (refer to Line4 in Alg.1).
• Regarding to the GYM environment, we agree that there are no obvious sub-tasks in GYM or similar environments. Instead of using random sample from the replay buffer (PPO or other online RL methods), we store the trajectories that bring the agent higher learning progress and do the experience replay based on these trajectories (similar to the idea in HER[1]).

[1] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, Wojciech Zaremba, Hindsight Experience Replay, NIPS 2017.

GNN explainer does not choose the sub-tasks for training.

What is the GNN Explainer’s training objective? If it is only trained to preserve the predictor’s accuracy, then it could just output the full graph. How exactly does the GNN explainer choose the distribution of subtasks to train on?

• We have already provided learning objectives for the GNN explainer in Appendix C in the original edition. And now, we move this part to Line.310-320. The purpose of the GNN explainer is to prune off unimportant edges (unimportant updates in policy). Due to space limitations and the fact that this is not much different from other GNN explainer methods from a technical perspective, we put this part in the appendix.
• We store the visualized policy update information and the corresponding RL evaluation progress into a buffer, and use samples from this buffer to train the GNN explainer. Therefore, GNN explainer does not need to choose the sub-tasks.

Multi-modal challenges.

The conclusion states that REVEAL-IT can’t adapt to multi-modal challenges.

• The structure and environment of RL policy are not restricted by REVEAL-IT. In theory, REVEAL-IT should be applicable to multi-modal challenges. However, we find this could make the paper presentation more complex. For instance, in ALFworld, two distinct policies are necessary to interact with the visual engine and the text engine. We are of the opinion that the problem will be overcomplicated if REVEAL-IT is employed in multiple policies/agents tasks. We are extremely appreciative of the valuable suggestions you have provided, as they are of significant importance to our future research. We also concur that including such a statement directly in the conclusion would likely lead to reader confusion, and we have made the necessary adjustments.

More explanations on Figure 2.

• The gray shaded regions. The gray shaded regions in Fig.2 represent the shared nodes that have significant updates across different sub-tasks. Please note the node number in Fig.2, we understand that the number could be too small to read; there is a larger version in the appendix.
• Do thicker connections indicate weights that were both selected by GNN explainer and had large updates in amplitude? Yes.
• the region triggered by evaluation. We have added a ReLU function in the first and third layers in the control policy so that we can check which nodes are activated during the evaluation phase.

The link to view the project on line 311 is broken.

• We checked the link and found it can work. Maybe this is caused by the error of the website.

Thank you for your responses.

Regarding to the GYM environment, we agree that there are no obvious sub-tasks in GYM or similar environments. Instead of using random sample from the replay buffer (PPO or other online RL methods), we store the trajectories that bring the agent higher learning progress and do the experience replay based on these trajectories

Thanks for explaining; this explanation appears to be missing from your paper. I think it's important to include it.

We have already provided learning objectives for the GNN explainer in Appendix C in the original edition. And now, we move this part to Line.310-320.

I find the description you provided in lines 310-320 to be unclear- it looks like you provide an equation for the traditional GNN explainer learning objective, but I do not see an equation for the objective used by REVEAL-IT

You did not answer my question from my review about Figure 2:

How were the portions of the policy that are common to several sub-tasks identified?

In addition, you did not address the main weaknesses I raised with this paper- the writing is still very difficult to follow, and I do not see how the your method provides any helpful human-interpretability to the deep RL models. Could the authors provide concrete examples of helpful, actionable insights they gained from looking at the outputs of the GNN Explainer?

Thanks for your response and further clarifications.

We visualize a value map based on the trained policy by REVEAL-IT in AFLworld to show the ability that the agent learns.

How exactly do you compute this value map? And what insights does this provide?

Thanks for the response. We are glad to hear that we have reached a consensus on the limitations of traditional XRL methods. However, we disagree with your point that the GNN explainer is not helpful for understanding the agent's behavior when the policy is large. Graph tasks widely adopt GNN pruning as a technique for explaining GNN, particularly in scenarios where the GNN network is too large. Pruning does not affect performance, and it can help humans understand the key nodes in the GNN. This is consistent with the purpose of REVEAL-IT. Through pruning, we can also understand the important nodes and corresponding weights in the policy, pay attention to their changes during training, and better understand the learning process of the agent. On the other hand, based on the understanding of the policy, REVEAL-IT also optimizes the task sequence for training, which achieves better performance and learning efficiency of RL.

Thank you for your time and valueable suggestions! We have updated the paper following your suggestions, please refer to the highlighted part in the rebuttal revision. Now, We address your main concerns as below:

The basic definitions in RL.

Question 2: what is the objective of the controller, and what purpose does the control policy serve? Question 3: does "updating the policy" equate to "updating the agent’s learning process"?

• The controller is the control policy $\pi_t$, which is used to control the RL agent to complete tasks in the environment.
• The learning objective of $\pi_t$ is following the standard setting in RL, i.e., we train it to maxmize the accumalted reward $G_T$, which is defined as: $G=\sum_{t=0}^{T} \gamma^t R_{t+1}$, where $T$ denotes the maximal timesteps for each trajectory.
• Yes. Updating the policy is equal to the learning process of the RL agent. We will change the expression here. Thank you for the suggestion!

The explanations provided by REVEAL-IT based on the activated nodes.

Question 1: what is the precise format of the explanation the authors intend to provide? Is the optimal training task sequence itself considered the explanation, as suggested in Section 5.2 (lines 429-471)? Question 4: could the authors elaborate on the terms “nodes linked to significant updates” and “activated nodes during the test” in Section 4.2, specifically how their correlation is analyzed?

• REVEAL-IT is able to provide explanations in complex environments and tasks differs that the other conventional explanation methods or causal RL methods CANNOT (they usually target simple 2D environments, such like GYM, roboschool).
• In the tesh phase, some nodes in the control policy will be actviated to control the agent to complete the task. These nodes are strongly related to the specific capabilities of the agent. REVEAL-IT explains in which sub-tasks the weights of the links to these nodes are trained and learned, so the update of these weights is "significant updates". Combined with the sub-task sequence, we can understand how the agent gradually learns to complete a complex task in a series of sub-tasks.
• Therefore, we can claim that REVEAL-IT explains the learning process of an RL agent based on the control policy, i.e., REVEAL-IT learns to highlight the important update in the weights of the control policy during the training. In Fig.2, the GNN explainer in REVEAL-IT learns to highlight which parts of the updated weights is important to the success of completing each sub-task (note the thicker gray lines in each graph in fig.2). By comparing the shared important udpates across different sub-tasks, we can understand if there is some shared abilities for the agent in different tasks (note the organe box in fig.2).

Reference effor

where is Figure 3 referenced in the main text?

• We discuss the Fig.3 in the paragraph starts from Line 429. We apologize for the latex reference error, and we have updated this part.

Thank you for the response. We address your concerns as below:

Compare with other RL techniques.

The authors claim that their framework provides explanations in complex environments where other XRL techniques fail. This claim is only partially true. Specifically, the paper needs to clarify why other XRL techniques (e.g., saliency maps, reward decomposition, causal models)

• The main reason we didn't include the comparison with other XRL techniques is that conterfacutal-based/causal-based methods cannot work in a complex environment (e.g., ALFworld). More specifically, saliency mapping, causal RL and counterfacutal methods require to intervening on the RL environment to generate counterfactual states/intervening data for conterfacual learning. In Atari games, this would be fine to work since the environment is quite simple. But intervening in a complex and continuous environment can be challenging.
• More specifically, we want to argue that we are not targeting the same problem as the saliency map. It is intuitive to use state-value to reflect and analyze the agent's behavior in a discrete environment. However, this is not the same thing in a complex and continuous environment, i.e., we have tried saliency mapping (the source code provided in https://arxiv.org/pdf/1912.05743) in the past several days per your request. We create a dataset collected from 134 AlfWorld environments across six different tasks. Note that we only collect 8 pixel figures for each task in each environment, and the dataset reaches 4.4 GB. We tried to train a generator to generate counterfactual states based on it but failed.
• We visualize a value map based on the trained policy by REVEAL-IT in AFLworld to show the ability that the agent learns. Please refer to https://anonymous.4open.science/r/temporary-log-E0F5/README.md to check. We will include this in the next version. We appreciate your suggestions, and we believe this would help in understanding the advantages of our method. We hope this reply can help you understand the target problem of REVEAL-IT.

Human can understand the agent's learning by comparing the visualized policy update across tasks.

it is unclear how users can semantically interpret the visualized sections of the policy weights. Specifically: How can users relate the "highlighted edges" (updated weights) to the RL agent's success?

• The most direct way to understand a deep learning framework is to explain how the network works, like Grad-CAM to highlight the important part in a figure for recognition. However, this is not easy to achieve for RL especially in complex environments/tasks. This is because we cannot directly map a specific section of the policy to an agent's behavior. On the other hand, it is quite cost to compute the state value for every instance from the agent's view in a visual environment.
• To understand how the policy learns to complete a whole task, we designed REVEAL-IT to visualize the policy update information for humans to understand what an ability the agent learns in a sub-task. To make this visualization simpler and easier to read, we deploy a GNN explainer to highlight the important update. Based on this, by comparing the visualized results across different sub-tasks, humans can understand which part of the policy maps to a specific ability to complete a necessary step for the whole task and which part of the policy corresponds to a sharing ability for diverse tasks.

Saliency map

the claim that "REVEAL-IT is distinguished from other explainable RL methods by its independence from external data and structure" is debatable. For instance, saliency maps also operate without imposing environment or algorithm-specific constraints. Understanding performance advantages:

• To our best knowledge, saliency methods are not designed to formalize an abstract human-understandable concept, and they do not provide a means to quantitatively compare semantically meaningful consequences of agent behavior. This leads to subjectivity in the conclusions drawn from saliency maps. Usually, in the RL community, the saliency map relies on the counterfacutal analysis to enhance its subjectivity.

REVEAL-IT does not do planning.

it is unclear whether the advantages stem from improved planning steps or better-learned low-level behaviours

• REVEAL-IT only optimizes the sub-task sequences for training rather than planning. We apologize if we misunderstand the meaning of "planning steps" here. If the "planning steps" means the sub-task planning, we think the two terms have the same meaning, since a better sub-task sequence will make the agent learn basic abilities more efficiently and results in a better performance and improved learning efficiency.