UTILITY: Utilizing Explainable Reinforcement Learning to Improve Reinforcement Learning

Weakness 1.6: "Line 247-257: This whole paragraph seems very ad hoc. Why should we discourage the misleading state-actions? Maybe under a different policy, they are actually necessary for optimal behavior? E.g., consider an agent that must cross some “bridge” to get to the goal. Now, because the agent is not optimal, state actions on the bridge are misleading. But if we restrict the agent from visiting them, it will never get to the goal! The causal entropy term is also not well motivated - why is exploration crucial here?"
Answer: Thanks for mentioning the misleading state-action pairs and entropy term. We discuss these two points respectively as follows.
For the misleading state-action pairs, we agree that there could be false positive, i.e., some misleading state-action pairs should not be avoided. To mitigate this issue, we can impose soft constraint on the misleading state-action pairs, i.e., $b>0$. In this case, we can still visit some misleading state-action pairs to achieve high cumulative reward. We choose $b=0$, $b=5$ and $b=20$ for comparisons. The values in the table below are the cumulative reward improvement.

The results above show that when $b=5$, the cumulative reward improvement will become larger. This shows that we have false positive misleading state-action pairs and a small positive $b$ can reduce the negative impact of those false positive misleading state-action pairs. When $b=20$, the cumulative improvement becomes much smaller because at this time, many misleading state-action pairs that should be avoided are not avoided. The results above indicate that while our misleading state-action pairs have some false positives, the majority is accurate.
For the entropy term, RL has a long history about the exploration and exploitation balance. In specific, if we always choose greedy policy under the current policy evaluation, the RL algorithm may only explore a small portion of the state-action space and thus miss the real optimal action. At the beginning, people usually use $\epsilon$-greedy policy to encourage exploration. Recently, soft Q-learning and soft actor-critic propose to add an entropy term to the RL objective to encourage exploration. In our case, the exploration is even more important because we already know that the previous policy $\pi\_A$ is suboptimal. Therefore, exploration is encouraged to explore real optimal actions.

Weakness 2: "The writing and presentation could be improved. For example, the paper talks a lot about explainability, but I did not find explainability to be relevant here so much. At some point, the “explainable RL” idea is replaced by standard IRL. I think the authors should tone down the XRL parts and focus instead on using IRL as a signal in RL."
Answer: Thanks for mentioning the XRL part. Our low-level explanation identifies misleading state-action pairs. This idea is widely used in XRL methods [D1,D2,D3,D4] where people want to explain the performance of the RL agent by identifying critical points. Therefore, the low-level explanation belongs to XRL. Our high-level explanation indeed uses IRL to learn a reward $\hat{r}$. However, this also belongs to XRL [D5] where the learned reward function $\hat{r}$ is used to explain the real reward function that the RL agent is actually optimizing.

[D1] Amir, Dan, and Ofra Amir. "Highlights: Summarizing agent behavior to people." International Conference on Autonomous Agents and Multiagent Systems, 2018.

[D2] Guo, Wenbo, et al. "Edge: Explaining deep reinforcement learning policies." Advances in Neural Information Processing Systems, 2021.

[D3] Cheng, Zelei, et al. "Statemask: Explaining deep reinforcement learning through state mask." Advances in Neural Information Processing Systems, 2023.

[D4] Jacq, Alexis, et al. "Lazy-MDPs: Towards Interpretable RL by Learning When to Act." International Conference on Autonomous Agents and Multiagent Systems, 2022.

[D5] Xie, Yuansheng, Soroush Vosoughi, and Saeed Hassanpour. "Towards interpretable deep reinforcement learning models via inverse reinforcement learning." International Conference on Pattern Recognition, 2022.

Thank you very much for your thoughtful consideration of our paper and the helpful comments and suggestions. We would hope that our answers will alleviate your concerns. We address your comments below. We first answer the six details of Weakness 1 and then answer Weakness 2.

Weakness 1.1: "Line 185: “At a low level, the RL agent is not optimal meaning that it visits some critical points that lead to the non-optimality.” - I don’t understand this. It could be that the agent is slightly suboptimal on all states. This statement seems very ad hoc. Why should it be suboptimal only on some critical points?"
Answer: Thanks for mentioning the critical points. The critical points are the most important points that lead to non-optimality, however, this does not mean that the original policy $\pi\_A$ is only sub-optimal on the critical points. The original policy $\pi\_A$ can be suboptimal on all the states, and we pick the top K most suboptimal ones as critical points. In very rare cases where the policy $\pi\_A$ is equally suboptimal on every state, we randomly pick K critical points. In conclusion, the critical points are the most suboptimal points, and thus are a subset of all the sub-optimal points. As long as the original policy $\pi\_A$ is suboptimal, it must be suboptimal on certain states (including the case of all states) and we pick the most suboptimal ones as critical points. Therefore, we believe that this statement is not very ad hoc.

Weakness 1.2: "Line 199: $\pi^{\ast}$ and $\pi\_A$ are distributions. What does $\pi^{\ast}-\pi\_A$ mean exactly? Why is it a metric?"
Answer: The metric in line 199 is $\pi^{\ast}(a|s)-\pi\_A(a|s)$. While $\pi^{\ast}$ and $\pi\_A$ are distributions over $\mathcal{S}\times\mathcal{A}$, $\pi^{\ast}(a|s)$ and $\pi\_A(a|s)$ are specific values (i.e., the probabilities of $\pi^{\ast}$ and $\pi\_A$ choosing the action $a$ at the state $s$). This metric is the probability difference between the optimal policy $\pi^{\ast}$ and the original policy $\pi\_A$ at the state-action pair $(s,a)$. This metric can test whether $(s,a)$ is misleading. In specific, the value of this metric is between -1 and 1. If $\pi^{\ast}(a|s)-\pi\_A(a|s)$ is close to -1, this means that $\pi^{\ast}(a|s)$ has low probability of choosing the action $a$ at the state $s$ while $\pi\_A(a|s)$ has high probability of choosing $a$ at $s$. Therefore, this state-action pair $(s,a)$ is identified as misleading because $\pi\_A$ chooses the state-action pair $(s,a)$ that the optimal policy $\pi^{\ast}$ is not likely to choose. The closer the metric $\pi^{\ast}(a|s)-\pi\_A(a|s)$ is to -1, the more misleading the state-action pair $(s,a)$ is.

Weakness 1.3: "Line 209: isn’t this a well known result? I.e., a policy is optimal iff it satisfies the Bellman equation."
Answer: Line 209 is different from Bellman optimality equation. The Bellman optimality equation is $Q\_r^{\pi}(s,a)=r(s,a)+\gamma E\_{s'\sim P(\cdot|s,a)}[\max\_{a'}Q\_r^{\pi}(s',a')]$ for any $(s,a)$. However, what line 209 proves is the case where $Q\_r^{\pi\_A}(s,a)=\max\_{a'}Q\_r^{\pi\_A}(s,a')$ for $(s,a)$ such that $\pi\_A(a|s)>0$. This is not the Bellman optimality equation. In our proof in Appendix B.1, we prove that this condition in line 209 can lead to Bellman optimality equation.

Thank you very much for your detailed reply. We fully understand your comment that avoiding misleading state-action pairs may limit the algorithm from finding an optimal policy and thus theory is needed to motivate problem (1). We address your comment in the following three parts. In part (1), we elaborate that our method (avoiding misleading state-action pairs) is inspired by monotonic policy improvement theorem because our method can improve the original policy $\pi_A$ by helping the policy satisfy the monotonic policy improvement theorem. In part (2), we discuss how our method can improve the policy in your mentioned example. In part (3), we discuss how we get the budget $b$ in the experiments.

Part (1): We would like to first emphasize that our goal is to improve the original policy $\pi_A$ (as shown in the paper title) instead of finding an optimal policy. We acknowledge that finding an optimal policy can improve $\pi_A$, however, improving $\pi_A$ does not necessarily mean that we have to find an optimal policy. We next elaborate why avoiding the misleading state-action pairs can improve $\pi_A$. In brief, we show that our method of avoiding the misleading state-action pairs can help $\pi_A$ satisfy monotonic policy improvement theorem, and thus help improve $\pi_A$.

To start with, we first recall the problem setting of improving $\pi_A$. We first run an RL algorithm (e.g., SAC) until convergence to get $\pi_A$. The converged policy $\pi_A$ is suboptimal, i.e., it is stuck in a sub-optimal stage without further improvement. According to monotonic policy improvement theorem, $\pi_A$ will improve if it chooses greedy actions according to its Q-function $Q_r^{\pi_A}$ at all the states. Given that $\pi_A$ stops improving, it means that $\pi_A$ chooses some nongreedy actions at some states. In our paper, we identify these as misleading state-action pairs, i.e., the state-action pairs where $\pi_A$ chooses nongreedy actions at the states. Here, we refer to the states of the misleading state-action pairs as critical states. Constraining these misleading state-action pairs means that we constrain the nongreedy actions $\pi_A$ originally chooses at the critical states. This constraint can help $\pi_A$ choose greedy actions because it eliminates some nongreedy actions and thus $\pi_A$ only needs to find greedy actions from smaller action sets at the critical states. Since this constraint can help $\pi_A$ find greedy actions, it can help improve $\pi_A$ according to monotonic policy improvement theorem.

In conclusion, we agree that the nongreedy actions of $\pi_A$ may be optimal actions of an optimal policy, and constraining these actions may limit the algorithm from finding an optimal policy. However, we aim to improve $\pi_A$ instead of finding an optimal policy, and constraining these nongreedy actions can help improve $\pi_A$ according to monotonic policy improvement theorem. This property that constraining the misleading state-action pairs can improve $\pi_A$ motivates us to avoid visiting misleading state-action pairs.

Part (2): In your given example, we agree that constraining the misleading state-action pairs cannot lead to an optimal policy. However, it can lead to a policy better than the original policy. The original policy achieves zero cumulative reward if it starts from A. By avoiding the misleading state-action pairs, the new policy will choose A->A1, B->B2, C->C2 and achieves the cumulative reward of 1. This new policy is better than the original policy. Moreover, this new policy is optimal if the initial state is B or C.

We agree that improving $\pi_A$ to an optimal policy is an interesting problem, however, this is not the problem we study. In practice, improving $\pi_A$ to an optimal policy could be a too ambitious and even unrealistic goal. For example, in the MuJoCo environment, it is even impossible to evaluate whether the obtained policy is optimal or not because we do not know the maximum cumulative reward a policy can achieve.

Part (3): The hyper-parameter $b$ needs to be manually tuned in the experiment. In our experiment, we simply choose $b=0$, i.e., hard constraint. In the ablation study in Table 2 (fifth column), it shows that we can still improve $\pi_A$ even if we simply avoid the misleading state-action pairs. This aligns with the theory we elaborate in part (1), i.e., simply avoiding the misleading state-action pairs may not lead to an optimal policy but can improve the original policy $\pi_A$ by encouraging $\pi_A$ to choose greedy actions. In our answer to Weakness 1.6, we also empirically show how different $b$ will affect the learning performance.

We really appreciate your detailed reply, and we are happy to discuss if you have further comments.

Thanks for your reply. We appreciate that you find our discussion helpful. Following your suggestion, we add the discussion to the newly uploaded version (highlighted in blue in lines 244-251). Your understanding about the misleading state-action pairs is correct. The misleading state-action pairs are not updated during the run of the algorithm. We next answer your three questions:

Question 1: Why not update the misleading states? In my example, if you would update the misleading states, two iterations of your method would converge to optimal policy.
Answer: The main reason is the practical issue of sample complexity. We agree that if we update the misleading state-action pairs when we have a new policy, it will help every learned policy find greedy actions and thus facilitate policy improvement during the whole run. However, finding misleading state-action pairs requires to significantly sample the environment. In specific, in order to find the misleading state-action pairs of a learned policy $\pi$, we need to learn a quite accurate $Q$-function $Q_r^{\pi}$. This requires to sample many episodes from the environment so that we can collect enough data to learn an accurate $Q_r^{\pi}$. In our scenario, we sample 100 episodes to learn $Q_r^{\pi_A}$ in the explanation part. If we want to update the misleading state-action pairs for every learned policy, it means that we need to sample 100 episodes every iteration just to find misleading state-action pairs. In our current algorithm, we only sample 10 episodes every iteration (10% of the samples needed to find misleading state-action pairs). Moreover, the baselines (e.g., SAC and the other algorithms that can improve SAC) in the experiment also sample 10 episodes every iteration. If we update the misleading state-action pairs, it would be an unfair comparison between our algorithm and SAC because our algorithm would require ten times as many samples as SAC per iteration and SAC typically does not sample many (e.g., 100) episodes per iteration due to the consideration of sample efficiency. Our method aims to improve RL under the constraint that we do not require additional samples during improvement. This is evident in Figure 2 where we plot the performance of our algorithm and the baselines under the same amount of environment interaction steps.
In fact, sample efficiency is an important problem in RL because sampling can be slow and expensive [D6,D7,D8]. Therefore, many recent RL algorithms (e.g., SAC, DDPG, TD3) are off-policy and use replay buffers to make sample efficient.
In conclusion, we agree that updating misleading state-action pairs can help improve the performance. However, this can significantly increase sampling overhead and thus is sample inefficient.

Question 2: In your experiments, if I understand correctly, the misleading states will only be calculated in the beginning of the run from some demonstration data, and not update afterwards. Where is this demonstration data coming from and what is it exactly?
Answer: In our paper, we aim to improve an RL agent whose original policy is $\pi_A$. In specific, we first run the RL agent until convergence to get $\pi_A$. Following XRL literature [D9,D10,D11], we only treat the RL agent as a black box (lines 155-160). In specific, we can only observe a set $\mathcal{D}$ of trajectories demonstrated by the RL agent (using $\pi_A$) where each trajectory is a sequence of state-action pairs $s_0,a_0,s_1,a_1,\cdots$. In the experiment, we first run SAC until convergence and use the converged policy to sample demonstrated trajectories where each trajectory is a sequence of state-action pairs. These demonstrated trajectories are $\mathcal{D}$, from which we learn misleading state-action pairs.

Thank you for the detailed and constructive suggestions. We will follow these seven suggestions and revise the paper accordingly. We really appreciate your time and feedback, which significantly help us improve the paper. It is a pleasure to discuss with you. Finally, we would like to thank you for recognizing the contributions of this work.

Best,
Authors

Dear Reviewer iSK2,

We would like to thank you again for your constructive suggestions and we are now revising the paper according to your advice. Could you please update the rating so that it is clearer for the AC to know that you recommend to accept the paper?

Best,
Authors

Thank you very much for your thoughtful consideration of our paper and the helpful comments and suggestions. We would hope that our answers will alleviate your concerns. We address your comments below:

Weakness 1: "It is unclear to me why the policy would learn the wrong reward function internally in the first place. I might have misunderstood something."
Answer: The reason is that the original policy $\pi\_A$ is sub-optimal. Our paper aims to improve the sub-optimal policy $\pi\_A$. The internal reward function $\hat{r}$ of $\pi\_A$ is the reward function, to which $\pi\_A$ is optimal. If the internal reward function is the ground truth reward function (i.e., $\hat{r}=r$), $\pi\_A$ would be an optimal policy because $\pi\_A$ is optimal to the internal reward function.

Weakness 2} "It is unclear why theorem 1 is true. I do not see reference to any proof."
Answer: The proof of Theorem 1 is in Appendix D.2. Here we provide the our proof logic of Theorem 1. The fundamental logic is that we first prove that the dual problem (i.e. the lower-level problem in (2)) has a unique optimal solution due to its strict convexity and then we prove that the primal problem (i.e. the lower-level problem in (1)) and the dual problem have the same set of optimal solutions. Since the optimal solution of the dual problem is unique, then the optimal solution of the primal problem is also unique.

Weakness 3: "I am unsure how the performance would translate to environments with high-dimensional features. For example, Atari environments. Is there a reason why only MuJoCo environments are chosen for experiments?"
Answer: We choose MuJoCo environments because it is a widely-used RL benchmark. It is interesting to explore how our algorithm performs in Atari, and we will do this in our future works.

Weakness 4: "It is unknown how useful these explanations are to human users, and who is the target audience of these explanations."
Answer: The explanations can help humans understand why the original policy $\pi\_A$ is sub-optimal and thus utilize the explanations to improve $\pi\_A$. Therefore, there are two sets of target audience. The first set is the people who are simply curious about why $\pi\_A$ is sub-optimal. Take Figure 1 as an example, our explanations are easy for nonexperts to understand. The second set is the people who want to improve RL. Our explanations help identify some mistakes made by $\pi\_A$, therefore, people can improve $\pi\_A$ by correcting those mistakes.

Weakness 5: "The main novelty is on improving model performance using explanations. The explanations themselves are not new to this work."
Answer: We agree that our main novelty is on improving RL performance and our explanations build on some high-level idea of XRL literature. However, our low-level explanation (i.e., finding misleading state-action pairs) is new. The XRL literature [C1,C2,C3] has the idea of identifying critical states that are most important to cumulative reward. However, these critical states do not explain why the original policy $\pi\_A$ is sub-optimal. Built on the "finding critical points" idea, our paper is the first one to explain why the original policy is sub-optimal by identifying "misleading" state-action pairs that lead to sub-optimality.

Question 1: "Definition 1: Why is the Q-function taken with respect to $\pi\_A$. Isn't $\pi\_A$ the suboptimal policy?"
Answer: $\pi\_A$ is the sub-optimal policy. Definition 1 aims to find misleading state-action pairs of policy $\pi\_A$. We agree that the ideal case is that we can compare the Q-functions between the optimal policy $\pi^{\ast}$ and $\pi\_A$, i.e., $Q\_r^{\pi^{\ast}}(s,a)-Q\_r^{\pi\_A}(s,a)$. However, this metric is infeasible because $\pi^{\ast}$ is unknown. We propose a feasible metric in definition 1 which only uses $\pi\_A$. As mentioned below Definition 1 in lines 209-213, we prove in Appendix B.1 that the policy $\pi\_{A}$ will be an optimal policy if $l(s,a)=0$ for all $(s,a)$ that $\pi\_A$ visits with nonzero probability. Therefore, any $(s,a)$ such that $l(s,a)>0$ can be regarded as a ``misleading" state-action pair that leads the policy $\pi\_A$ to be nonoptimal.

Question 2: "Line 245: What is the functional form of the reward shaping function?"
Answer: In the experiment, we consider linear shaping reward, i.e., $r\_{\theta}(s,a)=r(s,a)+\theta(s,a)(r(s,a)-\hat{r}(s,a))$. The shaping parameter $\theta$ is a neural network whose input is $(s,a)$ and output is a scalar.

Thank you very much for your thoughtful consideration of our paper and the helpful comments and suggestions. We would hope that our answers will alleviate your concerns. We address your comments below:

Weakness 1: "The notion of $l(s,a)$: The paper assumes access to environment interaction and hence I am considering that there is enough knowledge about the actions per given state. If that is so, the way $l(s,a)$ is defined by the paper in definition 1, is simply negative of advantage function for policy $\pi\_A$.
Answer: The loss function $l(s,a)$ is not the negative of advantage function. The advantage function of the policy $\pi\_A$ is $A\_r^{\pi\_A}(s,a)=Q\_r^{\pi\_A}(s,a)-V\_r^{\pi\_A}(s)$ where $V\_r^{\pi\_A}(s)=E\_{a\sim\pi(\cdot|s)}[Q\_r^{\pi\_A}(s,a)]$. However, the loss $l(s,a)=\max\_{a'}Q\_r^{\pi\_A}(s,a')-Q\_r^{\pi\_A}(s,a)$. Note that $V\_r^{\pi\_A}(s)=E\_{a\sim\pi(\cdot|s)}[Q\_r^{\pi\_A}(s,a)]\neq \max\_{a'}Q\_r^{\pi\_A}(s,a')$ unless the policy $\pi\_A$ is an optimal policy $\pi^{\ast}$ where $E\_{a\sim\pi^{\ast}(\cdot|s)}[Q\_r^{\pi^{\ast}}(s,a)]=\max\_{a'}Q\_r^{\pi^{\ast}}(s,a')$. However, the original policy $\pi\_A$ is only a suboptimal policy that we want to improve.

Weakness 2: "The choice of MuJoCo environment and plausible issues: MuJoCo environments allow the agent to complete an episode for full length while other environments might terminate early if certain states are reached. Subsequently, MuJoCo environments provide correlated "second-chances" to the agent to make similar mistakes and the misleading effect is easy to capture. The algorithm's performance depends highly on the coverage of the demonstrations given by the earlier policy. MuJoCo would end up allowing coverage over optimal state-action visitation distribution and hence the algorithm proposed in the paper would work with single low-level and high-level explanation. However, this will not hold in the other complex environments. Also, in the case of sparse environments, where the agent has not received enough feedback from the environment to capture the "misleading" state-actions, the algorithm might misattribute notion of misleading."
Answer: Thanks for expressing your concerns about (1) the misleading effect may be hard to capture for environments where the episode terminates early if certain states are reached; (2) the algorithm performance depends highly on the coverage of the demonstrations and may perform bad if demonstrations do not cover optimal state-action visitation distribution; and (3) the algorithm may fail in sparse environments where the agent does not receive enough feedback from the environment. We address your concerns respectively as follows:
For concern (1), we agree that if an agent makes similar mistakes multiple times, it is easier to find misleading state-action pairs. Therefore, for environments where the episode terminates early, it may be more difficult to find accurate misleading state-action pairs. However, this limitation does not bother the high-level explanation, i.e., the learned reward $\hat{r}$. In specific, we can always learn a reward $\hat{r}$ using ML-IRL, to which the original RL agent is optimal, regardless of whether the RL agent makes similar mistakes multiple times in the demonstrations or not. Therefore, as long as the original policy is not optimal, the learned reward $\hat{r}$ can capture this non-optimality and the comparison $r-\hat{r}$ quantifies the RL agent's misunderstanding about the real reward function $r$. Moreover, the ablation results in Table 2 show that the shaping reward (that uses the learned reward $\hat{r}$) is much more important than avoiding the misleading state-action pairs in terms of improving the RL performance. Therefore, even if the low level explanation may become less accurate when the episode terminates, our algorithm's performance will not degrade too much given that the performance highly relies on the shaping reward that uses $\hat{r}$ and we can always learn an accurate $\hat{r}$ using ML-IRL regardless of whether the episode terminates early or not. To better support our claim, we add an additional experiment on Lunar Lander of Gym where the episode terminates when the lander crashes or lands.

The results above show that our method (UTILITY) can still significantly outperform the baselines in environments where the episode terminates if certain states are reached.

Thank you very much for your thoughtful consideration of our paper and the helpful comments and suggestions. We would hope that our answers will alleviate your concerns. We address your comments below:

Weakness 1: "Can the authors add a discussion on stochastic environments and the impact of the same on their approach? I think the current approach has only been demonstrated in deterministic environments. Specifically, can the authors discuss how their theoretical guarantees and empirical results might change in stochastic environments, and whether any modifications to their approach would be needed?"
Answer: Thanks for mentioning stochastic environments. While our experiments are only conducted in deterministic MuJoCo environments, our method can work for stochastic environments. The reason is that our method is based on a standard MDP where the state transition function $P(\cdot|\cdot,\cdot)$ is stochastic and we do not assume that the transition function is deterministic. In fact, our current theoretical guarantees hold for stochastic environments because our proof considers stochastic environments. In specific, our proof uses $\int_{s'}P(s'|s,a)ds'$ to consider the next state $s'$ given the current state-action $(s,a)$. This is evident in the proof of every theoretical statement of this paper, e.g., line 951-960 in the proof of Theorem 1, lines 1013-1017 in the proof of Lemma 2, etc. To show that our method works for stochastic environments empirically, we conduct an additional experiment on Lunar Lander of Gym where the state transition is stochastic because the next state is affected by the wind and the wind effect is random. We provide the results below:

From the results above, we can see that our method (UTILITY) can outperform the baselines in stochastic environments.

Weakness 2: "The current method seems to need a lot of environment interaction. Can the authors detail the additional interaction needed for their approach?"
Answer: We quantify the environment interactions using the number of episodes. Each episode includes T-step environment interactions where T is the episode length. The environment interactions of our method come from two part: explanation (i.e., IRL and finding misleading state-action pairs) and improvement (solving problem (1)). We discuss these two parts respectively as follows:
For the explanation part, the IRL samples 500 episodes in total to learn a reward function where we run ML-IRL for 500 iterations and each iteration samples one episode. To find the misleading state-action pairs, we sample 100 episodes to learn the Q function. Therefore, we sample 600 episodes for the explanation.
For the improvement part (solving problem (1)), Algorithm 1 has three loops. In the inner loop, at each inner iteration, we sample 1 episode from the environment to compute the constrained soft Q function (line 5 in Algorithm 1). Therefore, the inner loop samples totally $\tilde{N}$ episodes from the environment where $\tilde{N}$ is the iteration number of the inner loop. Note that we also use a replay buffer to store previous data. In the middle loop, at each middle iteration, we sample 1 episode to compute $g_{\lambda;\theta}$ (line 8 in Algorithm). Therefore, each middle iteration requires $1+\tilde{N}$ sampled episodes, and the middle loop requires totally $\bar{N}(1+\tilde{N})$ episodes where $\bar{N}$ is the iteration number of the middle loop. At each outer iteration, we compute $\hat{\pi}\_{\hat{\lambda}(\theta);\theta}$ via $(\bar{N}-1)$-step soft policy iteration (line 11 in Algorithm 1) and each soft policy iteration requires to sample 1 episode. Moreover, we sample 5 episodes to compute $g_{\theta}$ (line 12 in Algorithm 1). Therefore, at each outer iteration, the total number of sampled episodes is $\bar{N}(1+\tilde{N})+(\bar{N}-1)+5$. Due to the warm start trick (discussed in Appendix F.6), we can choose very small $\bar{N}$ and $\tilde{N}$, i.e., $\tilde{N}=1$ and $\bar{N}=2$ (mentioned in lines 1585-1586). Therefore, the total episodes sampled at each outer iteration is 10. We run Algorithm 1 for 1000 outer iterations, therefore, the total number of sampled episodes is 10000. For a fair comparison, we run SAC for 1000 iterations and sample 10 episodes at each iterations, so that SAC also samples 10000 episodes in total.

In conclusion, solving problem (1) does not introduce additional interactions compared to SAC. The additional interactions of our method are the 600 episodes required for the explanation. Therefore, compared to standard SAC, our method requires 600/10,000=6% extra environment interactions.

Weakness 4: "The impact of the error in explanation on approach and final policy has not been covered, i.e., errors in reward learning and Q value error and behavior policy learning error. Can the authors address these with a discussion in the paper? Specifically, can the authors provide a sensitivity analysis showing how errors in each component (reward learning, Q-value estimation, behavior policy learning) impact the final performance with some discussion on potential mitigation strategies for these errors?"
Answer: Thanks for mentioning how the error in the explanation impacts the final performance. We respectively discuss the impacts of the error of $\hat{r}$ and the impact of the error of behavior policy $\hat{\pi}_A$ and Q value as follows.

For the error of the reward learning, suppose that we learn a reward $r'$ such that $||\hat{r}(s,a)-r'(s,a)||\_{\infty}\leq\epsilon$. Theorem 4 shows that if the state-action space is finite and the reward model $r\_\theta$ is linear and we obtain $\hat{r}$, our algorithm can achieve optimality, i.e., $\lim\_{N,\bar{N}\rightarrow\infty}J\_r(\hat{\pi}\_{\hat{\lambda}(\theta\_N);\theta\_N})-J^{\ast}=0$. If we only obtain $r'$ instead of $\hat{r}$, we can get that $\lim\_{N,\bar{N}\rightarrow\infty}J^{\ast}-J\_r(\hat{\pi}\_{\hat{\lambda}(\theta\_N);\theta\_N})\leq O(\epsilon)$. We provide the proof sketch as follows:
For simplicity, we use $\pi_{\hat{r}}$ to denote the learned policy under $\hat{r}$ (i.e., the optimal soft policy corresponding to $r\_{\theta}(r,r-\hat{r})$) and $\pi\_{r'}$ to denote the learned policy under $r'$ (i.e., the optimal soft policy corresponding to $r\_{\theta}(r,r-r')$). Therefore, $J\_r(\pi\_{\hat{r}})-J\_r(\pi\_{r'})\leq \frac{r\_{\max}}{1-\gamma}||\psi^{\pi\_{\hat{r}}}(s,a)-\psi^{\pi\_{r'}}(s,a)||\_{TV}$ where $r\_{\max}=\max\_{(s,a)\in\mathcal{S}\times\mathcal{A}}|r(s,a)|$, $|\cdot|\_{TV}$ is the total variation, and $\psi^{\pi}$ is the stationary state-action distribution of policy $\pi$. We first ignore the constraint caused by the misleading state-action pairs. Note that $\pi\_{\hat{r}}\propto \exp(Q\_{r\_{\theta}(r,r-\hat{r}),\pi\_{\hat{r}}}^{soft})$ and $\pi\_{r'}\propto \exp(Q\_{r\_{\theta}(r,r-r'),\pi\_{r'}}^{soft})$ (see the expression of the soft policy (equation 3)), from Lemma 3 in [A3] (also equation 64 in [A4]), we can get that $J\_r(\pi\_{\hat{r}})-J\_r(\pi\_{r'})\leq \frac{r\_{\max}}{1-\gamma}||\psi^{\pi\_{\hat{r}}}(s,a)-\psi^{\pi\_{r'}}(s,a)||\_{TV}\leq \frac{Cr\_{\max}}{1-\gamma}||Q\_{r\_{\theta}(r,r-\hat{r}),\pi\_{\hat{r}}}^{soft}(s,a)-Q\_{r\_{\theta}(r,r-r'),\pi\_{r'}}^{soft}(s,a)||\_{\infty}$ where $C$ is a positive constant. Since $\pi\_{r'}$ is the optimal soft policy corresponding to $r\_{\theta}(r,r-r')$, we have that $Q\_{r\_{\theta}(r,r-\hat{r}),\pi\_{\hat{r}}}^{soft}(s,a)-Q\_{r\_{\theta}(r,r-r'),\pi\_{r'}}^{soft}(s,a)\leq Q\_{r\_{\theta}(r,r-\hat{r}),\pi\_{\hat{r}}}^{soft}(s,a)-Q\_{r\_{\theta}(r,r-r'),\pi\_{\hat{r}}}^{soft}(s,a)$. Similarly, we have that $Q\_{r\_{\theta}(r,r-\hat{r}),\pi\_{\hat{r}}}^{soft}(s,a)-Q\_{r\_{\theta}(r,r-r'),\pi\_{r'}}^{soft}(s,a)\geq Q\_{r\_{\theta}(r,r-\hat{r}),\pi\_{r'}}^{soft}(s,a)-Q\_{r\_{\theta}(r,r-r'),\pi\_{r'}}^{soft}(s,a)$. Therefore we have that $J\_r(\pi\_{\hat{r}})-J\_r(\pi_{r'})\leq \frac{Cr\_{\max}}{1-\gamma}\max\{||Q\_{r\_{\theta}(r,r-\hat{r}),\pi\_{\hat{r}}}^{soft}(s,a)-Q\_{r\_{\theta}(r,r-r'),\pi\_{\hat{r}}}^{soft}(s,a)||\_{\infty},||Q\_{r\_{\theta}(r,r-\hat{r}),\pi\_{r'}}^{soft}(s,a)-Q\_{r\_{\theta}(r,r-r'),\pi\_{r'}}^{soft}(s,a)||\_{\infty}\}$. We first bound $||Q\_{r\_{\theta}(r,r-\hat{r}),\pi\_{\hat{r}}}^{soft}(s,a)-Q\_{r\_{\theta}(r,r-r'),\pi\_{\hat{r}}}^{soft}(s,a)||\_{\infty}$. Since $r\_\theta$ is linear, we have that $|r\_{\theta}(r(s,a),r(s,a)-\hat{r}(s,a))-r\_{\theta}(r(s,a),r(s,a)-r'(s,a))|=|[r(s,a)+\theta(s,a)(r(s,a)-\hat{r}(s,a))]-[r(s,a)+\theta(s,a)(r(s,a)-r'(s,a))]|=|\theta(s,a)(\hat{r}(s,a)-r'(s,a))|$ $\leq ||\theta(s,a)||\_{\infty}||\hat{r}(s,a)-r'(s,a)||\_{\infty}=O(\epsilon)$. Note that $||\theta(s,a)||\_\infty$ is finite because the state-action space is finite. Therefore $||Q\_{r\_{\theta}(r,r-\hat{r}),\pi\_{\hat{r}}}^{soft}(s,a)-Q\_{r\_{\theta}(r,r-r'),\pi\_{\hat{r}}}^{soft}(s,a)||\_{\infty}\leq \frac{1}{1-\gamma}||r\_{\theta}(r(s,a),r(s,a)-\hat{r}(s,a))-r\_{\theta}(r(s,a),r(s,a)-r'(s,a))||\_\infty = O(\epsilon)$. Similarly, we can get that $||Q\_{r\_{\theta}(r,r-\hat{r}),\pi\_{r'}}^{soft}(s,a)-Q\_{r\_{\theta}(r,r-r'),\pi\_{r'}}^{soft}(s,a)||\_{\infty}\leq O(\epsilon)$. Therefore, $J\_r(\pi\_{\hat{r}})-J\_r(\pi\_{r'})\leq O(\epsilon)$. This results hold for the case that the shaping parameter $\theta$ is same for $\hat{r}$ and $r'$, we will explore the case where $\theta$ is different.