PAGAR: Taming Reward Misalignment in Inverse Reinforcement Learning-Based Imitation Learning with Protagonist Antagonist Guided Adversarial Reward

5th line from the bottom ... inequality sign

The sign is correct, when $U_{r^-}(E)-max_\pi U_{r^-}(\pi) > U_{r^+}(E) - max_\pi U_{r^+}(\pi)$, reward function $r^-$ will be more 'optimal' than $r^+$. We use this inequality to show that we must consider the case where a sub-optimal reward function can align with the task while the optimal reward function is misaligned.

... non-optimal policy ...

In theory, the antagonist policy has to be optimal, while in practice, we have to train the antagonist policy together with the protagonist policy from scratch. The fact that VAIL alone not being able to produce successful episodes does not directly relate to whether running VAIL in the PAGAR framework can produce better results.
This is because the reward function in the PAGAR framework is optimized to increase the protagonist antagonist induced regret in Eq (2). This optimization incentivizes searching for a reward function that can makes the antagonist policy outperforms the protagonist policy. Given that inverse RL can have multiple solutions -- multiple reward functions aligned with the demonstrations, PAGAR-based IL avoids those reward functions that are difficult for any policy to achieve a high utility, since this will result in both the antagonist and the protagonist policy having low performance.

... maximize the likelihood for a MaxEnt-policy ...

Maximum entropy is a heuristic used to reduce reward ambiguity and not reward misalignment. In fact, MaxEnt cannot eliminate reward ambiguity even if we have infinite demonstrations ([Skalse. et al., 2022], which we cite in the first paragraph of Introduction).

Reward ambiguity and reward-task misalignment are two different concepts.
Reward ambiguity is the problem of many different reward functions in the reward hypothesis space can explain the expert demonstrations equally well. On the other hand, reward-task misalignment is the problem of policies that achieve a high utility under a reward function failing to finish the task (the well-known notion of reward hacking is a manifestation of this [Pan. et al., 2022], which we cite at the end of first paragraph).

Maximum entropy likelihood may lead to a policy that is highly probable in reproducing the demonstrated trajectories. It is equivalent to assuming that the underlying task is simply to generate the demonstrated trajectories with the highest probability. Our work does not make such an assumption. As shown by our definition of $R_{E,\delta}$, we include not only the optimal but also the sub-optimal solutions of inverse RL as candidate reward functions.

... The conditions used in Theorem 1 are not satisfiable ...

Firstly, in Definition 1, the intervals $S_r$ and $F_r$ are defined such that a policy's utility falling within $S_r$ or $F_r$ is deemed a sufficient but not necessary condition for determining success or failure. Put differently, if a policy $\pi$ achieves a utility in $S_r$, it is guaranteed to be successful, and if $\pi$ achieves a utility in $F_r$, it is guaranteed to be a failure. Note that $S_r$ and $F_r$ are not required to encompass all successful or failing policies' utilities. These intervals may also be very small for certain task. In the extreme scenario, $S_r$ could be a singleton, representing the maximum achievable policy utility under $r$, if the optimal policy under $r$ can successfully complete the task. Meanwhile, $F_r$ can be empty. The impact is that if all reward functions in the set $R$ have very small $S_r$ and $F_r$ intervals, $MinimaxRegret$ may have to achieve a low regret to produce a successful policy.

Also, the conditions about the width of S and F intervals are sufficient conditions for $MinimaxRegret$ to learn a successful policy. Suppose all the reward functions in the candidate reward function set are task-aligned. Then, even if they all match the description mentioned by the reviewer, i.e., no gap between $S_r$ and $F_r$, $MinimaxRegret$ can still induce a successful policy in the task.

Proof

• There must be a policy $\pi^+$ that makes $Regret(\pi^+, r)\leq|S_r|$ hold for all reward function $r$'s. Because its utility cannot be in the S interval under a reward function and in the F interval under another.
• For any policy $\pi^-$ that fails the task, $Regret(\pi^+, r)>|S_r|$ holds for all reward function $r$'s. Because if there exists an $r$ under which $Regret(\pi^-, r) \leq |S_r|$, $\pi^-$ is guaranteed to succeed in the task, contradicting the assumption.
• $MinimaxRegret$ will pick $\pi^+$ over $\pi^-$ as its solution because $Regret(\pi^+, r)<Regret(\pi^-, r)$ under all $r$'s

... Potentially related to this, it not clear to me how the

Table 1 shows an equivalent objective function of $MinimaxRegret$ It only describes the analytical solution of PAGAR. Our approach to PAGAR is not based on Table 1.

... $\delta$ is chosen to be strictly positive ...

We mentioned after Corollary 2 that $\delta$ cannot be larger than the negative minimal loss of inverse RL. If the minimal loss of inverse RL is non-negative, i.e., the expert policy is in the policy set, $\delta$ will be non-positive.

...the success of a policy or its performance can be deterministic ...

The 'success' here indicates the ability of the policy to finish the task, not whether the policy finishes the task in a single run. For instance, a success criterion can be "the probability of reaching the state must be higher than 'p'". If a reward function is well-designed, the success of this policy can be guaranteed if the policy's expected return exceeds a certain threshold.

... IQ-Learn should clearly outperform ...

In our continuous control experiments, the x-axis represents policy update iterations, not policy roll-out steps. IQ-learn's implementation is based on soft-actor-critic which is an off-policy algorithm that conducts several policy updates between consecutive interactions with the environment. The size of the replay buffer allocated for IQ-learn is consistent with the sample size used for PPO which is the RL algorithm used in our implementation.

... transfer experiment

The demonstrations are sampled in one environment, Algorithm 1 (reward learning, policy learning) is conducted in another environment.

If the agent reproduces the behavior of the expert and we assume that the expert solves the task, the agent will also solve the task.

A policy being able to reproduce the demonstrated trajectories does not imply the policy is ready for the task. In a stochastic environment, any stochastic policy can reproduce the demonstrated trajectories ... We have reiterated that the outcomes of individual attempts or instances do not solely determine success. If by reproduces the behavior you mean to recover the expert policy, the problem is that IRL can fail to recover expert policy for so many reasons ... We have cited the literature in the paper and in our rebuttal.

This claim is not substantiated and wrong ...

Our claim is substantiated and correct. We recommend revisiting Ziebart, Brian D., et al. Maximum entropy inverse reinforcement learning and searching for the keyword 'likelihood'. It is beyond doubt that max-ent IRL determines the reward weight by maximizing the likelihood of the max-ent policy generating the demonstrated trajectories. Max-Ent principle is not the only factor at play.

The underlying task would always be to maximize the reward function.

This does not contradict our argument. The policy is trained with a reward function that is learned based on a maximum likelihood objective function.

... at least if we assume the expert policy to be in the hypothesis space

Our paper never made such an assumption at all. It is common knowledge that IRL can be affected by various factors, including a limited number of expert demonstrations, expert policy not being in the hypothesis space, learning reward function in MDPs with different dynamics ... And our method is to mitigate the reward misalignment issue in challenging and varied conditions. While Theorem 3 implicitly covers the ideal condition where IRL can reach Nash Equilibrium, all other theories do not account for the ideal condition. Our experiments in MiniGrid aim to exhibit PAGAR's ability when those factors are present.

does not answer the question of how the set $R$ is chosen in the RL setting ... Is PAGAR meant to be applied in the RL setting ... Only applicable in practice to IL

We would like to clarify that Section 4.1, "RL with PAGAR", is about examining the conditions for $R$ under which $MinimaxRegret(R)$ can generate a policy to avoid task failure or guarantee task success. It does not deal with how to construct such an $R$ in an RL training scenario. It is meant to support the main objective of the paper, which is mitigating the reward alignment problem in IRL-based IL, since IRL-based IL involves inferring a reward from the demonstrations and performing RL with the inferred reward.

Applying PAGAR to IRL-based IL is equivalent to letting $R$ be the solution set of IRL. In this context, human demonstrations can be viewed as a supervision signal that help us identify $r$ that satisfies the conditions in Section 4.2 which are extensions of the conditions in Section 4.1 for a setting with expert demonstrations.

construction of the $R_E$ set

We do not directly construct this set; rather, we utilize the aforementioned conditions to restrict the exploration of $r$.
Our definition of $R_{E, \delta}$ is that for every $r\in R_{E,\delta}$, the difference between the optimal utility and the expert demonstration utility is within $\delta$. This difference equals the negative IRL loss under $r$. We can enforce selecting $r$ from $R_{E,\delta}$ by restricting the inverse RL loss of $r$ to be no greater than $-\delta$.

Example 1 ... parameterize $R_E$

We mention in the paragraph above Example 1 that $\delta$ cannot be larger than the negative minimal inverse RL loss. When $\delta$ reaches its maximum, $R_{E,\delta}$ equals the optimal solution set of inverse RL. In this case, we simplify the notation as $R_E$. We then use Example 1 to show that in some instances, inverse RL treats the entire reward space $R$ as its optimal solution set, in other words, $R_E=R$. The basis function is picked to illustrate the misalignment problem in inverse RL, as the choice of the basis functions can be arbitrary. However, by applying PAGAR, we can identify the ground-truth reward function.

Is the discussion at the bottom of pg 5 meant to be a construction of $R_E$ or just a non-constructive description

This discussion is about the formula in Table 1, which is an alternative of $MinimaxRegret$. It is equivalent to $MinimaxRegret$ in the sense that it induces the same result as $MinimaxRegret$. It is meant to provide more insights about PAGAR. It is not a constructive description.

Complexity of PAGAR

In Algorithm 1, we show that PAGAR involves optimizing two policies (protagonist and antagonist) and one reward function. Algorithm 1 leverages existing inverse RL algorithm to perform the reward optimization and the PPO algorithm for the policy optimization. Therefore, the complexity of PAGAR equals the complexity of the reward optimization algorithm plus the complexity of the PPO algorithm. In our experiments, we use the same reward optimization algorithms as those of GAIL and VAIL in PAGAR.

limited number of demonstrations (10)

The effective utilization of a limited number of demonstrations is widely recognized as a crucial metric for evaluating Inverse Reinforcement Learning (IRL) and Imitation Learning (IL) algorithms in the literature. It is often used to asess the performance of these algorithms, as demonstrated in notable works such as the GAIL paper [Ho. et al., 2016], the AIRL paper [Fu. et al., 2018], and various other references cited in our study. In GAIL [Ho. et al., 2016], which is a seminal work in imitation learning with generative adversarial networks, the algorithm demonstrates the ability to generate high-performance policies with just a single demonstration.

More demonstrations would reduce misalignment

As we mentioned in our introduction, a reward function aligning with the expert demonstrations does not imply aligning with the underlying task. These two concepts are orthogonal. Notice that the definition of Task-Reward Alignment (Definition 1) is independent of demonstrations. More demonstrations may help reduce reward ambiguity (i.e. multiple demonstration-aligned rewards), which, however, cannot be eliminated even with an infinite number of data [Ng & Russell (2000); Cao et al. (2021); Skalse et al. (2022a;b); Skalse & Abate (2022)].