Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment

... continuous environments are missing, e.g., MuJoCo.

We have included continuous control tasks in Appendix C.2.

... environment setups are too simple ... fewer than 100 states

The MiniGrid environments are challenging benchmarks for RL and IRL although they may look deceptively simple. The agent only makes partial observation of the environment -- the 7x7 cells in front of it. Furethermore, all the objects/walls/doors are randomly placed in every episode. As a result, it has far more than 100 states. In fact, due to its massive state space, it has become a standard benchmark for exploration-driven RL algorithms. For instance, AGAC [1], the SOTA exploration-driven RL technique, uses this environment as the main benchmark. It takes less than 10 million steps to attain high performance for a task in VizDoom (a more feature-rich, perception-based environment), but takes 200 million steps to only achieve a moderate performance on one of the MiniGrid environments. The MiniGrid Github repository also has a long list of papers that use MiniGrid in their empirical evaluations.

... not directly use some IRL algorithms as baselines to compare, e.g., MaxEnt IRL, ML-IRL, f-IRL, etc.

Our approach, named PAGAR-based IL, is an IL algorithm that uses existing IRL-based IL algorithm to specify the set $R_{E,\delta}$ as mentioned in line 172. We design the experiments mainly to show the compatibility of our algorithm with the IRL parts of the existing IRL-based IL benchmarks, and whether our algorithm can improve the imitation performance by using the same IRL parts as those of the benchmarks.

However, we appreciate the reviewrs' suggestions on trying other IRL algorithms. Therefore, we have incorporated our algorithm with the most advanced IRL/IL algorithm, RECOIL [2]. We have attached our results in our general response to all reviewers. Please refer to it for details.

[1] Flet-Berliac, Y., Ferret, J., Pietquin, O., Preux, P., Geist, M. Adversarially guided actor-critic. ICLR 2021

[2] Sikchi et al.; Dual RL: Unification and new methods for reinforcement and imitation learning, ICLR 2024

Thank you once again for your valuable comments. We hope that our previous response has addressed most of your concerns. To further support our rebuttal, we would like to share some additional experimental results that directly address the concerns that you raised. We would greatly appreciate it if you could re-evaluate our submission based on this response, or let us know if you have any other questions. Thank you!

(cont'd) ... directly use some IRL algorithms as baselines to compare, e.g., MaxEnt IRL, ML-IRL, f-IRL, etc...

We have conducted experiments combining PAGAR with f-IRL, and the results are summarized in our most recent General Response. This combination has achieved higher performance than f-IRL baseline with fewer exploration iterations in the environment across multiple standard IRL/IL benchmarks. These results will be presented in the appropriate format in the revised version of the paper.

Dear Reviewer,

Thank you once again for your valuable comments.

We hope that our explanation, along with the experiments in continuous control tasks with GAIL/VAIL in Appendix C.2, and the additional experiments with f-IRL (provided in the general responses) as well as the SOTA offline IRL/IL baseline RECOIL (also provided in the general responses) have addressed your concerns about experimental evaluation.

Please let us know if further clarification or additional evaluations would help in revisiting the score.

Your support would make a significant difference.

Thank you!

Best,

Authors

Thank you for your thoughtful review and for taking the time to consider our response. We’re pleased that our clarification was helpful.

We would like to reiterate our first point from the initial rebuttal that MiniGrid environment we use is a well-established RL benchmark with its own publication in NeurIPS 2023 [1]. As mentioned in our initial response, this environment is particularly complex even for SOTA IRL/IL algorithms due to its massive state space (much more than 100 states).

Given that the current score is quite close to the borderline, we would be grateful if you could revisit our explanations and the additional results.

In our revised version, we will not only include the additional experiments with f-IRL and RECOIL on continuous control tasks but also focus on presenting the main idea of the paper as follows to ensure our contributions are effectively communicated:

• Goal: Learn a policy to fulfill an unknown task that fits the description in Definition 1.
• Insight: Achieving high utility under task-aligned reward functions is essential for task fulfillment.
• Problem: The specific task-aligned reward function is unknown.
• Solution: By treating expert demonstrations as weak supervision, we learn a set of reward functions that encompass task-aligned rewards and then train a policy to achieve high utility across this set of reward functions.

Of course, we are happy to address any further questions.

[1] Chevalier-Boisvert, Maxime, et al. "Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks." NeurIPS 2023

The experimental campaign is concentrated on simple tasks... more complex tasks could prove scalability.

The MiniGrid tasks we employ are challenging benchmarks for RL and IRL although they may look deceptively simple. The agent only makes partial observation of the environment -- the 7x7 cells in front of it. All the objects/walls/doors are randomly placed after every episode. As a result, it has prohibitively large state space, which has made it a standard benchmark for exploration-driven RL algorithms. For instance, AGAC [1], the SOTA exploration-driven RL technique, uses this environment as the main benchmark. It takes less than 10 million steps to attain high performance for a task in VizDoom (a more feature-rich, perception-based environment), but takes 200 million steps to only achieve a moderate performance on one of the MiniGrid environments. The MiniGrid Github repository also has a long list of papers that use MiniGrid in their empirical evaluations.

Besides discrete navigation tasks, we have also included continuous control tasks, e.g., Mujoco, in Appendix C.2. Furthermore, we have experimentd on D4RL dataset and attached results later in this rebuttal.

How is $\Pi_{acc}$ spanned ...

$\Pi_{acc}$ is the set of policies that can accomplish the task. How it is spanned depends on what the underlying task is. For instance, in some reachability tasks, any policy that can reach a goal state with a higher probability than a threshold can be deemed acceptable. As mentioned in our general response, our definition of task is inspired by the NeurIPS 2021 best paper [2], which shows that this definition can characterize a wide range of real-world tasks.

Definition 2 task-relatedness is only induced by the definition of $\Pi_{acc}$, is this correct?

Whether a reward function is task-aligned is determined by $\Pi_{acc}$: a reward function $r$ is task-aligned if all the acceptable policies from $\Pi_{acc}$ achieve higher utilities under $r$ than all those not from $\Pi_{acc}$. Even if the expert policy $\pi_E$ belongs inside $\Pi_{acc}$, it does not guarantee that its optimal reward function $r_E$ is task-aligned.

... hyperparameter $k$ ... what defines task-relatedness is actually set as a hyperparameter in this framework.

The hyperparameter $k$ does not determine task-relatedness ('task-alignment'). It is used to determine the size of the candidate reward functions set $R_{E,k}$.

One way to assist understanding the role of $k$ is to draw an analogy to adversarially robust machine learning: the learning model should maintain high accuracy/low error not only at the training data points but also in the 'vicinity' of these points. As the range of adversarial perturbation is generally unknown, the range of this 'vicinity' is often set as a hyperparameter. But this hyperparamter does not determine which point is prone to be attacked.

...‘finding less overfitting reward functions’ ... regularization technique rather than an active search for task-aligned ...

We do not actively search for task-aligned reward functions. Our goal is to find an acceptable policy. However, the concept of task alignment is crucial as it is the backbone of Theorem 1 which justifies two key aspects of our approach:

1. Collect a set of reward functions, including IRL's sub-optimal rewards, to encompass task-aligned rewards, and 
2. Train a policy to attain high utility across this reward set. 

We have not seen literature supporting that "finding less overfitting reward functions" justifies either of these aspects.

Our formulation in Equation 3 is a constrained optimization problem, distinct from merely adding regularization to IRL/IL’s loss. The objective is to find a policy that minimizes worst-case regret while the constraint allows us to consider all reward functions with low IRL loss.

... breaking the storyline ...

The main idea of this paper can be summarized as follows:

• Goal: Learn a policy to fulfill an unknown task that fits the description in Definition 1.
• Insight: Only achieving high utility under task-aligned reward functions can ensure task fulfillment.
• Problem: Unknown which reward function is task-aligned
• Solution:
  1. Treating expert demos as weak supervision, learn a set of reward functions to encompass task-aligned rewards
  2. Train a policy to achieve high utility across this set of reward functions

the difference between r* and r_E

$r^*$ is the optimal reward function solved via IRL while $r_E$ is the expert reward function.

line 4: I guess it should be $\pi_P$ instead of $\pi_A$

Yes, we meant to update $\pi_P$ instead of $\pi_A$ in line 4.

new locomotion benchmark LocoMuJoCo

As mentioned earlier, we included continuous control tasks, e.g., Mujoco, in Appendix C.2. However, we appreciate the suggestion and will conduct experiments on LocoMujoco and update the results in the paper.

an ablation on $\delta$ would be interesting

In our experiments, we observe that when $\delta$ is too large, the algorithm does not induce any meaningful results. This has been described in line 164. We plan to include this observation in the updated version of the paper.

...similar methods using implicit rewards

We will add them to our related work section. In our general response, we have attached our experiemtns on combining our framework with RECOIL in [3]. This marks that compatibility with implicit reward and offline IL as a new feature of our method.

[1] Flet-Berliac, Y., Ferret, J., Pietquin, O., Preux, P., Geist, M. Adversarially guided actor-critic. ICLR 2021

[2] Abel, David, et al. On the expressivity of markov reward, NeurIPS 2021

[3] Sikchi et al.; Dual RL: Unification and new methods for reinforcement and imitation learning, ICLR 2024

Dear reviewer,

Thank you once again for your valuable comments. We hope that our explanation and the new experimental results on RECOIL and f-IRL with continuous control benchmarks (provided in the General Responses) have addressed your concerns. We would greatly appreciate it if you could re-evaluate our submission based on this response, or let us know if you have any further questions. Thank you!

Best,

Authors

We sincerely thank the reviewer for raising our score and appreciate the constructive feedback. We will incorporate the suggested improvements as outlined in our response.

In addition, we would like to highlight that the MiniGrid environment we use is a well-established RL benchmark with its own publication in NeurIPS 2023 [1]. As mentioned in our initial response, this environment is particularly complex, even for SOTA IRL/IL algorithms due to its massive state space.

Also, we need to clarify that our Task definition in Definition 1 is not intended to limit the scope of benchmarks. This definition inherits concepts from [2] . This abstract task definition is very general and encompasses many real-world examples, as also demonstrated in [2]. In fact, it is designed to address considerations from dimensions beyond simple classifications such as ‘continuous/discrete,’ ‘finite/infinite states,’ or ‘fixed/infinite horizon.’ We now demonstrate that our Task definition is expressive enough to characterize standard RL tasks (learning the optimal policy from a reward function $r$).

Proof: Given a reward function $r$ and a policy hypothesis set $\Pi$, the corresponding Task is $(\Pi, \preceq_{task}, \Pi_{acc})$ where the partial order $\preceq_{task}$ is defined as $\forall \pi_1,\pi_2\in \Pi,\pi_1\preceq \pi_2 \Leftrightarrow U_r(\pi_1)\leq U_r(\pi_2)$, and $\Pi_{acc}=\\{\pi|\pi'\preceq_{task} \pi\ \forall \pi'\in \Pi\\}$ if only considering the optimal policies.

Our theories are based on this task definition. Our algorithm can work on a variety of standard RL benchmarks, and our experimental results in MiniGrid and continuous control have demonstrated this.

[1] Chevalier-Boisvert, Maxime, et al. "Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks." NeurIPS 2023

[2] Abel, David, et al. On the expressivity of Markov reward, NeurIPS 2021

the basis of UED should be explained more.

Our Eq. 3, which minimizes the worst-case Protagonist Antagonist Induced Regret $Regret$, is inspired by UED in [1]. However, the core of our approach focuses on learning policy with a set of candidate reward functions rather than searching for diverse environments to test the robustness of policies as in [1]. We have developed novel theories for reward search.

Furthermore, as mentioned in line 47~52, our contribution is three-fold. Besides formulating Eq. 3 for reward search, our two other contributions are independent of UED.

• Our first contribution in establishing the concepets and properties of task-alignment that ultimately lead us to Theorem 1, which justifies the usage of Eq.3.
• Our third contribution is to develope a practical implementation for solving MinimaxRegret in Section 5.1.

Therefore, we cited [1] but omitted its details to focus on the distinct aspects of our work.

...additional experiments on continuous control tasks

In our main text (lines 226-234) and in the general response, we have explained why we did not focus on continuous tasks.

However, we did include continuous control tasks in Appendix C.2. Furthermore, we have conducted additional experiments on Offline IL with D4RL continuous control datasets. The details can be found in our general response

...E.g. section 5.1. could be moved to the appendix or integrated in a reduced version in the main text.

We intended for Section 5.1, especially Proposition 2, to highlight the difference between PAGAR-based IL and IRL-based IL: IRL-based IL uses a single reward function derived from an inner optimization problem for policy training, whereas our method employs a mixture of reward functions for policy training. We will consider restructuring our manuscript to provide better clarification rather than presenting all the information as it currently stands.

Missing } in line 270.

The } is in the middle of the equation.

[1] Dennis, M, et al., Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design, NeurIPS 20202.

Dear Reviewer,

Thank you once again for your valuable comments.

In our revised version, we will focus on presenting the main idea of the paper as follows to ensure our contributions are effectively communicated:

• Goal: Learn a policy to fulfill an unknown task that fits the description in Definition 1.
• Insight: Achieving high utility under task-aligned reward functions is essential for task fulfillment.
• Problem: The specific task-aligned reward function is unknown.
• Solution: By treating expert demonstrations as weak supervision, we learn a set of reward functions that encompass task-aligned rewards and then train a policy to achieve high utility across this set of reward functions.

We also hope that our experiments with GAIL/VAIL in continuous control tasks in Appendix C.2, and the additional experiments with the more recent IRL/IL baseline f-IRL (provided in the general responses) and SOTA offline IL technique RECOIL (also provided in the general responses) have addressed your concerns about experimental evaluation.

Please let us know if further clarification or additional evaluations would help in revisiting the score.

Your support would make a significant difference.

Thank you!

Best,

Authors

MiniGrid includes more challenging environments

As our approach builds upon existing IRL/IL methods (PAGAR-GAIL, PAGAR-VAIL), its performance is influenced by the integrated IRL/IL algorithms. We found that the more complex MiniGrid environments, such as ObstructedMaze and KeyCorridor, are too challenging for our integrated IRL/IL methods and the baselines to produce meaningful results. Consequently, we did not include these environments. Future work will focus on enhancing our method and the underlying IRL/IL algorithms to address these more challenging tasks.

... requires a set of candidate reward functions

We do not explicitly build this set of reward functions. We mention in Section 6.3 that we incorporate the $r\in R_{E,\delta}$ constraint into the objective function. Algorithm 1 shows that the policy optimization and reward searching processes are performed alternately, similar to those in GAN-based IRL/IL methods.

Comparing the time efficiency with prior work would be beneficial.

We appreciate this suggestion and will analyze the time efficiency in the updated version of the paper. The primary potential increase in time complexity arises from the need to optimize two policies, $\pi_A$ and $\pi_P$. However, these two policies can explore the environment and be optimized in parallel, mitigating the redundant time complexity. Additionally, the estimation of Equations 6 and 7 introduces some overhead: we evaluate policy $\pi_A$'s output along $\pi_P$'s samples and vice versa. This increased time complexity is linear in the size of the replay buffers of the two policies.

would the method show more advantages if only 1 demonstration

We did attempt to use only one demonstration. However, our approach, like the integrated GAN-based IRL/IL algorithms, suffers when only one demo is provided. Our conjecture is that the $R_{E,\delta}$ set can be overwhelmed by task-misaligned reward functions that overfit the single trajectory, leading to poor performance. Our approach aims to improve task alignment under general non-ideal situations, but it is not optimized to excel in one-shot IL across baselines.

In larger mazes or more challenging environments with complicated demonstrations, would the proposed approach still perform impressively with only 1 demo?

In environments like S9N1, IRL/IL baselines struggle with only one demo, while our approach shows better results. However, as tasks become more complex, the partial observability significantly hinders the IRL/IL baselines from producing meaningful results, which can affect the performance of our method, as mentioned earlier. We have tested our algorithm and the baselines in the KeyDoorS3 and ObstructedMaze environments but did not achieve satisfying results.

There is no analysis explaining why the proposed method performs better on MiniGrid (Figure 2) than on locomotion tasks

We discussed this briefly in lines 226-234. The intuition is that MiniGrid tasks, which are primarily about reachability, have a clear binary success or failure criterion. Such tasks present a distinct discrepancy between task-aligned and misaligned reward functions, as well as between acceptable and unacceptable policies. In contrast, locomotion tasks exhibit more quantitative and nuanced differences between policies. Our method aims to learn an acceptable policy, which is not necessarily an optimal policy under some 'ground truth' reward function -- in Proposition 2, we show that the policy is optimal under a mixture of rewards.

Another probable explanation is that we did not use high-quality demos in locomotion to simulate non-ideal situations, as shown by the expert's total reward in Table 2. In MiniGrid, even a low-quality demo ends up reaching the goal, and our algorithm captures this property. However, in locomotion, low-quality demos may exhibit different behaviors, such as low speed or perfect balance followed by a crash.

Dear Reviewer,

Thank you for recognizing the effort in our response.

Please let us know if further clarification or additional evaluations would help in revisiting the score.

Your support would make a significant difference.

Bests,

Authors

Thanks for the detailed response. I'd keep my score.