In-Trajectory Inverse Reinforcement Learning: Learn Incrementally Before an Ongoing Trajectory Terminates

Thanks for your constructive reviews. We believe that our discussion will lead to a better paper. Before addressing your comments, we would like to clarify the goal of IRL and what role the reward unidentifiability issue plays. IRL aims to learn a reward function that can explain the expert demonstrations. Explaining expert demonstrations means that the optimal policy of the learned reward generates the same trajectories as the expert demonstrations [C1,C2]. Current IRL methods can already learn a single reward function to explain expert demonstrations [C1,C7,C8] but cannot solve the reward unidentifiability issue in general cases. Reward unidentifiability means that even if we can learn a single reward to explain the demonstrations, we cannot guarantee that the learned reward is the expert reward because there are many different rewards that can explain the expert demonstrations. There are some papers studying reward identifiability, however, they all impose additional assumptions. For example, [C3] shows that the reward can be recovered up to a constant if we have demonstrations under different discount factors or different environments. [C4] quantifies the reward distance between the learned reward and the ground truth reward under the assumption that the MDP is linear. [C5,C6] study feasible reward set to identify a set of rewards that can explain the demonstrations, however, they do not solve the identifiability issue because the feasible reward set still includes multiple reward functions that can explain the expert demonstrations. In summary, if we want to learn a single reward to explain expert demonstrations, it is not necessary to solve reward unidentifiability issue nor study reward feasible sets. In fact, the reward unidentifiability issue in general IRL cases is still unsolved to IRL community.

Weakness 1: There is no clear ... this fact.
Answer: Thanks for mentioning (1) reward identifiability and feasible reward set, (2) what kind of reward is extracted, and (3) what the learned reward can be used for. We address these three comments one by one.

For the reward identifiability and feasible reward set, please refer to the first paragraph in this response that (i) it is sufficient for IRL to learn a single reward function to explain the demonstrations, (ii) reward identifiability and learning a feasible reward set are sufficient but not necessary for reward learning. Note that reward unidentifiability in general IRL cases is still unsolved to IRL community. In this paper, we empirically compare the learned reward and the expert reward in Appendix D.2, and the results show that the learned reward is close to the expert reward. As mentioned in Appendix F, we would like to explore the reward identifiability in future works.

We now discuss what kind of reward we extract. Lines 89-94 mention that the kind of reward we aim to learn is the reward with maximum likelihood. Maximum likelihood IRL [C8,C9] is a standard IRL approach to learn a reward function to explain the expert demonstrations. The intuition behind this is that we aim to learn a reward function whose corresponding optimal policy makes the expert demonstrations most likely (line 94), i.e., the corresponding optimal policy can generate trajectories that are same with the expert demonstrations. Therefore, the learned reward with maximum likelihood explains the expert demonstrations [C8,C9].

The learned reward can be used to explain the demonstrations. Moreover, compared to imitation learning that directly learns an imitating policy, the benefit of learning a reward is that the learned reward can be used for counterfactual analysis such as the estimation of optimal policies under different environment dynamics [C8].

Weakness 2: The presentation ... is poor.
Answer: Thanks for mentioning the presentation of the contents and contributions. We have a contribution statement in lines 38-53 that clearly summarizes our contributions: (1) we study a novel problem setting of in-trajectory learning, formulate it as a bi-level optimization problem, and propose a novel reward update mechanism to solve the problem; (2) we provide solid theoretical analysis to guarantee that the algorithm achieves sub-linear (local) regret where the input data is not i.i.d.

To improve the presentation of other contents, we will (i) add more explanations about problem (1) to highlight the kind of reward we learn, (ii) add the discussion on related works about reward identifiability and feasible reward set, (iii) add more discussions on the technical assumptions and the theoretical statements.

Question 1: What reward ... used for?
Answer: Please refer to the answer to weakness 1.

Question 2: Since in RL ... the entire horizon?
Weakness: Please refer to the global response.

[C1] Pieter Abbeel, and Andrew Y. Ng. "Apprenticeship learning via inverse reinforcement learning." ICML, 2004.

[C2] Saurabh Arora, and Prashant Doshi, "A survey of inverse reinforcement learning: Challenges, methods and progress." Artificial Intelligence, 2021.

[C3] Haoyang Cao, et al. "Identifiability in inverse reinforcement learning", NeurIPS, 2021.

[C4] Zihao Li, et al. "Reinforcement learning with human feedback: Learning dynamic choices via pessimism", arXiv preprint arXiv:2305.18438, 2023.

[C5] Alberto Maria Metelli, et al. "Towards theoretical understanding of inverse reinforcement learning", ICML, 2023.

[C6] Filippo Lazzati, et al. "Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms", arXiv preprint, arXiv:2402.15392, 2024.

[C7] Brian D. Ziebart, et al. "Maximum entropy inverse reinforcement learning." AAAI, 2008.

[C8] Siliang Zeng, et al. "Maximum-likelihood inverse reinforcement learning with finite-time guarantees." NeurIPS, 2022.

[C9] Shicheng Liu and Minghui Zhu. "Learning multi-agent behaviors from distributed and streaming demonstrations." NeurIPS, 2022.

Thanks for your constructive review. We believe that our discussion will lead to a stronger paper. We address your comments below:

Weakness 1: Assumption 1 assumes that the parameterized reward is smooth, which can be a strong assumption, especially for neural networks with non-smooth activation functions such as ReLU.
Answer: Thanks for mentioning Assumption 1. We agree that this assumption could be strong when the neural networks use ReLU activation. However, the smoothness assumption can be satisfied by using some smooth activation functions such as tanh and using linearly parameterized models. In our experiment, we use tanh as the activation function. Moreover, as mentioned in the paragraph under Assumption 1 (i.e., line 255), this assumption is widely adopted in RL [A1] and IRL [A2,A3]. It is interesting to relax this assumption, and we will explore it in the future.

Weakness 2: The meta-regularization part seems to be important to the performance. However, the meta-learning needs to pre-collect data for many related tasks, which can be difficult in real world.
Answer: Meta-learning requires to collect data for related tasks before training, however, data collection is not difficult in many real-world problems. In fact, meta-learning has been applied to many real-world problems, such as velocity modelling for self-driving where the data of related tasks (i.e., velocity trajectories in different roads) can be obtained from public data sets [A4], and object grasping of robotic arms where the data of related tasks (i.e., demonstrations of grasping objects from different locations) is generated by solving RL problems to grasp the objects [A5].
In Section 5.2, a real-world stock market example is used to evaluate our algorithm. Data collection is not difficult for this application. In particular, the demonstration data of related tasks is the investment trajectories of other investors with different preferences of taking risks. We can collect such information from public records [A6]. Moreover, our method can also be applied to [A4,A5]. Take [A5] as an example, we can first use their data sets to learn the meta-prior. Then, given a new demonstration trajectory of grasping an object from a new location, we can sequantially reveal this trajectory to the learner in an ongoing fashion.

Question 1: How the authors collect data for the meta-regularization in the experiment?
Answer: For the MuJoCo experiment, the different training tasks have different target velocity from 0 to 3 (line 328) and the reward function is $-|v-v_{\text{target}}|$ where $v$ is the current velocity and $v_{\text{target}}$ is the target velocity (line 326). Therefore, for each training task, we randomly sample the target velocity from 0 to 3, design the reward function correspondingly, and use the reward function to train an RL policy to generate demonstration trajectories. For the stock market environment, different tasks have different preferences of taking risks, which is captured by different turbulence threshold (line 362). The reward function is $p_1-p_2$ where $p_1$ quantifies the profit and $p_2$ quantifies the amount of trading stocks that are above the turbulence threshold. Therefore, for each training task, we randomly sample turbulence threshold from 30 to 60 (line 365), design the corresponding $p_2$, and train an RL agent under the current $p_1-p_2$ to generate investment demonstrations. For the active shooting experiment in Appendix D.5, different training tasks have different goal locations. For each training task, we randomly sample an $1\times 1$ goal area from the state space and train an RL algorithm to reach the goal to generate demonstrations.

[A1] Kaiqing Zhang ,Alec Koppel, Hao Zhu, and Tamer Basar. "Global convergence of policy gradient methods to (almost) locally optimal policies." SIAM Journal on Control and Optimization, 2020.

[A2] Siliang Zeng, Chenliang Li, Alfredo Garcia, and Mingyi Hong, “Maximum-likelihood inverse reinforcement learning with finite-time guarantees,” in Advances in Neural Information Processing Systems, 2022.

[A3] Ziwei Guan, Tengyu Xu, and Yingbin Liang. "When will generative adversarial imitation learning algorithms attain global convergence." In International Conference on Artificial Intelligence and Statistics, 2021.

[A4] Bo Yu, Xiangyu Feng, You Kong, Yuren Chen, Zeyang Cheng, and Shan Bao. "Using meta-learning to establish a highly transferable driving speed prediction model from the visual road environment." Engineering Applications of Artificial Intelligence, 2024.

[A5] Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. "One-shot visual imitation learning via meta-learning." In Conference on robot learning, 2017.

[A6] Xiaoyang Liu et al. "FinRL-Meta: Market environments and benchmarks for data-driven financial reinforcement learning." Advances in Neural Information Processing Systems, 2022.

Thanks for your insightful reviews. We believe that our discussion will lead a better paper. We address your comments below:

Weakness 1: In the experiments, I don’t understand why different algorithms have different initial points? When t=0, are you using a same random reward function or the prior-meta reward function to train your policy?
Answer: Thanks for mentioning the different initial points. There are two different initial points in the experiment. One initial point is the meta-prior reward and the other initial point is random reward. There are four algorithms where the algorithms with meta-regularization (i.e., MERIT-IRL and naive MERIT-IRL) use the same meta-prior reward function as the initialization and the algorithms without meta-regularization (i.e., IT-IRL and naive IT-IRL) use the same random reward function as the initialization. The different initial points show the benefit of meta-regularization where the algorithms with meta-regularization have higher initial cumulative reward than the algorithms without meta-regularization.

Question 1: Can the author explain why comparing the suffixes $\\{s_i', a_i'\\}$ and $\\{s_i'', a_i''\\}$ encourages the reward function to predict the future? Since you are completing the incomplete trajectory using the same policy, can reward function learn something even if both are generated by the same policy.
Answer: The suffices $\\{s\_i',a\_i'\\}\_{i\geq t}$ and $\\{s\_i'',a\_i''\\}\_{i\geq t}$ are both generated by the learned policy but they start from different initial state-action pairs where $\\{s\_i'',a\_i''\\}\_{i\geq t}$ starts from the expert state-action pair $(s_t^E,a_t^E)$ and $\\{s\_i',a\_i'\\}\_{i\geq t}$ starts from the learner state-action pair $(s_t',a_t')$. Consider the standard IRL where we have a complete expert trajectory $\\{s\_i^E,a\_i^E\\}\_{i\geq 0}$ to guide the learned trajectory $\\{s\_i',a\_i'\\}\_{i\geq 0}$. The best option to predict the future is to use the expert future (i.e., expert suffix $\\{s\_i^E,a\_i^E\\}\_{i\geq t}$) to guide the learner suffix $\\{s\_i',a\_i'\\}\_{i\geq t}$. However, in our setting, we do not have the future expert trajectory $\\{s\_i^E,a\_i^E\\}\_{i\geq t}$ so that we use the suffix $\\{s\_i'',a\_i''\\}\_{i\geq t}$ as an approximation of the future expert trajectory $\\{s\_i^E,a\_i^E\\}\_{i\geq t}$ to guide the learner suffix $\\{s\_i',a\_i'\\}\_{i\geq t}$. If the approximation suffix $\\{s\_i'',a\_i''\\}\_{i\geq t}$ is a good approximation of the expert suffix $\\{s\_i^E,a\_i^E\\}\_{i\geq t}$, the learned reward can be accurate for the entire horizon.

Now the question left is whether the approximation suffix $\\{s\_i'',a\_i''\\}\_{i\geq t}$ is a good approximation of $\\{s\_i^E,a\_i^E\\}\_{i\geq t}$. The answer is that the approximation error between $\\{s\_i'',a\_i''\\}\_{i\geq t}$ and $\\{s\_i^E,a\_i^E\\}\_{i\geq t}$ diminishes when $t$ increases. This is mathematically justified by the sub-linear regret in Theorems 1 and 2 because the regret will be at least linear if the approximation error is always above a certain constant. The intuition behind this is that when $t$ increases, we observe more portion of the expert trajectory and thus the learned reward $r_{\theta_t}$ improves. A better $r_{\theta_t}$ will lead to a better learned policy $\pi_t$ that generates trajectories closer to the expert trajectory, and thus the new approximation suffix $\\{s\_i'',a\_i''\\}\_{i\geq t+1}$ will be closer to the expert suffix $\\{s\_i^E,a\_i^E\\}\_{i\geq t+1}$, i.e., the approximation error becomes smaller. Therefore, by comparing the learner suffix $\\{s\_i',a\_i'\\}\_{i\geq t+1}$ and the approximation suffix $\\{s\_i'',a\_i''\\}\_{i\geq t+1}$, the learner suffix $\\{s\_i',a\_i'\\}\_{i\geq t+1}$ gets closer to the expert suffix $\\{s\_i^E,a\_i^E\\}\_{i\geq t+1}$.

Question 2: Just wondering how you complete the incomplete trajectories in MuJoCo using saved states, since the state in MuJoCo only includes partial information about the simulator.
Answer: In MuJoCo, the learned policy usually maps the observation to a distribution of actions. Given the current observation and current action, the simulator tells us the next observation. We agree that the observation only includes partial information about the simulator. However, the observation is not the physical state of the simulated robot. The physical state in MuJoCo is characterized by two vectors, called "qpos" and "qvel". The vector "qpos" stores all the current positions and orientations of the joints and bodies and the vector "qvel" stores all the current linear and angular velocities of the joints and bodies. The vectors "qpos" and "qvel" include the complete information of the simulated robot. Therefore, we save "qpos" and "qvel" instead of the observation to save a physical state of the simulated robot. In practice, we use the command "env.data.qpos" and "env.data.qvel" to obtain the "qpos" and "qvel" vectors of the current simulated robot and save these vectors. When we want to set the robot's physical state to a saved (qpos,qvel), we use the command "env.set_state(qpos,qvel)". These commands work for gym-v4.