RILe: Reinforced Imitation Learning

We are grateful for the reviewer’s insightful comments and are convinced that they have improved our paper. Below, we address each of the concerns:

1. Motivation and Advantages of the Teacher-Student Framework: • We thank the reviewer for the comment. GAIL does indeed have advantages over behavioral cloning, since it tries to give rewards to the learning agent in states which are not covered in data through a discriminator (which is trained in a supervised manner). However, long-term effects of given rewards are never considered by the discriminator, since it is trained in a supervised way to minimize a step-based loss. This results in a discriminator that does not consider long term effects of the reward signals it produces, and eventually results in an agent which just tries to exactly imitate expert state-action pairs, even if they are problematic. For instance, in the dataset, for some specific state-action pairs, if all samples in data have a great noise on them, then the discriminator expects the student to exactly match these pairs, which results in a poor policy. Moreover, a discriminator may not fully understand connections between different possible states, since it does not maximize a cumulative loss, and since the loss just depends on the current step. In RILe, we consider long-horizon effects of given rewards by training a teacher agent as a reward function, which is rewarded based on a trajectory of a student. This also allows the teacher to learn long term effects of the produced reward signals and relations between different states, since the long-horizon cost is minimized. By breaking the data-policy connection, we imply that the student does not just mimics expert behavior, but satisfies an intermediary agent that learns that even if some state-action pairs are not beneficial in the short-term, these pairs may offer long-term benefits. Furthermore, instead of training a discriminator, which just considers the current timestep similarity of the given state-action pair, we train a the teacher agent by rewarding it on the trajectory generated by the student. All in all, the alleviation of the data sensitivity comes from long-horizon optimization problem of the reward, and the trajectory based loss used in teacher training. We have updated the method section to more clearly explain this motivation.
2. Reward Structure and Use of Reinforcement Learning Algorithms: • We removed the ¡supervised learning over loss functions defined in Equations 11 and 12¿ sentence to avoid confusion. We were trying implying that any single-agent RL method can be used to learn the reward function, instead of learning it via a supervised discriminator.
3. Concerns About Atari Experiments: • Note that we utilized Ram observations without frame stacking. This resulted in signif- icantly poorer performance than previously reported results, since the dynamics of the environment are hardly understandable without temporal information. In case of Pong and BreakOut, the lack of temporal data led to policies generating scores that imply a possible lack of convergence. Therefore, we excluded the experiments involving those games from the paper. • The random sampling of batches might cause BC to require more time to converge with 100 trajectories compared to a single one. • There was a misinterpretation in reporting the standard deviation for baselines, which we have now rectified. Updated results with correct standard deviations are included. We utilized ten different testing seeds for both RILe and baseline methods. Five training trials with varied random seeds were conducted for each setup, selecting the most promising outcome for testing.

We would like to express our gratitude for the insightful and constructive feedback provided on our paper. We have revised the paper to address each of the points raised and outline the updates we have made in response to the reviewers’ feedback below:

1. Presentation and Soundness: • We have thoroughly investigated the soundness of our approach. We have updated the Section 4 for better explanation of RILe. Each component is explained in more detail, by also providing their own optimization problems. Then, RILe is explained as the combi- nation of the three presented components: the student agent, the teacher agent and the discriminator. Moreover, we demonstrated the ability of the teacher to guide student for mimicking expert behaviors through a lemma, and provided theoretical results for this in Appendix A. This lemma shows that the student policy should eventually perform correct distribution matching to satisfy the teacher. We also have introduce another lemma that presents that the discriminator successfully learns to discriminate expert state-action pairs, and we provided theoretical results for this in Appendix A.

2. Comparison to AIL/IRL: • In AIL, the learning agent is guided by a discriminator, which is defined the same way in RILe, however, in our case it is used to guide the teacher agent instead of the learning agent. Therefore, in AIL, the student tries to satisfy the discriminator directly. Since the discriminator just aims to minimize a step-based cross entropy loss, the long-term effects of generated rewards are never considered by the discriminator. This results in a myopic discriminator which does not consider long term effects of the reward signals in the policy, and eventually results in an agent which is guided to exactly imitate expert state- action pairs, even if they are noisy. For clarification, in both AIL and RILe, the student agent is long-horizoned, maximizing a cumulative reward. What we are comparing is the discriminator in GAIL and the teacher agent in RILe. The myopic optimization in AIL may also lead to failure in understanding connections between different possible states. In other words, since the discriminator in RILe is just step-based, it does not have any incentive to discover the relation between states, i.e., it does not know what the next-state of the student is given the current state-action pair. However, the teacher agent in RILe is trained via RL, which helps it to discover these state-relations along with helping to understand long-term effects of produced reward signals in the end policy. In contrast, IRL considers the effects of the deduced reward function in a more exten- sive manner, by incrementally updating the reward function and re-training a policy from scratch. This sequential re-training of the policy helps IRL to fully understand the effect of reward signals on the policy. However, this incremental update pipeline comes with a bur- den of computational in-feasibility. Because for each small update in the reward function, a new policy is trained from scratch to see the effects of the reward fully. In RILe, we combined the granular understanding of the effects of the reward function representation from IRL, along with the adversarial discriminator from AIL. The aim of RILe is making IRL computationally feasible, by guiding the reward function learning with an adversarial discriminator. This results in a reward function, i.e., teacher, that learns long term effects of the produced reward signals and relation between different states, since long- horizon cost is minimized via RL. Thanks to guidance from the adversarial discriminator, the reward function is learned at the same time with the student policy. In Section 4, we have added a subsection Intuition, where we discuss the intuition behind RILe. In this section, we put RILe into context with adversarial imitation learning and inverse reinforcement learning intuitively.

3. Online Sample Efficiency: • Return plots for the Humanoid task are included in the updated paper under Section 5, providing a clear comparison of the online sample efficiency between RILe and baselines. We selected Humanoid since it is more challenging and requires more samples for training. In the plots, we present that when the same learning agent is used, which is SAC, RILe has significantly better sample efficiency.

We thank the reviewer for taking the time to provide a detailed review of our paper. We greatly appreciate the insightful comments and have made several improvements to the manuscript. Please find below our responses to the issues raised:

1. Presentation and Notation Clarity: • We have thoroughly revised Section 4 to clarify the technical description of our method. First, we explained three components (the student agent, the teacher agent and the dis- criminator) along with their own optimization problems for clarity. The discriminator aims to understand whether a given state-action pair comes from an expert trajectory or not. It is defined as a feed-forward deep neural network. Given expert state-action pairs and other state-action pairs whose source is different than the expert data, the discriminator aims to discriminate expert pairs from others. The student agent aims to learn a policy by interacting with an environment in a standard RL setting, where for each of its actions the environment returns a new state. However, rather than from a hand-crafted reward function, the student agent receives its reward from the policy of the teacher agent. Therefore, the reward function is represented by the teacher policy. The student agent is guided by the actions of the teacher agent, i.e., the action of the teacher is the reward of the student. The teacher agent aims to guide the student to mimic expert behavior by operating as its reward mechanism. Therefore, the teacher agent learns a policy that produces adequate reward signals to guide the student agent, by interacting with an environment in a standard RL setting. The state of the teacher comprises the state-action pair of the student. It generates a scalar action which is given to the student agent as reward. The teacher agent’s reward function is defined with a reward approximating network RILe combines the three key components defined previously in order to converge to a student policy, which mimics expert behaviors. To achieve this goal, the discriminator optimization problem is tweaked as to allow it to discriminate expert state-action pairs when samples from student’s and expert’s trajectories are given. In other words, the discriminator aims to discriminate state-action pairs from expert and student agent. This reformulated dis- criminator is employed as the reward function of the teacher. Consequently, to increase its rewards, the teacher agent must encourage the student to generate state-action pairs that deceive the discriminator into perceiving them as originating from an expert. To prove that the student agent can learn expert-like behavior, we need to show that the teacher agent learns to give higher rewards to student experiences that match with the expert state-action pair distribution, as this would enable a student policy to eventually mimic expert behavior. Lemma 1: Given the discriminator, the teacher agent optimizes its policy via policy gra- dients to provide rewards that guide the student agent to match expert’s state-action dis- tributions. Lemma 2: The discriminator will converge to a function that estimates the probability of a state-action pair being generated by the expert policy, when trained on samples generated by both a student policy and an expert policy. Theoretical results for lemmas are provided in Appendix A.