Adversarial Imitation Learning via Boosting

Thank you for your review. We hope to address your concerns below:

How policies are ensembled

The way we set up our ensemble is as follows: whenever we evaluate our policy, we choose policy $\pi_i$ from our ensemble with weight proportional to $\alpha_i$, and then roll out said policy. A more detailed discussion of this sampling process can be found in [1] section 4.1. As you pointed out, we sample the policy with respect to some random number, but note that our sampling is weighted based on the policy weight ($\alpha_i$). Note that we are learning a boosted policy in the state-action occupancy measure space as detailed in section 2 of [1]. Finally, the weighting scheme is following the gradient boosting literature for RL including [1][2].

[1] Provably Efficient Maximum Entropy Exploration, Hazan et al. 2019

[2] Local Policy Search in a Convex Space and Conservative Policy Iteration as Boosted Policy Search, Scherrer and Geist 2014

Thank you for taking the time for a thorough review. We hope to address your concerns below:

[W1] More SOTA IL algorithms

We believe that we have covered a good range of SOTA off-policy, IL algorithms. Could the reviewer point us to any additional algorithms they would like to see?

[W2] Lack of Theoretical analysis

Thank you for this feedback. We agree with the reviewer that additional, in-depth analysis would only strengthen our work, but with this work we aimed to investigate and experimentally instantiate an algorithm that has the same principled off-policy treatment as ValueDICE and the practical scalability of DAC. Thus we focused on doing a thorough evaluation across the DeepMind Control Suite.

[W4] Lack of Ablation

Thank you for this critique. The practical instantiation of our algorithm is very close to DAC with the main difference being the boosted discriminator update. By comparing to DAC we felt that this was both ablating the most important contribution of AILBoost when compared to baselines. As for the ablation of the mixing parameter, we did not find this parameter to be so sensitive. Note that we used the same parameters across all experiments in a given task suite.

[Q1] Computational Cost

Of course, here is a discussion of the computation costs of our practical instantiation of Algorithm 1. We would like to clarify that in our practical instantiation of our algorithm, we incur negligible additional computational overhead over DAC. We maintain the same hyperparameters for the replay buffer and store the samples that we collected from each weak learner in the replay buffer of fixed size. The only additional data that we need to store are the weights of each sample that represent the weight of each weak learner. Finally, for both the policy and the discriminator, we found that warmstarting the following weak learner and discriminator with the previous models respectively worked well in practice. For the final evaluation, we evaluated both the ensemble and the final resulting policy and found both policies to have comparable performance. Thus, we could fully avoid storing the entire ensemble.

[Q2] How are the weights computed? What is the impact of replay buffer size?

For our experiments we implemented a fixed exponential weighting scheme that favors more recent weak learners. This is following gradient boosting work in RL [1] and maximum entropy RL [2]. We agree that it would be interesting to further investigate learning and adaptively update the ensemble weights during learning but did not find this to be essential in showing improvement over previous algorithms.

One of the hyperparameters we explored involved the number of weak learners contained in the replay buffer. Given our implementation, increasing the buffer size would have the effect of increasing the number of weak learners combined in the ensemble. We did not find this hyperparameter to be very sensitive, especially since we could tune the weighting scheme to effectively control the number of relevant learners.

[1] Local Policy Search in a Convex Space and Conservative Policy Iteration as Boosted Policy Search, Scherrer and Geist 2014

[2] Provably Efficient Maximum Entropy Exploration, Hazan et al. 2019

[Q3] Effect of Number of Expert Demonstrations

AILBoost, similar to other IL algorithms, exhibits improved performance with more and more expert data. We would most likely see diminishing returns once we have enough expert demonstrations to fully estimate $d^{\pi*}$. This, however, is not specific to AILBoost and is a property of all AIL algorithms.

[Q4] What is the key factor?

Of course, first when compared to DAC, we believe the key improvement was AILBoost’s proper handling of off-policy samples in the discriminator update. ValueDICE also addresses and fixes this concern through the DICE objective; however, we have found this objective to be more unstable and difficult to scale in practice. Furthermore, the ValueDICE objective, in principle, only works for deterministic environments. Thus we believe the key factors are the combination of more scalable DAC-like optimization with an improved treatment of off-policy samples.

[Q5] Limitations

Thank you for providing this feedback. On top of the slight memory overhead, a potential limitation of AILBoost could be settings where SAC-style off-policy learning may be very expensive. That is, since SAC updates during the collection of a trajectory, in settings where it is impossible to incur the latency costs of stopping mid-trajectory to update the policy, AILBoost would also suffer.

Thank you for taking the time for a thorough review. We hope to address your concerns below:

[W1] Noise in Policy Rollout

Please see response/question to [Q4].

[W2] Not Aligned with Adaboost

We would first like to clarify that AILBoost is a gradient boosting [1] algorithm rather than one closer to adaboost. Moreover, could the reviewer elaborate on the unrealizable case? As [2] showed, our RL procedure could be viewed as boosted policy search.

[1] Boosting Algorithms as Gradient Descent, Mason 1999

[2] Local Policy Search in a Convex Space and Conservative Policy Iteration as Boosted Policy Search, Scherrer and Geist 2014

[Q1] Dagger mixing:

We do not consider the mixing performed in Dagger because we do not assume access to an online interactive expert. Instead, in our setting, we only have access to expert demonstration data. We are performing weighted mixing of weak learners, not a weighted mixture between a weak learner(s) and an expert, because we do not have access to an expert policy. More specifically, we are learning a boosted policy in the state-action occupancy measure space as detailed in section 2 of [1].

[1] Provably Efficient Maximum Entropy Exploration, Hazan et al. 2019

[Q2] Smooth loss condition:

Please see [1], section 1.1 for their informal main theorem. For a more detailed discussion, please see Section 4 of [1].

[1] Provably Efficient Maximum Entropy Exploration, Hazan et al. 2019

[Q3] Off-policy versus on-policy:

The underlying reinforcement learning algorithm that our boosting algorithm builds on top of is soft actor-critic (SAC). SAC is an off-policy algorithm because it keeps around a replay buffer of past experiences to update the policy (which weren’t collected from the current/most recent policy), vs. on-policy algorithms that strictly update the policy from samples gathered from the current policy. In our algorithm, we keep around a replay buffer of past weak learners’ experiences that we weight to update our mixture policy, making our algorithm off-policy.

[Q4] Noise during policy rollout:

Could we ask the reviewer to clarify what they mean by noise in the policy rollout? If they are referring to the case where the off-policy rollouts that we store are, by chance, that deviates from the experts then we emphasize that this is not a bad case. By training a discriminator that witnesses these deviations we provide a strong learning signal and over the course of training where we iterate discriminator training and policy sampling, we would eventually correct any distribution shift that may occur.

[Q5] Learner weaker than expert:

It would be interesting to see experiments where the learner is truly weaker than the expert. We will leave this investigation for future work.

[Q6] Updating the weaker leaner:

In algorithm 2, we update the weak learner $\pi_{t+1}$ with our off-policy RL algorithm using the reward function from the discriminator. The discriminator reward function captures the difference between the mixture policy and the expert data. The new weak learner is trained to minimize this difference, essentially making the mixture policy more similar to the expert dataset.

[Q7] Uniform weight weak learner update:

Our boosting algorithm modifies the discriminator updates to address the fact that the original DAC does not compute the density ratio – which is needed to minimize the original divergence that we care about. For updating our weak learner, we follow the original soft actor-critic (SAC) procedure which uses uniform weighting when sampling data from the replay buffer [1].

[1] Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, Haarnoja et al. 2018

[Q8] Policy updates versus discriminator updates:

By 1000 policy updates to 100 discriminator updates, we are referring to the gradient update steps that we did for each phase of learning. That is, we took 1000 SAC updates for the policy for every 100 gradient discriminator updates.

[Q9] Mobil reference

Thank you for the reference; we will make sure to include this MoBIL observation in our final version.

Hi,

Thank you so much for your response. The definition of "off-policy" in our case is that our algorithm uses data that is not collected by the most recent policy to update our ensemble. This is enforced through a replay buffer which we sample from to do the policy update. All data that is collected by our most recent policy is added to the buffer, and can be resampled for future updates. Thus, we are able to update our ensemble through data that wasn't explicitly collected by rolling out said ensemble, making our algorithm off-policy. An example of an off-policy method is Q-learning, where online interaction with the environment is allowed. In continuous control, SAC is an off-policy algorithm, which we used in our work.

Offline reinforcement learning refers to when the agent cannot interact with the environment, which we think is what you are referring to. You are correct that our algorithm is not an offline RL/IL algorithm.

Best, Authors

Thank you for taking the time for a thorough review. We hope to address your concerns below:

[Q1] Presentation of Abstract and Introduction

We would like to thank the reviewers for the feedback on the presentation of our paper. We agree that focusing on the paper’s main contribution and removing additional details about existing work could increase the readability of the abstract and introduction. We will address these concerns in the final version of the paper.

[Q2] Choice of reverse KL

DAC optimizes the same divergence that AIRL [1] minimizes, which is the reverse KL (please see Section 4.1 of [2]). Taking recommendations from both [2] and [3], we chose to minimize the functional form for the reverse KL for our adversarial IL setting. Finally, we independently confirmed that we can reproduce DAC’s results with this functional form of the reverse KL.

[1] Learning Robust Rewards with Adversarial Inverse Reinforcement Learning, Fu et al. 2018

[2] A Divergence Minimization Perspective on Imitation Learning Methods, Ghasemipour et al. 2019

[3] Imitation Learning as f-Divergence Minimization, Ke et al. 2020

[Q3] Weighted replay buffer and boosting

We agree with the reviewer that our idea of incorporating a weighted replay buffer and boosting can be extended to any divergence minimization imitation learning algorithm. Note that DAC uses the reverse KL divergence in their algorithm, so we performed our experiments using this divergence for a fair comparison and thus we are improving DAC through boosting.

[Q4] Warm starting weak learner

When learning a weak learner in our boosting algorithm, we could either randomly initialize our weak learner or initialize our weak learner from our previous policy. On page 7, line 1, we are stating that our weak learners are initialized to the previous policy that was learned.

[Q5] Comparing training times

In terms of training time, the computational complexity remains the same – we still sample the same from the replay buffer for discriminator training (albeit weighted) and we only update the ensemble via SAC on the latest policy. The key difference is that we roll out with our ensemble, which doesn’t incur any computational overhead. We do make a point that there will be a slight increase in memory cost, which we found manageable at the end of Section 4.

[Q7] Figure Sizes

Thank you for pointing out the issues with figure 2,3,4. We will make sure to address these issues for the final version.