Blending Imitation and Reinforcement Learning for Robust Policy Improvement

We greatly appreciate your comprehensive review of our manuscript. Below, we address each of your questions in detail:

Q1: "I think including a thorough discussion about the relationship of the proposed method and MAMBA as well as limitations would go a long way to make this easier to understand."

A: Thank you for your advice! We have added a detailed paragraph in Appendix A.1 (additional notes on related work) where we discuss the relationship between the proposed method and MAMBA, as well as their limitations (see Appendix B.2 limitations of the prior art). Please see the updated PDF. In the meantime, we created the following table in hopes that it addresses your questions/concerns.

Q2: "...include some wall-time comparisons between the different methods.."

A: Thank you for the suggestion! We have added Table 4 in Appendix D.6 "Computing Infrastructure and Wall-Time Comparison" that provides the wall-time comparison that you requested.

Q3: "In the appendix, it says that the value ensembles are pre-trained before the main loop starts. That sounds like the proposed method uses more samples (and more compute) than the baselines. This should be mentioned in the main paper and discussed as a limitation."

A: Thank you for pointing this out. You are correct that the IL methods (including RPI) use slightly more samples compared to PPO-GAE as a result of pretraining the value function of the oracles. However, we emphasize that we use the same number of samples for RPI as we do for the IL+RL baselines (i.e., MAMBA, MAPS, Max-Aggregation, and LOKI) to ensure a fair comparison. We updated Section 7 (Baselines) to make this clear.

Q4: "The theory section borrows from MAMBA, which is only natural since these methods are closely related, but it would have been clearer to emphasize the points of departure from MAMBA more and perhaps move the parts that are more closely shared to the appendix?"

A: We appreciate your feedback! We agree with the reviewer that there are similarities with MAMBA and that the paper can be clearer regarding how RPI differs. Despite these similarities, we respectfully note that our theory is not directly borrowed from MAMBA. Different from MAMBA, our approach considers an oracle set that includes the learner policy, which we call extended oracle set. With this change, the (extended) oracle set that the algorithm work with is not static anymore (due to the improvement of learner's policy). This changes the theoretical analysis in a non-trivial manner. While Proposition 5.1 is marked as "adapted from MAMBA," our bound differs from MAMBA's for this reason. We updated the main paper to make this departure from MAMBA clear.

Thank you for your additional feedback! We are glad that our initial responses adequately answered most of your questions. We are also pleased that it has convinced you to change your score. We would like to take the opportunity to further address your remaining question.

Q4: "The theory section borrows from MAMBA, which is only natural since these methods are closely related, but it would have been clearer to emphasize the points of departure from MAMBA more and perhaps move the parts that are more closely shared to the appendix?"

I would still like to see improvements w.r.t. Q4 per your labeling. I believe this is the same issue as 6RE6's latest comment about the similarity of section 4 to MAMBA. As 6RE6 states, section 4 extends the definitions in MAMBA in a straightforward way. Including a clear discussion on how the definitions relate to corresponding ones MAMBA would be good.

A: We agree that section 4 has similarity to MAMBA, as both share the same problem setting. The main difference arises from lifting the static oracle set into our dynamic extended oracle set. However, to formally state the problem and establish our algorithm and analysis, we find it necessary to rigorously set up the notations and keep the work self-contained, which unavoidably overlaps with MAPS/MAMBA as a special case. In fact, many of the notations also inherit from MAMBA due to the shared problem setting, and we have moved most of the background discussion to Appendix B1 (“Additional algorithms for learning from multiple oracles”) for clarity. In addition, to make it clearer, we added a footnote in section 4 where we explicitly state, "When we exclude the learner’s policy from the extended oracle set, this reduces to the max-aggregation policy, which was used in MAMBA (Cheng et al., 2020)" to elucidate the connection with existing work (Please find details in the revised version.). Starting from Section 5, our approach clearly departs from MAMBA as it really focus on selecting between learner's policy and oracle policies, as well as the robust update in policy gradient.

We hope our responses have addressed your remaining questions. Again, we greatly appreciate your valuable time and input!

We appreciate your in-depth review of our work! Below please find our detailed responses to your questions below.

Q: "For reproducibility purposes,it would be great if the authors could include a table of algorithm hyperparameters used in the appendix..."

A: Thanks for the recommendation! In the revised version, we added a table that summarizes the hyper-parameter settings in Appendix D.5. Please check our updated version file for more details.

Q: "How would the following naive baseline perform? Initialize from the expert policy, and then finetune from there with RL. It seems like this baseline could perform quite well in the proposed experimental setup, given that PPO-GAE is one of the baselines and performs quite well."

A: We appreciate the suggestion! This baseline is indeed intriguing, and we have implemented and tested a similar one called MAPS-ALL (see Appendix E.1 that compares RPI against baselines with a data advantage). In MAPS-ALL, the learner policy has access to the initial pretrain transitions given by rolling-out the oracles, which is approximately twice the size accessible to the RPI algorithm. We initialize the learner policy using this extended offline data. We observe that such initialization enhances performance in the Pendulum environment (Figure 5 (c)). However, despite using significantly less data, RPI still outperforms MAPS-ALL in Walker-walk and Cartpole, and this difference is statistically significant.

In addition, we followed the approach of previous works, such as MAPS and MAMBA, in using the best-so-far plot. However, we observed that MAPS-ALL often does not perform as well as expected. The primary challenge arises from "catastrophic forget" or "performance drop", which, despite not being reflected in the plot, occurs when transitioning from offline training to online fine-tuning. This challenge is a common issue in offline-to-online reinforcement learning. The prevailing method to address catastrophic forgetting involves adopting offline pessimism to online pessimism, but this often leads to suboptimal policies. Another approach is addressing catastrophic forgetting through online reinforcement learning with offline data. In contrast to the offline-to-online approach, MAPS, our RPI, demonstrates greater robustness and stability.

We hope our response has addressed your concerns and bolstered the credibility of our work. If you have any further inquiries, please let us know. Thank you!

Thank you for the feedback on our work! Below please find our detailed responses to your questions.

Q1: "The policy improvement of perfect knowledge is similar to making ensembles of several imitated policies... analysis seems redundant."

A: We understand that our approach and ensembles of imitated policies may seem similar, but our active RL and IL strategy introduces novel technical challenges. Importantly, we do not assume "perfect knowledge" of the experts---unlike standard ensemble imitation, our approach requires state-wise differentiation to identify the most effective oracle in any given context (state). Moreover, a crucial aspect of our methodology is the active decision-making process, i.e., choosing between imitating the best policy or focusing on self-improvement. This aspect is particularly critical in scenarios where imitating a suboptimal policy can be detrimental, a challenge not addressed by existing ensemble-based approaches. Our learner policy is designed to determine which expert policy to follow based on the state, further differentiating our approach from traditional ensembles.

Q2: "The exploration in RPI is uniformly random sampling (line 3 of Alg.1). These seem trivial."

A: Please note that we are not claiming the uniform sampling of the roll-in horizon $t_e$ in Line 3 of Alg. 1 as novelty of this work. Rather, we focus on the robust policy selection and robust policy gradient algorithm, which ensures active exploration of RPI.

Selecting an oracle to imitate uniformly at random from the learner’s state distribution is a standard procedure in online imitation learning. As the reviewer correctly points out, this strategy, employed in several classic online IL algorithms like Dagger (Ross et al., 2011), Aggravate (Sun et al., 2017), THOR (Sun et al.,2018), MAMBA (Cheng et al.,2020), and MAPS (Liu et al.,2023), serves as a common practice for conducting imitation learning based on the learner's distribution, ensuring comparability with existing methodologies.

Q3: "The experimental setting is not clear and sufficient. How many demonstrates for imitation learning? How many online interactions for reinforcement learning? The x-axis of the curves is the training step, what does 100 training steps mean? And do the baselines use the same examples and online interactions?"

A: Thank you for raising these questions. We would like to note that most of these experimental details were presented in the appendix (D.3 setup D.4 implementation details and D.5 hyperparameter table) and in the source code---both included in the supplemental materials---for reproducibility of our results. In terms of your specific questions:

• We will explain the details of each training iteration as follows. For each training iteration of the RPI algorithm, the environment is queried for 4 episodes (roll-in + roll-out; this amounts to 4k steps for DMControl suite, 1.2k steps for MetaWorld tasks), followed by 2,048 steps (only rolling out the learner's policy). Thus, each training iteration involves 6,048 steps of environment interactions for DMControl suite (or 3,248 steps of interactions for MetaWorld).
• To elaborate on the learner's policy update within a single training iteration, we will give a little bit of implementation details: 1. We first prepare a set of transitions. This comes from the roll-in (see Alg. 1 line 4) and full roll-out (see Alg. 1 line 6) of the learner policy. 2. We then create a set of mini-batches (with size 128) by grouping the transitions. 3. We iteratively perform gradient updates on the mini-batches for 2 epochs (i.e., going over all of the mini-batches twice). This entire process makes up a single training step.
• All baselines were subjected to an equal number of environment interaction steps to maintain consistency, except for a minor difference in PPO-GAE (please see Appendix D.3 Setup).

We have updated the text in the main paper and appendix to make sure that the experimental details are clear.

Q4: " The selected tasks are easy. In the meta-world, as I know, button-press and window-close are not hard. What about bin-picking and door-open? In DMC, what about hopper or even humanoid?"

A: Please note that the tasks considered were variants of the standard setting, tailored towards the evaluation of imitation learning with multiple blackbox oracles:

• Button-press and window-close are originally dense-reward environments (see Appendix E.2 Meta-world experiments (dense reward)). However, we converted the environment to have a sparse reward, where a reward is given only when the window is closed or a button is pressed. Thus, most of the time, the learner doesn't receive any feedback. Compared to the original dense reward environment, this is significantly harder learning problem.

• We chose these tasks to facilitate comparisons with the most relevant work in online imitation learning as they are a superset of the standard baselines for this domain. Please refer to the following table, which compares tasks across methods, and kindly check our updated PDF for more details.

Q5: " Lack of baselines. Except for MAPS, all the baselines are from before 2020. I think there are more works than listed."

A: To the best of our knowledge, there are only a few existing approaches that focus on online imitation learining from multiple black-box oracles. We included the state-of-the-art MAPS [2023], along with all baselines used to evaluate MAPS, such as MAMBA [2020]. Additionally, we implemented several variants of the proposed algorithm to provide a comprehensive evaluation. To provide some further context, please refer to Appendix Table 2 for a detailed comparison to 17 other relevant methods, which explicitly points out the key differences in problem settings.

We believe we have included a sufficient set of relevant baselines for the problem setting that we consider, but would appreciate the reviewer's suggestion for any other relevant baselines.

Q6: "... why are the links in the paper invalid? (such as citation and equations)"

A: We appologize for the inconvenience. It was a result of generating the PDF of the main paper by removing the appendix pages from a version of the paper that included the main text and appendices as a single PDF. Please see a complete version of the paper (including the appendices) with working links in the supplementary materials section. We note that many of the questions that you raised regarding environment setup have been addressed there as well.

We hope these responses adequately address your concerns and clarify the contributions and methodologies of our work. We appreciate your constructive feedback and look forward to further discussions!

Thank you for the response. The experiment setup is more clear and sufficient. But I think the novelty is quite limited. I have raised my score to 5.

Thank you for your in-depth review of our work! We are especially grateful for your recognition of the beneficial impact that the publication of this work will have on the community. Below, please find our detailed responses to your questions.

Q1: "My first main concern is related to the contextualization of various parts of the paper to previous work. I think it could be made clearer which parts are inspired by previous work... I think it would be good to give credit to the general formulations in previous work where appropriate."

A: Thank you for the advice! We agree that the paper would benefit from a discussion that better places RPI in the context of existing work. We created the following table in an effort to clarify the relationship between RPI and what we believe are the most relevant existing methods. Additionally, we added a detailed paragraph to Appendix A.1 that discusses related work.

Q2: "My second main concern is the experimental setting seems very similar to that studied in Liu et al., since environments and baselines are relatively similar. However, it seems that the performance of the reported baselines in this paper differs significantly from the performance reported in the previous study of MAPS." "Did you use the original implementations for MAPS and MAMBA or did you reimplement the methods? Do you have an intuition why in the present comparisons these methods perform significantly worse than in the previous literature?"

A: Thanks for raising this concern! We utilized the original implementation for both MAPS and MAMBA baselines. However, in our main paper, we employed a different experimental setting (specifically, the amount of accessible transitions) to ensure a fairer evaluation compared to MAPS’ setting. Additionally, we utilized a different oracle set (see footnote on Page 8). To bolster the credibility of our work, we have addressed this potential concern in Appendix E.1, where we compare RPI to baselines with a data advantage, and named MAPS as MAPS-ALL (with additional data from the oracles’ pre-trained offline dataset for learner policy.). In this section, we demonstrate that MAPS-ALL exhibits overall performance similar to that of the original MAPS paper, with any differences caused by the difference in the oracle set. Notably, as a result, MAPS-ALL distinctly outperforms RPI only in the Pendulum environment. RPI’s performance remains comparable to MAPS-ALL in the Cheetah environment and significantly surpasses the MAPS-ALL baseline in the Walker-Walk and Cartpole environments, despite utilizing much less data (~50% less).

Thank you for the various clarifications, I appreciate it. I do think I have a better grasp of the manuscript now. I also read the other reviews as well as the responses to them.

I'm coming back to a few points here to clarify what I myself was referring to.

Q1: I was not concerned about the differences in manuscripts but rather the fact that section $4$ of the manuscript seems relatively similar in setup to the MAPS paper. In a sense, - and I'm exaggerating a little here - a large chunk of section $4$ could have been written as a summary of the MAPS paper it seems and one sentence at the end could have stated that now the learners policy is included and all functions are defined as such. The $\text{max}^+$ policy seems to be the same as the $\text{max}$ policy, $\text{max}^+$ aggregation seems to be an extension of $\text{max}$ aggregation and the $\text{max}^+$ following policy seems to be simply the definition of the $\text{max}$ following policy. In all cases, the learner's policy is added to achieve the "+".

Contextualization and Q6: To clarify my positive novelty point. I think the general idea of adding the learners policy to the policy set is sufficiently distinct from prior approaches in the domain. This does not necessarily mean that the idea of combining offline and online training is novel. It is quite prevalent in the offline RL literature as I will outline here.

I am familiar with the difference between online and offline learning. I apologize for not being more clear here. There is an abundance of literature in the offline RL community on how to leverage offline data to fine-tune and continually train a new policy by including data in replay buffers of off-policy methods [1], sampling techniques of offline and online data [2, 3, 4, 5] or simply continually fine-tuning online [6]. There is also theoretical results such as [7] on online and offline policy combination. Specifically, [5] works with the inclusion of the behavioral policy to a learning policy which seems very similar to the suggested approach. However, rather than building on a set of oracles, they use a single behavioral policy. The manuscript currently does not outline, how this work is related and how the settings are differentiated. To me it seems that they are very related which would limit both the novelty of the approach with respect to related work as well as retain the weakness of proper contextualization.

[1] Vecerik et al., Deep q-learning from demonstrations. Proceedings of the AAAI Conference on Artificial Intelligence, 2018. [2] Nair et al., Overcoming exploration in reinforcement learning with demonstrations. In Proceedings of the International conference on robotics and automation, 2018.
[3] Kalashnikov et al., Scalable deep reinforcement learning for vision-based robotic manipulation. In Proceedings of The 2nd Conference on Robot Learning, 2018.
[4] Hansen et al. MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrations. In Proceedings of the Eleventh International Conference on Learning Representations, 2023.
[5] Zhang et al, Policy expansion for bridging offline-to-online reinforcement learning. In Proceedings of the Eleventh International Conference on Learning Representations, 2023.
[6] Kostrikov et al. Offline reinforcement learning with implicit Q-learning. In Proceedings of International Conference on Learning Representations, 2022.
[7] Song et al. Hybrid RL: Using both offline and online data can make RL efficient. In Proceedings of the Eleventh International Conference on Learning Representations, 2023.

As such, as before I will retain optimism in the face of uncertainty and slightly favor acceptance but I will not increase my score.

• edit: formatting

Q: Are the settings similar?

A: From our perspective, our setting (the problem we aim to address) is quite different from theirs. Specifically, the approach to handling multiple suboptimal oracles and an offline dataset varies considerably. Even the treatment of multiple suboptimal oracles versus a single oracle is distinct; addressing the sample efficiency challenge to identify the state-wise optimal oracle is necessary when dealing with multiple black-box oracles, whereas a single oracle (behavioral policy) does not require tackling these challenges. To illustrate the distinction of RPI against these works, we added the referenced works [1,4~7] into Appendix A Table 2 (selective comparison against related works). Please find more details in revised version.

Q: Specifically, [5] works with the inclusion of the behavioral policy to a learning policy which seems very similar to the suggested approach. However, rather than building on a set of oracles, they use a single behavioral policy. The manuscript currently does not outline, how this work is related and how the settings are differentiated. To me it seems that they are very related which would limit both the novelty of the approach with respect to related work as well as retain the weakness of proper contextualization.

A: We agree that this paper is similar to ours in that they consider an expanded policy set that includes the learner policy as well as a policy that results from offline RL. The offline policy in the expanded set is frozen and only the learner policy will be updated with online training. One of the stark differences between our work and theirs is the strategy by which we collect online data. Although we perform MAMBA-style efficient exploration that consists of rolling-in the learner policy followed by an oracle roll-out, their work relies on a rather simple mixture of predicted actions (by each policy in the set) weighted by each Q value estimate. The other difference that you have already pointed out is that they have a single pre-trained policy (i.e., a single oracle) in addition to the learner policy, whereas we consider multiple complementary oracles.

We hope our responses have further addressed your two main concerns and clarified the remaining questions. Again, we are really appreciative of your valuable time and input!

[1] Vecerik et al., Deep q-learning from demonstrations. Proceedings of the AAAI Conference on Artificial Intelligence, 2018.

[2] Nair et al., Overcoming exploration in reinforcement learning with demonstrations. In Proceedings of the International conference on robotics and automation, 2018.

[3] Kalashnikov et al., Scalable deep reinforcement learning for vision-based robotic manipulation. In Proceedings of The 2nd Conference on Robot Learning, 2018.

[4] Hansen et al. MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrations. In Proceedings of the Eleventh International Conference on Learning Representations, 2023.

[5] Zhang et al, Policy expansion for bridging offline-to-online reinforcement learning. In Proceedings of the Eleventh International Conference on Learning Representations, 2023.

[6] Kostrikov et al. Offline reinforcement learning with implicit Q-learning. In Proceedings of International Conference on Learning Representations, 2022.

[7] Song et al. Hybrid RL: Using both offline and online data can make RL efficient. In Proceedings of the Eleventh International Conference on Learning Representations, 2023.