Dear Reviewer 48gS,

We would like to express our gratitude for your insightful feedback and acknowledgment of our work. We have carefully considered your questions and provided clarifications below, which will be incorporated into the paper to enhance clarity. We believe that these modifications and clarifications address the weakness you point out about confusion in the paper. Please let us know if all issues have been addressed, or if any further issues remain, as we would be happy to fix these!

If the expert intervenes for 5 steps, would all 5 transitions be labeled -1? Wouldn’t this result in actions taken by the expert being labeled -1 too?

Expert actions won’t be labeled as -1 when an expert intervenes for 5 steps, only the state and action pair immediately before the intervention begins is labeled -1, this was originally in algorihtm 2 box. The intuition here is that the -1 reward is assigned to the transition (state-action tuple) that most immediately resulted in the intervention. Our theoretical analysis in Section 6 shows that this leads to a good policy (that approximately maximizes the true reward) in theory, and our empirical evaluation shows this in practice. Expert actions are labeled as 0, similar to non-intervention transitions.

Could the else statement in line 7 of Algorithm 2 set the reward to the RL reward signal if it were available?

Yes, that could be a straightforward extension as you suggested. Though in our paper we focused on learning entirely from intervention feedback without knowledge of the ground truth reward. It’s reasonable to assume the sample efficiency of RLIF would be further improved if it could also access the true task reward.

I’m very confused by this sentence: “the performance of DAgger-like algorithms will be subject to the suboptimality of the experts, while our method can reach good performance even with suboptimal experts by learning from the expert’s decision of when to intervene.“ If no task reward is provided, how could the policy end up outperforming the expert?

A crucial assumption for RLIF to work is that the intervention signal itself should contain task-relevant information. For example, in the driving example in the introduction, it is easier for a safety driver to intervene when the car does something bad than it is for them to carry out a perfectly optimal sequence of intervention actions – the choice of when to intervene itself often carries a lot of information. In Section 5.1, we provide one example of an expert intervention model based on Q-values that would carry considerable information in this choice. we also ablate the performance of RLIF w.r.t. Q functions containing different level of task information; as you hypothesized, the least task-relevant information Q function has, the worse the performance, as in Appendix A.6. Of course, there is no free lunch, and our method is sensitive to the particular intervention strategy the human uses, as evidenced by the fact that the naive random intervention baseline in our experiments performs much worse. However, this also indicates that our method is effective at deriving a learning signal from the intervention strategy, and the real-world experiments illustrate that real humans do provide interventions that are informative.

How do you end up with a 110% expert if the expert level is w.r.t an expert trained to near optimality?

The statement about a 110% expert level refers to the expert's evaluation normalized score. In D4RL locomotion environments, we additionally learn expert policies from a curated dataset, so it can be possible that learned expert achieve normalized scores above 100. And we label them as greater than 100% in expert levels. This was also explained Section 5.2 - results and discussions. We’ll add a more detailed explanation in the final paper.

Dear reviewer guRK,

Thank you for your thoughtful feedback, we would like to address your concerns below.

DAgger in RL setting

We compare to DAgger under the same assumptions as our method -- namely, for both DAgger and RLIF, no ground truth reward function is provided, and the algorithm performs rollouts from the current policy with interventions provided by an expert. Could you elaborate on what kind of comparison you would like to see here? We are not aware of any prior work that uses DAgger in RL, but it's entirely possible we missed something (or misunderstood your comment), so if you can provide a reference to a relevant method we should have compared with, we can attempt to include such an evaluation.

RLIF reduced to a heuristic to define a reward function

Our method does not require the expert to define a reward function, but rather to provide interventions. While the way this is used to induce a reward function in RL, that is not a heuristic: we show that this leads to a reward signal that allows our method to recover a policy that is close to the optimal policy (under some assumptions).

compare with DAgger in RL

We are not sure what "using DAgger in RL" means -- could you provide a reference of a method that does this? The advantage of our method over DAgger is illustrated in the empirical comparisons. For example, Table 1 shows our method is 2-3x better than DAgger.

I apologize for the lack of clarity when I wrote my review. I am concerned with the lack of comparison with the common approach of warm starting the policy training with Dagger and proceeding with PPO. The heuristic determination of when to execute the expert is equivalent to defining an arbitrary reward function, so in my opinion that would be a fair comparison. I would like to know the author's opinion on why this is not the case.

Edit: Regardless of this discussion, which I do consider necessary for this paper, I think my rating is unnecessarily hard, so I'll correct it to a fairer score while waiting for a chance for the authors to respond to my question, which I hope is clearer now.

Dear reviewer,

We greatly appreciate your time and dedication to providing us with your valuable feedback. We hope we have addressed the concerns, but if there is anything else that needs clarification or further discussion, please do not hesitate to let us know.

Best, Authors

First of all, thank you so much again for your quick response.

Even in the case where the intervention is not heuristic, the reference value function has been learned from data, so a fair comparison might still include using e.g. IRL to learn a reward function and apply e.g. some on-policy algorithm like PPO after a Dagger warm start. I wouldn't want to propose such a costly experiment for the rebuttal, and, at the end of the day, the actual decision of when to call the expert in the proposed method is still a heuristic on top of the learned reference value function, so a heuristic could in principle also be used to define a competent reward function as a comparison, which should easily beat at least the random intervention baseline and would show how far a reasonable but not optimal heuristic reward function is from the one defined with the proposal.

In any case, as I mentioned above, since this seems to be of concern only to me, I won't argue against the paper based on that. Again, thank you for your responses.

I wish there had been a chance to further discuss what I consider a reasonable question about the usability of the proposal. It's not clear to me that designing a heuristic to call an expert is essentially easier than defining a reward function, but the limited discussion period (and my late receipt of the authors' answer) made it impossible for me to clarify this point.

In any case, I thank the authors for considering to include such an experiment in a potential final version and, since my concern doesn't seem to be relevant in the other reviews, I won't hold a strong opinion about the paper, which otherwise is well written and even shows real-world deployment of learned policies.

Dear Reviewer RarS,

Thank you for your thoughtful feedback, we would like to address your concerns below.

Regarding baselines:

our aim is to develop a reinforcement learning method that can operate under the same assumptions as interactive imitation learning (e.g., DAgger), hence we compare to DAgger, HG-DAgger, and BC. While IRL methods could in principle be adapted to this problem setting also, this would be an awkward fit, as to our knowledge IRL methods do not use interventions. However, we did conduct new experiments using IQL as an offline RL baseline across all simulation benchmarks, and the results are detailed in Table 1.

Regarding your question about intervention frequencies,

our real-world robot experiments with vision-based peg insertion and deformable object manipulation (in Section 5) already demonstrated the effectiveness of our method under both sparse and dense interventions, i.e., the user intervenes only a few times, but each intervention length is long vs an user intervenes multiple times for a short time during a rollout. In practical terms, the choice between sparse and dense interventions is problem-dependent, varying according to the specific needs of each scenario. We have also included a plot of the average intervention rate over environment steps in Figure 3, illustrating that at the onset of training, the intervention rate tends to be higher, gradually decreasing as the policy improves towards the end of the training process.

RLIF vs RLHF:

Thank you for your suggestion! RLHF indeed can be relevant to our work, we’ll add some related works in the final version.

Your engagement with our work is highly valued, and we welcome any further questions or insights you may have.

Table 1: IQL Baseline

Dear reviewer,

We greatly appreciate your time and dedication to providing us with your valuable feedback. We hope we have addressed the concerns, but if there is anything else that needs clarification or further discussion, please do not hesitate to let us know.

Best, Authors

Dear Reviewer 9AYW,

Thanks for your insightful feedback on our work, we would like to address your concerns below.

To answer your question regarding the theoretical results:

how the level of sub-optimality results in what sample complexity of the offline RL agent. The reviewer still enjoys reading the analysis framework proposed here.

Corollary 6.7 provides a sample complexity bound of learning a policy using the intervention reward $\tilde{r}$. Since the intervention strategy is generated by the expert $\pi^{\exp}$, which eventually affects the suboptimality gap in a linear relationship w.r.t. $\Delta_{ref}$, the visitation distribution gap between the reference policy $\pi^{ref}$ and the optimal policy $\pi^{\star}$.

Concerning your question about offline RL baselines

First, RLIF in principle can be integrated with any off-policy RL algorithm. We chose RLPD because of its good sample efficiency. That said, per your suggestion, we conducted additional experiments incorporating IQL as an offline RL baseline across all simulation benchmarks. Please find the detailed results in Table 1. It's worth noting that, in all our experiments, true rewards were not provided, as our algorithm is designed to learn an optimal policy with only intervention feedback.

Upon examination, the results from the IQL experiments indicate a performance that lags behind the outcomes achieved using the online algorithm RLPD. This observation aligns with our expectations because it’s slow for IQL to explore new actions.

Thank you for your continued engagement with our work, and we remain open to any further questions or suggestions you may have.

Table 1: IQL Baseline

Dear reviewer,

We greatly appreciate your time and dedication to providing us with your valuable feedback. We hope we have addressed the concerns, but if there is anything else that needs clarification or further discussion, please do not hesitate to let us know.

Best, Authors