Robot-Gated Interactive Imitation Learning with Adaptive Intervention Mechanism

Thank you for your effort to thoroughly review our paper and for your feedback. In response to your feedback, we have included qualitative evaluations and ablation studies that have strengthened the study.

Experimental Designs Or Analyses:

1.Qualitative evaluations are missing.

We include qualitative evaluations in Fig 1 of https://limewire.com/d/teXVP#Okg7PsYIne to show that when the car approaches the road boundaries, AIM successfully requests human intervention, while other uncertainty-based methods output a low uncertainty estimate and fail to signal the need for help.

Other Strengths And Weaknesses:

1.Line 053: why "fixed intervention criterion" would cause "fixed request rate"? The logic between such two sentences is not clear.

In Line 053, fixed intervention criterion refers to requesting human help when the uncertainty estimate exceeds a constant $\epsilon$, as in EnsembleDAgger, which uses the variance of actions in the policy ensemble as the uncertainty estimate.

According to Fig 4 of https://limewire.com/d/teXVP#Okg7PsYIne, EnsembleDAgger requests expert help at a fixed intervention rate after 1K steps, implying that some states’ uncertainty estimates always exceed $\epsilon$. In contrast, ThriftyDAgger adapts $\epsilon$ and avoids a fixed request rate.

Therefore, we revise Line 053 as follows:

"The uncertainty-based methods including (Menda et al., 2019; Kelly et al., 2019) request human help when the uncertainty estimate is larger than a fixed threshold. Without adjusting the threshold adaptively, the agent keeps requesting expert help even when it has successfully imitated the human expert."

2.Line 071: What do you mean by "dynamically adjusts to...agent policy"? That's quite confusing.

We revise Line 071 for clarity:

"Our learned intervention criterion adaptively shrinks the intervention rate as the agent becomes performing. Our learned intervention criterion can request human help at an appropriate rate based on the agent's performance during training."

We further explain Line 071 as follows. According to Line 184-188, the intervention frequency of AIM gradually drops down as the agent’s performance improves during training. The reason is that AIM’s Q value depends on the proportion of states that the agent’s action aligns with the expert’s action. During training, the average of AIM’s Q value decreases towards -1, leading to the shrink of the intervention rate. The shrink of the intervention rate is also shown in Fig 4 of https://limewire.com/d/teXVP#Okg7PsYIne . We can observe that our method’s intervention rate matches that of human-gated PVP even though it’s a robot-gated method, implying that our intervention mechanism adapts to the agent’s performance and resembles the human-gated intervention rule. In addition, our intervention rate is lower than other robot-gated baselines.

3.Line 295: "we report the success rate" or "we employ the success rate as the evaluation metric" would be better.

Thanks for your suggestion! We revise Line 295 as follows: "In MiniGrid, we employ the success rate as the agent’s evaluation metric."

Questions For Authors:

1. Could the authors demonstrate the value of their method in more complex tasks such as those involving physical environments, e.g. Mujoco tasks like Ant in Gymnasium.

We appreciate the reviewer’s interest in evaluating our method on physical tasks such as those in MuJoCo. However, tuning and training a near-optimal expert and all baseline methods demand time and resources. Moreover, human demonstrations for MuJoCo tasks like Ant are infeasible, as a human cannot reasonably demonstrate the complex multi-legged locomotion task. Our MetaDrive environment with a high-dimensional observation space generates diverse driving scenarios such as fixed or movable traffic vehicles, traffic cones, and warning triangles, which already offers a compelling and challenging benchmark. We plan to include experiments on physical tasks in future work.

2. Why the proposed Q function could help the agent receive sufficient human guidance at "safety-critical states"? Why the Q-function can emphasize the "safety-critical states"?

The key is that the proposed Q-function can classify states where the agent’s actions align with human actions and those where there is a significant action discrepancy.

By explicitly using an action difference function $f$ to label these states, the Q-function learns to emphasize those states where human guidance is most needed. Additionally, incorporating a TD loss propagates these signals to nearby states.

We also include an ablation study in Fig 3 of https://limewire.com/d/teXVP#Okg7PsYIne , showing that dropping the TD loss or replacing the Q-labeling by reward-labeling will damage the performance of AIM. This implies the effectiveness of our AIM Q-function design.

Thank you for reading our paper in detail and providing valuable suggestions. We summarize and respond to each question as follows:

Claims And Evidence:

However, the term "sufficient" in the third claim (Line 98) should be softened unless providing a proof.

We revise Line 98: “The expert demonstrations requested by AIM contain corrective actions in safety-critical states, so that they can assist a novice agent to imitate the expert’s policy.”

Relation To Broader Scientific Literature:

The awareness of human-centered literature and decades of work on this topic seems to be lacking. The paper does not discuss Confidence-based Autonomy (CBA).

Thanks for sharing relevant works! We will add these references in the revised version.

Other Strengths And Weaknesses:

The limitations section is quite short and does not fully address the many weaknesses of not doing a real user study.

We will add this to the limitation section: "This paper does not include real-human experiments or user studies, and human demonstrations may be imperfect or faulty."

Figures 4-5 show unconvincing results regarding the superiority of AIM vs. PVP.

Figures 4 and 5 show that AIM and PVP perform similarly in imitating the expert during safety-critical states and collect high-quality expert demonstrations. In short, AIM’s intervention rule behaves similarly with the human-gated PVP’s rule, even though AIM is a robot-gated interactive IL method.

In Figure 4 and 5, we do not aim to show that AIM’s intervention rule is better than PVP. According to Line 358-362 in Page 7, since PVP is a human-gated IL algorithm, the key advantage of AIM over PVP is that it requires fewer human cognitive efforts and expert-involved steps.

Questions for Authors:

Under what conditions of f in Equation 2 is the agent able to recover the expert policy?

We clarify in Line 164 (Page 3) that the expert’s intervention method follows Eq. 2: $I^{exp}(s, a_r, a_h) = f(a_r, a_h) = \mathbb{I}[\|a_r - a_h\|^2 > \epsilon]$ ($a_r$ is the robot action, $a_h$ is the human action). With sufficient expert demonstrations, we can bound the difference of the value functions of the expert policy and the student’s final learned policy by $\epsilon$ and the horizon $H$.

Must the f in Equation 2 conform to the equation embedded in the text from Menda et al., 2019 and Hoque et al., 2021a;b?

The function f does not need to conform to Menda et al. (2019) and Hoque et al. (2021). The general form of the intervention is based on the probability of the expert taking the robot’s action, which reduces to Eq. 2 if the expert follows a Gaussian policy. The proof is in Theorem 3.3 of “Guarded Policy Optimization with Imperfect Online Demonstrations” (Z Xue et al., 2023).

It is confusing to say that $Q_{\theta}^I(s, a_r)$ is equal to -1 or +1 by assignment (Lines 209-218). That implies that \gamma = 0 and Equation 4 is meaningless -- there is no TD-Error.

In Line 209-218, the proxy value assignment is a learning objective but not a hard constraint to the proxy value function. The AIM loss helps select human preferable actions at those states $s$ with human interventions, and TD loss propagates human preference to the states $s$ without human interventions.

Should it not be the reward function definition for -1 and +1 -- not the Q-function?

We include an ablation study in Fig. 3 of https://limewire.com/d/teXVP#Okg7PsYIne . The figure shows that dropping the TD loss or replacing the Q-labeling by reward-labeling will damage the performance of AIM.

Replacing the Q-labeling by reward-labeling fails in our setting because negative rewards cannot be matched with transitions from unsafe agent actions. In human-involved transitions $(s, a_h, s’)$, we can assign +1 since $s’ \sim P(s, a_h)$ is generated by the human action. However, for dangerous agent actions $a_r$, querying the environment to obtain the resulting state $s’’ \sim P(s, a_r)$ is not feasible.

It is unclear how this paper is presenting an approach to "shared autonomy." See this work by Reddy et al. (2018).

Thanks for pointing out that our definition of "shared autonomy" differs from Reddy et al. (2018), and we revise the terminology to "interactive imitation learning." While Reddy et al. apply human-AI shared control during both training and testing, we only use it in the training phase and evaluate the agent's performance without human involvement in the test phase.

Why are none of the baselines able to match the Neural Expert?

In Table 1, we require all the baselines to use no more than 2K expert-involved steps. These baselines can match the neural expert with 3.5K expert-involved steps. See Fig 2 of https://limewire.com/d/teXVP#Okg7PsYIne .

Why is the neural expert not at near 100% performance?

The neural expert is trained using Lagrangian PPO with 20M environment steps. MetaDrive safety environments can present challenging scenarios where a well-trained expert may fail.

Thank you for taking the time to carefully read through and understand our paper, and provide constructive feedback. We summarize and respond to each question as follows:

Other Strengths And Weaknesses:

1.Why L2 distance instead of other distance metrics? Is $a_h$ unique and deterministic in task like MetaDrive?

We use L2 distance because it is simple to implement and effectively identifies states where the agent's actions deviate significantly from expert behavior. Fig 3 in https://limewire.com/d/teXVP#Okg7PsYIne shows that using the L1 distance metric does not affect the performance. In MetaDrive, the expert action $a_h$ is stochastic, as the expert policy is a stochastic policy trained with Lagrangian PPO.

2.Elaborate more on the choice of fixing the threshold epsilon in Eq. 9.

The threshold $\epsilon$ controls the length of expert demonstrations after the agent requests help. We choose $\epsilon$ based on observations from the human replay buffer in warm-up steps (Eq. 8, Line 240). A small $\epsilon$ leads to overuse of expert help, while a large $\epsilon$ results in insufficient corrections, slowing down training.

Does an adaptive switch-to-agent threshold work and perform better?

An adaptive switch-to-agent threshold does not significantly improve the number of expert-involved steps. In our current setup, the agent requires only 5–10 steps of expert help before returning to self-exploration, which is nearly the minimum needed for safety.

3.Give several examples under safety-critical states, where other baselines robot-gated IL baselines fail to request for human help but AIM did.

In Fig. 1 of https://limewire.com/d/teXVP#Okg7PsYIne , when the car approaches the road boundaries, AIM successfully requests human intervention, while other uncertainty-based methods output a low uncertainty estimate and fail to signal the need for help.

4.Why did authors not compare AIM with Ensemble-Dagger and Thrifty-Dagger for mini-grid?

In Page 6 Line 310-314, we mentioned that the two methods rely on the action variance for uncertainty estimation, which doesn’t work well with discrete action spaces like MiniGrid. 'Action variance' refers to the variance of the output actions across the ensemble of policy networks.

To apply the two baselines to MiniGrid, we need to replace their variance-based uncertainty estimations by the entropy-based estimation, which is the entropy of the action distributions derived from the soft Q-value. Table 1 of https://limewire.com/d/teXVP#Okg7PsYIne shows that AIM reduces the expert-involved steps needed compared with the two baselines.

5.In page 7 line 374, the authors are claiming AIM’s mechanism saves training time and the total environment data usage according to figure 4, but results show that AIM is not the one with least total data usage.

Thanks for pointing out the misleading claim. In Figure 4, we show that AIM requires fewer environment samples than other robot-gated baselines (Thrifty-DAgger and Ensemble-DAgger) to approach expert actions in safety-critical states. In Table 1 and Table 2, we highlight that AIM reduces expert-involved steps and cognitive effort, though it may require more training time. Thus, we revise line 374 to:

"Compared with other robot-gated IIL baselines, AIM requires fewer environment data usage to imitate expert actions in safety-critical states."

6.How does the intervention rate change during the training/testing stage?

We visualize the overall intervention rate in the training stage in Fig. 4 of https://limewire.com/d/teXVP#Okg7PsYIne . Our method AIM’s intervention rate matches that of human-gated PVP and is lower than other robot-gated baselines. According to Line 281-284, in the test stage, we evaluate the agent’s performance without expert involvement, so there’s no intervention rate during testing.

7.Does introduce warm up stage to Ensemble/Thrifty-DAgger lead to better performance?

In our experiment (Table 1 and Table 2), we already introduced a warm up stage to all the baselines including Ensemble-DAgger, Thrifty-DAgger, and PVP for fairness. (i.e., demonstrating initial two trajectories as we do for AIM).

Other Comments Or Suggestions:

1.What’s $I^{exp}$, $\delta$ in the formula (1)?

$I^{exp}$ is the human-gated intervention criterion, where the expert decides whether to take over control at state s using human control signal $a_h$ when the agent outputs action $a_r$. The $\delta$ function in Eq. 1 is the Dirac delta distribution.

2.What’s the unit of the y axis for the plots in figure 2? How is uncertainty estimated?

In Figure 2, the y-axis represents the uncertainty estimation: $Var(a_n) - \varepsilon$, where $Var(a_n)$ is the variance of agent actions and $\varepsilon$ is the switch-to-human threshold in Ensemble-DAgger. Human help is requested when $Var(a_n) > \varepsilon$. We plot $\max(0, Var(a_n) - \varepsilon)$ to visualize the timesteps when human help is requested.