Conformalized Interactive Imitation Learning: Handling Expert Shift and Intermittent Feedback

B. Regarding additional baselines

We appreciate the suggestion of including additional interactive imitation learning baselines! In our original manuscript, we focused on comparing ConformalDAgger with EnsembleDAgger because the ensemble output variance provides a measurable interval for coverage, enabling direct comparison. As suggested by the reviewer, we included two relevant DAgger approaches: LazyDAgger[2] and SafeDAgger [3]. SafeDAgger relies on switching to human control when the predicted action discrepancy exceeds a threshold, while LazyDAgger initiates human intervention based on a safety classifier and returns to autonomous control only when the discrepancy falls below an additional context-switching-threshold [2]. Similar to Ensembles, Monte-Carlo Dropout approaches (e.g. UAIL [1]) can also be used to estimate epistemic uncertainty in deep networks; we chose EnsembleDAgger because of prior works empirically observed by Lakshminarayanan et al. [2017], Lee et al. [2015] that training ensemble networks with random initialization works well in practice for capturing this epistemic uncertainty. We refrained from adding ThriftyDAgger because the algorithm objective is mitigating human feedback to be below some budget. Conformal, Ensemble, Lazy, and Safe do not budget, or cap, the number of human interventions received, but rather focus on handling feedback through any human intervention necessary.

We have rerun experiments including these additional baselines and under broader environment shift conditions for the 4D Simulated Reaching Task. The results are discussed in the section below and figures are in the attached PDF (Figure 5 in Section 5.2 in revised manuscript).
 

Updated Results on Simulated 4DoF Reaching Task with Human Gated Probability (=0.2 and =0.5):

When the expert shifts and drifts, ConformalDAgger asks for more help immediately compared to EnsembleDAgger and SafeDAgger, enabling the algorithm to more quickly align to the expert, with fewer human interventions than LazyDAgger. LazyDAgger does successful align to the shifted strategy, but requests many more interventions overall.

Under environment shift, EnsembleDAgger is well-calibrated, boosting the number of interventions requested. This is intuitive as ensembles capture the variance in the training distribution and when the environment shifts, changing the training distribution, the algorithm flags high uncertainty. ConformalDAgger performs well on environment shift context, though asks fewer questions than EnsembleDAgger, converging to the shifted policy nearly as quickly as EnsembleDAgger.

When the human-gated probability is high (0.9), as expected, all approaches receive high quantities of human feedback, resulting in rapid converge to expert and environment shift.

[1] Y. Cui, D. Isele, S. Niekum and K. Fujimura, "Uncertainty-Aware Data Aggregation for Deep Imitation Learning," 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada, 2019, pp. 761-767, doi: 10.1109/ICRA.2019.8794025.

[2] Hoque, Ryan, et al. "Lazydagger: Reducing context switching in interactive imitation learning." 2021 IEEE 17th international conference on automation science and engineering (case). IEEE, 2021.

[3] Zhang, Jiakai, and Kyunghyun Cho. "Query-efficient imitation learning for end-to-end autonomous driving." arXiv preprint arXiv:1605.06450 (2016).

On the inclusion of experiments on simulated IL benchmarks (e.g. Robosuite)

We conducted additional experiments on Robosuite’s nut-assembly task, a benchmark for LazyDAgger and ThriftyDAgger. For consistency, we used the LazyDAgger thresholds from [1] and set the SafeDAgger threshold to match the safety value used in LazyDAgger. The nut-assembly task involved a hardcoded expert policy that sequentially rotates to pick up the nut, lifts it, moves it to the peg, and lowers it to place. We trained the base policy with 30 demonstration episodes.

We evaluated this task under two conditions: a No-Shift scenario, where the task remains unchanged during deployment, and an Environment-Shift scenario, where the peg location is altered during interactive deployment (highlighted in red in Figure \ref{fig:robomimic}).

A simulated human provided interventions by teleoperating the robot. We used 30 publicly available offline demonstrations (l{https://github.com/ryanhoque/thriftydagger} 2,687 state-action pairs) from a human supervisor to initialize the robot policy across all algorithms.

We found that ConformalDAgger identified the initial policy's nonperformance in both contexts, prompting more queries and achieving a marginally higher autonomous rollout success rate compared to other baselines. See Figure 13 in Appendix Section G for updated results and figure.

[1] Hoque, Ryan, et al. "Thriftydagger: Budget-aware novelty and risk gating for interactive imitation learning." (2021).

We thank the reviewer for the thoughtful and constructive feedback. With respect to weaknesses and questions, we provide our responses as follows. We have uploaded a revised manuscript with additional experiments, analysis, and baselines; we discuss the results below and refer the reader to the updated figures in the manuscript. We use the color blue to highlight updated text and figures in the manuscript.
 

A. Regarding expert shift motivation

We would first like to clarify – algorithmically, our IQT approach is agnostic to the underlying cause of distribution shift between the expert and the policy. All that it requires is accessing the expert’s labels intermittently to construct calibrated intervals on the base policy’s predictions. The reason the base policy is inconsistent with the expert labels can come from a variety of underlying causal factors. In our initial submission, we focused on expert shift – wherein the expert’s latent strategy changes over times – as one concrete cause of why the base policy disagrees with the expert.

More broadly, policy mispredictions during deployment may result from a multitude of reasons, including suboptimal or multimodal demonstrations, covariate shift (e.g., encountering new observations of the environment during policy execution), or expert shift. To show the generality of our IQT algorithm, we added a Environment Shift context, where the state distribution during deployment differs from training. We instantiated Environment Shift in both the toy 4D reaching task and the newly-added Robosuite benchmark Nut-Assembly task (Robosuite benchmark results can be found in Appendix Section G in revised manuscript).

• The expert is only selectively annotating states where the learner requests feedback. This is consistent with specifically online interactive imitation learning methods where the expert is in-the-loop during deployment rollouts.

• We have fixed the Section references.

• We applied IQT only to the end effector position because the Euclidean distance metric offers us an intuitive nonconformity score to use for IQT. For quaternion rotation, we require designing a score that operates in angular space. In that case, the representation of nonconformity would be over the union of position and rotation, so when you conformalize, your nonconformity score is operating on two different distance metrics. In practice, the suitable way to approach these different distance metrics is to run multiple IQT processes: one on positional uncertainty, and one on rotational. Due to this, we chose to calibrate only the xyz, the portions of the state that share the same representation to remove the confound of multiple IQT processes running. We hope to examine in future work multiple IQT processes handling these different nonconformity scores.

• Then, they should be conformalized independently. To make our hardware experiments as straightforward/simple

• We have updated the EnsembleDAgger hardware implementation to include the safety classifier. Our initial motivation for omitting the safety classifier on the hardware experiments was to isolate only the uncertainty quantification method, and evaluate how uncertainty-gated queries affect both approaches. In our rebuttal, we added in the safety classifier in the EnsembleDAgger implementation on hardware. See response to Reviewer YA6C for further details.

We added the hyperparameter sensitivity analysis in Section G of the updated Appendix.

We have added to the main text of the manuscript the hardware training details of EnsembleDAgger, including how action variance is computed over the position and rotation components of the policy’s prediction. The ensemble’s uncertainty threshold is 0.005, at approximately the mean of the distribution of ensembles variances (over position and orientation predicted by the policy) over the training data.

B. On Needing Sufficiently Frequent Human Feedback

In practice, it is true that if the feedback rate ($p_t$) is not high, the finite horizon (T) regret guarantee can be quite loose, as reflected in Proposition 1, which relies on the value of $p_t$. Intuitively, this makes sense, as if the model does not receive enough access to the ground truth labels, it will be challenging to ensure coverage, as the intervals must be adapted based on whether the model is correctly predicting the ground truth values.

In ConformalDAgger, the robot policy relies on having some, intermittent access to the ground truth values. In our simulated and hardware experiments, we control the level of human-gated feedback to ensure that we received a minimum of human feedback at a likelihood of 0.2. If the robot almost never receives expert feedback, the intervals may not be updated frequently enough, and may remain uncalibrated. In that context, leveraging ensemble disagreement and other sources of uncertainty in the policy can be combined with IQT to provide multiple avenues for uncertainty-prompted feedback. IQT can be combined with these other approaches to offer online, adaptive uncertainty quantification, in addition to the other mechanisms used in for EnsembleDAgger.

We thank the reviewer for the thoughtful and constructive feedback. We are encouraged by the reviewer’s positive feedback regarding the utility of conformal prediction-based uncertainty. With respect to weaknesses and questions, we provide our responses as follows. We have uploaded a revised manuscript with additional experiments, analysis, and baselines; we discuss the results below and refer the reader to the updated figures in the manuscript. We use the color blue to highlight updated text and figures in the manuscript.

A. Hyperparameter Selection

The hyperparameter selection is indeed an important hyperparameter that affects the algorithm performance. Our approach and all baselines require thresholds for hyperparameter selection

• For EnsembleDAgger, practitioners need to select an uncertainty threshold over the variance in ensemble predictions to flag the need for intervention, as well as a safety threshold which informs the classes (safe vs unsafe) to train the safety classifier.
• For ConformalDAgger, we need to select only an output level uncertainty threshold.

While there is no established rigorous method for selecting the threshold values for each algorithm, we included a new sensitivity analysis in the Expert Shift (Human-gated probability = 0.2) context. Due to the expensive training and demonstration collection time of the hardware experiments, we focused our hyperparameter sensitivity analysis in the 4D simulated reaching task.

Uncertainty threshold ablation for Conformal vs. Ensemble:

We first varied the uncertainty threshold ($\tau$) for the two methods that maintain an uncertainty estimate: ConformalDAgger and EnsembleDAgger. We varied the uncertainty threshold across three discrete values: [0.03, 0.06, 0.09]. We note that since EnsembleDAgger relies on a safety classifier, we held the safety threshold constant throughout all uncertainty threshold ablations at 0.03. The results are averaged over 3 seeds. On one hand, we find that when $\tau$ is low (0.03), Conformal and Ensemble both identify the shift and ask for help accordingly; however, for both algorithms this comes at the cost of a great deal of human feedback requested throughout every deployment episode. Even in the first episode in which there has been no expert shift, a low-threshold at 0.03 requires conformal and ensemble to ask questions 70-80% of the time. On the other hand, when the threshold is set too high, at $\tau = 0.09$, Ensemble is unable to identify the shift and continuously mispredict. Comparatively, we find that ConformalDAgger’s performance is less sensitive to this threshold choice, and is able to effectively adapt to the expert’s shifted policy.
 

Summary:

Comparatively, we find that ConformalDAgger’s performance is less sensitive to this threshold choice, and is able to effectively adapt to the expert’s shifted policy. This is much in line with the reviewer’s intuition that with rigorous online conformal prediction, the ConformalDAgger parameters are easier to tune, and more flexible in the choice of uncertainty threshold. We also compared hyperparameter sensitivity for our additional baselines (for these, see response to Reviewer YA6C).

Please refer to Appendix Section on Hyperparameter Sensitivity Analysis (Appendix Section B) for these figures and results.

Thank you for your feedback and revised score.

We have included the discussion in the Appendix Section B, and have uploaded the updated Appendix with the discussion included, with changes highlighted in blue.

We added a line in Algorithm 1, which explicitly writes the robot query triggering condition (in newly uploaded manuscript main text). The additional Line 7 in the Algorithm now reads "Compute robot query likelihood (e.g. $P(\text{obs}^r_t\mid o_t;\tau) = 1$ if $u(x_t; \pi^r) > \tau$, else 0.)" to indicate that the queries are triggered when the uncertainty is greater than $\tau$, and this threshold may for example, be a hard threshold, or soft (the soft threshold defined in the Leveraging Uncertainty: Asking for Help paragraph).

E. Regarding Expert Realizability

We agree with the reviewer that an outstanding challenge in the theory of DAgger is guarunteeing learning a high-quality policy from the expert when the expert is unrealizable. We assume, much like [1], that our expert is realizable. Whether or not this is true in practice with human experts is an open area of research. Nevertheless, similar to HG-DAgger[3] and EnsembleDAgger[3], our approach offers a practical approach to contend with these real-world challenges. We included this discussion in the manuscript.

[1] Spencer, Jonathan, et al. "Feedback in imitation learning: The three regimes of covariate shift." arXiv preprint arXiv:2102.02872 (2021).

[2] Menda, Kunal, Katherine Driggs-Campbell, and Mykel J. Kochenderfer. "Ensembledagger: A bayesian approach to safe imitation learning." 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2019.

[3] Kelly, Michael, et al. "Hg-dagger: Interactive imitation learning with human experts." 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.

F. Regarding Algorithm Runtimes

Quantifying uncertainty via IQT during policy rollout on an Apple M3 Max chip takes on average 0.0013 ms per timestep during interactive rollouts of the simulated 4DoF reaching task experiments.

Response to Minor Comments and Questions:

We included further clarification about the significance of Proposition 1 in Section 3. As included in the manuscript, we describe that Prop 1 guarantees IQT gives the $1-\alpha$ long-term empirical coverage frequency, even with intermittent ground truth access, regardless of the underlying data generation process.

We clarified line 7 of Algorithm 1 in the manuscript. The robot elects to ask questions with probability $P(obs^r_t|o_t)$. Alternatively, we clarify that the human may also intervene independently. The human intervenes independently with probability $P(obs^h_t|o_t)$. These samplings are drawn from two different probability distributions: the likelihood of asking for help, and the likelihood of the human independently giving feedback irrespective of the robot’s query for help.

The Appendix numbers have been fixed.

D. Ensemble Hyperparameter Selection

The hyperparameter selection is indeed an important hyperparameter that affects the algorithm performance. ConformalDAgger, EnsembleDAgger, LazyDAgger, and SafeDAgger all rely on a choice of threshold.

• For EnsembleDAgger, practitioners need to select an uncertainty threshold over the variance in ensemble predictions to flag the need for intervention, as well as a safety threshold which informs the classes (safe vs unsafe) to train the safety classifier. The safety threshold is needed for SafeDAgger.
• For ConformalDAgger, we need to select only an output level uncertainty threshold.
• For LazyDAgger, the safety threshold is needed, as well as a context switching threshold.

While there is no established rigorous method for selecting the threshold values for each algorithm, we included a new sensitivity analysis in the Expert Shift (Human-gated probability = 0.2) context.

Uncertainty threshold ablation for Conformal vs. Ensemble:

We first varied the uncertainty threshold ($\tau$) for the two methods that maintain an uncertainty estimate: ConformalDAgger and EnsembleDAgger. We varied the uncertainty threshold across three discrete values: [0.03, 0.06, 0.09]. We note that since EnsembleDAgger relies on a safety classifier, we held the safety threshold constant throughout all uncertainty threshold ablations at 0.03. The results are averaged over 3 seeds. On one hand, we find that when $\tau$ is low (0.03), Conformal and Ensemble both identify the shift and ask for help accordingly; however, for both algorithms this comes at the cost of a great deal of human feedback requested throughout every deployment episode. Even in the first episode in which there has been no expert shift, a low-threshold at 0.03 requires conformal and ensemble to ask questions 70-80% of the time. On the other hand, when the threshold is set too high, at $\tau = 0.09$, Ensemble is unable to identify the shift and continuously mispredict. Comparatively, we find that ConformalDAgger’s performance is less sensitive to this threshold choice, and is able to effectively adapt to the expert’s shifted policy.

Context-Switching threshold ablation for LazyDAgger:

LazyDAgger relies on a context-switching threshold, which we ablated with values of 0.1, 1, and 10, as in done in the LazyDagger paper [cite]. Similarly to the prior experiments, we hold the LazyDAgger safety threshold constant at 0.03. These values are multipliers on the safety threshold used to train the classifier and inform switches back to the robot after the safety classifier as flagged an unsafe state. We find that thresholds of 1 and 10 results in highly infrequent questions during the episodes where no shift has occurred, but that 0.1 sufficiently better balances the queries.

Safety threshold ablation for SafeDAgger:

SafeDAgger relies on a safety classifier threshold, which we ablated with values 0.01, 0.03, and 0.05. Because SafeDAgger differs from Ensemble and Lazy in that it only has one mechanism for inducing queries, we find that 0.03 and 0.05 are insufficient are generating queries when the safety classifier is used in isolation, and 0.01 sufficiently better balances the queries.

Summary:

All results are fully reflected in the revised manuscript. Please see Appendix Section on Hyperparameter Sensitivity Analysis (Appendix Section B) for these figures and results.

C. Hardware Experiments - EnsembleDAgger Safety Classifier

We have updated the EnsembleDAgger hardware implementation to include the safety classifier. Our initial motivation for omitting the safety classifier on the hardware experiments was to isolate only the uncertainty quantification method, and evaluate how uncertainty-gated queries affect both approaches. In our rebuttal, we added in the safety classifier in the EnsembleDAgger implementation on hardware.

Safety Classifier Training Details:

We trained the safety classifier as a fully connected network which takes in as inputs the Resnet-18 embeddings of the camera images and robot proprioceptive state. We will include training details in the manuscript. For the EnsembleDAgger threshold, we chose our threshold of 0.005 at the mean of the distribution of ensembles variances over the training data. For the safety classifier, we found that setting the safety threshold equal to the mean divergences between expert and learner policy was effective in achieving sufficient safety-prompted queries. Wolleb et al. 2022 takes an implicit-Ensemble by sampling multiple times from the same diffusion model, but its domain is not in the content of interactive imitation learning. We only use the paper’s notion of extracting an ensemble from multiple samples from the same diffusion model to produce our ensemble. We will clarify this in the revised manuscript.

Updated Hardware Experiment Details:

For the updated hardware experiments in the rebuttal, we trained on 50 demos for the base policy, and perfomed an interactive deployment with all methods (Conformal, Lazy, Safe, and Ensemble) on 24 interactive deployments. We used a buffer size of 6k rather than 12k, cutting it in half.

Updated Hardware Results:

We find that without shift, all approaches refrain from robot-gated human interventions. With expert shift, ConformalDAgger learns the shifted policy most saliently. When Lazydagger asks for help, it’s intervention profile is similar to ConformalDAgger, but because the algorithm doesn’t ask for consistently in each episode, it does not receive enough additional data to sufficiently adapt to the shifted policy. Figures are updated to reflect these results in the revised manuscript (Figure 6 in Section 6).

B. Regarding additional baselines

We appreciate the suggestion of including additional interactive imitation learning baselines! In our original manuscript, we focused on comparing ConformalDAgger with EnsembleDAgger because the ensemble output variance provides a measurable interval for coverage, enabling direct comparison. As suggested by the reviewer, we included two relevant DAgger approaches: LazyDAgger[2] and SafeDAgger [3]. SafeDAgger relies on switching to human control when the predicted action discrepancy exceeds a threshold, while LazyDAgger initiates human intervention based on a safety classifier and returns to autonomous control only when the discrepancy falls below an additional context-switching-threshold [2]. Similar to Ensembles, Monte-Carlo Dropout approaches (e.g. UAIL [1]) can also be used to estimate epistemic uncertainty in deep networks; we chose EnsembleDAgger because of prior works empirically observed by Lakshminarayanan et al. [2017], Lee et al. [2015] that training ensemble networks with random initialization works well in practice for capturing this epistemic uncertainty.

We have rerun experiments including these additional baselines and under broader environment shift conditions. The results are discussed in the section below and figures are in the attached PDF (Figure 5 in Section 5.2 in revised manuscript).
 

Updated Results on Simulated 4DoF Reaching Task with Human Gated Probability (=0.2 and =0.5):

When the expert shifts and drifts, ConformalDAgger asks for more help immediately compared to EnsembleDAgger and SafeDAgger, enabling the algorithm to more quickly align to the expert, with fewer human interventions than LazyDAgger. LazyDAgger does successful align to the shifted strategy, but requests many more interventions overall.

Under environment shift, EnsembleDAgger is well-calibrated, boosting the number of interventions requested. This is intuitive as ensembles capture the variance in the training distribution and when the environment shifts, changing the training distribution, the algorithm flags high uncertainty. ConformalDAgger performs well on environment shift context, though asks fewer questions than EnsembleDAgger, converging to the shifted policy nearly as quickly as EnsembleDAgger.

When the human-gated probability is high (0.9), as expected, all approaches receive high quantities of human feedback, resulting in rapid converge to expert and environment shift.

[1] Y. Cui, D. Isele, S. Niekum and K. Fujimura, "Uncertainty-Aware Data Aggregation for Deep Imitation Learning," 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada, 2019, pp. 761-767, doi: 10.1109/ICRA.2019.8794025.

[2] Hoque, Ryan, et al. "Lazydagger: Reducing context switching in interactive imitation learning." 2021 IEEE 17th international conference on automation science and engineering (case). IEEE, 2021.

[3] Zhang, Jiakai, and Kyunghyun Cho. "Query-efficient imitation learning for end-to-end autonomous driving." arXiv preprint arXiv:1605.06450 (2016).

We thank the reviewer for the thoughtful and constructive feedback. With respect to weaknesses and questions, we provide our responses as follows. We have uploaded a revised manuscript with additional experiments, analysis, and baselines; we discuss the results below and refer the reader to the updated figures in the manuscript. We use the color blue to highlight updated text and figures in the manuscript.
 

A. Regarding expert shift and environment shift.

We would first like to clarify – algorithmically, our IQT approach is agnostic to the underlying cause of distribution shift between the expert and the policy. All that it requires is accessing the expert’s labels intermittently to construct calibrated intervals on the base policy’s predictions. The reason the base policy is inconsistent with the expert labels can come from a variety of underlying causal factors. In our initial submission, we focused on expert shift – wherein the expert’s latent strategy changes over times – as one concrete cause of why the base policy disagrees with the expert.

More broadly, policy mispredictions during deployment may result from a multitude of reasons, including suboptimal or multimodal demonstrations, covariate shift (e.g., encountering new observations of the environment during policy execution), or expert shift. To show the generality of our IQT algorithm, we added a Environment Shift context, where the state distribution during deployment differs from training. We instantiated Environment Shift in both the toy 4D reaching task and the newly-added Robosuite benchmark Nut-Assembly task (Robosuite benchmark results can be found in Appendix Section G in revised manuscript). Results for the Simulated 4D Reaching Task are detailed in rebuttal point “B” below and in the revised manuscript (Figure 5 in Section 5.2 in revised manuscript main text).